Unique Dictionaries in Python Lists

Lists of dictionaries are common in Python applications. However, managing duplicate dictionaries can be challenging. This article discusses how to efficiently remove duplicates and obtain a list of unique dictionaries.

Consider a list of dictionaries:

L = [
    {'id': 1, 'name': 'john', 'age': 34},
    {'id': 1, 'name': 'john', 'age': 34},
    {'id': 2, 'name': 'hanna', 'age': 30}
]

Approaching the Problem

To deduplicate a list of dictionaries, a straightforward approach involves iterating over the list and comparing each dictionary to the others. However, this process can be computationally expensive for large lists.

Using a Temporary Dictionary

A more efficient solution leverages a temporary dictionary to handle the deduplication. The key of the dictionary is set to the id field of each dictionary, and the value is set to the dictionary itself. This operation effectively filters out duplicates because each unique id will correspond to only one dictionary entry.

Retrieving Unique Dictionaries

Once the temporary dictionary is populated, the values (which represent the unique dictionaries) can be retrieved using the values() method.

Python Implementation

Python 2.7:

{v['id']:v for v in L}.values()

Python 3:

list({v['id']:v for v in L}.values())

Python 2.5/2.6:

dict((v['id'],v) for v in L).values()

These concise solutions result in a list of unique dictionaries:

[
    {'id': 1, 'name': 'john', 'age': 34},
    {'id': 2, 'name': 'hanna', 'age': 30}
]

This approach efficiently removes duplicates by leveraging a temporary dictionary to identify and extract unique dictionary values.