Removing Duplicates from a List of Dictionaries

Duplication in a data collection can be a hindrance to efficient data processing. In Python programming, lists of dictionaries are commonly used to store tabular data. However, there may be instances where you need to remove duplicate dictionaries from such a list.

Consider the following list of dictionaries:

[
    {'id': 1, 'name': 'john', 'age': 34},
    {'id': 1, 'name': 'john', 'age': 34},
    {'id': 2, 'name': 'hanna', 'age': 30},
]

The goal is to obtain a list with only unique dictionaries, excluding the duplicates. To achieve this, we can employ a straightforward approach:

Creating a Temporary Dictionary with ID as Key

1. Create a temporary dictionary using a list comprehension, where the key for each dictionary is its 'id' field.
2. This step essentially maps each unique 'id' value to a specific dictionary.

Extracting Unique Dictionaries from Values

1. Obtain the values of the temporary dictionary using the values() method.
2. The result is a list of unique dictionaries, with duplicates removed.

Python Implementation

Here's how to implement this approach in Python:

def remove_duplicates_from_dicts(dict_list):
    dict_id_mapping = {v['id']: v for v in dict_list}
    return list(dict_id_mapping.values())

sample_list = [
    {'id': 1, 'name': 'john', 'age': 34},
    {'id': 1, 'name': 'john', 'age': 34},
    {'id': 2, 'name': 'hanna', 'age': 30},
]
print(remove_duplicates_from_dicts(sample_list))

This code will produce the following output:

[{'id': 1, 'name': 'john', 'age': 34}, {'id': 2, 'name': 'hanna', 'age': 30}]

By employing this strategy, you can effectively remove duplicate dictionaries from a list and obtain a new list with only unique elements.