How do I find duplicates between two lists in Python?

Python | Difference of two lists including duplicates

The ways to find difference of two lists has been discussed earlier, but sometimes, we require to remove only the specific occurrences of the elements as much they occur in other list. Let’s discuss certain ways in which this can be performed.

Method #1 : Using collections.Counter[]
The Counter method can be used to get the exact occurrence of the elements in the list and hence can subtract selectively rather than using the set and ignoring the count of elements altogether. Then the subtraction can be performed to get the actual occurrence.

# Python3 code to demonstrate # Difference of list including duplicates # Using collections.Counter[] from collections import Counter # initializing lists test_list1 = [1, 3, 4, 5, 1, 3, 3] test_list2 = [1, 3, 5] # printing original lists print["The original list 1 : " + str[test_list1]] print["The original list 2 : " + str[test_list2]] # Using collections.Counter[] # Difference of list including duplicates res = list[[Counter[test_list1] - Counter[test_list2]].elements[]] # print result print["The list after performing the subtraction : " + str[res]]

Output : The original list 1 : [1, 3, 4, 5, 1, 3, 3] The original list 2 : [1, 3, 5] The list after performing the subtraction : [1, 3, 3, 4]

Method #2 : Using map[] + lambda + remove[]
The combination of above functions can be used to perform this particular task. The map function can be used to link the function to all elements and remove removes the first occurrence of it. Hence doesn’t remove repeatedly. Works with Python2 only.

# Python code to demonstrate # Difference of list including duplicates # Using map[] + lambda + remove[] # initializing lists test_list1 = [1, 3, 4, 5, 1, 3, 3] test_list2 = [1, 3, 5] # printing original lists print["The original list 1 : " + str[test_list1]] print["The original list 2 : " + str[test_list2]] # Using map[] + lambda + remove[] # Difference of list including duplicates res = map[lambda x: test_list1.remove[x] if x in test_list1 else None, test_list2] # print result print["The list after performing the subtraction : " + str[test_list1]]

Output : The original list 1 : [1, 3, 4, 5, 1, 3, 3] The original list 2 : [1, 3, 5] The list after performing the subtraction : [1, 3, 3, 4]

Article Tags :

Python

Python Programs

Python list-programs

Read Full Article

1. Using set[] function

A simple solution is to convert both lists to set a data structure and then calculate the difference using the - operator.

1 2 3 4 5 6 7 8 9 10

if __name__ == '__main__': first = [2, 1, 3, 4, 1] second = [3, 4, 5] # The `set[]` function is used to create sets diff = list[set[first] - set[second]] print[diff]# prints [1, 2]

DownloadRun Code

How to compare two lists in Python

Python provides multiple ways to compare the two lists. Comparison is the process when the data items of are checked against another data item of list, whether they are the same or not.

The methods of comparing two lists are given below.

• The cmp[] function
• The set[] function and == operator
• The sort[] function and == operator
• The collection.counter[] function
• The reduce[] and map[] function

The cmp[] function

The Python cmp[] function compares the two Python objects and returns the integer values -1, 0, 1 according to the comparison.

Note - It doesn't use in Python 3.x version.

The set[] function and == operator

Python set[] function manipulate the list into the set without taking care of the order of elements. Besides, we use the equal to operator [==] to compare the data items of the list. Let's understand the following example.

Example -

Output:

The list1 and list2 are equal

Explanation:

In the above example, we have declared the two lists to be compared with each other. We converted those lists into the set and compared each element with the help of == operator. All elements are equal in both lists, then if block executed and printed the result.

The sort[] method with == operator

Python sort[] function is used to sort the lists. The same list's elements are the same index position it means; lists are equal.

Note - In the sort[] method, we can pass the list items in any order because we are sorting the list before comparison.

Let's understand the following example -

Example -

Output:

The list1 and list3 are not the same The list1 and list2 are not the same

The collection.counter[] function

The collection module provides the counter[], which compare the list efficiently. It stores the data in dictionary format : and counts the frequency of the list's items.

Note - The order of the list's elements doesn't matter in this function.

Example -

Output:

The lists list1 and list2 are not the same The lists list1 and list3 are the same

The reduce[] and map[]

The map[] function accepts a function and Python iterable object [list, tuple, string, etc] as an arguments and returns a map object. The function implements to each element of the list and returns an iterator as a result.

Besides, The reduce[] method implements the given function to the iterable object recursively.

Here, we will use both methods in combination. The map[] function would implement the function [it can be user-define or lambda function] to every iterable object and the reduce[] function take care of that would apply in recursive manner.

Note - We need to import the functool module to use the reduce[] function.

Let's understand the following example.

Example -

Output:

The list1 and list2 are not the same The list1 and list3 are the same

In this section, we have covered various methods of comparing two lists in Python.

Next TopicHow to convert int to string in Python

← prev next →

Difference of two lists including duplicates in Python

PythonServer Side ProgrammingProgramming

Sometimes we need to find the differences between two lists. It will also mean a mathematical subtraction in which the elements from the first list are removed if they are present in the second list. Duplicates are preserved. Below is the approach through which we can achieve this.

We can use the Counter method from the collections module which will keep track of the count of elements. A straight mathematical subtraction gives the desired result. In the final result the number of occurrences of an element between the first and the second list will decide the elements.

How to Remove Duplicates From a Python List?

Naive Method: Go over each element and check whether this element already exists in the list. If so, remove it. However, this takes a few lines of code.

Efficient Method: A shorter and more concise way is to create a dictionary out of the elements in the list to remove all duplicates and convert the dictionary back to a list. This preserves the order of the original list elements.

lst = ['Alice', 'Bob', 'Bob', 1, 1, 1, 2, 3, 3] print[list[dict.fromkeys[lst]]] # ['Alice', 'Bob', 1, 2, 3]

1. Convert the list to a dictionary with dict.fromkeys[lst].
2. Convert the dictionary into a list with list[dict].

Each list element becomes a new key to the dictionary. For example, the list [1, 2, 3] becomes the dictionary {1:None, 2:None, 3:None}. All elements that occur multiple times will be assigned to the same key. Thus, the dictionary contains only unique keys—there cannot be multiple equal keys.

As dictionary values, you take dummy values [per default].

Then, you convert the dictionary back to a list, throwing away the dummy values.

Here’s the code:

>>> lst = [1, 1, 1, 3, 2, 5, 5, 2] >>> dic = dict.fromkeys[lst] >>> dic {1: None, 3: None, 2: None, 5: None} >>> duplicate_free = list[dic] >>> duplicate_free [1, 3, 2, 5]

Related blog articles:

• Python Remove Duplicates From List of Lists
• Python List Remove
• The Ultimate Guide to Python Dictionaries!

Boolean Comparison

Short answer: The most Pythonic way to check if two ordered lists l1 and l2 are identical, is to use the l1 == l2 operator for element-wise comparison. If all elements are equal and the length of the lists are the same, the return value is True.

Problem: Given are two lists l1 and l2. You want to perform Boolean Comparison: Compare the lists element-wise and return True if your comparison metric returns True for all pairs of elements, and otherwise False.

Examples:

l1 = [1, 2, 3, 4, 5] l2 = [1, 2, 3] # compare[l1, l2] --> False l1 = [1, 2, 3, 4, 5] l2 = [1, 2, 3, 5, 4] # compare[l1, l2] --> False l1 = [1, 2, 3, 4, 5] l2 = [1, 2, 3, 4, 5] # compare[l1, l2] --> True

Let’s discuss the most Pythonic ways of solving this problem. Here’s a quick interactive code overview:

Exercise: Glance over all methods and run the code. What questions come to mind? Do you understand each method?

Read on to learn about each method in detail!

Method 1: Simple Comparison

Not always is the simplest method the best one. But for this particular problem, it is! The equality operator == compares a list element-wise—many Python coders don’t know this!

# 1. Simple Comparison def method_1[l1, l2]: return l1 == l2 l1 = [1, 2, 3, 4, 5] l2 = [1, 2, 3] print[method_1[l1, l2]] # False

So, if you just want to learn about the most Pythonic way to solve this problem, look no further.

But if you want to dive into the wonderful world of Python, learning about different interesting and powerful Python functions, read on!

Method 2: Simple For Loop

The following method is what you’d see from a coder coming from another programming language or from a beginner who doesn’t know about the equality operator on lists [see Method 1].

# 2. Simple For Loop def method_2[l1, l2]: for i in range[min[len[l1], len[l2]]]: if l1[i] != l2[i]: return False return len[l1] == len[l2] l1 = [1, 2, 3, 4, 5] l2 = [1, 2, 3] print[method_2[l1, l2]] # False

In the code, you iterate over all indices from 0 to the last position of the smallest list as determined by the part min[len[l1], len[l2]]. You then check if both elements at the same position are different. If they are different, i.e., l1[i] != l2[i], you can immediately return False because the lists are also different.

If you went through the whole loop without returning False, the list elements are similar. But one list may still be longer! So, by returning len[l1] == len[l2], you ensure to only return True if [1] all elements are equal and [2] the lists have the same length.

A lot of code to accomplish such a simple thing! Let’s see how a better coder would leverage the zip[] function to reduce the complexity of the code.

Method 3: zip[] + For Loop

The zip function takes a number of iterables and aggregates them to a single one by combining the i-th values of each iterable into a tuple for every i.

Let’s see how you can use the function to make the previous code more concise:

# 3. Zip + For Loop def method_3[l1, l2]: for x, y in zip[l1, l2]: if x != y: return False return len[l1] == len[l2] l1 = [1, 2, 3, 4, 5] l2 = [1, 2, 3] print[method_3[l1, l2]] # False

Instead of iterating over indices, you now iterate over pairs of elements [the ones zipped together]. If the lists have different sizes, the remaining elements from the longer list will be skipped. This way, element-wise comparison becomes simpler and no elaborate indexing schemes are required. Avoiding indices by means of the zip[] function is a more Pythonic way for sure!

Method 4: sum[] + zip[] + len[]

But true Python coders will often avoid a for loop and use a generator expression instead.

• You first create an iterable of Boolean values using the generator expression x == y for x, y in zip[l1, l2].
• Then, you sum up over the Boolean values [another trick of pro coders] to find the number of elements that are the same and store it in variable num_equal.
• Finally, you compare this with the length of both lists. If all three values are the same, both lists have the same elements and their length is the same, too. They are equal!

# 4. Sum + Zip + Len def method_4[l1, l2]: num_equal = sum[x == y for x, y in zip[l1, l2]] return num_equal == len[l1] == len[l2] l1 = [1, 2, 3, 4, 5] l2 = [1, 2, 3] print[method_4[l1, l2]] # False print[method_4[[1, 2], [1, 2]]] # True

From the methods except the first one using the == operator, this is the most Pythonic way due to the use of efficient Python helper functions like zip[], len[], and sum[] and generator expressions to make the code more concise and more readable.

You could also write this in a single line of code!

sum[x == y for x, y in zip[l1, l2]] == len[l1] == len[l2]

If you love Python one-liners, check out my new book Python One-Liners with internationally renowned publisher NoStarch press. [Amazon Link]

Method 5: map[] + reduce[] + len[]

The last method is just to train your functional programming skills.

# 5. map[] + reduce[] + len[] from functools import reduce def method_5[l1, l2]: equal = map[lambda x, y: x == y, l1, l2] result = reduce[lambda x, y: x and y, equal] return result and len[l1] == len[l2] l1 = [1, 2, 3, 4, 5] l2 = [1, 2, 3] print[method_5[l1, l2]] # False print[method_5[[1, 2, 3], [1, 2, 3]]] # True

The map[] function combines all pairs of elements to Boolean values [are the two elements equal?]. Thereduce[] function combines all Boolean values performing an and operation. Sure, you can also use the more concise variant using the all[] function:

Method 6: map[] + all[]

This is the same as the previous method—but using the all[] function instead of reduce[] to combine all Boolean values in a global and operation.

# 6. map[] + all[] def method_6[l1, l2]: result = all[map[lambda x, y: x == y, l1, l2]] return result and len[l1] == len[l2] l1 = [1, 2, 3, 4, 5] l2 = [1, 2, 3] print[method_5[l1, l2]] # False print[method_5[[1, 2, 3], [1, 2, 3]]] # True

If you want to learn something new every day, join my free Python email series for continuous improvement in Python and computer science.

Original article: The Most Pythonic Way to Check if Two Ordered Lists Are Identical

How to Compare Two Lists in Python using set[], cmp[] and difference[] Functions

While working with lists in Python, you might have encountered two lists which seem similar. To figure out the difference, you have to compare the data items of both lists. You can do this by using the set[], difference[] and sort[] methods.

In this article, we will understand how to compare two lists in Python.