Tutorial

Python Remove Duplicates from a List

Published on August 3, 2022
author

Pankaj

Python Remove Duplicates from a List

There are many ways to remove duplicates from a Python List.

Removing Duplicates from a List

Python list can contain duplicate elements. Let’s look into examples of removing the duplicate elements in different ways.

1. Using Temporary List

This is the brute-force way to remove duplicate elements from a list. We will create a temporary list and append elements to it only if it’s not present.

ints_list = [1, 2, 3, 4, 3, 2]

temp = []

for x in ints_list:
    if x not in temp:
        temp.append(x)

ints_list = temp

print(f'Updated List after removing duplicates = {temp}')

Output: Updated List after removing duplicates = [1, 2, 3, 4] Recommended Reading: Python f-strings

2. set() function

Python set doesn’t have duplicate elements. We can use the built-in set() function to convert the list to a set, then use the list() function to convert it back to the list.

ints_list = [1, 2, 3, 4, 3, 2]

ints_list1 = list(set(ints_list))
print(ints_list1)  # [1, 2, 3, 4]

3. List elements as Dictionary Keys

We know that dictionary keys are unique. The dict class has fromkeys() function that accepts an iterable to create the dictionary with keys from the iterable.

ints_list = [1, 2, 3, 4, 3, 2]

ints_list2 = list(dict.fromkeys(ints_list))
print(ints_list2)  # [1, 2, 3, 4]

The list count() method returns the number of occurrences of the value. We can use it with the remove() method to eliminate the duplicate elements from the list.

ints_list = [1, 2, 3, 4, 3, 2]

for x in ints_list:
    if ints_list.count(x) > 1:
        ints_list.remove(x)
print(ints_list)  # [1, 2, 3, 4]

NOTE: As pointed out in the below comment, using the count() function is not advisable while removing the element from the same iterator because it can lead to unwanted results. For example:

values = [87, 94, 45, 94, 94, 41, 65, 94, 41, 99, 94, 94, 94]

for x in values:
    if values.count(x) > 1:
        values.remove(x)
print(values)  # [87, 45, 65, 41, 99, 94, 94] - 94 is still present twice

5. List Comprehension

We can create a list from an iterable using the list comprehension. This technique is the same as using the temporary list and the for loop to remove the duplicate elements. But, it reduces the number of lines of the code.

int_list = [1, 2, 3, 4, 3, 2]
temp = []
[temp.append(x) for x in ints_list if x not in temp]
print(temp)  # [1, 2, 3, 4]

Best Way to Remove Duplicates from a List

If you don’t want duplicate elements, you should use Set. But, if you have to remove the duplicate values from a list, then I would prefer count() function because it doesn’t create another temporary set or list object. So, it’s more memory efficient.

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Learn more about our products

About the authors
Default avatar
Pankaj

author

While we believe that this content benefits our community, we have not yet thoroughly reviewed it. If you have any suggestions for improvements, please let us know by clicking the “report an issue“ button at the bottom of the tutorial.

Still looking for an answer?

Ask a questionSearch for more help

Was this helpful?
 
JournalDev
DigitalOcean Employee
DigitalOcean Employee badge
December 22, 2020

values = [87, 94, 45, 94, 94, 41, 65, 94, 41, 99, 94, 94, 94] def removeDuplicate(z): for i in z: if z.count(i)>1: z.remove(i) removeDuplicate(z) return z print(removeDuplicate(values)) ************Output************** [87, 45, 65, 41, 99, 94]

- vikram ram raut

    JournalDev
    DigitalOcean Employee
    DigitalOcean Employee badge
    December 2, 2020

    The count() method should never be used. The reason is you should never try to remove an element from a list while iterating on the list itself. The system may behave unexpectedly and skip over a particular item from the list as the list is constantly changing while the loop runs. Try the count() method on following list: values = [87, 94, 45, 94, 94, 41, 65, 94, 41, 99, 94, 94, 94] You’ll notice that the value 94 still stays in the final outcome. The best way is to use set method.

    - Meet

      JournalDev
      DigitalOcean Employee
      DigitalOcean Employee badge
      June 27, 2020

      how can i remove duplicates using two for loop by comparing the list member and if match occur then simply remove the particular element

      - RANA SIDHDHARAJSINH SAHDEVSINH

        JournalDev
        DigitalOcean Employee
        DigitalOcean Employee badge
        May 28, 2020

        ints_list = [1,1,1,1] for x in ints_list: if ints_list.count(x) > 1: ints_list.remove(x) print(ints_list) # Output: [1,1]

        - chirag maliwal

          JournalDev
          DigitalOcean Employee
          DigitalOcean Employee badge
          May 7, 2020

          how can i remove duplicate from a list without using a for loop or while loop or any in-build function in python do’not use list comprehension as well.

          - Rajeev

            Try DigitalOcean for free

            Click below to sign up and get $200 of credit to try our products over 60 days!

            Sign up

            Join the Tech Talk
            Success! Thank you! Please check your email for further details.

            Please complete your information!

            Featured on Community

            Get our biweekly newsletter

            Sign up for Infrastructure as a Newsletter.

            Hollie's Hub for Good

            Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

            Become a contributor

            Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

            Welcome to the developer cloud

            DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

            Learn more