Python Forum
Find duplicate files in multiple directories
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Find duplicate files in multiple directories
#1
Hello,

This snippet allows to find duplicate files in multiple directories and count the number of duplicates.
import os

dir1 = '/...path1/'
dir2 = '/...path1/'
dir3 = '/...path3/'
dir4 = '/...path4/'
dir5 = '/...path5/'

list1 = [dir1, dir2, dir3, dir4, dir5]

files = []
for folder in list1:
    files = files + [f for f in os.listdir(folder)]

dup = {x for x in files if files.count(x) > 1}

for item in sorted(dup):
    print('{}\t{}'.format(item, files.count(item)))
What I'm looking for is to create a dataset (maybe a dictionary) where each duplicate file is associated with the locations where the duplicates of the file occur.
Any ideas ?
Thanks.
Reply


Messages In This Thread
Find duplicate files in multiple directories - by Pavel_47 - Dec-25-2022, 02:32 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Trying to generating multiple json files using python script dzgn989 4 428 May-10-2024, 03:09 PM
Last Post: deanhystad
  [SOLVED] Loop through directories and files one level down? Winfried 3 493 Apr-28-2024, 02:31 PM
Last Post: Gribouillis
  Organization of project directories wotoko 3 649 Mar-02-2024, 03:34 PM
Last Post: Larz60+
  python convert multiple files to multiple lists MCL169 6 1,834 Nov-25-2023, 05:31 AM
Last Post: Iqratech
  splitting file into multiple files by searching for string AlphaInc 2 1,125 Jul-01-2023, 10:35 PM
Last Post: Pedroski55
  Merging multiple csv files with same X,Y,Z in each Auz_Pete 3 1,391 Feb-21-2023, 04:21 AM
Last Post: Auz_Pete
  Listing directories (as a text file) kiwi99 1 947 Feb-17-2023, 12:58 PM
Last Post: Larz60+
  unittest generates multiple files for each of my test case, how do I change to 1 file zsousa 0 1,065 Feb-15-2023, 05:34 PM
Last Post: zsousa
  rename same file names in different directories elnk 0 799 Nov-04-2022, 05:23 PM
Last Post: elnk
  Python: re.findall to find multiple instances don't work but search worked Secret 1 1,330 Aug-30-2022, 08:40 PM
Last Post: deanhystad

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020