Python Forum
Counting Duplicates in large Data Set
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Counting Duplicates in large Data Set
#3
Use a set instead of a tuple. Should be faster. But I don't think this as a possible solution.

"Simple solution" is not going to work because the numbers are staggering. I am not surprised that 1 million combinations did not produce a single 1, 2, 3, 4, 5, 6. One million is a pretty small sample size when there are almost 14 million combinations. For the kind of test you propose I would suggest 1 billion combinations. and how are you going to store the 320 MB Counter dictionary?
Reply


Messages In This Thread
Counting Duplicates in large Data Set - by jmair - Dec-06-2022, 01:52 PM
RE: Counting Duplicates in large Data Set - by deanhystad - Dec-06-2022, 07:15 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Add group number for duplicates atomxkai 2 1,254 Dec-08-2022, 06:08 AM
Last Post: atomxkai
  Reading large crapy text file in anaconda to profile data syamatunuguntla 0 918 Nov-18-2022, 06:15 PM
Last Post: syamatunuguntla
  Searching Module to plot large data G_rizzle 0 1,546 Dec-06-2021, 08:00 AM
Last Post: G_rizzle
  Pandas Indexing with duplicates energerecontractuel 3 3,016 Mar-07-2019, 12:57 AM
Last Post: scidam
  How to filter specific rows from large data file Ariane 7 8,441 Jun-29-2018, 02:43 PM
Last Post: gontajones
  jupyter pandas remove duplicates help okl 3 7,649 Feb-25-2018, 01:11 PM
Last Post: glidecode

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020