Which is Faster, List or SQL

[AceScottie]

Maybe you can "partition" your list somehow and process it in smaller batches?

map/reduce with external computers?  *giddy excitement*

If you're doing this more than once, then you're probably storing the data somewhere.  Like a file maybe?  If you're reading it every time, adding new things, and repeating, every single day, with whatever you generate needing to be available for the future... I think you should go with a db.  Even something small like sqlite.  Then the db can handle picking a few at random for you to use to generate new urls, you can quickly check if those generated urls already exist, add them to the existing tables, and move on.  

This sounds a lot like the sort of problem a database is designed to solve.

I do store the results in a csv file, however i only read once and simply append any non matched urls (however this doesnt always work)

Its ok for the first 1000 URLs but after that it starts missing a few and duplicating a few, i run a second code that uses sets to clean it up after and i dropped ftom 154k to 150k when i last ran it.

I will look into creating a db as it will be easier to access multiple times rather than using a file.

to give you more of a scope this is for a demo of network usage on a cluster computer. going to use 10-20 Raspberry Pis' and generate as much traffic from differenct sources and protocols as possible.