Which is Faster, List or SQL

[nilamo]

Maybe you can "partition" your list somehow and process it in smaller batches?

map/reduce with external computers?  *giddy excitement*

If you're doing this more than once, then you're probably storing the data somewhere.  Like a file maybe?  If you're reading it every time, adding new things, and repeating, every single day, with whatever you generate needing to be available for the future... I think you should go with a db.  Even something small like sqlite.  Then the db can handle picking a few at random for you to use to generate new urls, you can quickly check if those generated urls already exist, add them to the existing tables, and move on.  

This sounds a lot like the sort of problem a database is designed to solve.