Mar-16-2017, 06:29 PM
I have a list of over 150,000 urls.
each time the code runs it selects the next url in the list and then generates a few urls from it.
It then has to compare these generated urls to every oth url in the list to make sure they are not the same.
Simply put which would be faster, a list in python. or using an sql database, or is there an alternative which is much faster.
There may be a few million item in the list eventually and it will keep growing every time the code runs.
each time the code runs it selects the next url in the list and then generates a few urls from it.
It then has to compare these generated urls to every oth url in the list to make sure they are not the same.
Simply put which would be faster, a list in python. or using an sql database, or is there an alternative which is much faster.
There may be a few million item in the list eventually and it will keep growing every time the code runs.