My script is working correctly when i have only one thread.
when I make 50 threads, the output result in 'table' seems to have missing entries in each row ! for example column 'Average Volume' sometimes have missing results for some index. this issue is sporadic !
I am sure that each thread write in different index i.e the threads doesn't overwrite each others !
when I make 50 threads, the output result in 'table' seems to have missing entries in each row ! for example column 'Average Volume' sometimes have missing results for some index. this issue is sporadic !
I am sure that each thread write in different index i.e the threads doesn't overwrite each others !
table = pd.DataFrame(index =tickers,columns = some_columns) def processData(q,table): while not q.empty(): ticker = q.get() # if bad_condition: q.task_done() continue try: if bad_condition: q.task_done() continue #######A lot of code here table.loc[ticker,'Price']=lastPrice table.loc[ticker,'Shares Outstanding']=sharesOutstanding table.loc[ticker,'Capital']=Capital table.loc[ticker,'Average Volume']=averageVolume except urllib.error.HTTPError: print(ticker,'doesnt exist on yahoo finance') except urllib.error.URLError: print(ticker,'yahoo finance has issue') q.task_done() return True num_theads = 50 q = Queue(maxsize=0) for ticker in table.index: q.put(ticker) for i in range(0,num_theads): worker = Thread(target=processData, args=(q,table)) worker.setDaemon(True) worker.start() q.join() table.to_csv('result.csv')