Python Forum
multithreading issue with output
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
multithreading issue with output
#1
My script is working correctly when i have only one thread.
when I make 50 threads, the output result in 'table' seems to have missing entries in each row ! for example column 'Average Volume' sometimes have missing results for some index. this issue is sporadic !
I am sure that each thread write in different index i.e the threads doesn't overwrite each others !

table = pd.DataFrame(index =tickers,columns = some_columns)

def processData(q,table):
    while not q.empty():
        ticker = q.get()
        #
        if  bad_condition:
            q.task_done()
            continue
              
        try:
                if bad_condition:
                    q.task_done()
                    continue
                
#######A lot of code here               
                

                table.loc[ticker,'Price']=lastPrice
                table.loc[ticker,'Shares Outstanding']=sharesOutstanding
                table.loc[ticker,'Capital']=Capital
                table.loc[ticker,'Average Volume']=averageVolume
                
        except urllib.error.HTTPError:
            print(ticker,'doesnt exist on yahoo finance')
        except urllib.error.URLError:
            print(ticker,'yahoo finance has issue')
        q.task_done()
    return True

num_theads = 50
q = Queue(maxsize=0)
for ticker in table.index:
    q.put(ticker)
for i in range(0,num_theads):
    worker = Thread(target=processData, args=(q,table))
    worker.setDaemon(True)
    worker.start()    
q.join()
table.to_csv('result.csv') 
Reply


Messages In This Thread
multithreading issue with output - by mr_byte31 - Sep-05-2019, 12:45 PM
RE: multithreading issue with output - by stullis - Sep-06-2019, 03:22 AM
RE: multithreading issue with output - by mr_byte31 - Sep-06-2019, 01:53 PM
RE: multithreading issue with output - by mr_byte31 - Sep-06-2019, 03:20 PM
RE: multithreading issue with output - by stullis - Sep-11-2019, 12:04 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  multithreading Hanyx 4 1,331 Jul-29-2022, 07:28 AM
Last Post: Larz60+
Question Problems with variables in multithreading Wombaz 2 1,335 Mar-08-2022, 03:32 PM
Last Post: Wombaz
  Multithreading question amadeok 0 1,784 Oct-17-2020, 12:54 PM
Last Post: amadeok
  How can i add multithreading in this example WoodyWoodpecker1 3 2,516 Aug-11-2020, 05:30 PM
Last Post: deanhystad
  matplotlib multithreading catosp 0 2,954 Jul-03-2020, 09:33 AM
Last Post: catosp
  Multithreading dynamically syncronism Rodrigo 0 1,537 Nov-08-2019, 02:33 AM
Last Post: Rodrigo
  Locks in Multithreading Chuonon 0 1,848 Oct-03-2019, 04:16 PM
Last Post: Chuonon
  Multithreading alternative MartinV279 1 2,802 Aug-01-2019, 11:41 PM
Last Post: scidam
  Output issue twinpiques 6 3,192 Jul-29-2019, 11:24 PM
Last Post: Yoriz
  using locks in multithreading in python3 srm 2 3,676 Jul-13-2019, 11:35 AM
Last Post: noisefloor

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020