Python Forum
Why thread and queue only process the last row
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Why thread and queue only process the last row
#1
CSv
date,ref_num,desc,debit,credit,payee_id,code,name

    2019-01-31,L00001,john,30.00,,150,500,johnkino
    2019-01-31,L00001,john,,30.00,150,600,johnkino
    2019-01-30,L00002,john,30.00,,150,500,johnkino
    2019-01-30,L00002,john,,30.00,150,600,johnkino
import csv
from itertools import groupby
import threading
from queue import Queue

def postRequest():
    print('thread starting')
    while True:
        item_to_process = q.get()
        print(item_to_process)
        q.task_done()

q = Queue()

with open("test.csv", "r") as csv_ledger:
    r = csv.DictReader(csv_ledger)
    data = [ dict(d) for d in r ]
    groups = {}

    for k, g in groupby(data, lambda r: (r[ 'ref_num' ])):
        items = [ ]
        for i in g:
            chart_of_account = {k: v for k, v in i.items() if k in [ 'name', 'code' ]}
            item = {k: v for k, v in i.items() if k in [ 'debit', 'credit', 'desc', 'payee_id' ]}
            item.update({'chart_of_account': chart_of_account})
            items.append(item)
        groups.update({
            "date": i[ 'date' ],
            "desc": i[ 'desc' ],
            "ref_num": k,
            "items": items
        })
        print(groups) <--- correct structure
        q.put(groups)


for i in range(1):
    t = threading.Thread(target=postRequest)
    t.start()

print('Main thread waiting')
q.join()
print('Done.')
Result
2019-01-30,L00002,john,,30.00,150,600,johnkino
2019-01-30,L00002,john,,30.00,150,600,johnkino
2019-01-30,L00002,john,,30.00,150,600,johnkino
2019-01-30,L00002,john,,30.00,150,600,johnkino
The data of groups is the complete and correct data structure which is loop every row but the result always loops for the last row only after passing to queue for processing.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Process doesn't work but Thread work ! mr_byte31 4 2,635 Oct-18-2021, 06:29 PM
Last Post: mr_byte31
  task queue Valon1981 8 3,609 Jul-07-2020, 07:41 AM
Last Post: freeman
  kill thread or process asap, even during time.sleep nanok66 4 2,937 Apr-29-2020, 10:13 AM
Last Post: nanok66
  Error SQLite objects created in a thread can only be used in that same thread. binhduonggttn 3 15,588 Jan-31-2020, 11:08 AM
Last Post: DeaD_EyE
  Queue in Pygame constantin01 1 3,692 Jan-07-2020, 04:02 PM
Last Post: metulburr
  How to sharing object between multiple process from main process using Pipe Subrata 1 3,672 Sep-03-2019, 09:49 PM
Last Post: woooee
  Queue maxsize mr_byte31 2 4,558 Sep-03-2019, 07:02 PM
Last Post: mr_byte31
  multiprocessing.Queue.join_thread() says "thread" Skaperen 0 2,167 Jul-29-2019, 12:45 AM
Last Post: Skaperen
  Queue.Queue() would not reduce capacity after get() yuan8421 9 11,095 Jan-02-2018, 09:38 PM
Last Post: Windspar
  Good way to have a worker queue accessible by multiple process? cheater 2 2,509 Dec-21-2017, 09:30 PM
Last Post: wavic

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020