Python Forum
Why thread and queue only process the last row
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Why thread and queue only process the last row
#1
CSv
date,ref_num,desc,debit,credit,payee_id,code,name

    2019-01-31,L00001,john,30.00,,150,500,johnkino
    2019-01-31,L00001,john,,30.00,150,600,johnkino
    2019-01-30,L00002,john,30.00,,150,500,johnkino
    2019-01-30,L00002,john,,30.00,150,600,johnkino
import csv
from itertools import groupby
import threading
from queue import Queue

def postRequest():
    print('thread starting')
    while True:
        item_to_process = q.get()
        print(item_to_process)
        q.task_done()

q = Queue()

with open("test.csv", "r") as csv_ledger:
    r = csv.DictReader(csv_ledger)
    data = [ dict(d) for d in r ]
    groups = {}

    for k, g in groupby(data, lambda r: (r[ 'ref_num' ])):
        items = [ ]
        for i in g:
            chart_of_account = {k: v for k, v in i.items() if k in [ 'name', 'code' ]}
            item = {k: v for k, v in i.items() if k in [ 'debit', 'credit', 'desc', 'payee_id' ]}
            item.update({'chart_of_account': chart_of_account})
            items.append(item)
        groups.update({
            "date": i[ 'date' ],
            "desc": i[ 'desc' ],
            "ref_num": k,
            "items": items
        })
        print(groups) <--- correct structure
        q.put(groups)


for i in range(1):
    t = threading.Thread(target=postRequest)
    t.start()

print('Main thread waiting')
q.join()
print('Done.')
Result
2019-01-30,L00002,john,,30.00,150,600,johnkino
2019-01-30,L00002,john,,30.00,150,600,johnkino
2019-01-30,L00002,john,,30.00,150,600,johnkino
2019-01-30,L00002,john,,30.00,150,600,johnkino
The data of groups is the complete and correct data structure which is loop every row but the result always loops for the last row only after passing to queue for processing.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Process doesn't work but Thread work ! mr_byte31 4 2,556 Oct-18-2021, 06:29 PM
Last Post: mr_byte31
  task queue Valon1981 8 3,517 Jul-07-2020, 07:41 AM
Last Post: freeman
  kill thread or process asap, even during time.sleep nanok66 4 2,868 Apr-29-2020, 10:13 AM
Last Post: nanok66
  Error SQLite objects created in a thread can only be used in that same thread. binhduonggttn 3 15,391 Jan-31-2020, 11:08 AM
Last Post: DeaD_EyE
  Queue in Pygame constantin01 1 3,640 Jan-07-2020, 04:02 PM
Last Post: metulburr
  How to sharing object between multiple process from main process using Pipe Subrata 1 3,619 Sep-03-2019, 09:49 PM
Last Post: woooee
  Queue maxsize mr_byte31 2 4,501 Sep-03-2019, 07:02 PM
Last Post: mr_byte31
  multiprocessing.Queue.join_thread() says "thread" Skaperen 0 2,143 Jul-29-2019, 12:45 AM
Last Post: Skaperen
  Queue.Queue() would not reduce capacity after get() yuan8421 9 10,963 Jan-02-2018, 09:38 PM
Last Post: Windspar
  Good way to have a worker queue accessible by multiple process? cheater 2 2,484 Dec-21-2017, 09:30 PM
Last Post: wavic

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020