Python Forum
How can I make this code more efficient and process faster?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How can I make this code more efficient and process faster?
#1
The main function simulates receiving as much data as possible and then forwarding it to other subprocesses for processing. There will actually be multiple subprocesses, and only one subprocess is tested here. The data calculation in the subprocess cannot affect the data reception, and the calculation time is uncertain and may be longer. Below is my test code. I want to know how to optimize it to make it faster.
import multiprocessing
import threading
import collections
import time

def doCount(data, num):
    num=num+1
    #print("doCount:",time.time_ns())
    time.sleep(0.000001)

def threadWorker(d,event):
    tstart = time.time_ns()
    num = 0
    while True:
        if d:
            data = d.popleft()
            if data is None:
                break
            doCount(data,num)
        else:
            event.clear()
            event.wait()

    tend = time.time_ns()
    print('thread-recv:', (tend - tstart) / 1000000)

def processWorker(conn):
    d = collections.deque()
    event = threading.Event()
    t = threading.Thread(target=threadWorker, args=(d,event), daemon=True)
    t.start()

    cstart = time.time_ns()
    while True:
        try:
            data = conn.recv()
            d.append(data)
            event.set()
            if data is None:
                break
        except EOFError:
            break

    cend = time.time_ns()
    print('recv:',(cend- cstart)/1000000)
    print('thread-wait:', time.time_ns())
    t.join()
    print('thread-end:', time.time_ns())

def main():
    read_conn, write_conn = multiprocessing.Pipe(duplex=False)

    p = multiprocessing.Process(target=processWorker, args=(read_conn,))
    p.start()

    pstart = time.time_ns()
    for i in range(100000):
        write_conn.send(
            {"code": f"code {i}", "time": time.time_ns(), "type": 1, "num": 100000, "aaa": "ddd", "bbbb": "dddd",
             "ccc": "dddd", "fff": "dddd", "ggg": "dddd", "hhh": "dddd", "iii": "dddd"})

    write_conn.send(None)
    write_conn.close()

    pend = time.time_ns()
    print('send:', (pend - pstart) / 1000000)

    p.join()

if __name__ == '__main__':
    main()
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Make code run faster: point within polygon lookups Bennygib 2 570 Apr-19-2025, 09:33 AM
Last Post: Larz60+
  hi need help to make this code work correctly atulkul1985 5 2,140 Nov-20-2023, 04:38 PM
Last Post: deanhystad
  newbie question - can't make code work tronic72 2 1,644 Oct-22-2023, 09:08 PM
Last Post: tronic72
  A more efficient code titanif 2 1,322 Oct-17-2023, 02:07 PM
Last Post: deanhystad
  Cleaning my code to make it more efficient BSDevo 13 4,037 Sep-27-2023, 10:39 PM
Last Post: BSDevo
  how to make bot that sends instagram auto password reset code kraixx 2 2,848 Mar-04-2023, 09:59 PM
Last Post: jefsummers
  Make code non-blocking? Extra 0 2,125 Dec-03-2022, 10:07 PM
Last Post: Extra
  Making a function more efficient CatorCanulis 9 3,810 Oct-06-2022, 07:47 AM
Last Post: DPaul
  faster code for my code kucingkembar 19 7,093 Aug-09-2022, 09:48 AM
Last Post: DPaul
  How to make app run only when process is aviable Mawixy 1 1,806 Apr-19-2022, 03:45 PM
Last Post: Axel_Erfurt

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020