Python Forum
How can I make this code more efficient and process faster?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How can I make this code more efficient and process faster?
#1
The main function simulates receiving as much data as possible and then forwarding it to other subprocesses for processing. There will actually be multiple subprocesses, and only one subprocess is tested here. The data calculation in the subprocess cannot affect the data reception, and the calculation time is uncertain and may be longer. Below is my test code. I want to know how to optimize it to make it faster.
import multiprocessing
import threading
import collections
import time

def doCount(data, num):
    num=num+1
    #print("doCount:",time.time_ns())
    time.sleep(0.000001)

def threadWorker(d,event):
    tstart = time.time_ns()
    num = 0
    while True:
        if d:
            data = d.popleft()
            if data is None:
                break
            doCount(data,num)
        else:
            event.clear()
            event.wait()

    tend = time.time_ns()
    print('thread-recv:', (tend - tstart) / 1000000)

def processWorker(conn):
    d = collections.deque()
    event = threading.Event()
    t = threading.Thread(target=threadWorker, args=(d,event), daemon=True)
    t.start()

    cstart = time.time_ns()
    while True:
        try:
            data = conn.recv()
            d.append(data)
            event.set()
            if data is None:
                break
        except EOFError:
            break

    cend = time.time_ns()
    print('recv:',(cend- cstart)/1000000)
    print('thread-wait:', time.time_ns())
    t.join()
    print('thread-end:', time.time_ns())

def main():
    read_conn, write_conn = multiprocessing.Pipe(duplex=False)

    p = multiprocessing.Process(target=processWorker, args=(read_conn,))
    p.start()

    pstart = time.time_ns()
    for i in range(100000):
        write_conn.send(
            {"code": f"code {i}", "time": time.time_ns(), "type": 1, "num": 100000, "aaa": "ddd", "bbbb": "dddd",
             "ccc": "dddd", "fff": "dddd", "ggg": "dddd", "hhh": "dddd", "iii": "dddd"})

    write_conn.send(None)
    write_conn.close()

    pend = time.time_ns()
    print('send:', (pend - pstart) / 1000000)

    p.join()

if __name__ == '__main__':
    main()
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  hi need help to make this code work correctly atulkul1985 5 1,868 Nov-20-2023, 04:38 PM
Last Post: deanhystad
  newbie question - can't make code work tronic72 2 1,487 Oct-22-2023, 09:08 PM
Last Post: tronic72
  A more efficient code titanif 2 1,143 Oct-17-2023, 02:07 PM
Last Post: deanhystad
  Cleaning my code to make it more efficient BSDevo 13 3,553 Sep-27-2023, 10:39 PM
Last Post: BSDevo
  how to make bot that sends instagram auto password reset code kraixx 2 2,550 Mar-04-2023, 09:59 PM
Last Post: jefsummers
  Make code non-blocking? Extra 0 1,988 Dec-03-2022, 10:07 PM
Last Post: Extra
  Making a function more efficient CatorCanulis 9 3,443 Oct-06-2022, 07:47 AM
Last Post: DPaul
  faster code for my code kucingkembar 19 6,378 Aug-09-2022, 09:48 AM
Last Post: DPaul
  How to make app run only when process is aviable Mawixy 1 1,690 Apr-19-2022, 03:45 PM
Last Post: Axel_Erfurt
  Process finished with exit code 137 (interrupted by signal 9: SIGKILL) erdemath 2 12,007 Apr-18-2022, 08:40 PM
Last Post: erdemath

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020