Python Forum
How to multiprocessing Api post request
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to multiprocessing Api post request
#1
I successfully post each record one by one from a csv file. However, I'm trying to implement multiprocessing to make it more efficient to handle large data file in the future.
    ENDPOINT_URL = 'https://example.com'
    headers = {'Api-key': '123abc'}
    
    with open("student.csv", "r") as csv_ledger:
        r = csv.DictReader(csv_ledger)
        data = [dict(d) for d in r ]
        groups = {}
    
        for k, g in groupby(data, lambda r: (r['name'])):
            #My data mapping
    
            #for loop to post each record
            post_api = requests.post(ENDPOINT_URL, json=groups, headers=headers)
Is there any easy way to do the multiprocessing for api request?
Reply
#2
here's a couple of resources to read: https://pymotw.com/3/concurrent.futures/
https://docs.python.org/3/library/concur...tures.html
https://www.blog.pythonlibrary.org/2016/...es-module/
There are other asyncio methods, but this is one of the easiest to use.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  how can I correct the Bad Request error on my curl request tomtom 8 4,968 Oct-03-2021, 06:32 AM
Last Post: tomtom
  ImportError: cannot import name 'Request' from 'request' abhishek81py 1 3,860 Jun-18-2020, 08:07 AM
Last Post: buran
  Unable post request to AWS elasticsearch service Rupini 0 1,867 May-18-2020, 08:27 AM
Last Post: Rupini
  Download data from webpage after POST request AlDe 0 2,175 Feb-02-2019, 06:26 AM
Last Post: AlDe
  Python automation GET/POST/REQUEST Raki 1 3,059 Sep-22-2017, 06:24 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020