Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Making several POST requests
#1
Hello!

I'm attempting to stage some data for testing purposes via API.
I've written a script that essentially does 2 things:
A) Generate n unique payloads (this is stored as a list of dicts, where each dict is the unique payload).
B) Make a post request for each dict in the above mentioned list.

Here is some pseudo code to help give a rough idea:
# Python 3
import requests

unique_objects = 500
url = # the endpoint
headers = {
    # all required headers
}

def create_payload(num):
    payload = []
    for i in range(0, num):
        # create a unique dict and append to payload
    return payload

def make_request(payload):
    for i in payload:
        response = request.post(url, headers = headers, json = i)
        # additional handling for response

p = create_payload(unique_objects)
make_request(p)


In this example, a list is created containing 500 unique dictionaries, then 500 POST requests are made to the same endpoint (1x for each unique payload in the list).
This process takes only a few minutes to create the list and make all requests.

The problem:
I need to test against a much larger set of data. So I will need to make tens of thousands of POST requests, given the previously observed completion time - this would likely take hours.

Unfortunately, there is not currently a way for me to make a bulk POST request - so I will need to make a request for each unique payload I'm attempting to send off.

I am trying to find a way to responsibly make multiple requests to this endpoint at one time, so that making tens of thousands of POST requests does not take hours to complete.


I've done some searching and have found posts where people mention using aiohttp / asyncio / grequests / etc.
I have attempted to review docs, and work examples into my code, however I'm not seeing any performance increases.
I'm honestly a bit lost when it comes to async requests.
Any help would be appreciated!
Reply
#2
It looks like what you want is session objects.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Async for making requests DangDuong 3 991 Mar-04-2024, 03:50 AM
Last Post: sarahlly
  POST requests - different requests return the same response Default_001 3 1,901 Mar-10-2022, 11:26 PM
Last Post: Default_001
  requests.post() does work Alto 1 1,993 Aug-13-2021, 07:58 AM
Last Post: ndc85430
  requests issue with post on dot_net api Heinrich 1 2,437 Jan-23-2020, 04:28 AM
Last Post: Larz60+
  requests post/get to HTML form mrdominikku 1 2,300 Nov-03-2019, 07:12 PM
Last Post: Larz60+
  Error in requests.post debanilroy 3 5,364 Sep-18-2018, 06:15 PM
Last Post: snippsat
  How do i loop through list of data from CSV file and post requests in aspx dynamics w Prince_Bhatia 1 6,042 Nov-09-2017, 02:53 PM
Last Post: heiner55
  [SOLVED] requests returning HTTP 404 when I follow a link after I do a POST JChris 9 27,594 Nov-14-2016, 02:23 PM
Last Post: JChris

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020