Python Forum
requests_futures.sessions retry list of HTTP 429
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
requests_futures.sessions retry list of HTTP 429
#1
Got this python code that works great sending parallel http requests. When i get a HTTP 429 (Too Many Requests) it is appended to a list. The question is how can i rerun these urls?

or is there a better way.


# Yield successive n-sized 
# chunks from l. 
def divide_chunks(l, n): 
    # looping till length l
    for i  in range(0, len(l), n):
        yield l[i:i + n]
    # yield l[i:i + n] 
  
# How many elements each 
# list should have 
n = 5

lst = []
dlist = []
try:
    with FuturesSession(max_workers=10) as session:
        urls = list(divide_chunks(my_list, n))
        for items in urls:
            futures = [session.get(url[2]) for url in items]
        for future in futures:
            r = future.result()
            if r.status_code == 429:
                lst.append((r.url, divide_chunks(my_list, n )))
            elif r.status_code == 200:
                dict = r.json()
                dlist.append(dict)
        print (dlist)
except requests.ConnectionError as e:
		print (e)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
Question Use function, retry until valid Ashcora 8 1,469 Jan-06-2023, 10:14 AM
Last Post: Ashcora
  How to ignore "Retrying (Retry(total=2, connect=2, read=5, redirect=5, status=None))" const 3 2,712 Mar-26-2022, 08:55 AM
Last Post: ndc85430
  Add http to a list on a text file Blue Dog 5 2,157 Sep-23-2021, 06:40 PM
Last Post: snippsat
  Retry After Exception Harshil 2 2,259 Aug-09-2020, 05:32 PM
Last Post: Harshil
  ModuleNotFoundError: No module named 'http.client'; 'http' is not a package abhishek81py 1 15,470 Jun-25-2020, 08:58 AM
Last Post: buran
  How can I fix this code so if i want to retry, it will go through the same loop SteampunkMaverick12 1 1,895 Apr-15-2019, 07:36 PM
Last Post: ichabod801
  problem to use multi-threads in a code for telnet sessions anna 6 6,103 Jun-27-2018, 10:02 AM
Last Post: anna

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020