Python Forum
HTTP request inside a for loop.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
HTTP request inside a for loop.
#1
I am writing a web scraping script where I have a list of urls to scrap and I iterate through these urls using a for loop. These urls will return the same page structure but with different data hence I am using the same code to scrap all these urls. My problem is that when I run the script, it returns a 404 html error for all the urls in the list and I think its because the for loop is running faster than the 'session.get()' can return the pages. I may be wrong, please advice what you think the problem is. Please see code below.

session = requests.session()
url_list = list(current_urls_set)
if len(url_list) > 0:
   payload = {
       'UserName': 'myemail',
       'Password': 'mypassword'
   }
   session.post('website url here', data=payload)
   rfq_dir = 'C:/Projects/TenderBot_Python/Tenders_and_RFQs/cpt/RFQs/{}'.format(datetime.today().strftime('%d-%m-%Y'))
   if not os.path.exists(rfq_dir):
       os.mkdir(rfq_dir)
   for url in url_list:
       data_list = url.split(',')
       closing_date = datetime.strptime(data_list[1], '%m/%d/%Y %I:%M:%S %p')
       if closing_date > datetime.now():
           rfqHtml = session.get(websit url here{}'.format(data_list[3].strip()))
           print(rfqHtml.text)
Reply
#2
Never mind this, I had the urls wrong.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Triggering a ps1 script in remote windows server via http python request jasveerjassi 1 319 Jan-26-2024, 07:02 PM
Last Post: deanhystad
  Variable definitions inside loop / could be better? gugarciap 2 375 Jan-09-2024, 11:11 PM
Last Post: deanhystad
  How to create a variable only for use inside the scope of a while loop? Radical 10 1,526 Nov-07-2023, 09:49 AM
Last Post: buran
  Help adding a loop inside a loop Extra 31 4,337 Oct-23-2022, 12:16 AM
Last Post: Extra
  Help request for loop example Jamie_Python 1 1,375 Nov-22-2021, 02:01 PM
Last Post: Marbelous
  how can I correct the Bad Request error on my curl request tomtom 8 4,968 Oct-03-2021, 06:32 AM
Last Post: tomtom
  Adding to the dictionary inside the for-loop - weird behaviour InputOutput007 5 2,651 Jan-21-2021, 02:21 PM
Last Post: InputOutput007
  why print('\n') produced 2 new lines instead of 1 - Located inside a FOR loop JulyFire 2 2,467 Jan-10-2021, 01:50 AM
Last Post: JulyFire
  Creating a variables inside FOR loop zazas321 5 4,040 Sep-16-2020, 04:42 PM
Last Post: Naheed
  How to to tie the execution of one process to another inside a loop in Python ignorant_wanderer 0 2,018 Jul-11-2020, 03:44 AM
Last Post: ignorant_wanderer

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020