Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 HTTP request inside a for loop.
I am writing a web scraping script where I have a list of urls to scrap and I iterate through these urls using a for loop. These urls will return the same page structure but with different data hence I am using the same code to scrap all these urls. My problem is that when I run the script, it returns a 404 html error for all the urls in the list and I think its because the for loop is running faster than the 'session.get()' can return the pages. I may be wrong, please advice what you think the problem is. Please see code below.

session = requests.session()
url_list = list(current_urls_set)
if len(url_list) > 0:
   payload = {
       'UserName': 'myemail',
       'Password': 'mypassword'
   }'website url here', data=payload)
   rfq_dir = 'C:/Projects/TenderBot_Python/Tenders_and_RFQs/cpt/RFQs/{}'.format('%d-%m-%Y'))
   if not os.path.exists(rfq_dir):
   for url in url_list:
       data_list = url.split(',')
       closing_date = datetime.strptime(data_list[1], '%m/%d/%Y %I:%M:%S %p')
       if closing_date >
           rfqHtml = session.get(websit url here{}'.format(data_list[3].strip()))
snippsat wrote May-08-2017, 10:18 AM:
Fix formatting,use Ctrl+Shift+v when copy in code.
Never mind this, I had the urls wrong.
nilamo likes this post

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  ModuleNotFoundError: No module named 'http.client'; 'http' is not a package abhishek81py 1 153 Jun-25-2020, 08:58 AM
Last Post: buran
  ImportError: cannot import name 'Request' from 'request' abhishek81py 1 146 Jun-18-2020, 08:07 AM
Last Post: buran
  Updating a matrix in a time interval inside a for loop vp1989 4 218 May-17-2020, 07:15 PM
Last Post: vp1989
  Loop inside loop inside loop L0RTK1PAN1DZ3 4 235 Mar-16-2020, 10:44 AM
Last Post: buran
  Is it possible to avoid 2 loops inside another loop? SvetlanaofVodianova 2 337 Nov-27-2019, 02:30 PM
Last Post: Gribouillis
  Loop through tags inside tags in Selenium/Python xpack24 1 802 Oct-23-2019, 10:15 AM
Last Post: Larz60+
  plotting inside the loop is not working jenya56 4 642 Apr-10-2019, 08:11 PM
Last Post: perfringo
  EXE file inside the loop iwsyang 1 747 Nov-01-2018, 07:09 PM
Last Post: j.crater
  msvcrt.getch prints twice inside loop ReallyBigTeeth 5 2,704 Sep-04-2018, 06:54 AM
Last Post: ReallyBigTeeth
  For looping over a list, editing the list from inside the loop? Krookroo 3 1,701 Sep-04-2017, 05:08 PM
Last Post: Krookroo

Forum Jump:

Users browsing this thread: 1 Guest(s)