Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 HTTP request inside a for loop.
#1
I am writing a web scraping script where I have a list of urls to scrap and I iterate through these urls using a for loop. These urls will return the same page structure but with different data hence I am using the same code to scrap all these urls. My problem is that when I run the script, it returns a 404 html error for all the urls in the list and I think its because the for loop is running faster than the 'session.get()' can return the pages. I may be wrong, please advice what you think the problem is. Please see code below.

session = requests.session()
url_list = list(current_urls_set)
if len(url_list) > 0:
   payload = {
       'UserName': 'myemail',
       'Password': 'mypassword'
   }
   session.post('website url here', data=payload)
   rfq_dir = 'C:/Projects/TenderBot_Python/Tenders_and_RFQs/cpt/RFQs/{}'.format(datetime.today().strftime('%d-%m-%Y'))
   if not os.path.exists(rfq_dir):
       os.mkdir(rfq_dir)
   for url in url_list:
       data_list = url.split(',')
       closing_date = datetime.strptime(data_list[1], '%m/%d/%Y %I:%M:%S %p')
       if closing_date > datetime.now():
           rfqHtml = session.get(websit url here{}'.format(data_list[3].strip()))
           print(rfqHtml.text)
snippsat wrote May-08-2017, 10:18 AM:
Fix formatting,use Ctrl+Shift+v when copy in code.
Quote
#2
Never mind this, I had the urls wrong.
nilamo likes this post
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Is it possible to avoid 2 loops inside another loop? SvetlanaofVodianova 2 173 Nov-27-2019, 02:30 PM
Last Post: Gribouillis
  Loop through tags inside tags in Selenium/Python xpack24 1 138 Oct-23-2019, 10:15 AM
Last Post: Larz60+
  plotting inside the loop is not working jenya56 4 390 Apr-10-2019, 08:11 PM
Last Post: perfringo
  EXE file inside the loop iwsyang 1 437 Nov-01-2018, 07:09 PM
Last Post: j.crater
  msvcrt.getch prints twice inside loop ReallyBigTeeth 5 2,035 Sep-04-2018, 06:54 AM
Last Post: ReallyBigTeeth
  For looping over a list, editing the list from inside the loop? Krookroo 3 1,434 Sep-04-2017, 05:08 PM
Last Post: Krookroo
  List data-member of object not updating inside loop. Sagar 2 1,138 Aug-30-2017, 01:07 PM
Last Post: Sagar

Forum Jump:


Users browsing this thread: 1 Guest(s)