Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 scraping multiple pages of a website.
#11
Thank you all, I try your one first buran and it worked great. I am working on your now Larz60+, I hope both are you will not mind if I don't understand something that I can ask you to explain it to me. Thanks for all your help. I have a lot of work to now.
Quote
#12
Hi Buran, I took you code apart peace by peace. I know every bit of it now, but for this.
Tell what this does how it works
url = '{}{}'.format(base_url, letter)
what does '{} {}' this do
I try to find .format in the requests Mod. but could not.
what is and what does .format do
I know what base_url is and I think letter is the letter at the end of the web page address. If you can, I need to know everything about this line of code.
Thank you
Renny
Quote
#13
url = '{}{}'.format(base_url, letter) is a method used to concatenate strings, combining
"https://www.usa.gov/federal-agencies/" and "a" to create "https://www.usa.gov/federal-agencies/a"

For a discussion of various string concatenation methods, see: https://softwareengineering.stackexchang...catenation

The following code (taken from this thread) compares two different concatenation methods. The method used by Buran seems to be preferred for readability for all but the simplest of concatenations.
from string import ascii_lowercase

base_url  = 'https://www.usa.gov/federal-agencies/'
for letter in ascii_lowercase:
    url_one = '{}{}'.format(base_url, letter)
    url_two = base_url + letter
    print("{}  {}".format(letter, url_one))
    print("{}  {}".format(letter, url_two))    
Lewis
To paraphrase: 'Throw out your dead' code. https://www.youtube.com/watch?v=grbSQ6O6kbs Forward to 1:00
Quote
#14
Lewis already explained, but here is the docs. Read about string-formating mini language

https://docs.python.org/3.6/library/stri...formatting

in 3.6+ one can use f-strings that Larz60+ used in his post#2 https://python-forum.io/Thread-scraping-...4#pid49344
Quote
#15
Thank you all, nice Doc buran, I enjoy reading it. I have try it on other site and it worked fine but for one. I did get it to work on that one. Thank you all so much.
renny
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  scraping from a website that hides source code PIWI_Protein 1 92 Mar-27-2020, 05:08 PM
Last Post: Larz60+
  Scraping not moving to the next pages in a website jithin123 0 67 Mar-23-2020, 06:10 PM
Last Post: jithin123
  Scraping from multiple URLS to print in a single line. jb89 4 193 Jan-29-2020, 06:12 AM
Last Post: perfringo
  Looping through multiple pages with changing url Qaruri 2 217 Jan-17-2020, 01:55 PM
Last Post: Qaruri
  Scrapping javascript website with Selenium where pages randomly fail to load JuanJuan 14 736 Dec-27-2019, 12:32 PM
Last Post: JuanJuan
  Random Loss of Control of Website When Scraping bmccollum 0 219 Aug-30-2019, 04:04 AM
Last Post: bmccollum
  MaxRetryError while scraping a website multiple times kawasso 6 3,503 Aug-29-2019, 05:25 PM
Last Post: kawasso
  How to handle tables splitted across multiple web pages ankitjindalbti 2 385 Jun-02-2019, 07:33 AM
Last Post: ankitjindalbti
  scraping with multiple iframe jansky 1 1,306 Nov-09-2018, 11:12 AM
Last Post: snippsat
  Scraping external URLs from pages Apook 5 1,420 Jul-18-2018, 06:42 PM
Last Post: nilamo

Forum Jump:


Users browsing this thread: 1 Guest(s)