Python Forum
scraping multiple pages of a website.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
scraping multiple pages of a website.
#11
Thank you all, I try your one first buran and it worked great. I am working on your now Larz60+, I hope both are you will not mind if I don't understand something that I can ask you to explain it to me. Thanks for all your help. I have a lot of work to now.
Reply
#12
Hi Buran, I took you code apart peace by peace. I know every bit of it now, but for this.
Tell what this does how it works
url = '{}{}'.format(base_url, letter)
what does '{} {}' this do
I try to find .format in the requests Mod. but could not.
what is and what does .format do
I know what base_url is and I think letter is the letter at the end of the web page address. If you can, I need to know everything about this line of code.
Thank you
Renny
Reply
#13
url = '{}{}'.format(base_url, letter) is a method used to concatenate strings, combining
"https://www.usa.gov/federal-agencies/" and "a" to create "https://www.usa.gov/federal-agencies/a"

For a discussion of various string concatenation methods, see: https://softwareengineering.stackexchang...catenation

The following code (taken from this thread) compares two different concatenation methods. The method used by Buran seems to be preferred for readability for all but the simplest of concatenations.
from string import ascii_lowercase

base_url  = 'https://www.usa.gov/federal-agencies/'
for letter in ascii_lowercase:
    url_one = '{}{}'.format(base_url, letter)
    url_two = base_url + letter
    print("{}  {}".format(letter, url_one))
    print("{}  {}".format(letter, url_two))    
Lewis
To paraphrase: 'Throw out your dead' code. https://www.youtube.com/watch?v=grbSQ6O6kbs Forward to 1:00
Reply
#14
Lewis already explained, but here is the docs. Read about string-formating mini language

https://docs.python.org/3.6/library/stri...formatting

in 3.6+ one can use f-strings that Larz60+ used in his post#2 https://python-forum.io/Thread-scraping-...4#pid49344
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#15
Thank you all, nice Doc buran, I enjoy reading it. I have try it on other site and it worked fine but for one. I did get it to work on that one. Thank you all so much.
renny
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Need help opening pages when web scraping templeowls 1 272 Feb-29-2024, 06:45 PM
Last Post: snippsat
  Scrape table from multiple pages Nhattanktnn 1 823 Jun-07-2023, 09:35 AM
Last Post: Larz60+
Information Web-scraping, multiple webpages Pabloty92 1 1,246 Dec-28-2022, 02:09 PM
Last Post: Yoriz
  web scraping for new additions/modifed website? kingoman123 4 2,184 Apr-14-2022, 04:46 PM
Last Post: snippsat
  Scraping lender data from Ren Ren Dai website using Python. I will pay for that 200$ Hafedh_2021 1 2,724 May-18-2021, 08:41 PM
Last Post: snippsat
  Scraping all website text using Python MKMKMKMK 1 2,051 Nov-26-2020, 10:35 PM
Last Post: Larz60+
  Web scrap multiple pages anilacem_302 3 3,783 Jul-01-2020, 07:50 PM
Last Post: mlieqo
  scraping multiple pages from table bandar 1 2,648 Jun-27-2020, 10:43 PM
Last Post: Larz60+
  Beginner help - Leap Year Issue Feb 29 and multiple pages warriordazza 3 2,668 May-10-2020, 01:14 AM
Last Post: warriordazza
  Scraping a Website (HELP) LearnPython2 1 1,708 May-08-2020, 03:20 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020