Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scrape multiple urls LXML
#2
Can show a example of one way to do it.
from lxml import html
import requests

url_lst = ['https://www.cnn.com', 'https://python-forum.io']
for page in url_lst:
    page = requests.get(page)
    tree = html.fromstring(page.content)
    title = tree.xpath('//title')
    print('-' * 20)
    print(title[0].text)
Output:
-------------------- CNN International - Breaking News, US News, World News and Video -------------------- Python Forum
Reply


Messages In This Thread
Scrape multiple urls LXML - by santdoyle - Oct-26-2019, 01:46 PM
RE: Scrape multiple urls LXML - by snippsat - Oct-26-2019, 09:53 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Scrape table from multiple pages Nhattanktnn 1 872 Jun-07-2023, 09:35 AM
Last Post: Larz60+
  BeautifulSoup not parsing other URLs giddyhead 0 1,204 Feb-23-2022, 05:35 PM
Last Post: giddyhead
  Need logic on how to scrap 100K URLs goodmind 2 2,645 Jun-29-2020, 09:53 AM
Last Post: goodmind
  scrape data 1 go to next page scrape data 2 and so on alkaline3 6 5,219 Mar-13-2020, 07:59 PM
Last Post: alkaline3
  Scraping from multiple URLS to print in a single line. jb89 4 3,382 Jan-29-2020, 06:12 AM
Last Post: perfringo
  Need to Verify URLs; getting SSLError rahul_goswami 0 2,224 Aug-20-2019, 10:17 AM
Last Post: rahul_goswami
  Scrap text out of td table from URLS Gochix2020 4 5,689 Aug-03-2019, 02:56 AM
Last Post: Larz60+
  Regex URLs Django 2.1 sterion66 0 2,755 Nov-04-2018, 10:22 AM
Last Post: sterion66
  Scraping external URLs from pages Apook 5 4,218 Jul-18-2018, 06:42 PM
Last Post: nilamo
  Scrape multiple lines with regex greetings 2 3,067 Jul-04-2018, 09:09 PM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020