Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scrape multiple urls LXML
#1
Hi i want scrape multiple urls using LXML. The code im using allow me scrape one url a a time, but i want to do it in bulk. I tried to doit with a for loop, but i couldnt. The single url code is:

from lxml import html
import requests
page = requests.get('https://miwebsite.com')
tree = html.fromstring(page.content)
Titulo = tree.xpath('//div[@class="tituladorVIVO"]/text()')
Fuentes = tree.xpath('//audio/@src')
print ('Fuentes: ', Fuentes)
print ('Titulo', Titulo)
How can i do it with a list of urls. Regards
Reply
#2
Can show a example of one way to do it.
from lxml import html
import requests

url_lst = ['https://www.cnn.com', 'https://python-forum.io']
for page in url_lst:
    page = requests.get(page)
    tree = html.fromstring(page.content)
    title = tree.xpath('//title')
    print('-' * 20)
    print(title[0].text)
Output:
-------------------- CNN International - Breaking News, US News, World News and Video -------------------- Python Forum
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Scrape table from multiple pages Nhattanktnn 1 823 Jun-07-2023, 09:35 AM
Last Post: Larz60+
  BeautifulSoup not parsing other URLs giddyhead 0 1,169 Feb-23-2022, 05:35 PM
Last Post: giddyhead
  Need logic on how to scrap 100K URLs goodmind 2 2,570 Jun-29-2020, 09:53 AM
Last Post: goodmind
  scrape data 1 go to next page scrape data 2 and so on alkaline3 6 5,089 Mar-13-2020, 07:59 PM
Last Post: alkaline3
  Scraping from multiple URLS to print in a single line. jb89 4 3,305 Jan-29-2020, 06:12 AM
Last Post: perfringo
  Need to Verify URLs; getting SSLError rahul_goswami 0 2,160 Aug-20-2019, 10:17 AM
Last Post: rahul_goswami
  Scrap text out of td table from URLS Gochix2020 4 5,579 Aug-03-2019, 02:56 AM
Last Post: Larz60+
  Regex URLs Django 2.1 sterion66 0 2,721 Nov-04-2018, 10:22 AM
Last Post: sterion66
  Scraping external URLs from pages Apook 5 4,148 Jul-18-2018, 06:42 PM
Last Post: nilamo
  Scrape multiple lines with regex greetings 2 3,021 Jul-04-2018, 09:09 PM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020