Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Youtube page scraping
#1
Hello all,

The page I would like to scrape is a user videos page on youtube. All of the video titles in the page are nested under the same xpath which is: //*[@id='video-title'] What I would like my script to do is to create a list of titles on a loaded page and then print it, however it does not seem to work. Any advice?

from lxml import html
import requests

page = requests.get('https://www.youtube.com/user/numberphile/videos')

tree = html.fromstring(page.content)

title = tree.xpath("//*[@id='video-title']/text()")

titles = []

for f in title:
    titles.append(f)

print(titles)
Reply
#2
Turn of JavaScript in browser and see how many videos you see Think
If you had looked at what page.content return,
you would have seen that's there no id='video-title' at all.

Selenium have more about it here.
Look at YouTube Data API v3,if can get data that way.
A post about usage.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  I am scraping a web page but got an Error Sarmad54 3 1,418 Mar-02-2023, 08:20 PM
Last Post: Sarmad54
  Scraping the page without distorting content oleglpts 5 2,443 Dec-16-2021, 05:08 PM
Last Post: oleglpts
  Scraping a page with log in data (security, proxies) iamaghost 0 2,103 Mar-27-2021, 02:56 PM
Last Post: iamaghost
  Scraping .aspx page Larz60+ 21 50,853 Mar-18-2021, 10:16 AM
Last Post: Larz60+
  Scraping Whole Page Source GJG 1 2,101 Jan-13-2021, 03:19 PM
Last Post: GJG
  use Xpath in Python :: libxml2 for a page-to-page skip-setting apollo 2 3,580 Mar-19-2020, 06:13 PM
Last Post: apollo
  Python, youtube scraping. Jardz04 2 2,308 Dec-12-2019, 01:55 PM
Last Post: Jardz04
  Scraping next page of LinkedIn jobs RiteshMahto 6 6,287 Dec-09-2019, 09:43 PM
Last Post: Larz60+
  Scraping data from ebay seller page yuvalta 3 5,948 Sep-25-2019, 04:22 AM
Last Post: sandramoraes
  Web Page not opening while web scraping through python selenium sumandas89 4 10,000 Nov-19-2018, 02:47 PM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020