Strange BS4 Scraping Issue

[digitalmatic7]

1 2 3 4 5 6 7 8 9 10 11 12

from bs4 import BeautifulSoup import re import requests   url = "http://www.foxnews.com/politics/2018/01/14/trump-says-newspaper-misquoted-him-on-kim-jung-un-points-to-audio-tape.html" scrape = requests.get(url, headers={"user-agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"}) html = scrape.content soup = BeautifulSoup(html, 'html.parser')   z = soup.find_all(text=re.compile(r"livefyre_comment_stream", re.IGNORECASE))   print(z)

I'm having this strange problem where BS4 isn't showing any results..

If you open the URL manually and check the source code you can find "livefyre_comment_stream"

Can someone explain where I'm going wrong?

---

Solved.

1	id=re.compile

[wavic]

What is "livefyre_comment_stream"? A class, an id?

The text attribute is to search in the text of the already rendered webpage. What you see in the browser.

If "livefyre_comment_stream" is a class for example:

1	z = soup.sellect(".livefyre_comment_stream")

z is not descriptive variable name.