Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Simple newbie Q
#1
I'm new to python so hope this is really simple.

import bs4 as bs
import urllib .request

saurce = urllib .request.urlopen("https://www.premierleague.com/fixtures").read()
soup = bs.BeautifulSoup(saurce,'lxml')

print(soup.get_text())

I thought this returns all the text on the page. However it doesn't, it returns the text in the header and foot of the page. Much different than following the video using a different page. What should I be doig differently to get the text in the main part of the page?

Many thanks
Quote
#2
Here's a super simple example using requests (use instead of urllib2)

to get requests package:
pip install requests
import requests
from bs4 import BeautifulSoup
response = requests.get('https://www.premierleague.com/fixtures')
if response.status_code == 200:
    soup = BeautifulSoup(response.text, 'lxml')
    print(soup)
else:
    print('Problem downloading status code: {}'.format(response.status_code))
Quote
#3
Probably the page content is generated by JS. So urllib can't do anything here. Test the script with a webpage that is static. This one for example: https://fishshell.com/docs/current/tutorial.html
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Quote
#4
Thanks. The code didn't do the trick so will explode the link provided.
Quote
#5
I tried it by myself. It's difficult because the page is rendered by JavaScript and the CSS-Selector I tried, did not bring any results.

There is a lib, which is using BeautifulSoap, but with a better API: https://github.com/kennethreitz/requests-html
There you have the method html.render() which should start a hidden Chronium instance in the background, to render the JavaScripts.
My code examples are always for Python >=3.6.0
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Newbie, looking for advices. stambus 2 573 Sep-03-2018, 06:52 AM
Last Post: KirkmanJ
  Regarding guidance for a Newbie in Python and Django ravindersingh13 3 1,655 May-20-2017, 07:59 PM
Last Post: micseydel
  General Python code for Newbie Ramakrishna 22 5,021 Apr-29-2017, 02:34 PM
Last Post: buran
  Newbie Doubt arshidkv12 8 2,191 Mar-20-2017, 06:33 AM
Last Post: wavic

Forum Jump:


Users browsing this thread: 1 Guest(s)