Python Forum
Help Scraping links and table from link
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Help Scraping links and table from link
#7
Some tips you should not have open file in the loop,same for *https://www.sbostats.com which is a value that don't change.
So a example like this i use with open(close file object automatic).
from bs4 import BeautifulSoup
from bs4.dammit import EncodingDetector
import requests

parser = 'html.parser'  # or 'lxml' (preferred) or 'html5lib', if installed
resp = requests.get("https://www.sbostats.com/soccer/league/italy/serie-a")
http_encoding = resp.encoding if 'charset' in resp.headers.get('content-type', '').lower() else None
html_encoding = EncodingDetector.find_declared_encoding(resp.content, is_html=True)
encoding = html_encoding or http_encoding
soup = BeautifulSoup(resp.content, parser, from_encoding=encoding)
table = soup.find_all('table',attrs={'class':'updated_next_results_table'})

table = table[0]
tr = table.find_all('tr')
base_url = '*https://www.sbostats.com'
with open('matches.txt', 'a') as fp:
    for row in tr:
        if row.text == None:
            pass
        if row.find('a') == None:
            pass
        else:
            #print(' '.join(row.text.replace('STATS', '-').split()[:3]))
            #print(f"{base_url}{row.find('a')['href']}\n")
            fp.write(f"{' '.join(row.text.replace('STATS', '-').split()[:3])}\n")
            fp.write(f"{base_url}{row.find('a')['href']}\n\n")
Output:
Verona - Napoli *https://www.sbostats.com/soccer/stats?country=italy&league=serie-a"e=1.50&direction=away&id=NDAxMTg3OA== Torino - Inter *https://www.sbostats.com/soccer/stats?country=italy&league=serie-a"e=1.83&direction=away&id=NDAxMTg3OQ== Sassuolo - Lazio *https://www.sbostats.com/soccer/stats?country=italy&league=serie-a"e=2.30&direction=away&id=NDAxMTg4MA== .....
I like output better like this,but you can just change to have all one line as in your example.

Quote:i edited the link for my needs i have only to understand how to remove all numbers of odds
Also i guess that you have tested all this in the loop,this how i testet only one value(interactive interpret) then added to loop.
>>> tr[2]
<tr> <td class="widget-results__team-details ovf updated_m130"> <span class="widget-results__team-name match-name" data-original-title="Verona" data-placement="bottom" data-toggle="tooltip">Verona</span> </td> <td class="widget-results__score text-center limitstats"> <a class="btn btn-primary btn-xs" href='/soccer/stats?country=italy&amp;league=serie-a"e=1.50&amp;direction=away&amp;id=NDAxMTg3OA=='>STATS</a> </td> <td class="widget-results__team-details ovf updated_m130 text-right"> <div class="row"> <div class="col-sm-3"> </div> <div class="col-sm-9"> <span class="widget-results__team-name match-name" data-original-title="Napoli" data-placement="bottom" data-toggle="tooltip"> Napoli </span> </div> </div> </td> <td class="widget-results__quote"> <span class="" style="">6.50</span> </td> <td class="widget-results__quote"> <span class="">4.00</span> </td> <td class="widget-results__quote"> <span class="match_fav" style="">1.50</span> </td> </tr>
>>> 
>>> ' '.join(tr[2].text.replace('STATS', '-').split())
'Fiorentina - Empoli 1.44 4.33 7.50'
>>> # Remove odds
>>> ' '.join(tr[2].text.replace('STATS', '-').split()[:3])
'Verona - Napoli'
Reply


Messages In This Thread
RE: Help Scraping links and table from link - by snippsat - Oct-10-2023, 11:33 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Scraping data from table into existing dataframe vincer58 1 3,164 Jan-09-2022, 05:15 PM
Last Post: vincer58
  Need help scraping wikipedia table bborusz2 6 4,901 Dec-01-2020, 11:31 PM
Last Post: snippsat
  Web Scraping Inquiry (Extracting content from a table in asubdomain) DustinKlent 3 4,863 Aug-17-2020, 10:10 AM
Last Post: snippsat
  Scraping a dynamic data-table in python through AJAX request filozofo 1 5,080 Aug-14-2020, 10:13 AM
Last Post: kashcode
  scraping multiple pages from table bandar 1 3,510 Jun-27-2020, 10:43 PM
Last Post: Larz60+
  get link and link text from table metulburr 5 8,350 Jun-13-2019, 07:50 PM
Last Post: snippsat
  webscrapping links and then enter those links to scrape data kirito85 2 4,350 Jun-13-2019, 02:23 AM
Last Post: kirito85
  Error while scraping links with beautiful soup mgtheboss 4 10,291 Dec-22-2017, 12:41 PM
Last Post: mgtheboss
  Web scraping "fancy" table acehole60 2 5,890 Dec-16-2016, 09:17 AM
Last Post: acehole60

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020