The following will show all data available by calling show_detail
and then get the item of interest.
The show detail explains how the index or tr[6] and td[0] and td[1] were determined.
I use requests which is better than urlopen
The css_select value is obtained in the browser (I use firefox),
and then get the item of interest.
The show detail explains how the index or tr[6] and td[0] and td[1] were determined.
I use requests which is better than urlopen
The css_select value is obtained in the browser (I use firefox),
- place cursor over item of interest,
- right click selected text and choose inspect element
- in inspect window, move cursor over <table tag
- right click
- select copy
- select css selector
- paste to code soup.select(paste here)
from bs4 import BeautifulSoup import os import requests import NewPrettifyPage import sys class Weather: def __init__(self): self.pp = NewPrettifyPage.PrettifyPage() def show_detail(self, trs): for n, tr in enumerate(trs): tds = tr.find_all('td') for n1, td in enumerate(tds): print(f"\n--------------------- tr_{n}, td_{n1} ---------------------") print(f"{td}\ntext: {td.text.strip()}") def scrape_weather_info(self, metar_link): response = requests.get(metar_link) if response.status_code == 200: soup = BeautifulSoup(response.content, 'lxml') table = soup.select('#awc_main_content_wrap > table:nth-child(3)')[0] trs = table.find_all('tr') self.show_detail(trs) item_of_interest = trs[6] tds = item_of_interest.find_all('td') print(f"\nitem_of_interest: {tds[0].text.strip()} {tds[1].text.strip()}") if __name__ == '__main__': os.chdir(os.path.abspath(os.path.dirname(__file__))) sw = Weather() sw.scrape_weather_info('https://www.aviationweather.gov/metar/data?ids=kbwi&format=decoded&date=&hours=0')output:
Output:--------------------- tr_0, td_0 ---------------------
<td align="right" width="130px"><span style="color: #3333CC; font-weight: bold">METAR for:</span></td>
text: METAR for:
--------------------- tr_0, td_1 ---------------------
<td>KBWI (Baltimore-Washington, MD, US) </td>
text: KBWI (Baltimore-Washington, MD, US)
--------------------- tr_1, td_0 ---------------------
<td align="right" valign="top"><span style="color: #9999CC; font-weight: bold">Text:</span></td>
text: Text:
--------------------- tr_1, td_1 ---------------------
<td style="background-color: #CCCCCC; font-weight: bold">KBWI 301454Z 12008KT 10SM SCT020 OVC042 23/16 A3029 RMK AO2 SLP255 SCT020 V BKN T02280161 53008</td>
text: KBWI 301454Z 12008KT 10SM SCT020 OVC042 23/16 A3029 RMK AO2 SLP255 SCT020 V BKN T02280161 53008
--------------------- tr_2, td_0 ---------------------
<td align="right"><span style="color: #9999CC; font-weight: bold">Temperature:</span></td>
text: Temperature:
--------------------- tr_2, td_1 ---------------------
<td> 22.8°C ( 73°F)</td>
text: 22.8°C ( 73°F)
--------------------- tr_3, td_0 ---------------------
<td align="right"><span style="color: #9999CC; font-weight: bold">Dewpoint:</span></td>
text: Dewpoint:
--------------------- tr_3, td_1 ---------------------
<td> 16.1°C ( 61°F) [RH = 66%]</td>
text: 16.1°C ( 61°F) [RH = 66%]
--------------------- tr_4, td_0 ---------------------
<td align="right"><span style="color: #9999CC; font-weight: bold">Pressure (altimeter):</span></td>
text: Pressure (altimeter):
--------------------- tr_4, td_1 ---------------------
<td>30.29 inches Hg (1025.8 mb) [Sea level pressure: 1025.5 mb]</td>
text: 30.29 inches Hg (1025.8 mb) [Sea level pressure: 1025.5 mb]
--------------------- tr_5, td_0 ---------------------
<td align="right"><span style="color: #9999CC; font-weight: bold">Winds:</span></td>
text: Winds:
--------------------- tr_5, td_1 ---------------------
<td>from the ESE (120 degrees) at 9 MPH (8 knots; 4.1 m/s)</td>
text: from the ESE (120 degrees) at 9 MPH (8 knots; 4.1 m/s)
--------------------- tr_6, td_0 ---------------------
<td align="right"><span style="color: #9999CC; font-weight: bold">Visibility:</span></td>
text: Visibility:
--------------------- tr_6, td_1 ---------------------
<td>10 or more sm (16+ km)</td>
text: 10 or more sm (16+ km)
--------------------- tr_7, td_0 ---------------------
<td align="right"><span style="color: #9999CC; font-weight: bold">Ceiling:</span></td>
text: Ceiling:
--------------------- tr_7, td_1 ---------------------
<td>4200 feet AGL</td>
text: 4200 feet AGL
--------------------- tr_8, td_0 ---------------------
<td align="right" valign="top"><span style="color: #9999CC; font-weight: bold">Clouds:</span></td>
text: Clouds:
--------------------- tr_8, td_1 ---------------------
<td> scattered clouds at 2000 feet AGL, overcast cloud deck at 4200 feet AGL</td>
text: scattered clouds at 2000 feet AGL, overcast cloud deck at 4200 feet AGL
item_of_interest: Visibility: 10 or more sm (16+ km)