Python Forum
BeautifulSoup returning text as N/A
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
BeautifulSoup returning text as N/A
#1
I'm trying to get the text value from the below html, so I should be getting FFT606, but when I run my Python code I'm getting n/a. Please help.

<span>id="highlighted_callsign">FFT606</span>
from bs4 import BeautifulSoup
import requests

url = 'https://globe.adsbexchange.com/?icao=ad627d'

r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')
div = soup.findAll('div',attrs={'id':'layout_container'})
for d in div:
    s = (soup.find('span'))
    print(s.text)
Output:
n/a
Reply
#2
I tried with Selenium web driver, and I'm able to get text values on that, but I prefer to use BeautifulSoup.
Reply
#3
I made some changes to my code, at least now I can "see" what I need, but still not sure how to get the text values. The text I need is showing as n/a. Please see output.

from bs4 import BeautifulSoup
import requests
import lxml

url = 'https://globe.adsbexchange.com/?icao=a7d3d3'

r = requests.get(url)
soup = BeautifulSoup(r.content, 'lxml')
main_div = soup.find('div', attrs={'id': 'infoblock-container'})
div = main_div.findAll(class_='infoBlockSection')
for divs in div:
    infoData = divs.findAll('div', class_='infoData')
    print(infoData, '\n')
Output:
[] 

[] 

[] 

[<div class="infoData"><span id="selected_registration">n/a</span></div>, <div class="infoData">
<span id="selected_country" title="Country of registration">n/a</span>
</div>, <div class="infoData"><span id="selected_dbFlags">n/a</span></div>, <div class="infoData">
<span id="selected_squawk1"></span>
</div>] 

[] 

[] 

[<div class="infoData">
<span id="selected_vert_rate">n/a</span>
</div>, <div class="infoData">
<span id="selected_track1">n/a</span>
</div>, <div class="infoData">
<span id="selected_position">n/a</span>
</div>, <div class="infoData">
<span id="selected_sitedist2">n/a</span>
</div>] 

[<div class="infoData">
<span id="selected_source">n/a</span>
</div>, <div class="infoData">
<span id="selected_rssi1">n/a</span>
</div>, <div class="infoData">
<span id="selected_message_rate">n/a</span>
</div>, <div class="infoData" id="selected_message_count">
</div>, <div class="infoData">
<span id="selected_seen_pos">n/a</span>
</div>, <div class="infoData">
<span id="selected_seen">n/a</span>
</div>] 

[<div class="infoData">
<span id="selected_nav_altitude">n/a</span>
</div>, <div class="infoData">
<span id="selected_nav_heading">n/a</span>
</div>] 

[<div class="infoData">
<span id="selected_ws">n/a</span>
</div>, <div class="infoData">
<span id="selected_wd">n/a</span>
</div>, <div class="infoData">
<span id="selected_temp">n/a</span>
</div>] 

[<div class="infoData">
<span id="selected_speed2">n/a</span>
</div>, <div class="infoData">
<span id="selected_tas">n/a</span>
</div>, <div class="infoData">
<span id="selected_ias">n/a</span>
</div>, <div class="infoData">
<span id="selected_mach">n/a</span>
</div>] 

[<div class="infoData">
<span id="selected_altitude2"></span>
</div>, <div class="infoData">
<span id="selected_baro_rate">n/a</span>
</div>, <div class="infoData">
<span id="selected_altitude_geom">n/a</span>
</div>, <div class="infoData fourColumnSection4">
<span id="selected_geom_rate">n/a</span>
</div>, <div class="infoData">
<span id="selected_nav_qnh">n/a</span>
</div>] 

[<div class="infoData">
<span id="selected_track2">n/a</span>
</div>, <div class="infoData">
<span id="selected_true_heading">n/a</span>
</div>, <div class="infoData">
<span id="selected_mag_heading">n/a</span>
</div>, <div class="infoData">
<span id="selected_mag_declination">n/a</span>
</div>, <div class="infoData">
<span id="selected_trackrate">n/a</span>
</div>, <div class="infoData">
<span id="selected_roll">n/a</span>
</div>] 

[<div class="infoData">
<span id="selected_nav_modes">n/a</span>
</div>, <div class="infoData">
<span id="selected_version">n/a</span>
</div>, <div class="infoData">
<span id="selected_category">n/a</span>
</div>] 

[<div class="infoData">
<span id="selected_nac_p">n/a</span>
</div>, <div class="infoData">
<span id="selected_sil">n/a</span>
</div>, <div class="infoData">
<span id="selected_nac_v">n/a</span>
</div>, <div class="infoData">
<span id="selected_nic_baro">n/a</span>
</div>, <div class="infoData">
<span id="selected_rc">n/a</span>
</div>] 
Reply
#4
(Sep-08-2021, 11:27 PM)tantony Wrote: tried with Selenium web driver, and I'm able to get text values on that, but I prefer to use BeautifulSoup.
As this site use a lot JavaScript,Selenium is the easiest way can still use BS as can pass page source to it.
Also not loading Browser helps with --headless.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import time

#--| Setup
options = Options()
options.add_argument("--headless")
browser = webdriver.Chrome(executable_path=r'C:\cmder\bin\chromedriver.exe', options=options)
#--| Parse or automation
url = 'https://globe.adsbexchange.com/?icao=a7d3d3'
browser.get(url)
soup = BeautifulSoup(browser.page_source, 'lxml')
time.sleep(3)
main_div = soup.find('div', attrs={'id': 'infoblock-container'})
div = main_div.findAll(class_='infoBlockSection')
for divs in div:
    infoData = divs.findAll('div', class_='infoData')
    print(infoData, '\n')
Output:
[<div class="infoData"><span id="selected_registration">N603QS</span></div>, <div class="infoData"> <span id="selected_country" title="Country of registration">United States</span> </div>, <div class="infoData"><span id="selected_dbFlags">none</span></div>, <div class="infoData"> <span id="selected_squawk1">3764</span> </div>] [] [] [<div class="infoData"> <span id="selected_vert_rate">n/a</span> </div>, <div class="infoData"> <span id="selected_track1">n/a</span> </div>, <div class="infoData"> <span id="selected_position">41.075°, -73.709°</span> </div>, <div class="infoData"> <span id="selected_sitedist2">n/a</span> </div>] .....
Reply
#5
Thank you, the reason I want to use BS4 is because wouldn't using Selenium use memory? What does not running in headless mode do?
Reply
#6
It will use some more memory,but it's not much so should not make any problems.
Not running headless it will load a visible browser,using headless feel just like running BS just a little more running time.
Reply
#7
(Sep-09-2021, 12:56 PM)snippsat Wrote: It will use some more memory,but it's not much so should not make any problems.
Not running headless it will load a visible browser,using headless feel just like running BS just a little more running time.

Ok thank you so much, now I can move on to the next portion of my project
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Python BeautifulSoup gives unusable text? dggo666 0 1,405 Oct-29-2021, 05:12 AM
Last Post: dggo666
  Python 3.9 : BeautifulSoup: 'NoneType' object has no attribute 'text' fudgemasterultra 1 8,815 Mar-03-2021, 09:40 AM
Last Post: Larz60+
  Selenium returning web element instead of desired text newbie_programmer 1 5,142 Dec-11-2019, 06:37 AM
Last Post: Malt
  BeautifulSoup 'NoneType' object has no attribute 'text' bmccollum 9 14,505 Sep-14-2018, 12:56 PM
Last Post: bmccollum

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020