(Sep-08-2021, 11:27 PM)tantony Wrote: tried with Selenium web driver, and I'm able to get text values on that, but I prefer to use BeautifulSoup.As this site use a lot JavaScript,Selenium is the easiest way can still use BS as can pass page source to it.
Also not loading Browser helps with
--headless
.from selenium import webdriver from selenium.webdriver.chrome.options import Options from selenium.webdriver.common.keys import Keys from bs4 import BeautifulSoup import time #--| Setup options = Options() options.add_argument("--headless") browser = webdriver.Chrome(executable_path=r'C:\cmder\bin\chromedriver.exe', options=options) #--| Parse or automation url = 'https://globe.adsbexchange.com/?icao=a7d3d3' browser.get(url) soup = BeautifulSoup(browser.page_source, 'lxml') time.sleep(3) main_div = soup.find('div', attrs={'id': 'infoblock-container'}) div = main_div.findAll(class_='infoBlockSection') for divs in div: infoData = divs.findAll('div', class_='infoData') print(infoData, '\n')
Output:[<div class="infoData"><span id="selected_registration">N603QS</span></div>, <div class="infoData">
<span id="selected_country" title="Country of registration">United States</span>
</div>, <div class="infoData"><span id="selected_dbFlags">none</span></div>, <div class="infoData">
<span id="selected_squawk1">3764</span>
</div>]
[]
[]
[<div class="infoData">
<span id="selected_vert_rate">n/a</span>
</div>, <div class="infoData">
<span id="selected_track1">n/a</span>
</div>, <div class="infoData">
<span id="selected_position">41.075°, -73.709°</span>
</div>, <div class="infoData">
<span id="selected_sitedist2">n/a</span>
</div>]
.....