Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 page impossible to scrap? :O
#1
Hi guys,

i have found page where i cant scrap the shortest flight

https://www.skyscanner.net/transport/fli...ref=home#/

how it come? it is possible to block site from scrapping?

page = 'https://www.skyscanner.net/transport/flights/mpm/tyoa/191008/191015/?adults=1&children=0&adultsv2=1&childrenv2=&infants=0&cabinclass=economy&rtn=1&preferdirects=false&outboundaltsenabled=false&inboundaltsenabled=false&ref=home#/'
r = requests.get(page)
content = (r.text)
soup = BeautifulSoup(content, 'html.parser')
test = soup.find_all(class_='BpkTicket_bpk-ticket__paper__2gPSe BpkTicket_bpk-ticket__main__J31fH BpkTicket_bpk-ticket__main--padded__WIbjx BpkTicket_bpk-ticket__main--horizontal__2MgwA BpkTicket_bpk-ticket__paper--with-notches__19yQc'):
print(test)
I guess flight seeker sites works with some kind of refresh data, hence, its not visible in requests? am i right? In this case i would need some sleep/wait function, right?
Quote
#2
If they use JavaScript you may need to use Selenium
Check our tutorial - https://python-forum.io/Thread-Web-scraping-part-2
look for God dammit JavaScript, why do i not get all content and next
zarize likes this post
Quote
#3
Thanks buran,

as always helpful! :)

now time to learn captcha solving! :D
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  use Xpath in Python :: libxml2 for a page-to-page skip-setting apollo 2 372 Mar-19-2020, 06:13 PM
Last Post: apollo
  Scrap a dynamic span hefaz 0 577 Mar-07-2020, 02:56 PM
Last Post: hefaz
  scrap by defining 3 functions zarize 0 260 Feb-18-2020, 03:55 PM
Last Post: zarize
  Skipping anti-scrap zarize 0 309 Jan-17-2020, 11:51 AM
Last Post: zarize
  Cannot get selenium to scrap past the first two pages newbie_programmer 0 959 Dec-12-2019, 06:19 AM
Last Post: newbie_programmer
  Scrap data from not standarized page? zarize 4 600 Nov-25-2019, 10:25 AM
Last Post: zarize
  Scrap a value from website harsush 1 379 Aug-29-2019, 01:57 PM
Last Post: snippsat
  Scrap text out of td table from URLS Gochix2020 4 1,267 Aug-03-2019, 02:56 AM
Last Post: Larz60+
  scrap macrotrends mr_byte31 7 2,017 Aug-02-2019, 12:02 AM
Last Post: mr_byte31
  Scrap arbitrage odds -help Gochix2020 3 637 Jul-31-2019, 10:45 AM
Last Post: Gochix2020

Forum Jump:


Users browsing this thread: 1 Guest(s)