Jul-11-2020, 02:43 PM
@HarleyQuin
This is a very clever approach, I will probably be using preset headers from now. I would probably never even consider effects websites might have on "robot" users.
@snippsat
Your code works well and I can definitely continue from here. Frankly, I have no idea what was different on my attempt this time, since using requests.get() and B.S. gave good results on my first try.
Given your examples with B.S. and Selenium, can Selenium replace B.S. entirely for use with scraping/navigating websites? In that case, I will stick to Selenium down the road, to avoid overhead and invest in learning one tool well instead.
This is a very clever approach, I will probably be using preset headers from now. I would probably never even consider effects websites might have on "robot" users.
@snippsat
Your code works well and I can definitely continue from here. Frankly, I have no idea what was different on my attempt this time, since using requests.get() and B.S. gave good results on my first try.
Given your examples with B.S. and Selenium, can Selenium replace B.S. entirely for use with scraping/navigating websites? In that case, I will stick to Selenium down the road, to avoid overhead and invest in learning one tool well instead.
