Python Forum
Web scraping cookie in URL blocks selenium
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Web scraping cookie in URL blocks selenium
#1
Hey guys,

i want to scrape data on a one-page website (just one URL). I am pretty new to programming. I am sorry for that ;)
So i guess it is a website generated the whole data via Java. I want to scrape some data from the Website. So navigation to reach my wanted page to scrape is already my problem.

I worked with selenium, now i stuck that the website is adding a cookie to my URL and blocks further search.
it adds: jsessionid=89F10E908FEB575216C17BE0432E19B9 to the URL.
How can i block the website form that? Or how can i remove it without changing my website?
If i refresh is still in the URL, if i load a new page (trigger search again, its gone!
Or else, how do i have to name the button to be able to click it now with selenium?
Currently, its: ( driver.find_element(By.CLASS_NAME, 'search').click() )

I am using selenium, because i have to interact with a Java button in text (There is Text in a table sheet to click. Than you see the page with data)
<button class="linkBeg" name="showBeg" type="submit" value="0d7fb682-e537-4732-8441-7fc50d9a3a6c" title="Detailansicht">
Bauer, Oliver </button>
but i can't manage to "click" or trigger that button with MechanicalSoup. If there is a way to do it with Scrapy or BS/MechaniclSoup i would be also happy to know.

Many thanks in advance!
Joerg
Reply
#2
Don't use find_by_class use the find_by_xpath or find_by_selector options. You can also clear cookies or use selenium in "incognito" mode Difficult to give good advice without an example code though
Reply
#3
Thanks for your answer,

I looked around and the issue was i didn't paused my code, so it had been to many requests to quickly.
I did set it on rest one second after each request, and now it's all working.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Scraping div tags with selenium, need help hfakoor2 1 1,032 Mar-12-2023, 08:31 AM
Last Post: hfakoor2
  Django Two blocks of dynamic content on one page iFunKtion 5 4,340 Jul-04-2019, 02:31 AM
Last Post: noisefloor
  Web scraping (selenium (i think)) Larz60+ 10 6,140 Jan-27-2019, 02:57 AM
Last Post: Larz60+
  Web Page not opening while web scraping through python selenium sumandas89 4 10,004 Nov-19-2018, 02:47 PM
Last Post: snippsat
  web scraping with selenium and bs4 Prince_Bhatia 2 3,721 Sep-18-2018, 10:59 AM
Last Post: Prince_Bhatia
  scraping javascript websites with selenium DoctorEvil 1 3,315 Jun-08-2018, 06:40 PM
Last Post: DoctorEvil
  Combining selenium and beautifulsoup for web scraping sumandas89 3 11,581 Jan-30-2018, 02:14 PM
Last Post: metulburr
  web scraping using selenium sumandas89 3 3,542 Jan-05-2018, 01:45 PM
Last Post: metulburr
  Error in Selenium: CRITICAL:root:Selenium module is not installed...Exiting program. AcszE 1 3,587 Nov-03-2017, 08:41 PM
Last Post: metulburr
  cookie operation x64 1 4,343 Oct-10-2016, 07:05 PM
Last Post: micseydel

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020