Python Forum
Selenium - "Element is not currently visible and may not be manipulated"
Thread Rating:
  • 2 Vote(s) - 2 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Selenium - "Element is not currently visible and may not be manipulated"
#1
Hi,

I have been working on the following code to scrape data from the website for my research. However, I got the following error message. "errorMessage":"Element is not currently visible and may not be manipulated"

When I try out each line of the code on the Python interpreter, they would work fine. However, when I save the code as scrapeEcoData_weekly.py and run the file, that's when I get the error message.

from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.webdriver.common.by import By
import time

driver = webdriver.PhantomJS(executable_path='C:/Users/AppData/Local/Programs/Python/Python35-32/myScript/Misc/PhantomJS')
driver.get("https://www.dailyfx.com/calendar")

weeklyView = driver.find_element_by_xpath('//*[@class="btn btn-default webinar-hover dfx-has-spinner hidden-xs"]')
weeklyView.click()
time.sleep(10)
pageSource = driver.page_source
bsObj = BeautifulSoup(pageSource, "html.parser")
tblTag = bsObj.findAll("table", {"class":"table dfx-calendar-table tab-pane fade in active "})[0]
tRows = tblTag.findAll("tr")
for index, item in enumerate(tRows):
print("index: ", index, " ", "item: ", item.get_text())
I tried to locate the "Weekly View" button element using the code instead. The codes would run fine. I took a screenshot of the website after weeklyView.click(), the weekly page didn't show up. It was still showing the daily page as if the button was never clicked on.

weeklyView = driver.find_element_by_xpath('//*[contains(., "Weekly View")]
However, if someone could help me with this.
Thank you!
Reply
#2
first, I use python's builtin urllib.request to get a page:
Here's a module you can take code from, or save it as GetUrl.py
and just import into your program

# GetUrl - Fetch files or web pages from internet
#
# Author: Larry McCaig (Larz60+)
import urllib.request as ur
import os
from time import sleep
import sys


class GetUrl:
    def __init__(self, returndata=False):
        self.returndata = returndata

    def get_url(self, url, ret=False, tofile=None, bin=False):
        rdata = None
        head, tail = os.path.split(url)
        try:
            if tofile:
                if os.path.exists(tofile):
                    os.remove(tofile)
                if bin:
                    with open(tofile, 'wb') as f:
                        rdata = ur.urlopen(url).read()
                        f.write(rdata)
                else:
                    with open(tofile, 'w') as f:
                        rdata = ur.urlopen(url).read().decode('utf8')
                        f.write(rdata)
                sleep(.5)
            if ret:
                return rdata
        except:
            print("Unexpected error:", sys.exc_info()[0])
I don't know which browser you use, mine is firefox, so I'll give instructions on how I'd go about debugging the error

In firefox, select tools->Web Developer->Page Source
The page html will come up
save it locally to a file (file->save as->filename.html

In your code, instead of fetching the page, open it and read into 'pageSource'
    with open(filename.html, 'r') as f:
        pagesource = f.read()
Now you can play around with it as much as you like without reloading each time.
You can also load the html file into an editor so you can manually search for variables.

Not an immediate fix for your problem, but perhaps will help with debugging
Reply
#3
(Oct-26-2016, 11:41 PM)Larz60+ Wrote: Now you can play around with it as much as you like without reloading each time.
You can also load the html file into an editor so you can manually search for variables.

Not an immediate fix for your problem, but perhaps will help with debugging
Little on wrong track here Lars60+,
because this site use JavaScript heavily and he shall also push a button(which load ouical.js).
The problem is that urllib(or Requests as you should have used) can not get this info.

Quote: It was still showing the daily page as if the button was never clicked on.
 FireFox web driver has a click() method,but when you load page trough PhantomJS(has no click() method) has to use other tool.
jQuery selector can be used and execute bye PhantomJS.

You can try this.
#weeklyView = driver.find_element_by_xpath('//*[@class="btn btn-default webinar-hover dfx-has-spinner hidden-xs"]')
#weeklyView.click()
driver.execute_script("$('#calendarbody > div:nth-child(12) > div.col-sm-2.calendar-home-weekly > button').click()")
Your indention is wrong in last two lines but this i guess is a copy fault.
The default path that 3.5 use is horrible,i would choose C:\Python35 so same as i had for 10-year ago C:\Python25 Shh python
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Selenium suddenly fails to find element Pavel_47 3 6,197 Sep-04-2022, 11:06 AM
Last Post: Pavel_47
  Clicking on element not triggering event in Selenium Python (Event Key is not in data dkaeloredo 2 4,230 Feb-16-2020, 05:50 AM
Last Post: dkaeloredo
  Selenium locating an element. JokerTux 3 2,637 Dec-28-2019, 08:50 AM
Last Post: snippsat
  Selenium returning web element instead of desired text newbie_programmer 1 5,139 Dec-11-2019, 06:37 AM
Last Post: Malt
  How to get visible text of Select Drop down ankitjindalbti 2 2,709 Jun-03-2019, 12:35 AM
Last Post: ankitjindalbti
  Python Selenium getting table element trengan 2 8,504 Dec-31-2018, 03:02 PM
Last Post: trengan
  Click Element if displayed using Selenium and Python giaco__mar 1 3,485 Dec-27-2018, 06:19 PM
Last Post: metulburr
  Selenium stale element reference test 1 2,737 Sep-19-2018, 10:19 PM
Last Post: test
  Simple Element Check Code in Selenium Not Working digitalmatic7 1 2,978 Feb-18-2018, 06:53 AM
Last Post: metulburr
  Selenium - Googlemaps element not visible Barnettch3 3 5,570 Jan-15-2018, 08:07 PM
Last Post: Barnettch3

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020