Python code for exporting table using Selenium

[gj31980]

I am looking to download the tables in this URL for different selections like Crude, Oil etc using Python but I am a beginner and I tried using Selenium to get the table but I don't know how to export this format to a file like excel or csv. I am not even sure if this is the correct approach.

http://www.jodidb.org/TableViewer/tableV...rtId=93904

Below is my code so far

1 2 3 4 5 6 7 8

from selenium import webdriver import chromedriver_binary import pandas as pd driver = webdriver.Chrome()   driver.get('http://www.jodidb.org/TableViewer/tableView.aspx?ReportId=93904')       elem = driver.find_element_by_class_name('ExplorerContentPane').click()

Any help is appreciated.

[Larz60+]

you are missing a lot of code.
you need to set options, start selenium, do your work and then stop selenium.
start with the following two tutorials (which will only take a short time to complete)
web scraping part 1
web scraping part 2 (this is main one for selenium)

[gj31980]

Thank you for the links. They were useful. I was able to simulate most of the filters I need to change using Selenium but unfortunately the final step involves clicking on download to excel or csv and that link opens a new window. Not sure how to go about handling that. I tried find_element_by_xpath on the download but that doesnt open the new tab. Please help.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67

from selenium import webdriver from selenium.webdriver.common.keys import Keys from selenium.webdriver.common.action_chains import ActionChains from selenium.webdriver.chrome.options import Options from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC import time from bs4 import BeautifulSoup import requests import pandas as pd        #browser = webdriver.Chrome() #--| Setup chrome_options = Options() #chrome_options.add_argument("--headless") chrome_options.add_argument('--disable-gpu') chrome_options.add_argument('--log-level=3') browser = webdriver.Chrome(chrome_options=chrome_options) # chrome_options.add_argument("headless") # browser = webdriver.Chrome(options=chrome_options)   browser.get('http://www.jodidb.org/TableViewer/tableView.aspx?ReportId=93904')     wait = WebDriverWait(browser, 30)     elem = wait.until(EC.element_to_be_clickable((By.ID, 'DnDTmp')))   elem.click() #SelectAll(0) elem_selectAllNone = browser.find_element_by_xpath('//*[@id="w_itemSelContainer"]/table[2]/tbody/tr[2]/td[3]') elem_selectAllNone.click()   elem_selectcheck = wait.until(EC.element_to_be_clickable((By.ID, 'SelectRange')))  elem_selectcheck.click()   elem_startdate = wait.until(EC.element_to_be_clickable((By.ID, 'WD_DateLower')))  elem_startdate.send_keys('12/2019')   elem_enddate = browser.find_element_by_id('WD_DateUpper') elem_enddate.send_keys('05/2020')   elem_search = browser.find_element_by_xpath('//*[@id="w_itemSelContainer"]/table[1]/tbody/tr/td[2]/input[1]') elem_search.click()     elem_SelectAllAll = browser.find_element_by_xpath('//*[@id="w_itemSelContainer"]/table[2]/tbody/tr[1]/td[3]/a')   # driver.execute_script("arguments[0].click();", element) elem_SelectAllAll.click()   elem_showtbl = browser.find_element_by_xpath('//*[@id="cell4"]/table[2]/tbody/tr[1]/td[4]/input') elem_showtbl.click()   # Save the window opener (current window)   elem_menudwnload = browser.find_element_by_xpath('//*[@id="MenuCell_Download"]') elem_menudwnload.click()       #time.sleep(30) #seconds browser.close()

[metulburr]

My post in the tutorial 2 explains a little about switching to new tabs.

but unfortunately the final step involves clicking on download to excel or csv and that link opens a new window

When i run your code and select download csv file it opens a new tab, not a new window. When a new tab opens you can run

1	browser.switch_to.window(browser.window_handles[1])

to switch to the next tab and select download from it.

As for getting the xpath, i was able to get it from expanding the html and reselecting inspect to again get the correct xpath.

Put this at the end of your script and it downloads for me

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

# Save the window opener (current window)    elem_menudwnload = browser.find_element_by_xpath('//*[@id="MenuCell_Download"]') elem_menudwnload.click()    drop_down_download = browser.find_element_by_xpath('//*[@id="ActDiv"]/table/tbody/tr/td[5]/a/img') drop_down_download.click() csv = browser.find_element_by_xpath('//*[@id="MenuCell_DownloadDiv"]/p[3]/nobr/a') csv.click() browser.switch_to.window(browser.window_handles[1]) wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="cell4"]/input[1]'))) download = browser.find_element_by_xpath('//*[@id="cell4"]/input[1]') download.click()     time.sleep(3000) #seconds browser.close()

Im assuming you have enough knowledge to modify it enough to change to excel if you want that instead.

[gj31980]

Thank you so much for your help!!! That worked perfectly.
I was not able to get below by inspecting myself but I will figure it out.

1 2	drop_down_download = browser.find_element_by_xpath('//*[@id="ActDiv"]/table/tbody/tr/td[5]/a/img') drop_down_download.click()