Python Forum
Suggestion request for scrapping html table
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Suggestion request for scrapping html table
#1
Hi,

I am struggling to scrap a html table.
When the date is selected if there is data, the table will be displayed, i am scrapping it using the xpath
table I'd and no issues with this.

But if the selected date has no data, no table will be displayed. Here my code stops and waiting for the table to appear.

How to specify timeout and continue without exceptions.
Reply
#2
Please show latest code (working or not).
Reply
#3
----snip-----
driver.get("https://website/list.php");

num_rows = 0
num_cols = 0
num_rows = len(driver.find_elements(By.XPATH,'/html/body/form/div[4]/div/table/tbody/tr'))
num_cols = len(driver.find_elements(By.XPATH, '//*[@id="gv_output"]/tbody/tr[2]/td'))
print("Num rows is {0} , num_col is {1}" .format(num_rows, num_cols))
table_output_reading = []
if num_rows > 1:
    for i in range (2,num_rows):
        table_zone = driver.find_element(By.XPATH, "/html/body/form/div[4]/div/table/tbody/tr["+str(i)+"]/td[3]").text
        table_product = driver.find_element(By.XPATH, "/html/body/form/div[4]/div/table/tbody/tr["+str(i)+"]/td[4]").text
        table_price = driver.find_element(By.XPATH, "/html/body/form/div[4]/div/table/tbody/tr["+str(i)+"]/td[5]").text
        table_output_reading.append(str(table_zone)+str(table_product)+str(table_price))
        print("table lists is", table_output_reading)

--snip----
This works fine if there is a table listed.

Suppose when i select another date, and on that date if there is no data no html table is displayed.
and this script here waits endlessly for the table
i tried giving num_rows >1 not working
The table id is gv_output

I want this script to run only if table is present

thanks in advance
Reply
#4
What do the error traceback look like?
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Problem with scrapping Website giddyhead 1 1,533 Mar-08-2024, 08:20 AM
Last Post: AhanaSharma
  python web scrapping mg24 1 226 Mar-01-2024, 09:48 PM
Last Post: snippsat
  How can I ignore empty fields when scrapping never5000 0 1,339 Feb-11-2022, 09:19 AM
Last Post: never5000
  web scrapping through Python Naheed 2 2,559 May-17-2021, 12:02 PM
Last Post: Naheed
  Website scrapping and download santoshrane 3 4,234 Apr-14-2021, 07:22 AM
Last Post: kashcode
  HTML multi select HTML listbox with Flask/Python rfeyer 0 4,483 Mar-14-2021, 12:23 PM
Last Post: rfeyer
  Using Python request without selenium on html form with javascript onclick submit but eraosa 0 3,120 Jan-09-2021, 06:08 PM
Last Post: eraosa
  Newbie help with lxml scrapping chelsealoa 1 1,813 Jan-08-2021, 09:14 AM
Last Post: Larz60+
  Scrapping Sport score laplacea 1 2,208 Dec-13-2020, 04:09 PM
Last Post: Larz60+
  How to export to csv the output of every iteration when scrapping with a loop efthymios 2 2,227 Nov-30-2020, 07:46 PM
Last Post: efthymios

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020