Python Forum
'For' command and filling excel table with pandas by doing web-scrabbing
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
'For' command and filling excel table with pandas by doing web-scrabbing
#1
With selenium library, I can open and read some information from a website, but while transferring these data read to an excel table, I can't fill all table with 'for command' because at the end of 'for command' and after saving data frame as 'csv', I see, for the only last i number, the data was put in processed, for other 'i's, filling the table was skipped. I don't know the reason for that.

How can I fill table for all i numbers?

My script is:
from selenium import webdriver
import pandas as pd

inpExcelFile = 'sites of svbilgi kizilirmak.csv'
websites = pd.read_csv(inpExcelFile)
#csvfile can be downloaded from 'https://www.evernote.com/shard/s355/sh/3ba4c779-803c-4f5a-b402-a30ec973c48e/75f2c9aea94454e1'

driver = webdriver.Chrome(executable_path=r'D:\chromedriver.exe')

for i in range(1,3): 
    driver.get(str(websites['sitelist'][i]))
    code_and_name=driver.find_element_by_id("Span1").text
    approximate_elevation = driver.find_element_by_xpath("//table[@id='Table3']/tbody/tr[3]/td[2]").text
    df = pd.DataFrame(columns=['code_and_name', 'approximate_elevation']
    df.at[i,'code_and_name']= str(code_and_name)
    df.at[i,'approximate_elevation']= str(approximate_elevation)

df.to_csv('observed_data_list.csv', encoding='ISO-8859-9')
Reply
#2
Your df has been reinitialized every time in for loop. So just put it in front of for:

df = pd.DataFrame(columns=['code_and_name', 'approximate_elevation'])
for i in range(3):
    driver.get(str(websites['sitelist'][i]))
    code_and_name = driver.find_element_by_id("Span1").text
    approximate_elevation = driver.find_element_by_xpath(
        "//table[@id='Table3']/tbody/tr[3]/td[2]").text
    df.at[i, 'code_and_name'] = str(code_and_name)
    df.at[i, 'approximate_elevation'] = str(approximate_elevation)
Output:

,code_and_name,approximate_elevation
0,D15A200 İNCEĞİZ AYVACA D.,"960,00 m."
1,D15A257 TUTMAÇ BAŞÖREN D.,"1379,00 m."
2,D15A014 ÇEŞME ARKI YEŞİLHİSAR Ç.,"1131,00 m."
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  page navigation & form filling rudolphyaber 0 1,603 Mar-13-2019, 06:31 PM
Last Post: rudolphyaber

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020