Python Forum
Scraping problems. Pls help with a correct request query.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Scraping problems. Pls help with a correct request query.
#3
You should use selenium for this.
You will also have to install proper driver (chromedriver or gekodriver)
(I use firefox, so gekodriver)

The following code is almost correct, you can finish:
import selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from bs4 import BeautifulSoup
from pathlib import Path
import os
import PrettifyPage
import time


class FetchEaton:
    def __init__(self):
        os.chdir(os.path.abspath(os.path.dirname(__file__)))
        url = 'http://www.eatonpowersource.com/cross-reference/'
        self.pp = PrettifyPage.PrettifyPage()
        self.get_xref_data(url)
    
    def get_xref_data(self, url):
        browser = self.start_browser()
        browser.get(url)
        time.sleep(1)
        searchbox = browser.find_element(By.XPATH, '//*[@id="Criteria_CompetitorPartNumber"]')
        searchbox.clear()
        searchbox.send_keys('0330D0')
        btn = browser.find_element(By.XPATH, '/html/body/div[5]/section/div/section/div[1]/div[1]/span/aside/form/div[1]/div/span/button/i').click()
        time.sleep(2)
        table = browser.find_element(By.XPATH, '/html/body/div[5]/section/div/section/div[1]/div[2]/div[3]/div/div[1]/div/div/table')
        allRows = table.find_elements(By.TAG_NAME, 'tr');
        for row in allRows:
            cells = row.find_elements(By.TAG_NAME, 'td');
            for cell in cells:
                print(cell.text)
        self.stop_browser(browser)

    def start_browser(self):
        caps = webdriver.DesiredCapabilities().FIREFOX
        caps["marionette"] = True
        return webdriver.Firefox(capabilities=caps)

    def stop_browser(self, browser):
        browser.close()


if __name__ == '__main__':
    FetchEaton()
Output:
Output:
Filtration Hydac 0330D003BHHC Filtration V0334B2H03 Filtration Hydac 0330D003BNHC Filtration V0332B2C03 Filtration Hydac 0330D005BHHC Filtration V0334B2H10 Filtration Hydac 0330D005BNHC Filtration V0332B2C05 Filtration Hydac 0330D010BHHC Filtration V0334B2H10 Filtration Hydac 0330D010BNHC Filtration V0332B2C10
Reply


Messages In This Thread
RE: Scraping problems. Pls help with a correct request query. - by Larz60+ - Sep-30-2019, 08:14 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  POST request with form data issue web scraping hoff1022 1 2,728 Aug-14-2020, 10:25 AM
Last Post: kashcode
  Scraping a dynamic data-table in python through AJAX request filozofo 1 3,926 Aug-14-2020, 10:13 AM
Last Post: kashcode
  The correct POST request abhie_lp 5 3,049 Jun-05-2020, 07:27 AM
Last Post: buran
  Scraping problems with Python requests. gtlhbkkj 1 1,907 Jan-22-2020, 11:00 AM
Last Post: gtlhbkkj
  Scraping problems. Pls help with a correct request query. gtlhbkkj 0 1,533 Oct-09-2019, 12:00 PM
Last Post: gtlhbkkj
  web scraping to csv formatting problems bluethundr 4 2,839 Jul-04-2019, 02:00 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020