Python Forum
Python SSL web page scraping
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python SSL web page scraping
#2
There are severals big problems here.
Should not be needed now to say that you should be using Python 2.7,it has been dead💀 for 3-years now.
For the scraing part the url leads to a login page and not the main page,so copy content will only copy the login page.
So this is far away from working,to give some hint i would use Selenuim to do login to get to main page.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from bs4 import BeautifulSoup
import time

#--| Setup
options = Options()
#options.add_argument("--headless")
ser = Service(r"C:\cmder\bin\chromedriver.exe")
browser = webdriver.Chrome(service=ser, options=options)
#--| Parse or automation
url = 'https://edd.telstra.com/telstra/'
browser.get(url)
browser.implicitly_wait(2)
user_name = browser.find_element(By.CSS_SELECTOR, '#Username')
user_name.send_keys('login_name')
bus_id = browser.find_element(By.CSS_SELECTOR, '#Business\ ID')
bus_id.send_keys('999')
So start like this then push login button to get to main page,then copy content may work or not at all.
It depends what main page data has available without requesting something.
[Image: 9awKCM.png]
Reply


Messages In This Thread
Python SSL web page scraping - by Vadanane - Jan-13-2023, 09:34 AM
RE: Python SSL web page scraping - by snippsat - Jan-13-2023, 04:11 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Problem with scraping the Title from a web page Wagner822 0 695 Jun-29-2022, 11:31 PM
Last Post: Wagner822
Brick Javascript based web page scraping amjadraza26 1 1,467 Oct-21-2021, 09:36 AM
Last Post: Larz60+
  scraping a table from an http page vchealy 1 1,740 Jun-10-2021, 09:48 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020