Python Forum
Problem with logging in on website - python w/ requests
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Problem with logging in on website - python w/ requests
#7
(Sep-23-2020, 12:35 PM)GoldeNx Wrote: Selenium is too slow for me. That's why I try to write this with requests
Can run Selenium in headless(not loading browser) then is faster.
When logged in can send source browser.page_source to BS for parse with lxml.
This is way easier the trying to do this with Requests/BS on a JavaScript heavy site.
Also when have .page_source can do serval operation at same speed.
Example setup bid_size and price_sales or more will have good same speed after the setup and catch of .page_source is done.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import time, sys

#--| Setup
options = Options()
options.add_argument("--headless")
options.add_argument('--disable-gpu')
#options.add_argument('--log-level=3')
browser = webdriver.Chrome(executable_path=r'C:\cmder\bin\chromedriver.exe', options=options)
#--| Parse or automation
browser.get('https://www.morningstar.com/stocks/XOSL/XXL/quote.html')
#time.sleep(1)
soup = BeautifulSoup(browser.page_source, 'lxml')
bid_size = soup.select('div.dp-value.price-down.ng-binding.ng-scope')
price_sales = soup.select('li:nth-child(9) > div > div.dp-value.ng-binding')
print(price_sales[0].text.strip())
print(bid_size[0].text.strip())
Output:
0.49 25.20×1,000
Reply


Messages In This Thread
RE: Problem with logging in on website - python w/ requests - by snippsat - Sep-25-2020, 10:52 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Extracting content from a website using Python? SandraYokum 2 501 May-27-2024, 03:30 AM
Last Post: Davidleo
  Problem with scrapping Website giddyhead 1 1,785 Mar-08-2024, 08:20 AM
Last Post: AhanaSharma
  Retrieve website content using Python? Vadanane 1 1,432 Jan-16-2023, 09:55 AM
Last Post: Axel_Erfurt
  POST requests - different requests return the same response Default_001 3 2,084 Mar-10-2022, 11:26 PM
Last Post: Default_001
  I want to create an automated website in python mkdhrub1 2 2,621 Dec-27-2021, 11:27 PM
Last Post: Larz60+
  Python to build website Methew324 1 2,334 Dec-15-2020, 05:57 AM
Last Post: buran
  Scraping all website text using Python MKMKMKMK 1 2,176 Nov-26-2020, 10:35 PM
Last Post: Larz60+
  Python Webscraping with a Login Website warriordazza 0 2,727 Jun-07-2020, 07:04 AM
Last Post: warriordazza
  How to perform a successful login(signin) through Requests in Python Kalet 1 2,442 Apr-24-2020, 01:44 AM
Last Post: Larz60+
  Python tool based on website? zarize 2 2,565 Mar-21-2020, 02:25 PM
Last Post: zarize

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020