Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Extracting Data from Calendar
#3
(Feb-15-2020, 08:35 PM)snippsat Wrote: Look at Web-scraping part-2 God dammit JavaScript, why do i not get all content.
So Selenium would work.

Here a way it can be done if looking at site and try to understand what's going on.
So same cookie and token in a Ajax call do work for different dates,then it can be done like this(the harder way Doh if not using Selenium).
Parsing summary data at bottom of page.
import requests
from bs4 import BeautifulSoup

def stock_data(date):
    cookies = {
        '__cfduid': 'd9beb9712d05492d153761becc05fa2f01581793084',
        'XSRF-TOKEN': 'eyJpdiI6Ilc2VTVTRmd2OWJQQlwvakxmRHMzMGNRPT0iLCJ2YWx1ZSI6IkRLMHQ0Y1MzVUY5bVl4d2UwWVNrSlBGS3hNSmJaXC85MjZTNDNCUFlEM3V3ajV6RDFzS1dxYWY4SXFINFJFUVY5IiwibWFjIjoiYmFmZmVhMzRlNDcxZWU0N2FmN2EzOTQ1MmU1NmEwYTI1MzViM2UyOTA3ZTZhM2Y0YWVjMzYwMjQ2ZDkwYWI0ZSJ9',
        'sharesansar_session': 'eyJpdiI6IkxINjZUakgzdTR3eWplbWVXSThhUEE9PSIsInZhbHVlIjoiM3FqdFIxMnpBbmt5UFFVZXZlN2NNbWpYbkZ1WVljcG5YSGRFNVNpQ1RIV0laRW5DcVZONG1VSXJSTEdLS1NVWCIsIm1hYyI6IjJlMjEwN2RhNWExYzczYjNiMzhmMTkzMmM0ODc4YmFkZWFmNmFiNTMxZjYzNTAxYjM4NGJlMGFlMzM3ZDM1YzcifQ%3D%3D',
    }

    headers = {
        'Connection': 'keep-alive',
        'Accept': '*/*',
        'X-Requested-With': 'XMLHttpRequest',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36',
        'Sec-Fetch-Site': 'same-origin',
        'Sec-Fetch-Mode': 'cors',
        'Referer': 'https://www.sharesansar.com/today-share-price',
        'Accept-Encoding': 'gzip, deflate, br',
        'Accept-Language': 'nb-NO,nb;q=0.9,no;q=0.8,nn;q=0.7,en-US;q=0.6,en;q=0.5',
    }

    params = (
        ('_token', 'nhM8Q4PLiX1hZu1K8W8yCqfP0iF9IsEgnO90e8dT'),
        ('sector', 'all_sec'),
        ('date', date),
    )

    response = requests.get('https://www.sharesansar.com/ajaxtodayshareprice', headers=headers, params=params, cookies=cookies)
    return response

def parse_data(response, date):
    soup = BeautifulSoup(response.content, 'lxml')
    numb_of_compaines = soup.find_all('h4', class_="text-right")
    total_traded_shares = soup.find_all('h4', class_="text-right")
    print(f'-----| {date} |-----')
    print(numb_of_compaines[0].text)
    print(total_traded_shares[1].text)

if __name__ == '__main__':
    for day in range(1,4):
        date = f'2020-02-1{day}'
        response = stock_data(date)
        parse_data(response, date)
Output:
-----| 2020-02-11 |----- Total number of Compaines: 175 Total Traded Shares: 3,996,877 -----| 2020-02-12 |----- Total number of Compaines: 173 Total Traded Shares: 3,986,769 -----| 2020-02-13 |----- Total number of Compaines: 171 Total Traded Shares: 3,720,085

Thank you so so so much, it had been months since I had been trying to do this. I had tried to learn selenium but I failed to do so, anyways thank you so so much.
Reply


Messages In This Thread
Extracting Data from Calendar - by AgileAVS - Feb-15-2020, 04:22 PM
RE: Extracting Data from Calendar - by snippsat - Feb-15-2020, 08:35 PM
RE: Extracting Data from Calendar - by AgileAVS - Feb-16-2020, 01:06 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
Question Online calendar for booking. SpongeB0B 6 3,527 Nov-15-2023, 11:27 AM
Last Post: Woki
  Extracting html data using attributes WiPi 14 5,518 May-04-2020, 02:04 PM
Last Post: snippsat
  Follow Up: Web Calendar based Extraction AgileAVS 0 1,513 Feb-23-2020, 05:39 AM
Last Post: AgileAVS
  Google Calendar iCal Albireo 2 5,518 Mar-15-2017, 03:47 PM
Last Post: Albireo

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020