Python Forum
How to avoid open and save a url every time I run code
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to avoid open and save a url every time I run code
#1
Hi I'm new to Python, BeautifulSoup and PyCharm, like to know how to write code so I don't have run this every time I run a test
import requests
from bs4 import BeautifulSoup
url = 'https://finance.yahoo.com/quote/BARC.L/key-statistics?p=BARC.L'
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')
:When I'm just testing/learning using 1 liners such as
print(soup.select_one('td span').get_text())
(Also according to my location I'm using an rtl language, but in fact I'm writing in English (ltr
How do I change this editor to ltr?
Thanks for any help
Reply
#2
the way to avoid open, is to save, so subject
Quote:How to avoid open and save a url every time I run code
is confusing.

At any rate, I use a class that I wrote a while ago, if you pass the get_page method a filename, it will do one of two things:
if a cached file has already been saved, if will load it. Otherwise it will fetch the webpage and cache it.
If no filename is passed, it will always fetch from web.

Save in file named GetPage.py
import requests
import time


class GetPage:
    def __init__(self):
        pass

    def get_page(self, url, filename=None):
        if filename is not None and filename.exists():
            with filename.open('rb') as gp:
                return gp.read()
        else:
            response = requests.get(url)
            if response.status_code == 200:
                if filename is not None:
                    try:
                        with filename.open('wb') as gp:
                            gp.write(response.content)
                    except FileNotFoundError:
                        print(f"error type: {type(filename)} saving {filename.name}, {filename.resolve()}")
                return response.content
            else:
                return response.status_code
example use example.py:
from GetPage import GetPage
import os
from pathlib import Path


gp = GetPage()

def get_this_page():
    os.chdir(os.path.abspath(os.path.dirname(__file__)))
    url = 'https://python-forum.io/Thread-How-to-avoid-open-and-save-a-url-every-time-I-run-code'
    homepage = Path('.')
    savefile = homepage / 'MySavedPage.html'
    page = gp.get_page(url, savefile)


if __name__ == '__main__':
    get_this_page()
Reply
#3
How do I change this this editor to ltr - without leaving the country?
Thanks for your reply, sorry for confusing subject line.
Is there a way to save the url content to a variable and pass that variable to a subroutine and just execute the subroutine. Sorry for dumb question new to Python

Thanks for any help
Reply
#4
when you fetch a page with requests, the page is already in a variable text/content.
so long as you don't call requests again, that data remains there.
the code that I gave you transparently saves the data to a file, and reads from that file next time get_page is called.
Why do you have a hesitation about using a file.
Variable work, so long as you don't exit your program.
Reply
#5
Have you tried get any values from that site?
It will not work very well,some hint why in part-2...JavaScript Think
A site like this is not easy to start with if you doing this as training.
There is an API for Yahoo Finance(which now is paid was free before),there are Python packages that may still work like yfinance.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Open/save file on Android frohr 0 315 Jan-24-2024, 06:28 PM
Last Post: frohr
  how to save to multiple locations during save cubangt 1 543 Oct-23-2023, 10:16 PM
Last Post: deanhystad
  how do I open two instances of visual studio code with the same folder? SuchUmami 3 880 Jun-26-2023, 09:40 AM
Last Post: snippsat
  Avoid third party functions to wrote my python code into system debug-log? mark 9 2,197 Apr-09-2022, 08:41 PM
Last Post: mark
  How do I open the Source code of a library? JaneTan 1 2,263 Aug-18-2021, 02:12 AM
Last Post: Larz60+
  Assistance with running a few lines of code at an EXACT time nethatar 5 3,241 Feb-24-2021, 10:43 PM
Last Post: nilamo
  Stumped by my own code (ratio & epoch-time calculation). MvGulik 2 2,122 Dec-30-2020, 12:04 AM
Last Post: MvGulik
  Code taking too much time to process ErPipex 11 4,910 Nov-16-2020, 09:42 AM
Last Post: DeaD_EyE
  What is the run time complexity of this code and please explain? samlee916 2 2,292 Nov-06-2020, 02:37 PM
Last Post: deanhystad
  The count variable is giving me a hard time in this code D4isyy 2 1,967 Aug-09-2020, 10:32 PM
Last Post: bowlofred

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020