I wan't to Download all .zip Files From A Website (Project AI) - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: I wan't to Download all .zip Files From A Website (Project AI) (/thread-12450.html) |
RE: I wan't to Download all .zip Files From A Website (Project AI) - eddywinch82 - Oct-24-2018 Hi Larz60+, Could you look at this code again for me ? When you have some spare time ? I am keen to download the .zip Files, I want from the Website in question. Eddie RE: I wan't to Download all .zip Files From A Website (Project AI) - Larz60+ - Oct-24-2018 Quote:Could you look at this code again for meI'll bet that 'this code' has changed since the last post. re-post the complete code again, and don't touch it until you get a reply. It's impossible to work on a moving target. RE: I wan't to Download all .zip Files From A Website (Project AI) - eddywinch82 - Oct-24-2018 Hi Larz60+, Here is the current Python Code I have, it hasn't been updated, since you last messaged me in early September :- import sys import getpass import hashlib import requests import os from bs4 import BeautifulSoup BASE_URL = 'https://www.flightsim.com/' LOGIN_PAGE = 'https://www.flightsim.com/vbfs/login.php?do=login' def do_login(credentials): session = requests.Session() session.get(BASE_URL) req = session.post(LOGIN_PAGE, params={'do': 'login'}, data=credentials) if req.status_code != 200: print('Login not successful') sys.exit(1) # session is now logged in return session def get_credentials(): username = input('Username: ') password = getpass.getpass() password_md5 = hashlib.md5(password.encode()).hexdigest() return { 'cookieuser': 1, 'do': 'login', 's': '', 'securitytoken': 'guest', 'vb_login_md5_password': password_md5, 'vb_login_md5_password_utf': password_md5, 'vb_login_password': '', 'vb_login_password_hint': 'Password', 'vb_login_username': username, } credentials = get_credentials() session = do_login(credentials) class ScrapeUrlList: def __init__(self): self.fpath = os.fspath self.ziplinks = [] def get_url(self, url): page = None response = requests.get(url) if response.status_code == 200: page = response.content else: print(f'Cannot load URL: {url}') # pass return page def get_catalog(self): base_url = 'https://www.flightsim.com/vbfs' with self.fpath.links.open('w') as fp: 'C:\Python37\Lib' baseurl = self.fpath.base_catalog_url for pageno in range(1, 254): url = f'https://www.flightsim.com/vbfs/fslib.php?searchid=65893537&page={pageno}' print(f'url: {url}') page = self.get_url(self.fpath.base_catalog_url) if page: soup = BeautifulSoup(page, 'lxml') zip_links = soup.find_all('div', class_="fsc_details") for link in zip_links: fp.write(f"{link.find('a').text}, {base_url}/{link.find('a').get('href')}") input() else: print(f'No page: {url}') def main(): sul = ScrapeUrlList() sul.get_catalog() if __name__ == '__main__': main()Regards Eddie Winch RE: I wan't to Download all .zip Files From A Website (Project AI) - Larz60+ - Oct-24-2018 I'll look at this after dinner this evening. May not respond until my A.M. (EDT) RE: I wan't to Download all .zip Files From A Website (Project AI) - eddywinch82 - Oct-24-2018 Many thanks Larz60+, I appreciate you doing that for me, very much. Best Regards Eddie Winch Hi Larz60+, Could the .zip Files be saved, as the outer-most folder name in the zip (so the name would be LNI-Lion Mentari Airlines.zip) for example ? Is it fairly easy, to write a simple pipeline, to rename the files in this way ? The Zip file names start pai, i.e. paicm027.zip, so can't be indentified easily, like that. I am on holiday from tomorrow, and back on Sunday. I can access the Python Forums, on the internet on my phone and type reply's to posts and messages, but I won't have access to my computer therefore, until Sunday evening. RE: I wan't to Download all .zip Files From A Website (Project AI) - Larz60+ - Oct-25-2018 Haven't got to this yet. Forgot I have an event this evening, and been spending the day preparing for that. No promise on resolution, but I will get to it. RE: I wan't to Download all .zip Files From A Website (Project AI) - eddywinch82 - Oct-26-2018 Thankyou so much Larz60+, I very much appreciate all your time and effort, on this coding problem. Eddie RE: I wan't to Download all .zip Files From A Website (Project AI) - Larz60+ - Oct-26-2018 I looked at this for a bit. Sorry, but I can't continue, too frustrating. I'v already spent too much time on it. Perhaps another mod will be willing to pick up. RE: I wan't to Download all .zip Files From A Website (Project AI) - eddywinch82 - Oct-28-2018 Thanks Larz60+ for all your efforts for me. I understand, your help is much appreciated. Yes can anyone else help me, with this coding problem ? |