I wan't to Download all .zip Files From A Website (Project AI) - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html) +--- Thread: I wan't to Download all .zip Files From A Website (Project AI) (/thread-12450.html) |
RE: I wan't to Download all .zip Files From A Website (Project AI) - eddywinch82 - Sep-01-2018 I have had the Login Problem sorted, I paid a company for that to help me, but now when I run the modified code :- I get an unexpected Indent Error, on page 64 Here is the latest Python Code :- import sys import getpass import hashlib import requests import os from bs4 import BeautifulSoup BASE_URL = 'https://www.flightsim.com/' LOGIN_PAGE = 'https://www.flightsim.com/vbfs/login.php?do=login' def do_login(credentials): session = requests.Session() session.get(BASE_URL) req = session.post(LOGIN_PAGE, params={'do': 'login'}, data=credentials) if req.status_code != 200: print('Login not successful') sys.exit(1) # session is now logged in return session def get_credentials(): username = input('Username: ') password = getpass.getpass() password_md5 = hashlib.md5(password.encode()).hexdigest() return { 'cookieuser': 1, 'do': 'login', 's': '', 'securitytoken': 'guest', 'vb_login_md5_password': password_md5, 'vb_login_md5_password_utf': password_md5, 'vb_login_password': '', 'vb_login_password_hint': 'Password', 'vb_login_username': username, } credentials = get_credentials() session = do_login(credentials) class ScrapeUrlList: def __init__(self): self.fpath = os.fspath self.ziplinks = [] def get_url(self, url): page = None response = requests.get(url) if response.status_code == 200: page = response.content else: print(f'Cannot load URL: {url}') # pass return page def get_catalog(self): base_url = 'https://www.flightsim.com/vbfs' with self.fpath.links.open('w') as fp: 'C:\Python37\Lib' baseurl = self.fpath.base_catalog_url for pageno in range(1, 254): url = f'https://www.flightsim.com/vbfs/fslib.php?searchid=65893537&page={pageno}' print(f'url: {url}') page = self.get_url(self.fpath.base_catalog_url) if page: soup = BeautifulSoup(page, 'lxml') zip_links = soup.find_all('div', class_="fsc_details") for link in zip_links: fp.write(f"{link.find('a').text}, {base_url}/{link.find('a').get('href')}") input() else: print(f'No page: {url}') def main(): sul = ScrapeUrlList() sul.get_catalog() if __name__ == '__main__': main()Does anyone know, how I can sort this Problem out ? Which lines of the Code need Identing ? Or does more Code, need altering ? Eddie RE: I wan't to Download all .zip Files From A Website (Project AI) - Larz60+ - Sep-01-2018 this is basic python, why is there code after the colon? this is not what was given to you as it would never have run: with self.fpath.links.open('w') as fp: 'C:\Python37\Lib'I'd guess remove all after the colon. You do keep backup copies, right? RE: I wan't to Download all .zip Files From A Website (Project AI) - eddywinch82 - Sep-01-2018 The company I paid to help me with the Code, said I should put the Path to the os Module after that Colon, This is the Traceback Error I get without that Path, after the Colon on Line 63, when I run the Code :-
RE: I wan't to Download all .zip Files From A Website (Project AI) - Larz60+ - Sep-01-2018 good luck ... I guess you're using a new kind of Python RE: I wan't to Download all .zip Files From A Website (Project AI) - eddywinch82 - Sep-01-2018 I am using Python 3.7 I think they weren't sure, what I was trying to achieve with the coding. So I am assuming it was a misundertanding, on their part. I will have to pay them again, to sort out the rest of the Code, although I am thinking maybe I misunderstood their Instructions ? RE: I wan't to Download all .zip Files From A Website (Project AI) - Larz60+ - Sep-01-2018 You said the code ran. The code you show above would never run. The error that I showed you is real. You really ought to sit down and take a basic python tutorial as you are showing lack of basic skills RE: I wan't to Download all .zip Files From A Website (Project AI) - eddywinch82 - Sep-01-2018 You are Right Larz60+ I do, I can't afford to Keep paying people to help me. The Company I have paid to help me with this code, have sorted the Login Problem they say. But not the rest of the Code, I meant the code ran until that Traceback Error, I could Login, but then the Error occured. RE: I wan't to Download all .zip Files From A Website (Project AI) - snippsat - Sep-01-2018 (Sep-01-2018, 07:41 PM)eddywinch82 Wrote: The Company I have paid to help me with this code, have sorted the Login Problem they say.You have to use session in rest code after log in,is not used now. Can show how to do test. import sys import getpass import hashlib import requests import os, time from bs4 import BeautifulSoup BASE_URL = 'https://www.flightsim.com/' LOGIN_PAGE = 'https://www.flightsim.com/vbfs/login.php?do=login' def do_login(credentials): session = requests.Session() session.get(BASE_URL) req = session.post(LOGIN_PAGE, params={'do': 'login'}, data=credentials) if req.status_code != 200: print('Login not successful') sys.exit(1) # session is now logged in return session def get_credentials(): username = 'aaaa' password = 'xxxxxx' password_md5 = hashlib.md5(password.encode()).hexdigest() return { 'cookieuser': 1, 'do': 'login', 's': '', 'securitytoken': 'guest', 'vb_login_md5_password': password_md5, 'vb_login_md5_password_utf': password_md5, 'vb_login_password': '', 'vb_login_password_hint': 'Password', 'vb_login_username': username, } credentials = get_credentials() session = do_login(credentials) # Check that logged in time.sleep(3) url_in = 'https://www.flightsim.com/vbfs/forum.php' resonse = session.get(url_in) print(resonse) soup = BeautifulSoup(resonse.content, 'lxml') welcome = soup.find('h2', class_="forumtitle") print(welcome.text) So example in line 54 it most be session.get(url) If you paid they should help you to get started and have some working code that download some zip files. RE: I wan't to Download all .zip Files From A Website (Project AI) - eddywinch82 - Sep-01-2018 I paid 42 Dollars, I must have been scammed, 14 Dollars an Hour they said, and said would take 3 hours to sort out. Thanks for helping me again Snippsat. Unless, they meant the 3 hours just to sort out, the Login Problem ? Although skilled Programmers, wouldn't take that long normally right ? When I run that Code snippsat, it does indeed say :- <Response [200]> Newcomer Services After a few Seconds RE: I wan't to Download all .zip Files From A Website (Project AI) - eddywinch82 - Sep-03-2018 I have found out each of the Download Links, are like this :- https://dfs2.flightsim.com/download.php?fn=paiah082.zip And the first bit after the equals Sign, for all Project AI Aircraft Zip Files, is always pai |