I wan't to Download all .zip Files From A Website (Project AI)

[eddywinch82]

I have had the Login Problem sorted, I paid a company for that to help me, but now when I run the modified code :-

I get an unexpected Indent Error, on page 64

Here is the latest Python Code :-

import sys
import getpass
import hashlib
import requests
import os
from bs4 import BeautifulSoup


BASE_URL = 'https://www.flightsim.com/'

LOGIN_PAGE = 'https://www.flightsim.com/vbfs/login.php?do=login'


def do_login(credentials):
    session = requests.Session()
    session.get(BASE_URL)
    req = session.post(LOGIN_PAGE, params={'do': 'login'}, data=credentials)
    if req.status_code != 200:
        print('Login not successful')
        sys.exit(1)
    # session is now logged in
    return session


def get_credentials():
    username = input('Username: ')
    password = getpass.getpass()
    password_md5 = hashlib.md5(password.encode()).hexdigest()
    return {
        'cookieuser': 1,
        'do': 'login',
        's': '',
        'securitytoken': 'guest',
        'vb_login_md5_password': password_md5,
        'vb_login_md5_password_utf': password_md5,
        'vb_login_password': '',
        'vb_login_password_hint': 'Password',
        'vb_login_username': username,
    }


credentials = get_credentials()
session = do_login(credentials)


class ScrapeUrlList:
    def __init__(self):
        self.fpath = os.fspath
        self.ziplinks = []

    def get_url(self, url):
        page = None
        response = requests.get(url)
        if response.status_code == 200:
            page = response.content
        else:
            print(f'Cannot load URL: {url}')
            # pass
        return page

    def get_catalog(self):
        base_url = 'https://www.flightsim.com/vbfs'
        with self.fpath.links.open('w') as fp: 'C:\Python37\Lib'
            baseurl = self.fpath.base_catalog_url
            for pageno in range(1, 254):
                url = f'https://www.flightsim.com/vbfs/fslib.php?searchid=65893537&page={pageno}'
                print(f'url: {url}')
                page = self.get_url(self.fpath.base_catalog_url)
            if page:
                soup = BeautifulSoup(page, 'lxml')
                zip_links = soup.find_all('div', class_="fsc_details")
                for link in zip_links:
                    fp.write(f"{link.find('a').text}, {base_url}/{link.find('a').get('href')}")
                input()
            else:
                print(f'No page: {url}')


def main():
    sul = ScrapeUrlList()
    sul.get_catalog()


if __name__ == '__main__':
    main()

Does anyone know, how I can sort this Problem out ?

Which lines of the Code need Identing ? Or does more Code, need altering ?

Eddie

[Larz60+]

this is basic python, why is there code after the colon? this is not what was given to you as it would never have run:

with self.fpath.links.open('w') as fp: 'C:\Python37\Lib'

I'd guess remove all after the colon.
You do keep backup copies, right?

[eddywinch82]

The company I paid to help me with the Code, said I should put the Path to the os Module
after that Colon,

This is the Traceback Error I get without that Path, after the Colon on Line 63, when I run the Code :-

Error:
Warning (from warnings module):
  File "C:\Python37\lib\getpass.py", line 100
    return fallback_getpass(prompt, stream)
GetPassWarning: Can not control echo on the terminal.
Warning: Password input may be echoed.
Password: duxforded1
Traceback (most recent call last):
  File "C:\Users\Edward\Desktop\Python 3.7\flightsimscrape.py", line 85, in <module>
    main()
  File "C:\Users\Edward\Desktop\Python 3.7\flightsimscrape.py", line 81, in main
    sul.get_catalog()
  File "C:\Users\Edward\Desktop\Python 3.7\flightsimscrape.py", line 63, in get_catalog
    with self.fpath.links.open('w') as fp:  # Please change this code according to os path
AttributeError: 'builtin_function_or_method' object has no attribute 'links'

[Larz60+]

good luck ... I guess you're using a new kind of Python

[eddywinch82]

I am using Python 3.7 I think they weren't sure, what I was trying to achieve with the coding. So I am assuming it was a misundertanding, on their part. I will have to pay them again, to sort out the rest of the Code, although I am thinking maybe I misunderstood their Instructions ?

[Larz60+]

You said the code ran.
The code you show above would never run.
The error that I showed you is real.
You really ought to sit down and take a basic python tutorial as you are showing lack of basic skills

[eddywinch82]

You are Right Larz60+ I do, I can't afford to Keep paying people to help me.
The Company I have paid to help me with this code, have sorted the Login Problem they say.

But not the rest of the Code, I meant the code ran until that Traceback Error, I could Login, but then the Error occured.

[snippsat]

The Company I have paid to help me with this code, have sorted the Login Problem they say.

You have to use session in rest code after log in,is not used now.
Can show how to do test.

import sys
import getpass
import hashlib
import requests
import os, time
from bs4 import BeautifulSoup

BASE_URL = 'https://www.flightsim.com/'
LOGIN_PAGE = 'https://www.flightsim.com/vbfs/login.php?do=login'

def do_login(credentials):
    session = requests.Session()
    session.get(BASE_URL)
    req = session.post(LOGIN_PAGE, params={'do': 'login'}, data=credentials)
    if req.status_code != 200:
        print('Login not successful')
        sys.exit(1)
    # session is now logged in
    return session

def get_credentials():
    username = 'aaaa'
    password = 'xxxxxx'
    password_md5 = hashlib.md5(password.encode()).hexdigest()
    return {
        'cookieuser': 1,
        'do': 'login',
        's': '',
        'securitytoken': 'guest',
        'vb_login_md5_password': password_md5,
        'vb_login_md5_password_utf': password_md5,
        'vb_login_password': '',
        'vb_login_password_hint': 'Password',
        'vb_login_username': username,
    }

credentials = get_credentials()
session = do_login(credentials)

# Check that logged in
time.sleep(3)
url_in = 'https://www.flightsim.com/vbfs/forum.php'
resonse = session.get(url_in)
print(resonse)
soup = BeautifulSoup(resonse.content, 'lxml')
welcome = soup.find('h2', class_="forumtitle")
print(welcome.text)

Output:
<Response [200]>
Newcomer Services

So example in line 54 it most be session.get(url)

If you paid they should help you to get started and have some working code that download some zip files.

[eddywinch82]

I paid 42 Dollars, I must have been scammed, 14 Dollars an Hour they said, and said would take 3 hours to sort out.

Thanks for helping me again Snippsat.

---

Unless, they meant the 3 hours just to sort out, the Login Problem ? Although skilled Programmers, wouldn't take that long normally right ?

---

When I run that Code snippsat, it does indeed say :-

<Response [200]>
Newcomer Services

After a few Seconds

[eddywinch82]

I have found out each of the Download Links, are like this :-

https://dfs2.flightsim.com/download.php?fn=paiah082.zip

And the first bit after the equals Sign, for all Project AI Aircraft Zip Files, is always pai