Python Forum
I wan't to Download all .zip Files From A Website (Project AI)
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
I wan't to Download all .zip Files From A Website (Project AI)
#61
Hi Larz60+,

Could you look at this code again for me ? When you have some spare time ?
I am keen to download the .zip Files, I want from the Website in question.

Eddie
Reply
#62
Quote:Could you look at this code again for me
I'll bet that 'this code' has changed since the last post.
re-post the complete code again, and don't touch it until you get a reply.
It's impossible to work on a moving target.
Reply
#63
Hi Larz60+,

Here is the current Python Code I have, it hasn't been updated, since you last messaged me in early
September :-

import sys
import getpass
import hashlib
import requests
import os
from bs4 import BeautifulSoup
 
 
BASE_URL = 'https://www.flightsim.com/'
 
LOGIN_PAGE = 'https://www.flightsim.com/vbfs/login.php?do=login'
 
 
def do_login(credentials):
    session = requests.Session()
    session.get(BASE_URL)
    req = session.post(LOGIN_PAGE, params={'do': 'login'}, data=credentials)
    if req.status_code != 200:
        print('Login not successful')
        sys.exit(1)
    # session is now logged in
    return session
 
 
def get_credentials():
    username = input('Username: ')
    password = getpass.getpass()
    password_md5 = hashlib.md5(password.encode()).hexdigest()
    return {
        'cookieuser': 1,
        'do': 'login',
        's': '',
        'securitytoken': 'guest',
        'vb_login_md5_password': password_md5,
        'vb_login_md5_password_utf': password_md5,
        'vb_login_password': '',
        'vb_login_password_hint': 'Password',
        'vb_login_username': username,
    }
 
 
credentials = get_credentials()
session = do_login(credentials)
 
 
class ScrapeUrlList:
    def __init__(self):
        self.fpath = os.fspath
        self.ziplinks = []
 
    def get_url(self, url):
        page = None
        response = requests.get(url)
        if response.status_code == 200:
            page = response.content
        else:
            print(f'Cannot load URL: {url}')
            # pass
        return page
 
    def get_catalog(self):
        base_url = 'https://www.flightsim.com/vbfs'
        with self.fpath.links.open('w') as fp: 'C:\Python37\Lib'
            baseurl = self.fpath.base_catalog_url
            for pageno in range(1, 254):
                url = f'https://www.flightsim.com/vbfs/fslib.php?searchid=65893537&page={pageno}'
                print(f'url: {url}')
                page = self.get_url(self.fpath.base_catalog_url)
            if page:
                soup = BeautifulSoup(page, 'lxml')
                zip_links = soup.find_all('div', class_="fsc_details")
                for link in zip_links:
                    fp.write(f"{link.find('a').text}, {base_url}/{link.find('a').get('href')}")
                input()
            else:
                print(f'No page: {url}')
 
 
def main():
    sul = ScrapeUrlList()
    sul.get_catalog()
 
 
if __name__ == '__main__':
    main()
Regards

Eddie Winch
Reply
#64
I'll look at this after dinner this evening.
May not respond until my A.M. (EDT)
Reply
#65
Many thanks Larz60+, I appreciate you doing that for me, very much.

Best Regards

Eddie Winch


Hi Larz60+,

Could the .zip Files be saved, as the outer-most folder name in the zip (so the name would be LNI-Lion Mentari Airlines.zip) for example ? Is it fairly easy, to write a simple pipeline, to rename the files in this way ? The Zip file names start pai, i.e. paicm027.zip, so can't be indentified easily, like that.

I am on holiday from tomorrow, and back on Sunday. I can access the Python Forums, on the internet on my phone and type reply's to posts and messages, but I won't have access to my computer therefore, until Sunday evening.
Reply
#66
Haven't got to this yet. Forgot I have an event this evening, and been spending the day preparing for that.
No promise on resolution, but I will get to it.
Reply
#67
Thankyou so much Larz60+, I very much appreciate all your time and effort, on this coding problem. Eddie
Reply
#68
I looked at this for a bit. Sorry, but I can't continue, too frustrating. I'v already spent too much time on it.
Perhaps another mod will be willing to pick up.
Reply
#69
Thanks Larz60+ for all your efforts for me. I understand, your help is much appreciated. Yes can anyone else help me, with this coding problem ?
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Website scrapping and download santoshrane 3 4,257 Apr-14-2021, 07:22 AM
Last Post: kashcode
  Login and download an exported csv file within a ribbon/button in a website Alekhya 0 2,617 Feb-26-2021, 04:15 PM
Last Post: Alekhya
  Cant Download Images from Unsplash Website firaki12345 1 2,259 Feb-08-2021, 04:15 PM
Last Post: buran
  Download some JPG files and make it a single PDF & share it rompdeck 5 5,583 Jul-31-2020, 01:15 AM
Last Post: Larz60+
  download pdf file from website m_annur2001 1 2,956 Jun-21-2019, 05:03 AM
Last Post: j.crater
  Access my webpage and download files from Python Pedroski55 7 5,530 May-26-2019, 12:08 PM
Last Post: snippsat
  Download all secret links from a map design website fyec 0 2,813 Jul-24-2018, 09:08 PM
Last Post: fyec
  I Want To Download Many Files Of Same File Extension With Either Wget Or Python, eddywinch82 15 14,347 May-20-2018, 06:05 PM
Last Post: eddywinch82

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020