I wan't to Download all .zip Files From A Website (Project AI)

[eddywinch82]

Hi guys, snippsat I tried logging in with selenium, instead of requests, i.e. I used import selenium and I can't with that module either, I get the same error message, I got when running with requests with the codes. Also I have tried running both yours and Larz60 's codes for getting the File Path Data etc, and both have Syntax error, when I run them in Python. I am assuming that the coding worked for you both, in both cases ?

I have checked the coding, and I have copied both codes correctly.

Also snippsat you said "Or write a code that goes through all pages(simple page system 2,3,4, etc...) and download." How do I do that ?

---

Hi snippsat, I have attempted to adapt the coding you did for me, a while back for the Project AI Website .zip Files I wanted to download, But it hasn't worked, where am I going wrong ? Here is the adapted Code :-

from bs4 import BeautifulSoup
import requests
from tqdm import tqdm, trange
from itertools import islice
 
def all_planes():
    '''Generate url links for all planes'''
    url = 'https://www.flightsim.com/vbfs/fslib.php?do=search&fsec=62'
    url_get = requests.get(url)
    soup = BeautifulSoup(url_get.content, 'lxml')
    td = soup.find_all('td', width="50%")
    plain_link = [link.find('a').get('href') for link in td]
    for ref in tqdm(plain_link):
         url_file_id = 'https://www.flightsim.com/vbfs/fslib.php?searchid=65857709{}'.format(ref)
         yield url_file_id
 
def download(all_planes_pages):
    '''Download zip for 1 plain,feed with more url download all planes'''
    # A_300 = next(all_planes())  # Test with first link
    last_253 = islice(all_planes_pages(), 0, 253)
    for plane_page_url in last_253:
        url_get = requests.get(plane_page_url)
        soup = BeautifulSoup(url_get.content, 'lxml')
        td = soup.find_all('td', class_="text", colspan="2")
        zip_url = 'https://www.flightsim.com/vbfs/fslib.php?do=copyright&fid={}'
        for item in tqdm(td):
            zip_name = item.text
            zip_number = item.find('a').get('href').split('=')[-1]
            with open(zip_name, 'wb')  as f_out:
                down_url = requests.get(zip_url.format(zip_number))
                f_out.write(down_url.content)
 
if __name__ == '__main__':
    download(all_planes_pages)

Eddie