Same Data Showing Several Times With Beautifulsoup Query

[eddywinch82]

Hi there,

I have the following Python Code :-

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

import pandas as pd import requests import numpy as np from bs4 import BeautifulSoup import xlrd import re   pd.set_option('display.max_rows', 500) pd.set_option('display.max_columns', 500) pd.set_option('display.width', 1000)   res3 = requests.get("https://web.archive.org/web/20220521203053/https://www.military-airshows.co.uk/press22/bbmfschedule2022.htm")     soup3 = BeautifulSoup(res3.content,'lxml')   BBMF_2022 = []   #BBMF_elem = soup3.find_all('a', string=re.compile(r'between|Flypast'))     for item in soup3.find_all('a', string=re.compile(r'between|Flypast')):     li1 = item.find_parent().text     #li2 = li1.find_previous().font     #print(link)     print(li1)     #print(li2)           #BBMF_2022.append(li1)     #check if links are in dataframe #df = pd.DataFrame(BBMF_2022, columns=['BBMF_2022'])   #df

The issue I have is when I run the Code, the Data is printed for 15 Entries from May 28th to May 29th, several times,
I am not sure why that is the case ? Could someone suggest for me the reason why ? And tell me what I need to change in the Code, so
that that Data is printed only once and not several times ? I have tried to Scrape Data from a Website, where entries contain the word between or Flypast.

When I use the following piece of Code instead :-

1 2 3 4 5 6 7 8 9 10 11 12

for item in soup3.find_all('a', string=re.compile(r'between|Flypast')):     li1 = item.find_parent().text     #li2 = li1.find_previous().font     #print(link)     #print(li1)     #print(li2)            BBMF_2022.append(li1)     df = pd.DataFrame(BBMF_2022, columns=['BBMF_2022'])     df

The first entry for the 28th May, is printed out in the DataFrame 15 times ! instead of 15 seperate Entries I mentioned before.

Any help would be much appreciated.

Best Regards

Eddie Winch ))

[Larz60+]

You are using a redirected url, instead use: https://python-forum.io/thread-37342.html ?

This code will get all data and save as a json file, without any filtering. You can add filters, and any other data you need

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91

import requests from bs4 import BeautifulSoup import os import json import sys   class airshowdata:     def __init__(self):         self.airshow_details = {}         self.cd = CreateDict()         self.jsonfile = 'airshow.json'       def get_links(self):         url = 'https://www.military-airshows.co.uk/press22/bbmfschedule2022.htm'           res3 = requests.get(url)         if res3.status_code == 200:             soup3 = BeautifulSoup(res3.content,'lxml')         else:             print(f"Cannot load page {url}")             sys.exit(-1)           links = soup3.find_all('a')         for link in links:             anode = self.cd.add_node(self.airshow_details, link.text.strip())             self.cd.add_cell(anode, 'url', link.get('href'))           with open(self.jsonfile, 'w') as fp:             json.dump(self.airshow_details, fp)           # following not needed and can be removed (displays dictionary contents)         self.cd.display_dict(self.airshow_details)     class CreateDict:     """     CreateDict.py - Contains methods to simplify node and cell creation within                     a dictionary       Usage:             new_dict(dictname) - Creates a new dictionary instance with the name             contained in dictname           add_node(parent, nodename) - Creates a new node (nested dictionary)             named in nodename, in parent dictionary.           add_cell(nodename, cellname, value) - Creates a leaf node within node             named in nodename, with a cell name of cellname, and value of value.           display_dict(dictname) - Recursively displays a nested dictionary.       Requirements:         Python standard library:             os           Author: Larz60+  -- May 2019.     """     def __init__(self):         os.chdir(os.path.abspath(os.path.dirname(__file__)))       def new_dict(self, dictname):         setattr(self, dictname, {})       def add_node(self, parent, nodename):         node = parent[nodename] = {}         return node       def add_cell(self, nodename, cellname, value):         cell =  nodename[cellname] = value         return cell       def display_dict(self, dictname, level=0):         indent = " " * (4 * level)         for key, value in dictname.items():             if isinstance(value, dict):                 print(f'\n{indent}{key}')                 level += 1                 self.display_dict(value, level)             else:                 print(f'{indent}{key}: {value}')             if level > 0:                 level -= 1     def main():     airs = airshowdata()     airs.get_links()     if __name__ == '__main__':     main()

[eddywinch82]

Many thanks for that Code Larz60+, its very much appreciated by me, thankyou for taking the time to type it. I chose
the web.archive link, because the Data is from a week ago, from that Website, the 21st May Data was removed from the Website the other day.

Does anyone have any idea, how I can change my Code, to solve the issue I am having with it ?

Any help would be very much appreciated.

Regards

Eddie Winch ))