writing data to a csv-file

[DeaD_EyE]

Code like this:

    a = t[0]
    b = t[1]
    c = t[2]
    d = t[3]
    e = t[4]
    f = t[5]
    g = t[4]

Code like this is an indication that the design is not correct. Use data structures to represent your data.
You're assigning elements from a list to names and then you create from this a new list.


Your data is 2-dimensional. The first dimension is the index (rows) and the second dimension are the columns (td-data).

Create an empty list, which is later your whole dataset.
For each tag you need the text or an attribute. Putting a whole tag object into pandas will not work.

td_results = []
for i in range(1, 100):
    url = "https://my-website.com/webid={}".format(i)
    s = session.get(url, headers=headers, cookies=cookies)
 
    soup = bs(s.text, 'html.parser')
    data = soup.find_all('td') 
    td_results.append(column.text for column in soup.find_all('td')) # <- this here is the critical part
    # he could find something or not
    # and the amount of td elements can be different


print(td_results)
df = pdDataFrame(td_results)

So if you know all pages do have the same structure and you know for example, that you need the first 10 element, then you can use the subscription method.

Example to get the first 10 elements:

td_results.append(column.text for column in soup.find_all('td')[:10])