Capturing BS4 values into DF and writing to CSV

[cubangt]

I have the below logic that im getting results from, but when righting to the csv, each data value is being written to row instead of collectively into columns and rows.

import requests
from bs4 import BeautifulSoup
import pandas as pd

productdetails = []

headers = {
    'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36'
}

df2 = pd.DataFrame(columns=['Description', 'Price1', 'Price2'],index=range(1))

for x in range(1,180):
    response = requests.get(f'https://www.site.com/c/mens/mens-footwear?&page_{x}', verify=False, headers=headers)
    soup = BeautifulSoup(response.content,'lxml')

    element_list = soup.find_all('div',class_='product-content')
    for element in element_list:
        for link in element.find_all('a', class_='product-card-simple-title'):
            productdetails.append("Description: " + link.get_text().strip())
            for price in element.find_all('span',class_='sr-only'):
                productdetails.append("Price1: " + price.get_text().strip().replace('\n', '').replace(' ','').replace('dollars','.').replace('cents',''))
                if len(element.find_all('span',class_='sr-only')) == 2:
                    productdetails.append("Price2: " + price.get_text().strip().replace('\n', '').replace(' ','').replace('dollars','.').replace('cents',''))

print(productdetails)

Im currently working on checking if there are more than 1 prices, so that i can add that as 2nd price in the dataframe.


CSV data currently:

0,Description: Brooks Men's Adrenaline GTS 23 Running Shoes
1,Price1: 139.99
2,Description: Nike Men's Revolution 6 Next Nature Running Shoes
3,Price1: 22.37
4,Price2: 44.97

Expected CSV data:

Description, Price1, Price2
Brooks Men's Adrenaline GTS 23 Running Shoes, 139.99,
Nike Men's Revolution 6 Next Nature Running Shoes,22.37,44.97