Python Forum
How to export to csv the output of every iteration when scrapping with a loop
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to export to csv the output of every iteration when scrapping with a loop
#1
Hi,
I am very new to Python. I am trying to scrape a website and I have create a small code for this:

select = Select(character.find_element_by_id('character-template-choice'))
options = select.options
for index in range(0, len(options) - 0):
    select.select_by_index(index)
    option1 = character.find_elements_by_class_name('pc-stat-value')
    power = []
    for c in option1:
        power.append("Power:" + c.text)
    option2 = character.find_elements_by_class_name(
        'unit-stat-group-stat-label')
    skills = []
    for a in option2:
        skills.append(a.text)
    option3 = character.find_elements_by_class_name(
        'unit-stat-group-stat-value')
    values = []
    for b in option3:
        values.append(b.text)
    test = [power, skills, values]
The issue I have is when I try to export "test" list of lists to csv I only get the last iteration. I want to get the data for every iteration but I don't know how to do so. Can someone help me?

Thank you
Larz60+ write Nov-26-2020, 10:25 PM:
Please post all code, output and errors (it it's entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.

I fixed for you this time. Please use bbcode tags on future posts.
Reply
#2
Please, when posting code, provide enough to be able to run.

if you haven't already done so, import csv at top of script

# untested code, you may have to correct any errors.
select = Select(character.find_element_by_id('character-template-choice'))
options = select.options
savefile = 'savefile.txt' # Change to your desired file name
with open(savefile, 'w') as fp":
    csvwriter = csv.writer(fp)
    for index in range(0, len(options) - 0):
    select.select_by_index(index)
    option1 = character.find_elements_by_class_name('pc-stat-value')
    power = []
    for c in option1:
        power.append("Power:" + c.text)
    option2 = character.find_elements_by_class_name(
        'unit-stat-group-stat-label')
    skills = []
    for a in option2:
        skills.append(a.text)
    option3 = character.find_elements_by_class_name(
        'unit-stat-group-stat-value')
    values = []
    for b in option3:
        values.append(b.text)
    test = [power, skills, values]
        csvwriter.writerow(test)
efthymios likes this post
Reply
#3
Thank you for the help with the code. I will use the editor for future posts. I haven't posted before so I didn't know how I should do it properly.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Problem with scrapping Website giddyhead 1 1,662 Mar-08-2024, 08:20 AM
Last Post: AhanaSharma
  python web scrapping mg24 1 383 Mar-01-2024, 09:48 PM
Last Post: snippsat
  How can I ignore empty fields when scrapping never5000 0 1,416 Feb-11-2022, 09:19 AM
Last Post: never5000
  Suggestion request for scrapping html table Vkkindia 3 2,086 Dec-06-2021, 06:09 PM
Last Post: Larz60+
  Scrapy does not show all data in iteration loop georgekasa 0 2,018 Jul-31-2021, 09:10 AM
Last Post: georgekasa
  web scrapping through Python Naheed 2 2,656 May-17-2021, 12:02 PM
Last Post: Naheed
  Website scrapping and download santoshrane 3 4,397 Apr-14-2021, 07:22 AM
Last Post: kashcode
  Newbie help with lxml scrapping chelsealoa 1 1,886 Jan-08-2021, 09:14 AM
Last Post: Larz60+
  Scrapping Sport score laplacea 1 2,302 Dec-13-2020, 04:09 PM
Last Post: Larz60+
  Web scrapping - Stopped working peterjv26 2 3,119 Sep-23-2020, 08:30 AM
Last Post: peterjv26

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020