Python Forum
How to export to csv the output of every iteration when scrapping with a loop
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to export to csv the output of every iteration when scrapping with a loop
#1
Hi,
I am very new to Python. I am trying to scrape a website and I have create a small code for this:

select = Select(character.find_element_by_id('character-template-choice'))
options = select.options
for index in range(0, len(options) - 0):
    select.select_by_index(index)
    option1 = character.find_elements_by_class_name('pc-stat-value')
    power = []
    for c in option1:
        power.append("Power:" + c.text)
    option2 = character.find_elements_by_class_name(
        'unit-stat-group-stat-label')
    skills = []
    for a in option2:
        skills.append(a.text)
    option3 = character.find_elements_by_class_name(
        'unit-stat-group-stat-value')
    values = []
    for b in option3:
        values.append(b.text)
    test = [power, skills, values]
The issue I have is when I try to export "test" list of lists to csv I only get the last iteration. I want to get the data for every iteration but I don't know how to do so. Can someone help me?

Thank you
Larz60+ write Nov-26-2020, 10:25 PM:
Please post all code, output and errors (it it's entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.

I fixed for you this time. Please use bbcode tags on future posts.
Reply
#2
Please, when posting code, provide enough to be able to run.

if you haven't already done so, import csv at top of script

# untested code, you may have to correct any errors.
select = Select(character.find_element_by_id('character-template-choice'))
options = select.options
savefile = 'savefile.txt' # Change to your desired file name
with open(savefile, 'w') as fp":
    csvwriter = csv.writer(fp)
    for index in range(0, len(options) - 0):
    select.select_by_index(index)
    option1 = character.find_elements_by_class_name('pc-stat-value')
    power = []
    for c in option1:
        power.append("Power:" + c.text)
    option2 = character.find_elements_by_class_name(
        'unit-stat-group-stat-label')
    skills = []
    for a in option2:
        skills.append(a.text)
    option3 = character.find_elements_by_class_name(
        'unit-stat-group-stat-value')
    values = []
    for b in option3:
        values.append(b.text)
    test = [power, skills, values]
        csvwriter.writerow(test)
efthymios likes this post
Reply
#3
Thank you for the help with the code. I will use the editor for future posts. I haven't posted before so I didn't know how I should do it properly.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Problem with scrapping Website giddyhead 1 1,638 Mar-08-2024, 08:20 AM
Last Post: AhanaSharma
  python web scrapping mg24 1 338 Mar-01-2024, 09:48 PM
Last Post: snippsat
  How can I ignore empty fields when scrapping never5000 0 1,399 Feb-11-2022, 09:19 AM
Last Post: never5000
  Suggestion request for scrapping html table Vkkindia 3 2,042 Dec-06-2021, 06:09 PM
Last Post: Larz60+
  Scrapy does not show all data in iteration loop georgekasa 0 2,006 Jul-31-2021, 09:10 AM
Last Post: georgekasa
  web scrapping through Python Naheed 2 2,631 May-17-2021, 12:02 PM
Last Post: Naheed
  Website scrapping and download santoshrane 3 4,340 Apr-14-2021, 07:22 AM
Last Post: kashcode
  Newbie help with lxml scrapping chelsealoa 1 1,870 Jan-08-2021, 09:14 AM
Last Post: Larz60+
  Scrapping Sport score laplacea 1 2,269 Dec-13-2020, 04:09 PM
Last Post: Larz60+
  Web scrapping - Stopped working peterjv26 2 3,092 Sep-23-2020, 08:30 AM
Last Post: peterjv26

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020