Python Forum
python selenium - get info from nested divs
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
python selenium - get info from nested divs
#1
Hi, I'm new to selenium and am trying to grab information from the html below. The info I want is the student name located at the a_href (below as John Doe), the number of questions answered in the div "td questions-num"(425 shown below), the time spent in the div "td time-spent" (2hr 0min) and last practiced in the "last-practiced-text" div. It then has these details again in the divs below again <div class="tr student-info">...</div>. If there is a resource that you think might help me learn this I would also appreciate it if you mention it.

<div id="bd"> 
    <div class="report-title">...</div>
    <div class="all-students-report" id="yui_3_18_1_1_1614259351527_178">
        <div class="usage-summary">...</div>
        <div class="students-header">...</div>
        <div class="students-table" id="yui_3_18_1_1_1614259351527_177">
            <div class="tbody" id="yui_3_18_1_1_1614259351527_176">
                <div class="tr student-info" id="yui_3_18_1_1_1614259351527_175">
                    <div class="student-info-container" id="yui_3_18_1_1_1614259351527_174">
                        <div class="student-avatar-and-name" id="yui_3_18_1_1_1614259351527_173">
                            <div class="td student-avatar-column">...</div>
                            <div class="td student-name student-name-column" colspan="1" id="yui_3_18_1_1_1614259351527_172">
                                <a href="/analytics/student-details?teacherId=125756982#student=125757254" id="yui_3_18_1_1_1614259351527_170">John Doe</a>
                            </div>
                        </div>
                        <div class="td questions-num">
                            <div class="questions-num-pencil-icon"></div>
                            <span class="questions-num-data">425 questions</span>
                        </div>
                        <div class="td time-spent">
                            <div class="time-spent-clock-icon"></div>
                            <span class="time-spent-data">2 hr 0 min</span>
                        </div>
                        <div class="td last-active">
                            <div class="last-active-calendar-icon"></div>
                            <span class="last-active-data"><span class="last-practiced-text">Practised</span>
                            yesterday
                        </span>
                    </div
                    ></div>
                    <div class="td skill-row skill-name-column">...
                    </div>
                    <div class="tr student-info">...</div>
                    <div class="tr student-info">...</div>
                    <div class="tr student-info">...</div>
                    <div class="tr student-info">...</div>
I have tried to grab the details requested but the best I've done after I log onto the site is:

#select report
driver.get("https://uk.ixl.com/analytics/students-quickview?teacherId=125756982")
time.sleep(5)           

wait = WebDriverWait(driver, 10)

wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'date-range'))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, '//div[text()="Last 7 days"]'))).click()

student_data = wait.until(EC.presence_of_element_located((By.CLASS_NAME, 'tbody')))
for items in student_data:
    name = items.find_element_by_classname("student-name")
    print(name.text)
but I get a message saying 'WebElement' object is not iterable. So basically, I would appreciate if if someone could advise how I grab the information requested, thanks
Reply
#2
Hi,
You can try to identify the elements with their index, then illiterate on the index to the last one. Use this to identify by index
name=driver.find_element_by_xpath("(//*[contains(@class, 'student-name')])[1]")
print(name.text)
note the index [1]? you can replace it with a variable and illiterate with a For loop,else use a while loop(not soo elegant but can solve your problem)
x=1

while 1:
    try:
        name=driver.find_element_by_xpath("(//*[contains(@class, 'student-name')])[x]")
        print(name.text)
        #increment x by 1
        x+=1
    except Exception as e:
        break
damian0612 likes this post
Reply
#3
(Feb-25-2021, 09:50 PM)law Wrote: Hi,
You can try to identify the elements with their index, then illiterate on the index to the last one. Use this to identify by index
name=driver.find_element_by_xpath("(//*[contains(@class, 'student-name')])[1]")
print(name.text)
note the index [1]? you can replace it with a variable and illiterate with a For loop,else use a while loop(not soo elegant but can solve your problem)
x=1

while 1:
    try:
        name=driver.find_element_by_xpath("(//*[contains(@class, 'student-name')])[x]")
        print(name.text)
        #increment x by 1
        x+=1
    except Exception as e:
        break

fantastic, that has helped a lot, I can use
count = len(driver.find_elements_by_xpath("(//*[contains(@class, 'student-name')])"))
to find how many there are and use it to loop through the elements I need that amount of times, thanks very much
Reply
#4
Awesome, glad to have been of help
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  how to iterate through a divs tag yokaso 3 3,083 Nov-01-2019, 07:48 AM
Last Post: buran
  Error in Selenium: CRITICAL:root:Selenium module is not installed...Exiting program. AcszE 1 3,584 Nov-03-2017, 08:41 PM
Last Post: metulburr
  Web Crawler: How to find all divs that starts with... amandacstr 3 6,439 Oct-01-2016, 02:15 PM
Last Post: amandacstr

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020