Python Forum
Extracting links from website with selenium bs4 and python
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Extracting links from website with selenium bs4 and python
#1
Okay so.

Heading might seem like this question ahs already been asked but I had no luck finding an answer for it.

I need help about making link extracting program with python.

Actually It works. It finds all elements on a webpage. Takes their href="" and puts it in array. Then it exports it in csv file. Which is what I want.

But I can't get a hold of one thing.

Website is dynamic so I am using Selenium webdriver to get JavaScript result.

Code for program is pretty simple. I open website with webdriver and then get it's content. Then I get all links with

results = driver.find_elements_by_tag_name('a')
Then I lop throught results with for loop and get href with

result.get_attribute("href")
I store results in array and then print them out.

But problem is that I can't get name of the links.

<a href="https://www.google.com">This leads to Google</a>
Is there any way to get 'This leads to Google' string.

I need it for every link that is stored in array.

Thank you for your time

UPDATE

As it seems it only get's dynamic links. I just notices this. This is really strange now. For hard coded items it returns empty string. For dynamic link it returns it's name.

Okay so. Answer was using
get_attribude("textContent")
It returns string with name.
Reply
#2
what is the URL?
have you seen:
Web scraping 1 & 2:
https://python-forum.io/Thread-Web-Scraping-part-1
https://python-forum.io/Thread-Web-scraping-part-2
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Retrieve website content using Python? Vadanane 1 1,204 Jan-16-2023, 09:55 AM
Last Post: Axel_Erfurt
  All product links to products on a website MarionStorm 0 1,057 Jun-02-2022, 11:17 PM
Last Post: MarionStorm
  I want to create an automated website in python mkdhrub1 2 2,313 Dec-27-2021, 11:27 PM
Last Post: Larz60+
  Python to build website Methew324 1 2,195 Dec-15-2020, 05:57 AM
Last Post: buran
  Scraping all website text using Python MKMKMKMK 1 2,052 Nov-26-2020, 10:35 PM
Last Post: Larz60+
  Python Webscraping with a Login Website warriordazza 0 2,571 Jun-07-2020, 07:04 AM
Last Post: warriordazza
  Dynamic links with selenium EvilDodo 1 1,990 Apr-04-2020, 03:18 PM
Last Post: ndc85430
  Python tool based on website? zarize 2 2,437 Mar-21-2020, 02:25 PM
Last Post: zarize
  Extracting all the links on a website randeniyamohan 1 4,503 Jan-09-2020, 04:47 PM
Last Post: Clunk_Head
  Scrapping javascript website with Selenium where pages randomly fail to load JuanJuan 14 7,062 Dec-27-2019, 12:32 PM
Last Post: JuanJuan

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020