Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Extracting links from website with selenium bs4 and python
Okay so.

Heading might seem like this question ahs already been asked but I had no luck finding an answer for it.

I need help about making link extracting program with python.

Actually It works. It finds all elements on a webpage. Takes their href="" and puts it in array. Then it exports it in csv file. Which is what I want.

But I can't get a hold of one thing.

Website is dynamic so I am using Selenium webdriver to get JavaScript result.

Code for program is pretty simple. I open website with webdriver and then get it's content. Then I get all links with

results = driver.find_elements_by_tag_name('a')
Then I lop throught results with for loop and get href with

I store results in array and then print them out.

But problem is that I can't get name of the links.

<a href="">This leads to Google</a>
Is there any way to get 'This leads to Google' string.

I need it for every link that is stored in array.

Thank you for your time


As it seems it only get's dynamic links. I just notices this. This is really strange now. For hard coded items it returns empty string. For dynamic link it returns it's name.

Okay so. Answer was using
It returns string with name.
what is the URL?
have you seen:
Web scraping 1 & 2:

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Python Webscraping with a Login Website warriordazza 0 160 Jun-07-2020, 07:04 AM
Last Post: warriordazza
  Dynamic links with selenium EvilDodo 1 212 Apr-04-2020, 03:18 PM
Last Post: ndc85430
  Python tool based on website? zarize 2 448 Mar-21-2020, 02:25 PM
Last Post: zarize
  Extracting all the links on a website randeniyamohan 1 965 Jan-09-2020, 04:47 PM
Last Post: Clunk_Head
  Scrapping javascript website with Selenium where pages randomly fail to load JuanJuan 14 1,128 Dec-27-2019, 12:32 PM
Last Post: JuanJuan
  Create .exe file for Python flask website. vintysaw 4 2,594 Nov-18-2019, 07:56 AM
Last Post: tonycstech
  webscrapping links and then enter those links to scrape data kirito85 2 656 Jun-13-2019, 02:23 AM
Last Post: kirito85
  Python + request from specific website - please help hoff1022 8 1,036 Feb-14-2019, 06:52 PM
Last Post: buran
  Download all secret links from a map design website fyec 0 926 Jul-24-2018, 09:08 PM
Last Post: fyec
  hi new at python , trying to get urls from website dviry 6 1,767 Feb-24-2018, 07:34 PM
Last Post: metulburr

Forum Jump:

Users browsing this thread: 1 Guest(s)