Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Extracting links from website with selenium bs4 and python
Okay so.

Heading might seem like this question ahs already been asked but I had no luck finding an answer for it.

I need help about making link extracting program with python.

Actually It works. It finds all elements on a webpage. Takes their href="" and puts it in array. Then it exports it in csv file. Which is what I want.

But I can't get a hold of one thing.

Website is dynamic so I am using Selenium webdriver to get JavaScript result.

Code for program is pretty simple. I open website with webdriver and then get it's content. Then I get all links with

results = driver.find_elements_by_tag_name('a')
Then I lop throught results with for loop and get href with

I store results in array and then print them out.

But problem is that I can't get name of the links.

<a href="">This leads to Google</a>
Is there any way to get 'This leads to Google' string.

I need it for every link that is stored in array.

Thank you for your time


As it seems it only get's dynamic links. I just notices this. This is really strange now. For hard coded items it returns empty string. For dynamic link it returns it's name.

Okay so. Answer was using
It returns string with name.
what is the URL?
have you seen:
Web scraping 1 & 2:

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Create .exe file for Python flask website. vintysaw 2 81 Nov-13-2019, 08:50 AM
Last Post: vintysaw
  webscrapping links and then enter those links to scrape data kirito85 2 341 Jun-13-2019, 02:23 AM
Last Post: kirito85
  Python + request from specific website - please help hoff1022 8 749 Feb-14-2019, 06:52 PM
Last Post: buran
  Download all secret links from a map design website fyec 0 686 Jul-24-2018, 09:08 PM
Last Post: fyec
  hi new at python , trying to get urls from website dviry 6 1,381 Feb-24-2018, 07:34 PM
Last Post: metulburr
  Error in Selenium: CRITICAL:root:Selenium module is not installed...Exiting program. AcszE 1 1,054 Nov-03-2017, 08:41 PM
Last Post: metulburr
  Selenium to pick data from csv and enter into website Prince_Bhatia 1 2,371 Sep-08-2017, 10:58 AM
Last Post: hbknjr
  How to integrate a python algorithm in a Django website daekch 2 2,206 Mar-21-2017, 10:58 AM
Last Post: Jeanne

Forum Jump:

Users browsing this thread: 1 Guest(s)