Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Extracting links from website with selenium bs4 and python
#1
Okay so.

Heading might seem like this question ahs already been asked but I had no luck finding an answer for it.

I need help about making link extracting program with python.

Actually It works. It finds all elements on a webpage. Takes their href="" and puts it in array. Then it exports it in csv file. Which is what I want.

But I can't get a hold of one thing.

Website is dynamic so I am using Selenium webdriver to get JavaScript result.

Code for program is pretty simple. I open website with webdriver and then get it's content. Then I get all links with

results = driver.find_elements_by_tag_name('a')
Then I lop throught results with for loop and get href with

result.get_attribute("href")
I store results in array and then print them out.

But problem is that I can't get name of the links.

<a href="https://www.google.com">This leads to Google</a>
Is there any way to get 'This leads to Google' string.

I need it for every link that is stored in array.

Thank you for your time

UPDATE

As it seems it only get's dynamic links. I just notices this. This is really strange now. For hard coded items it returns empty string. For dynamic link it returns it's name.

Okay so. Answer was using
get_attribude("textContent")
It returns string with name.
Quote
#2
what is the URL?
have you seen:
Web scraping 1 & 2:
https://python-forum.io/Thread-Web-Scraping-part-1
https://python-forum.io/Thread-Web-scraping-part-2
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Create .exe file for Python flask website. vintysaw 2 81 Nov-13-2019, 08:50 AM
Last Post: vintysaw
  webscrapping links and then enter those links to scrape data kirito85 2 341 Jun-13-2019, 02:23 AM
Last Post: kirito85
  Python + request from specific website - please help hoff1022 8 749 Feb-14-2019, 06:52 PM
Last Post: buran
  Download all secret links from a map design website fyec 0 686 Jul-24-2018, 09:08 PM
Last Post: fyec
  hi new at python , trying to get urls from website dviry 6 1,381 Feb-24-2018, 07:34 PM
Last Post: metulburr
  Error in Selenium: CRITICAL:root:Selenium module is not installed...Exiting program. AcszE 1 1,054 Nov-03-2017, 08:41 PM
Last Post: metulburr
  Selenium to pick data from csv and enter into website Prince_Bhatia 1 2,371 Sep-08-2017, 10:58 AM
Last Post: hbknjr
  How to integrate a python algorithm in a Django website daekch 2 2,206 Mar-21-2017, 10:58 AM
Last Post: Jeanne

Forum Jump:


Users browsing this thread: 1 Guest(s)