Python Forum
hi new at python , trying to get urls from website
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
hi new at python , trying to get urls from website
#1
hello,
i'm new at python ,and i'm trying to get urls from website and write it to csv file


this is my code :

from urllib.request import urlopen
from bs4 import BeautifulSoup

html = urlopen("https://XXX") # Insert your URL to extract
bsObj = BeautifulSoup(html.read(), "lxml")
bsObj.title.text # check if im right website
for link in bsObj.find_all('a'):
    print(link.get('href'))
	 
	 cookieProcessor = urllib3.request.HTTPCookieProcessor()

from pyquery import PyQuery
import requests
payload = {'inUserName': 'XX', 'inUserPass': 'XX'}
url = 'XXX'
from bs4 import BeautifulSoup
import requests
url = 'XXX'
def links(url):
    html = requests.get(url).content
    bsObj = BeautifulSoup(html, 'lxml')

    finalLinks = set()
    for link in links:
        finalLinks.add(link.attrs['href'])
		  links = bsObj.findAll('a')

    for link in links :
        print (link[0].text, file=open("e:\dvir.csv", "href"))
my question is :
1. im not doing it right - what is the error with my code ?
2. how to export all my links to csv ?

thanks
Reply


Messages In This Thread
hi new at python , trying to get urls from website - by dviry - Feb-23-2018, 05:01 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Retrieve website content using Python? Vadanane 1 1,293 Jan-16-2023, 09:55 AM
Last Post: Axel_Erfurt
  BeautifulSoup not parsing other URLs giddyhead 0 1,217 Feb-23-2022, 05:35 PM
Last Post: giddyhead
  I want to create an automated website in python mkdhrub1 2 2,469 Dec-27-2021, 11:27 PM
Last Post: Larz60+
  Python to build website Methew324 1 2,254 Dec-15-2020, 05:57 AM
Last Post: buran
  Scraping all website text using Python MKMKMKMK 1 2,107 Nov-26-2020, 10:35 PM
Last Post: Larz60+
  Need logic on how to scrap 100K URLs goodmind 2 2,659 Jun-29-2020, 09:53 AM
Last Post: goodmind
  Python Webscraping with a Login Website warriordazza 0 2,628 Jun-07-2020, 07:04 AM
Last Post: warriordazza
  Python tool based on website? zarize 2 2,511 Mar-21-2020, 02:25 PM
Last Post: zarize
  Scrape multiple urls LXML santdoyle 1 3,579 Oct-26-2019, 09:53 PM
Last Post: snippsat
  Need to Verify URLs; getting SSLError rahul_goswami 0 2,234 Aug-20-2019, 10:17 AM
Last Post: rahul_goswami

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020