Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
[Scrapy] web scrape help
#1
Hello,
downloaded scrapy and went through the tutorials and still trying to understand the selector
aspect of scraping. So I thought scrape a different quotes web page:
website
I created a new project and spider:
# -*- coding: utf-8 -*-
import scrapy


class InspiderSpider(scrapy.Spider):
    name = 'inspider'
    allowed_domains = ['https://www.keepinspiring.me/famous-quotes/']
    start_urls = ['https://www.keepinspiring.me/famous-quotes//']

    def parse(self, response):
        for quotes in response.css('div.author-quotes'):
            yield {
                'text': quotes.css('span.text::text').extract_first(),
                'author': quotes.css('span.quote-author-name::text').extract_first()
            }
I can extract the authors but no luck on the quote.
output:
Output:
{"text": null, "author": "-Dr. Suess"}, {"text": null, "author": "-Marilyn Monroe"}, {"text": null, "author": null}, {"text": null, "author": "-Stephen King"}, {"text": null, "author": "-Mark Caine"}, {"text": null, "author": "-Helen Keller"}, .....
when I examine the quote element and copy xpath I get:
Output:
//*[@id="entry-4812"]/div/div[1]/div[6]/text()
any help appreciated,
Joe
Reply


Messages In This Thread
[Scrapy] web scrape help - by joe_momma - Sep-30-2019, 05:18 PM
RE: [Scrapy] web scrape help - by stranac - Sep-30-2019, 08:21 PM
RE: [Scrapy] web scrape help - by joe_momma - Oct-01-2019, 12:44 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  scrape data 1 go to next page scrape data 2 and so on alkaline3 6 5,436 Mar-13-2020, 07:59 PM
Last Post: alkaline3
  Scrapy-cut: Advanced Cookiecutter Scrapy Templating scriptso 2 4,768 Feb-02-2017, 07:57 PM
Last Post: scriptso

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020