Python Forum
Python Scrapy Date Extraction Issue
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python Scrapy Date Extraction Issue
#1
I am struggling with how to properly extract an oddly formatted date that exists on the page in one location using Python Scrapy.

What needs to be changed to have the date included on every output row in yyyy-mm-dd format?

Problematic code lines:

data2 = response.xpath('//span[@class="tab"]/text()').get().replace(". ", "-")
date = datetime.datetime.strptime(data2, "%d-%m-%Y").strftime("%Y-%m-%d")
Sample output appears to contain one character for date. Example: {'match_id': '1893065', 'date': '0'}

Here is the full spider.
import scrapy import datetime import re from datetime import timedelta

class Tennis_ExplorerSpider(scrapy.Spider):
    name = 'tennis_explorer'
    allowed_domains = ['tennisexplorer.com']

    def daterange(start_date, end_date):
        for n in range(int((end_date - start_date).days)):
            yield start_date + timedelta(n)
    
    start_date = datetime.datetime.today() - datetime.timedelta(days=1)
    end_date = datetime.datetime.today() + datetime.timedelta(days=1)    
    start_urls = []
    start_url='https://www.tennisexplorer.com/matches/?type=all&year='
    for single_date in daterange(start_date, end_date):
        start_urls.append(single_date.strftime(start_url+"%Y&month=%m&day=%d&timezone=-6"))

    
    def parse(self, response): 
            #Extracting the content using xpath            
            self.logger.debug('callback "parse": got response %r' % response)
            data = response.xpath('//table[@class="result"]//a[contains(@href,"match-detail")]/@href').extract()
            match_id =[re.sub('^.+=','',el) for el in data]

            data2 = response.xpath('//span[@class="tab"]/text()').get().replace(". ", "-")
            date = datetime.datetime.strptime(data2, "%d-%m-%Y").strftime("%Y-%m-%d")
            
            #Give the extracted content row wise
            for item in zip(match_id, date):
                #create a dictionary to store the scraped info
                scraped_info = {
                    'match_id' : item[0],
                    'date' : item[1]
                }

                #yield or give the scraped info to scrapy
                yield scraped_info
Reply


Messages In This Thread
Python Scrapy Date Extraction Issue - by tr8585 - Aug-05-2020, 02:06 AM
RE: Python Scrapy Date Extraction Issue - by tr8585 - Aug-05-2020, 04:32 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Python Scrapy tr8585 2 2,518 Aug-04-2020, 04:11 AM
Last Post: tr8585
  Article Extraction - Wordpress svzekio 7 5,584 Jul-10-2020, 10:18 PM
Last Post: steve_shambles
  Follow Up: Web Calendar based Extraction AgileAVS 0 1,598 Feb-23-2020, 05:39 AM
Last Post: AgileAVS
  Python - Scrapy Baggelhsk95 0 2,367 Apr-24-2019, 01:07 PM
Last Post: Baggelhsk95
  fb data extraction error periraviteja 1 2,313 Jan-05-2019, 01:07 AM
Last Post: stullis
  Python Scrapy ebay API Baggelhsk95 0 3,294 Nov-21-2018, 11:22 AM
Last Post: Baggelhsk95
  Python scrapy scraped_items Baggelhsk95 2 3,005 Nov-13-2018, 08:30 AM
Last Post: Baggelhsk95
  Python - Scrapy - CSS selector Baggelhsk95 1 5,695 Nov-07-2018, 04:45 PM
Last Post: stranac
  Python - Scrapy - Contains Baggelhsk95 3 4,694 Oct-27-2018, 03:42 PM
Last Post: stranac
  Python - Scrapy Login in Baggelhsk95 3 5,023 Oct-23-2018, 04:24 PM
Last Post: stranac

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020