Jan-23-2020, 02:19 PM
Dear Members,
I want to pass both name and link_only from parse to parse_country. I know we use the meta function to transfer. Could you please tell me how can I change the meta function below to transfer both items? With the following code, I get blank cells in the CSV file for the link_only column.
In my actual project, I have 7/8 items to transfer from one parse to another. Please suggest me how to make it
I want to pass both name and link_only from parse to parse_country. I know we use the meta function to transfer. Could you please tell me how can I change the meta function below to transfer both items? With the following code, I get blank cells in the CSV file for the link_only column.
In my actual project, I have 7/8 items to transfer from one parse to another. Please suggest me how to make it
import scrapy import logging class CountriesSpider(scrapy.Spider): name = 'countries' allowed_domains = ['www.worldometers.info'] start_urls = ['https://www.worldometers.info/world-population/population-by-country/'] def parse(self, response): countries=response.xpath("//td/a") for country in countries: name=country.xpath(".//text()").get() link_only=country.xpath(".//@href/text()").get() link=country.xpath(".//@href").get() yield response.follow(url=link, callback=self.parse_country, meta={'country_name': name, 'link_only':link_only}) def parse_country(self, response): name=response.request.meta['country_name'] link_only=response.request.meta['link_only'] rows = response.xpath("(//table[@class='table table-striped table-bordered table-hover table-condensed table-list'])[1]/tbody/tr") for row in rows: year=row.xpath(".//td[1]/text()").get() population=row.xpath(".//td[2]/strong/text()").get() yield{ 'country_name': name, 'link_only': link_only, 'year': year, 'population': population }