AttributeError: 'str' object has no attribute 'xpath' - nazmulfinance - Nov-10-2019
Dear Members,
I am very new to python. I am using the following code to extract the details of each product.
I am collecting product names from the original page and using each product link, I am collecting price, SKU, and frame information from the linked pages. However, I am getting an attribute error for the web-link command. One possible problem could be that the parsed link includes "https://www.glassesshop.com".
I think the main problem is href="https://www.glassesshop.com/eyeglasses/fz1370". The code would work if the link were href="/eyeglasses/fz1370"
Full python code:
import scrapy
class GlassSpider(scrapy.Spider):
name = 'glass'
allowed_domains = ['www.glassesshop.com']
start_urls = ['https://www.glassesshop.com/bestsellers/']
def parse(self, response):
names=response.xpath("//p[@class='pname col-sm-12']/a")
for name in names:
name=name.xpath(".//text()").get()
link=name.xpath(".//@href").get()
yield response.follow(url=link, callback=self.parse_glass, meta={'glass_name': name})
def parse_glass(self, response):
name=response.request.meta['glass_name']
price=response.xpath("//span[@class='product-price-original']/text()").get()
sku=response.xpath("//ul[@class='col-12 col-sm-6 default-content']/li[1]/text()").get()
frame=response.xpath("//a[@class='col01']/text()").get()
yield{
'glass_name': name,
'price': price,
'sku': sku,
'frame': frame
} I started working on python this week. I would highly appreciate if you can mention what specific change I should make to my codes to correct the attribute error
RE: AttributeError: 'str' object has no attribute 'xpath' - Larz60+ - Nov-11-2019
please always show complete and unmodified error traceback (in error tags) it contains valuable error debugging info.
RE: AttributeError: 'str' object has no attribute 'xpath' - nazmulfinance - Nov-11-2019
Error: (virtual_workspace) C:\Users\nazmu\projects\nextpage>scrapy crawl glass
2019-11-11 09:43:01 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: nextpage)
2019-11-11 09:43:01 [scrapy.utils.log] INFO: Versions: lxml 4.4.1.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.7.0, Python 3.7.5 (default, Oct 31 2019, 15:18:51) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 19.0.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.8, Platform Windows-10-10.0.18362-SP0
2019-11-11 09:43:01 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'nextpage', 'NEWSPIDER_MODULE': 'nextpage.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['nextpage.spiders']}
2019-11-11 09:43:02 [scrapy.extensions.telnet] INFO: Telnet Password: 5f97a9c6d439d439
2019-11-11 09:43:02 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2019-11-11 09:43:03 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2019-11-11 09:43:03 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2019-11-11 09:43:03 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2019-11-11 09:43:03 [scrapy.core.engine] INFO: Spider opened
2019-11-11 09:43:03 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-11-11 09:43:03 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2019-11-11 09:43:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.glassesshop.com/robots.txt> (referer: None)
2019-11-11 09:43:06 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.glassesshop.com/bestsellers> (referer: None)
2019-11-11 09:43:06 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.glassesshop.com/bestsellers> (referer: None)
Traceback (most recent call last):
File "C:\Users\nazmu\.conda\envs\virtual_workspace\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
yield next(it)
File "C:\Users\nazmu\.conda\envs\virtual_workspace\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 29, in process_spider_output
for x in result:
File "C:\Users\nazmu\.conda\envs\virtual_workspace\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 339, in <genexpr>
return (_set_referer(r) for r in result or ())
File "C:\Users\nazmu\.conda\envs\virtual_workspace\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in <genexpr>
return (r for r in result or () if _filter(r))
File "C:\Users\nazmu\.conda\envs\virtual_workspace\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in <genexpr>
return (r for r in result or () if _filter(r))
File "C:\Users\nazmu\projects\nextpage\nextpage\spiders\glass.py", line 14, in parse
link=name.xpath(".//@href").get()
AttributeError: 'str' object has no attribute 'xpath'
2019-11-11 09:43:06 [scrapy.core.engine] INFO: Closing spider (finished)
2019-11-11 09:43:06 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 457,
'downloader/request_count': 2,
'downloader/request_method_count/GET': 2,
'downloader/response_bytes': 29529,
'downloader/response_count': 2,
'downloader/response_status_count/200': 2,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2019, 11, 11, 14, 43, 6, 312496),
'log_count/DEBUG': 2,
'log_count/ERROR': 1,
'log_count/INFO': 9,
'response_received_count': 2,
'robotstxt/request_count': 1,
'robotstxt/response_count': 1,
'robotstxt/response_status_count/200': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'spider_exceptions/AttributeError': 1,
'start_time': datetime.datetime(2019, 11, 11, 14, 43, 3, 577779)}
2019-11-11 09:43:06 [scrapy.core.engine] INFO: Spider closed (finished)
(virtual_workspace) C:\Users\nazmu\projects\nextpage>
RE: AttributeError: 'str' object has no attribute 'xpath' - buran - Nov-11-2019
on the previous line
name=name.xpath(".//text()").get() you override name from the loop with the text. So on the next line (link=name.xpath(".//@href").get() ) name is just a str . use something different as variable name, not name
RE: AttributeError: 'str' object has no attribute 'xpath' - nazmulfinance - Nov-11-2019
Thank you Buran. I appreciate your help. The solution worked perfectly.
|