Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Scraping data saving to DB error with Cursor
#1
So ive been working on a scrapying project and up until now everything has been working great and as expected. Now that we have successfully scrapped the website we wanted, we wanted to port it over to a database to store the data instead writing to a csv file.

My attempt has failed due to a cursor error within my scrapy pipelines.py file.

here is my code: (Below the code is the error im getting)
    def _init_(self):
        try:
            self.mariadb_connection = mariadb.connect(user='test', password='test', database='test')
            self.cursor = self.mariadb_connection.cursor()
            
        except mariadb.Error as e:
            raise e
    
#    def close_spider(self, spider):
#        ## clean up when spider is closed
#        self.client.close()
        
    def process_item(self, item, spider):
        try:
            [b]self.cursor.execute[/b]( "INSERT INTO test (style,brand,description,price,compurl,reviewcount,reviewrating,model,packsize,spechandle,specbladelength,specoveralllength,specweight) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", 
            (item['style'],
            item['brand'],                    
            item['description'],
            item['price'],
            item['compurl'],
            item['reviewcount'],
            item['reviewrating'],
            item['model'],
            item['spechandle'],
            item['specbladelength'],
            item['specoveralllength'],
            item['specweight'],
            item['packsize'])) 
            self.conn.commit()
            
        except mariadb.Error as e:
            print("Error %d: %s" % (e.args[0], e.args[1]))
            return item
AttributeError: 'DsgDbPipeline' object has no attribute 'cursor'

I went ahead and moved the connection lines around and got it to work..
Basically move the connection and cursor down into the process_item and that worked..
Quote
#2
The reason DsgDbPipeline object has no attribute cursor is because the init did not happen because
def _init_(self):
should have double underscores
def __init__(self):
Quote
#3
so should i re-enable the init properly formatted and run it that way from now on?
Quote
#4
Yes, if you fix the init it will work as originally intended.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Web Scraping Error : Not getting expected result adminravi 4 128 Oct-08-2019, 09:53 AM
Last Post: snippsat
  Scraping data from ebay seller page yuvalta 3 2,064 Sep-25-2019, 04:22 AM
Last Post: sandramoraes
  Python Scraping Error ZenWoR 1 463 Sep-15-2018, 08:23 PM
Last Post: snippsat
  Web Scraping and data collection. LordDeanUK 5 1,131 Feb-08-2018, 05:39 PM
Last Post: andrew23
  Error while scraping item price from online store mgtheboss 2 1,318 Jan-12-2018, 06:42 PM
Last Post: snippsat
  Scraping data from a web page where same class name applied multiple times sumandas89 1 4,328 Dec-30-2017, 11:03 AM
Last Post: buran
  Error while scraping links with beautiful soup mgtheboss 4 2,890 Dec-22-2017, 12:41 PM
Last Post: mgtheboss

Forum Jump:


Users browsing this thread: 1 Guest(s)