Scraping data saving to DB error with Cursor

Scraping data saving to DB error with Cursor - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: Scraping data saving to DB error with Cursor (/thread-18504.html)

Scraping data saving to DB error with Cursor - cubangt - May-20-2019

So ive been working on a scrapying project and up until now everything has been working great and as expected. Now that we have successfully scrapped the website we wanted, we wanted to port it over to a database to store the data instead writing to a csv file.

My attempt has failed due to a cursor error within my scrapy pipelines.py file.

here is my code: (Below the code is the error im getting)

    def _init_(self):
        try:
            self.mariadb_connection = mariadb.connect(user='test', password='test', database='test')
            self.cursor = self.mariadb_connection.cursor()
            
        except mariadb.Error as e:
            raise e
    
#    def close_spider(self, spider):
#        ## clean up when spider is closed
#        self.client.close()
        
    def process_item(self, item, spider):
        try:
            [b]self.cursor.execute[/b]( "INSERT INTO test (style,brand,description,price,compurl,reviewcount,reviewrating,model,packsize,spechandle,specbladelength,specoveralllength,specweight) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)", 
            (item['style'],
            item['brand'],                    
            item['description'],
            item['price'],
            item['compurl'],
            item['reviewcount'],
            item['reviewrating'],
            item['model'],
            item['spechandle'],
            item['specbladelength'],
            item['specoveralllength'],
            item['specweight'],
            item['packsize'])) 
            self.conn.commit()
            
        except mariadb.Error as e:
            print("Error %d: %s" % (e.args[0], e.args[1]))
            return item

AttributeError: 'DsgDbPipeline' object has no attribute 'cursor'

I went ahead and moved the connection lines around and got it to work..
Basically move the connection and cursor down into the process_item and that worked..

RE: Scraping data saving to DB error with Cursor - Yoriz - May-20-2019

The reason DsgDbPipeline object has no attribute cursor is because the init did not happen because

def _init_(self):

should have double underscores

def __init__(self):

RE: Scraping data saving to DB error with Cursor - cubangt - May-20-2019

so should i re-enable the init properly formatted and run it that way from now on?

RE: Scraping data saving to DB error with Cursor - Yoriz - May-20-2019

Yes, if you fix the init it will work as originally intended.