So ive been working on a scrapying project and up until now everything has been working great and as expected. Now that we have successfully scrapped the website we wanted, we wanted to port it over to a database to store the data instead writing to a csv file.
My attempt has failed due to a cursor error within my scrapy pipelines.py file.
here is my code: (Below the code is the error im getting)
AttributeError: 'DsgDbPipeline' object has no attribute 'cursor'
I went ahead and moved the connection lines around and got it to work..
Basically move the connection and cursor down into the process_item and that worked..
My attempt has failed due to a cursor error within my scrapy pipelines.py file.
here is my code: (Below the code is the error im getting)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
def _init_( self ): try : self .mariadb_connection = mariadb.connect(user = 'test' , password = 'test' , database = 'test' ) self .cursor = self .mariadb_connection.cursor() except mariadb.Error as e: raise e # def close_spider(self, spider): # ## clean up when spider is closed # self.client.close() def process_item( self , item, spider): try : [b] self .cursor.execute[ / b]( "INSERT INTO test (style,brand,description,price,compurl,reviewcount,reviewrating,model,packsize,spechandle,specbladelength,specoveralllength,specweight) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)" , (item[ 'style' ], item[ 'brand' ], item[ 'description' ], item[ 'price' ], item[ 'compurl' ], item[ 'reviewcount' ], item[ 'reviewrating' ], item[ 'model' ], item[ 'spechandle' ], item[ 'specbladelength' ], item[ 'specoveralllength' ], item[ 'specweight' ], item[ 'packsize' ])) self .conn.commit() except mariadb.Error as e: print ( "Error %d: %s" % (e.args[ 0 ], e.args[ 1 ])) return item |
I went ahead and moved the connection lines around and got it to work..
Basically move the connection and cursor down into the process_item and that worked..