+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Store a python list of lists in a database to use for searches later on (/thread-19245.html)
Store a python list of lists in a database to use for searches later on - klllmmm - Jun-20-2019
When I scarp a web page I'm getting data similar to below lists.
MainID = [[1]] AddrID = [['1'], ['2']] Address1 = [['Lane 1'], ['Lane 2']] Address2 = [['State 1'], ['State 2']]Example of the table view shown in the attached picture
I'm thinking of how to store in a database in order to do searches afterward.
Eg. If my keyword (eg. "Lane 1") matches database column "Address1" I should be able to get an entire data point.
I'm not sure what is the database software suitable for this.
Do I have to get elements of lists and save in several columns and create SQLite/PostgreSQL type database or can I save this data in a MongoDB or similar NoSQL type database?
Appreciate if you can give some inputs
RE: Store a python list of lists in a database to use for searches later on - ODIS - Jun-20-2019
You can save the addresses to the separate table (one address per row). Use indexes on columns where you'll do the searching and you're ready to go. Easy to implement in any DB engine.
RE: Store a python list of lists in a database to use for searches later on - noisefloor - Jun-20-2019
Hi,
generally speaking, you can store anything into any data base. Just sometimes it's difficult to make data structures fit, thus some data bases don't make too much sense for certain structures.
The decision should also be based on which values you want to be able to query in the end.
In case you want to use a RDBMS, I would suggest two tables: one for the MainID and one for the address, whereas the tables may have a one-to-many relation.
Plan B would be to transfer the data set to JSON - which is easy - and store in a data base which accepts JSON. Like MongoDB or PostgreSQL.
Regards, noisefloor
RE: Store a python list of lists in a database to use for searches later on - buran - Jun-20-2019
In addition to everything said so far - it looks these nested lists are just result of you mishandling the information when scrape it from web. Also it's not clear what your data look like indeed. e.g. is AddrId=1, Line 1 and State1 part of the same address?