Python Forum
Store a python list of lists in a database to use for searches later on
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Store a python list of lists in a database to use for searches later on
#1
When I scarp a web page I'm getting data similar to below lists.

MainID = [[1]]   
AddrID = [['1'], ['2']]    
Address1 = [['Lane 1'], ['Lane 2']]
Address2 = [['State 1'], ['State 2']]
Example of the table view shown in the attached picture

I'm thinking of how to store in a database in order to do searches afterward.

Eg. If my keyword (eg. "Lane 1") matches database column "Address1" I should be able to get an entire data point.

I'm not sure what is the database software suitable for this.

Do I have to get elements of lists and save in several columns and create SQLite/PostgreSQL type database or can I save this data in a MongoDB or similar NoSQL type database?

Appreciate if you can give some inputs

Attached Files

Thumbnail(s)
   
Reply
#2
You can save the addresses to the separate table (one address per row). Use indexes on columns where you'll do the searching and you're ready to go. Easy to implement in any DB engine.
Reply
#3
Hi,

generally speaking, you can store anything into any data base. Just sometimes it's difficult to make data structures fit, thus some data bases don't make too much sense for certain structures.
The decision should also be based on which values you want to be able to query in the end.

In case you want to use a RDBMS, I would suggest two tables: one for the MainID and one for the address, whereas the tables may have a one-to-many relation.

Plan B would be to transfer the data set to JSON - which is easy - and store in a data base which accepts JSON. Like MongoDB or PostgreSQL.

Regards, noisefloor
Reply
#4
In addition to everything said so far - it looks these nested lists are just result of you mishandling the information when scrape it from web. Also it's not clear what your data look like indeed. e.g. is AddrId=1, Line 1 and State1 part of the same address?
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Webscrape from my webpage and store in database and send to grafana Armond 2 874 Jul-12-2023, 06:29 PM
Last Post: Armond
  List all possibilities of a nested-list by flattened lists sparkt 1 878 Feb-23-2023, 02:21 PM
Last Post: sparkt
  user input values into list of lists tauros73 3 1,025 Dec-29-2022, 05:54 PM
Last Post: deanhystad
  returning a List of Lists nafshar 3 1,014 Oct-28-2022, 06:28 PM
Last Post: deanhystad
  Creating list of lists, with objects from lists sgrinderud 7 1,564 Oct-01-2022, 07:15 PM
Last Post: Skaperen
  store all variable values into list and insert to sql_summary table mg24 3 1,097 Sep-28-2022, 09:13 AM
Last Post: Larz60+
  How to store the resulting Doc objects into a list named A xinyulon 1 1,853 Mar-08-2022, 11:49 PM
Last Post: bowlofred
  How to efficiently average same entries of lists in a list xquad 5 2,070 Dec-17-2021, 04:44 PM
Last Post: xquad
  Populate Dropdown list from Database TommyAutomagically 4 4,351 Nov-02-2021, 05:23 PM
Last Post: ndc85430
  sorting a list of lists by an element leapcfm 3 1,805 Sep-10-2021, 03:33 PM
Last Post: leapcfm

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020