Python Forum
How and where to store a data for path tree?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How and where to store a data for path tree?
#1
Hello.
I am developing a small scrapper that allows to save images from different urls and save it on many different folders. I have the skeleton for directory hierarchy(6-8 folders and 3-6 subfolders) and basic names, but each session I need to create new root folder with current date, and some custom text from text input.

I want that my folder structure would be easy to edit from the outside of script. Also I want to be able in future to add a feature where I could visually create directory tree with using wxpython treectrls.

My question is what's the best way to store all path tree data? So
1) It would be easy and comprehensively to edit it outside of a script
2) It could be quickly converted and handled by python.


Maybe xml, or json or a simple python file with lists(but it looks too hard to read a hierarchy for me, maybe I wrong)?
I couldn't find the solution or common pattern at google.
Reply
#2
I use relative path's all the time, using pathlib
I keep all of my paths in one file for each project, and name it ProjectPaths.py (replace 'Project' with actual name, like 'IllinoisPaths.py'
I then usually define paths for each type of file, csv, json, html, txt etc.
Note that the directory will be created if it doesn't exist.
It will not overwrite existing directories of same name.

example:
# BusinessPaths.py

from pathlib import Path
import os


class BusinessPaths:
    def __init__(self):
        os.chdir(os.path.abspath(os.path.dirname(__file__)))
        self.homepath = Path('.')
        self.rootpath = self.homepath / '..'

        self.datapath = self.rootpath / 'data'
        self.datapath.mkdir(exist_ok=True)

        self.dbpath = self.datapath / 'database'
        self.dbpath.mkdir(exist_ok=True)

        self.htmlpath = self.datapath / 'html'
        self.htmlpath.mkdir(exist_ok=True)

        self.idpath = self.datapath / 'Idfiles'
        self.idpath.mkdir(exist_ok=True)
        
        self.jsonpath = self.datapath / 'json'
        self.jsonpath.mkdir(exist_ok=True)

        self.prettypath = self.datapath / 'pretty'
        self.prettypath.mkdir(exist_ok=True)

        self.textpath = self.datapath / 'text'
        self.textpath.mkdir(exist_ok=True)

        self.tmppath = self.datapath / 'tmp'
        self.tmppath.mkdir(exist_ok=True)

        self.base_url = 'http://searchctbusiness.ctdata.org/'

        self.cities_json = self.jsonpath / 'cities.json'
        self.city_list_url = 'https://ctstatelibrary.org/cttowns/counties'
        self.raw_city_file = self.tmppath / 'raw_city.html'
        # self.cities_text = self.textpath / 'cities.txt'

        self.company_master_json = self.jsonpath / 'CompanyMaster.json'

        self.CompanyMasterDb = self.dbpath / 'CompanyMaster.db'

        self.company_main = self.jsonpath / 'CompanyMain.json'
        self.company_detail = self.jsonpath / 'CompanyDetail.json'
        self.company_principals = self.jsonpath / 'CompanyPrincipals.json'
        self.company_agents = self.jsonpath / 'CompanyAgents.json'
        self.company_filings = self.jsonpath / 'CompanyFilings.json'


if __name__ == '__main__':
    BusinessPaths()
So if I want to open a file to dump some data in my json directory into a file names 'mystuff.json' it's as simeple as:
from BusinessPaths import BusinessPaths
import json

sampledict = {
    'one': 1,
    'two': 2,
    'three': 3
}

bpaths = BusinessPaths()

jsonfile = bpaths.jsonpath / 'junk.json'

with jsonfile.open('w') as fp:
    json.dump(sampledict, fp)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How can users store data temporarily in flask app? darktitan 6 2,858 Mar-21-2022, 06:38 PM
Last Post: darktitan
  Extract json-ld schema markup data and store in MongoDB Nuwan16 0 2,412 Apr-05-2020, 04:06 PM
Last Post: Nuwan16
  how to store data from the soup in the loop? zarize 1 1,637 Mar-02-2020, 04:17 PM
Last Post: Larz60+
  Cant set api response as tree in Element tree hey_arnold 4 3,634 Mar-04-2019, 03:25 PM
Last Post: dsarin

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020