Create Dynamic nested Dictionaries

[Larz60+]

I just posted this on thread: https://python-forum.io/Thread-how-to-cr...ested-dict

It's useful enough that I'm posting it here as well:

#  Author: Larz60+ Nov 22, 2018
import os


class CreateDict:
    def __init__(self):
        os.chdir(os.path.abspath(os.path.dirname(__file__)))

    def new_dict(self, dictname):
        setattr(self, dictname, {})

    def add_node(self, parent, nodename):
        node = parent[nodename] = {}
        return node
    
    def add_cell(self, nodename, cellname, value):
        cell =  nodename[cellname] = value
        return cell

    def display_dict(self, dictname, level=0):
        indent = " " * (4 * level)
        for key, value in dictname.items():
            if isinstance(value, dict):
                print(f'\n{indent}{key}')
                level += 1
                self.display_dict(value, level)
            else:
                print(f'{indent}{key}: {value}')
            if level > 0:
                level -= 1


def testit():
    cd = CreateDict()

    cd.new_dict('CityList')

    boston = cd.add_node(cd.CityList, 'Boston')
    bos_resturants = cd.add_node(boston, 'Resturants')

    spoke = cd.add_node(bos_resturants, 'Spoke Wine Bar')
    cd.add_cell(spoke, 'Addr1', '89 Holland St')
    cd.add_cell(spoke, 'City', 'Sommerville')
    cd.add_cell(spoke, 'Addr1', '02144')
    cd.add_cell(spoke, 'Phone', '617-718-9463')

    highland = cd.add_node(bos_resturants, 'Highland Kitchen')
    cd.add_cell(highland, 'Addr1', '150 Highland Ave')
    cd.add_cell(highland, 'City', 'Sommerville')
    cd.add_cell(highland, 'ZipCode', '02144')
    cd.add_cell(highland, 'Phone', '617-625-1131')

    print(f'\nCityList Dictionary')
    cd.display_dict(cd.CityList)
    print(f'\nraw data: {cd.CityList}')

if __name__ == '__main__':
    testit()

Test results:

Output:
Boston

    Resturants

        Spoke Wine Bar
            Addr1: 02144
            City: Sommerville
            Phone: 617-718-9463

        Highland Kitchen
            Addr1: 150 Highland Ave
            City: Sommerville
            ZipCode: 02144
            Phone: 617-625-1131

raw data: {'Boston': {'Resturants': {'Spoke Wine Bar': {'Addr1': '02144', 'City': 'Sommerville', 'Phone': '617-718-9463'}, 'Highland Kitchen': {'Addr1': '150 Highland Ave', 'City': 'Sommerville', 'ZipCode': '02144', 'Phone': '617-625-1131'}}}}

[Gribouillis]

I like PyFilesystem (the fs module) very much, one reason being its extensibility. I created a simple abstract filesystem which manipulates a dictionary in memory. You can use it to create your dict like so

from dictfs import DictFs

def main():
    cd = DictFs()
    d = cd.makedirs('Boston/Restaurants/Spoke Wine Bar')
    with d.open('Addr1','w') as f: f.write('89 holland St')
    with d.open('City','w') as f: f.write('Sommerville')
    with d.open('ZipCode','w') as f: f.write('02144')
    with d.open('Phone','w') as f: f.write('617-718-9463')
    d = cd.makedirs('Boston/Restaurants/Highland Kitchen')
    with d.open('Addr1','w') as f: f.write('150 Highland Ave')
    with d.open('City','w') as f: f.write('Sommerville')
    with d.open('ZipCode','w') as f: f.write('02144')
    with d.open('Phone','w') as f: f.write('617-625-1131')
    print(cd.as_dict())
    
if __name__ == '__main__':
    main()

Output:
{'Boston': {'Restaurants': {'Spoke Wine Bar': {'Addr1': b'89 holland St', 'City': b'Sommerville', 'Phone': b'617-718-9463', 'ZipCode': b'02144'}, 'Highland Kitchen': {'Addr1': b'150 Highland Ave', 'City': b'Sommerville', 'Phone': b'617-625-1131', 'ZipCode': b'02144'}}}}

See the source of dictfs.py

An interesting feature is that pyfilesystem can transparently copy hierarchies from one filesystem to another. The above dictionary can be turned into a directory on disk by adding this into main()

    import fs.osfs
    with fs.osfs.OSFS('tmp') as tmp:
        target = tmp.makedir('dummy', recreate=True)
        fs.copy.copy_fs(cd, target)

Notice that my DictFs class stores byte strings instead of strings. It is a limitation, but it ensures interoperability with other forms of file systems.

[Larz60+]

I use pathlib a lot, I always make all references to a single root node, so I am able to redirect to any base directory and use the same code, with the new file structure automatically reconstructed.

I have not tried on AWS or anything other than local filesystems, but it, being fully Object based, i keep finding new and useful thing that I can do with it.

I am curious if os now uses PyFilesystem as there are new commands like os.fspath that use the fs prefix.
I use that command to condition pathlib paths for JSON files.

I guess I need to spend some time investigating this package and yours as well.

FYI - Frankly AWS scares the hell out of me. I know someone who just started to play with it at his office, went home for the weekend and came back on Monday facing a six digit bill from Amazon. He was able to quickly get the charges reversed, but this is frightening.

[Gribouillis]

I am curious if os now uses PyFilesystem as there are new commands like os.fspath that use the fs prefix.

No Python itself doesn't depend on a third party module such as PyFileSystem. The role of PyFileSystem is to offer a unified API for different objects, so that the same code can be used in different contexts. Consider a slightly better version of the above code

from dictfs import DictFs

def fill(root):
    d = root.makedirs('Boston/Restaurants/Spoke Wine Bar')
    d.settext('Addr1', '89 Holland St')
    d.settext('City', 'Sommerville')
    d.settext('ZipCode', '02144')
    d.settext('Phone', '617-718-9463')
    d = root.makedirs('Boston/Restaurants/Highland Kitchen')
    d.settext('Addr1', '150 Highland Ave')
    d.settext('City', 'Sommerville')
    d.settext('ZipCode', '02144')
    d.settext('Phone', '617-625-1131')

def main():
    with DictFs() as root:
        fill(root)
        root.tree()

if __name__ == '__main__':
    main()

Output:
└── Boston
    └── Restaurants
        ├── Highland Kitchen
        │   ├── Addr1
        │   ├── City
        │   ├── Phone
        │   └── ZipCode
        └── Spoke Wine Bar
            ├── Addr1
            ├── City
            ├── Phone
            └── ZipCode

This code acts in memory on a python dictionary. I only need to replace DictFs() by fs.osfs.OSFS('/tmp') or fs.tarfs.TarFS('foo.tgz', write=True) to have the same code operate on a directory on disk or a gunzipped tar file. The same code can manipulate a remote FTP or dropbox hierarchy.

Of course, it may remain some filesystem specific code, for example if I want to call root.as_dict(), which is defined only for my DictFs class, but clearly, using the unified API, it would be quite easy to define a function as_dict that would apply to any of these filesystems. PyFileSystem is a great step towards code reusability!

[Larz60+]

nice!

[Larz60+]

Is there a method to convert from fs.subfs.SubFS to dict?

[Gribouillis]

I don't think there is a built-in way, but it can be done easily using DictFs. In the following example, I suppose that I have a directory foo/dummy on my disk:

def main2():
    import fs.copy
    with fs.open_fs('osfs://tmp') as root:
        dummy = root.opendir('dummy') # this is a fs.subfs.SubFS
        target = DictFs()
        fs.copy.copy_dir(dummy, '', target, 'dummy')
        mydict = target.as_dict()['dummy']
        print(mydict)

if __name__ == '__main__':
    main2()

Output:
{'Boston': {'Restaurants': {'Spoke Wine Bar': {'ZipCode': b'02144', 'Addr1': b'89 holland St', 'Phone': b'617-718-9463', 'City': b'Sommerville'}, 'Highland Kitchen': {'ZipCode': b'02144', 'Addr1': b'150 Highland Ave', 'Phone': b'617-625-1131', 'City': b'Sommerville'}}}}

It gives me the idea to improve the as_dict() method by adding an optional path argument. Soon an improved version on the gist!

The great thing is that one can copy files and directory from one fs to another!

---

EDIT: I upgraded the dictfs gist. One can now use a path argument in as_dict(), for example cd.as_dict('Boston/Restaurants'). There are also new constructors from_fs() and from_dir(). They can be used to create a DictFS from a different filesystem (including fs.subfs.SubFS), for example

with fs.open_fs('osfs://tmp') as root:
    dfs = DictFs.from_dir(root, 'dummy')
    print(dfs.as_dict())

Remember however that DictFs is a first implementation. It is not at all optimized nor is it seriously tested. Also note that PyFileSystem has an implementation of a memory file system if that's what you need.

[Larz60+]

Glad I could move your neurons!

Remember however that DictFs is a first implementation. It is not at all optimized nor is it seriously tested

OK, so perhaps I'll test a bit.
You can never get enough.
I created a point of sale system for a chain store back in the 80's that had been tested, retested and tested again. After running in 169 stores on as many as twenty cash registers in each store, I got a call --

We received an error message. "There's a roach in this software ... Call Mom"

. This was an error message I embedded in a place where 'this never can happen' but it did.
So you can never be sure. Glad it wasn't the space shuttle!

[DeaD_EyE]

We need to change the laws in Germany. I want to marry Python.