Posts: 11,871
Threads: 474
Joined: Sep 2016
Nov-23-2018, 04:06 AM
(This post was last modified: Nov-23-2018, 04:06 AM by Larz60+.)
I just posted this on thread: https://python-forum.io/Thread-how-to-cr...ested-dict
It's useful enough that I'm posting it here as well:
# Author: Larz60+ Nov 22, 2018
import os
class CreateDict:
def __init__(self):
os.chdir(os.path.abspath(os.path.dirname(__file__)))
def new_dict(self, dictname):
setattr(self, dictname, {})
def add_node(self, parent, nodename):
node = parent[nodename] = {}
return node
def add_cell(self, nodename, cellname, value):
cell = nodename[cellname] = value
return cell
def display_dict(self, dictname, level=0):
indent = " " * (4 * level)
for key, value in dictname.items():
if isinstance(value, dict):
print(f'\n{indent}{key}')
level += 1
self.display_dict(value, level)
else:
print(f'{indent}{key}: {value}')
if level > 0:
level -= 1
def testit():
cd = CreateDict()
cd.new_dict('CityList')
boston = cd.add_node(cd.CityList, 'Boston')
bos_resturants = cd.add_node(boston, 'Resturants')
spoke = cd.add_node(bos_resturants, 'Spoke Wine Bar')
cd.add_cell(spoke, 'Addr1', '89 Holland St')
cd.add_cell(spoke, 'City', 'Sommerville')
cd.add_cell(spoke, 'Addr1', '02144')
cd.add_cell(spoke, 'Phone', '617-718-9463')
highland = cd.add_node(bos_resturants, 'Highland Kitchen')
cd.add_cell(highland, 'Addr1', '150 Highland Ave')
cd.add_cell(highland, 'City', 'Sommerville')
cd.add_cell(highland, 'ZipCode', '02144')
cd.add_cell(highland, 'Phone', '617-625-1131')
print(f'\nCityList Dictionary')
cd.display_dict(cd.CityList)
print(f'\nraw data: {cd.CityList}')
if __name__ == '__main__':
testit() Test results:
Output: Boston
Resturants
Spoke Wine Bar
Addr1: 02144
City: Sommerville
Phone: 617-718-9463
Highland Kitchen
Addr1: 150 Highland Ave
City: Sommerville
ZipCode: 02144
Phone: 617-625-1131
raw data: {'Boston': {'Resturants': {'Spoke Wine Bar': {'Addr1': '02144', 'City': 'Sommerville', 'Phone': '617-718-9463'}, 'Highland Kitchen': {'Addr1': '150 Highland Ave', 'City': 'Sommerville', 'ZipCode': '02144', 'Phone': '617-625-1131'}}}}
Posts: 4,498
Threads: 69
Joined: Jan 2018
Nov-24-2018, 08:15 AM
(This post was last modified: Nov-24-2018, 08:18 AM by Gribouillis.)
I like PyFilesystem (the fs module) very much, one reason being its extensibility. I created a simple abstract filesystem which manipulates a dictionary in memory. You can use it to create your dict like so
from dictfs import DictFs
def main():
cd = DictFs()
d = cd.makedirs('Boston/Restaurants/Spoke Wine Bar')
with d.open('Addr1','w') as f: f.write('89 holland St')
with d.open('City','w') as f: f.write('Sommerville')
with d.open('ZipCode','w') as f: f.write('02144')
with d.open('Phone','w') as f: f.write('617-718-9463')
d = cd.makedirs('Boston/Restaurants/Highland Kitchen')
with d.open('Addr1','w') as f: f.write('150 Highland Ave')
with d.open('City','w') as f: f.write('Sommerville')
with d.open('ZipCode','w') as f: f.write('02144')
with d.open('Phone','w') as f: f.write('617-625-1131')
print(cd.as_dict())
if __name__ == '__main__':
main() Output: {'Boston': {'Restaurants': {'Spoke Wine Bar': {'Addr1': b'89 holland St', 'City': b'Sommerville', 'Phone': b'617-718-9463', 'ZipCode': b'02144'}, 'Highland Kitchen': {'Addr1': b'150 Highland Ave', 'City': b'Sommerville', 'Phone': b'617-625-1131', 'ZipCode': b'02144'}}}}
See the source of dictfs.py
An interesting feature is that pyfilesystem can transparently copy hierarchies from one filesystem to another. The above dictionary can be turned into a directory on disk by adding this into main()
import fs.osfs
with fs.osfs.OSFS('tmp') as tmp:
target = tmp.makedir('dummy', recreate=True)
fs.copy.copy_fs(cd, target) Notice that my DictFs class stores byte strings instead of strings. It is a limitation, but it ensures interoperability with other forms of file systems.
Posts: 11,871
Threads: 474
Joined: Sep 2016
Nov-24-2018, 11:15 AM
(This post was last modified: Nov-24-2018, 11:16 AM by Larz60+.)
I use pathlib a lot, I always make all references to a single root node, so I am able to redirect to any base directory and use the same code, with the new file structure automatically reconstructed.
I have not tried on AWS or anything other than local filesystems, but it, being fully Object based, i keep finding new and useful thing that I can do with it.
I am curious if os now uses PyFilesystem as there are new commands like os.fspath that use the fs prefix.
I use that command to condition pathlib paths for JSON files.
I guess I need to spend some time investigating this package and yours as well.
FYI - Frankly AWS scares the hell out of me. I know someone who just started to play with it at his office, went home for the weekend and came back on Monday facing a six digit bill from Amazon. He was able to quickly get the charges reversed, but this is frightening.
Posts: 4,498
Threads: 69
Joined: Jan 2018
Nov-25-2018, 11:57 AM
(This post was last modified: Nov-25-2018, 03:43 PM by Gribouillis.)
Larz60+ Wrote:I am curious if os now uses PyFilesystem as there are new commands like os.fspath that use the fs prefix. No Python itself doesn't depend on a third party module such as PyFileSystem. The role of PyFileSystem is to offer a unified API for different objects, so that the same code can be used in different contexts. Consider a slightly better version of the above code
from dictfs import DictFs
def fill(root):
d = root.makedirs('Boston/Restaurants/Spoke Wine Bar')
d.settext('Addr1', '89 Holland St')
d.settext('City', 'Sommerville')
d.settext('ZipCode', '02144')
d.settext('Phone', '617-718-9463')
d = root.makedirs('Boston/Restaurants/Highland Kitchen')
d.settext('Addr1', '150 Highland Ave')
d.settext('City', 'Sommerville')
d.settext('ZipCode', '02144')
d.settext('Phone', '617-625-1131')
def main():
with DictFs() as root:
fill(root)
root.tree()
if __name__ == '__main__':
main() Output: └── Boston
└── Restaurants
├── Highland Kitchen
│ ├── Addr1
│ ├── City
│ ├── Phone
│ └── ZipCode
└── Spoke Wine Bar
├── Addr1
├── City
├── Phone
└── ZipCode
This code acts in memory on a python dictionary. I only need to replace DictFs() by fs.osfs.OSFS('/tmp') or fs.tarfs.TarFS('foo.tgz', write=True) to have the same code operate on a directory on disk or a gunzipped tar file. The same code can manipulate a remote FTP or dropbox hierarchy.
Of course, it may remain some filesystem specific code, for example if I want to call root.as_dict() , which is defined only for my DictFs class, but clearly, using the unified API, it would be quite easy to define a function as_dict that would apply to any of these filesystems. PyFileSystem is a great step towards code reusability!
Posts: 11,871
Threads: 474
Joined: Sep 2016
Posts: 11,871
Threads: 474
Joined: Sep 2016
Is there a method to convert from fs.subfs.SubFS to dict?
Posts: 4,498
Threads: 69
Joined: Jan 2018
Nov-25-2018, 06:52 PM
(This post was last modified: Nov-25-2018, 06:54 PM by Gribouillis.)
I don't think there is a built-in way, but it can be done easily using DictFs. In the following example, I suppose that I have a directory foo/dummy on my disk:
def main2():
import fs.copy
with fs.open_fs('osfs://tmp') as root:
dummy = root.opendir('dummy') # this is a fs.subfs.SubFS
target = DictFs()
fs.copy.copy_dir(dummy, '', target, 'dummy')
mydict = target.as_dict()['dummy']
print(mydict)
if __name__ == '__main__':
main2() Output: {'Boston': {'Restaurants': {'Spoke Wine Bar': {'ZipCode': b'02144', 'Addr1': b'89 holland St', 'Phone': b'617-718-9463', 'City': b'Sommerville'}, 'Highland Kitchen': {'ZipCode': b'02144', 'Addr1': b'150 Highland Ave', 'Phone': b'617-625-1131', 'City': b'Sommerville'}}}}
It gives me the idea to improve the as_dict() method by adding an optional path argument. Soon an improved version on the gist!
The great thing is that one can copy files and directory from one fs to another!
EDIT: I upgraded the dictfs gist. One can now use a path argument in as_dict(), for example cd.as_dict('Boston/Restaurants') . There are also new constructors from_fs() and from_dir() . They can be used to create a DictFS from a different filesystem (including fs.subfs.SubFS), for example
with fs.open_fs('osfs://tmp') as root:
dfs = DictFs.from_dir(root, 'dummy')
print(dfs.as_dict()) Remember however that DictFs is a first implementation. It is not at all optimized nor is it seriously tested. Also note that PyFileSystem has an implementation of a memory file system if that's what you need.
Posts: 11,871
Threads: 474
Joined: Sep 2016
Nov-25-2018, 07:54 PM
(This post was last modified: Nov-25-2018, 07:54 PM by Larz60+.)
Glad I could move your neurons!
Quote:Remember however that DictFs is a first implementation. It is not at all optimized nor is it seriously tested
OK, so perhaps I'll test a bit.
You can never get enough.
I created a point of sale system for a chain store back in the 80's that had been tested, retested and tested again. After running in 169 stores on as many as twenty cash registers in each store, I got a call -- Quote:We received an error message. "There's a roach in this software ... Call Mom"
. This was an error message I embedded in a place where 'this never can happen' but it did.
So you can never be sure. Glad it wasn't the space shuttle!
Posts: 2,021
Threads: 9
Joined: May 2017
We need to change the laws in Germany. I want to marry Python.
|