Python Forum
Is rememberme.memory() accurate for recursive structures?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Is rememberme.memory() accurate for recursive structures?
#1
I'm trying to measure the size of my recursively defined data structures, but I'm getting wildly large results.
Perhaps it's the tool that I'm using rememberme.memory(). I chose this because sys.getsizeof() does not work on recursively defined structures.
I estimate that the max size of the BST presented should be around 512 bytes plus a bit for python's overhead if any, but I'm getting results 16 times larger.
Is rememberme.memory() the correct tool for this job?

class Node:
    def __init__(self, data):
        self.data = data
        self.next = None
        self.previous = None

    def has_next(self):
        if self.next != None:
            return True
        else:
            return False

    def has_previous(self):
        if self.previous != None:
            return True
        else:
            return False

    def __str__(self):
        result = ""
        if self.previous != None:
            result += str(self.previous) + " "
        result += str(self.data)
        if self.next != None:
            result += " " + str(self.next)
        return result

class BST:
    def __init__(self):
        self.root = None

    def add(self, data):
        if self.root == None:
            self.root = Node(data)
        else:
            current_node = self.root
            while current_node != None:
                current_node = self.add_h(current_node, data)

    def add_h(self, node, data):
        if data > node.data:
            if node.next == None:
                node.next = Node(data)
                print('leaf node:', memory(node.next))
            else:
                return node.next
        if data < node.data:
            if node.previous == None:
                node.previous = Node(data)
                print('leaf node:', memory(node.previous))
            else:
                return node.previous

    def __str__(self):
        return str(self.root)

from rememberme import memory
from sys import getsizeof
        
print('BST')
bst = BST()
print('BST, rememberme', memory(bst), 'bytes')
print('BST getsizeof', getsizeof(bst), 'bytes')
bst.add(7)
print('BST, rememberme', memory(bst), 'bytes')
print('BST getsizeof', getsizeof(bst), 'bytes')
print('values:', bst)
bst.add(3)
print('BST, rememberme', memory(bst), 'bytes')
print('BST getsizeof', getsizeof(bst), 'bytes')
print('values:', bst)
bst.add(4)
print('BST, rememberme', memory(bst), 'bytes')
print('BST getsizeof', getsizeof(bst), 'bytes')
print('values:', bst)
bst.add(9)
print('BST, rememberme', memory(bst), 'bytes')
print('BST getsizeof', getsizeof(bst), 'bytes')
print('values:', bst)

print('\nList')
v_list = [3, 4, 7, 9]
print('rememberme', memory(v_list), 'bytes')
print('getsizeof', getsizeof(v_list), 'bytes')
print('values:', v_list)

print('\nDictionary')
v_dict = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
print('rememberme', memory(v_dict), 'bytes')
print('getsizeof', getsizeof(v_dict), 'bytes')
print('values:', v_dict)
Here are the results:
Output:
BST BST, rememberme 3991 bytes BST getsizeof 48 bytes BST, rememberme 7444 bytes BST getsizeof 48 bytes values: 7 leaf node: 4047 BST, rememberme 7624 bytes BST getsizeof 48 bytes values: 3 7 leaf node: 4047 BST, rememberme 7804 bytes BST getsizeof 48 bytes values: 3 4 7 leaf node: 4047 BST, rememberme 7984 bytes BST getsizeof 48 bytes values: 3 4 7 9 List rememberme 232 bytes getsizeof 120 bytes values: [3, 4, 7, 9] Dictionary rememberme 344 bytes getsizeof 232 bytes values: {'a': 1, 'b': 2, 'c': 3, 'd': 4}
I just don't understand how a 4 node BST can be nearly 8kb.
Reply
#2
You could perhaps add a __slots__ member to your classes, especially the nodes and see if it changes anything (I don't know rememberme.memory()).
Reply
#3
(Dec-17-2020, 10:39 AM)Gribouillis Wrote: You could perhaps add a __slots__ member to your classes, especially the nodes and see if it changes anything (I don't know rememberme.memory()).

With slots:
Output:
BST BST, rememberme 3700 bytes BST getsizeof 40 bytes BST, rememberme 7460 bytes BST getsizeof 40 bytes values: 7 leaf node: 4354 BST, rememberme 7544 bytes BST getsizeof 40 bytes values: 3 7 leaf node: 4354 BST, rememberme 7628 bytes BST getsizeof 40 bytes values: 3 4 7 leaf node: 4354 BST, rememberme 7712 bytes BST getsizeof 40 bytes values: 3 4 7 9 List rememberme 232 bytes getsizeof 120 bytes values: [3, 4, 7, 9] Dictionary rememberme 344 bytes getsizeof 232 bytes values: {'a': 1, 'b': 2, 'c': 3, 'd': 4}
Without slots:
Output:
BST BST, rememberme 3991 bytes BST getsizeof 48 bytes BST, rememberme 7444 bytes BST getsizeof 48 bytes values: 7 leaf node: 4047 BST, rememberme 7624 bytes BST getsizeof 48 bytes values: 3 7 leaf node: 4047 BST, rememberme 7804 bytes BST getsizeof 48 bytes values: 3 4 7 leaf node: 4047 BST, rememberme 7984 bytes BST getsizeof 48 bytes values: 3 4 7 9 List rememberme 232 bytes getsizeof 120 bytes values: [3, 4, 7, 9] Dictionary rememberme 344 bytes getsizeof 232 bytes values: {'a': 1, 'b': 2, 'c': 3, 'd': 4}
According to rememberme, the slots provide about an 8% improvement, but that improvement diminishes and n increases.
Thanks for bringing that up, I was unaware of __slots__ and I enjoyed reading about them.
In that reading I came across the pympler module and its asizeof.asizeof() function. It provides what appears to be a more accurate size reading:
Output:
BST BST, rememberme 3700 bytes BST getsizeof 40 bytes BST asizeof 40 bytes BST, rememberme 7460 bytes BST getsizeof 40 bytes BST asizeof 40 bytes values: 7 rememberme leaf node: 4354 asizeof leaf node: 104 BST, rememberme 7544 bytes BST getsizeof 40 bytes BST asizeof 40 bytes values: 3 7 rememberme leaf node: 4354 asizeof leaf node: 104 BST, rememberme 7628 bytes BST getsizeof 40 bytes BST asizeof 40 bytes values: 3 4 7 rememberme leaf node: 4354 asizeof leaf node: 104 BST, rememberme 7712 bytes BST getsizeof 40 bytes BST asizeof 40 bytes values: 3 4 7 9 List rememberme 232 bytes getsizeof 120 bytes asizeof 248 bytes values: [3, 4, 7, 9] Dictionary rememberme 344 bytes getsizeof 232 bytes asizeof 584 bytes values: {'a': 1, 'b': 2, 'c': 3, 'd': 4}
The only issue with asizeof is that it does not look like it works with slots. I suppose that means it's completely dependent on the object's __dict__ or I messed up using it.

Output:
BST BST, rememberme 3700 bytes BST getsizeof 40 bytes BST asizeof 40 bytes BST, rememberme 7460 bytes BST getsizeof 40 bytes BST asizeof 40 bytes values: 7 rememberme leaf node: 4354 asizeof leaf node: 104 BST, rememberme 7544 bytes BST getsizeof 40 bytes BST asizeof 40 bytes values: 3 7 rememberme leaf node: 4354 asizeof leaf node: 104 BST, rememberme 7628 bytes BST getsizeof 40 bytes BST asizeof 40 bytes values: 3 4 7 rememberme leaf node: 4354 asizeof leaf node: 104 BST, rememberme 7712 bytes BST getsizeof 40 bytes BST asizeof 40 bytes values: 3 4 7 9 List rememberme 232 bytes getsizeof 120 bytes asizeof 248 bytes values: [3, 4, 7, 9] Dictionary rememberme 344 bytes getsizeof 232 bytes asizeof 584 bytes values: {'a': 1, 'b': 2, 'c': 3, 'd': 4}
I'm going to read some more and maybe post it as a bug report on the author's git

edit:
It works with slots, I just needed to make some adjustments
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Cant's get accurate unbiased variance. Alex009988 1 2,249 Aug-13-2019, 02:55 PM
Last Post: ichabod801

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020