Performance degradation with IO class inheritance

[wsygzyx]

Hi all!

I recently tried to build a new IO class from python built-in IO classes. I noticed some performance degradation for these new classes. I wondering if anyone knows what might be the possible cause/fix for this.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

from _io import FileIO, BufferedReader   class FileIO_new(FileIO): () class BufferedReader_new(BufferedReader): ()   def built_in_io():     # a 3.5 MB file     raw = FileIO('filename', 'r')     buffer = BufferedReader(raw)     lines = buffer.readlines()     buffer.close() def new_io():     raw = FileIO_new('filenmae','r')     buffer = BufferedReader_new(raw)     lines = buffer.readlines()     buffer.close()   >>>%timeit built_in_io() 1.79 ms ± 14.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) >>>%timeit new_io() 5.25 ms ± 87 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

[DeaD_EyE]

It's really slower. If you use a ramdisk, you've lesser problems with caching and buffers during testing.
But I got the similar results.

Here my testcode:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

import io import os import shlex import timeit from pathlib import Path from shutil import rmtree from subprocess import call from contextlib import contextmanager   RAMDISK = Path("ramdisk") TESTFILE = RAMDISK / "testfile.bin"   def mount_ramfs():     RAMDISK.mkdir(exist_ok=True)     cmd = shlex.split(f"sudo mount -t ramfs ramfs {RAMDISK}")     call(cmd)     def umount_ramfs():     cmd = shlex.split(f"sudo umount {RAMDISK}")     call(cmd)     RAMDISK.rmdir()     def make_test_file():     with TESTFILE.open("wb") as fd:         fd.write(os.urandom(4 * 1024 ** 2))     @contextmanager def with_testfile():     mount_ramfs()     make_test_file()     try:         yield TESTFILE     except Exception as e:         print(repr(e))     umount_ramfs()     ## testcode ## class FileIO_new(io.FileIO): pass class BufferedReader_new(io.BufferedReader): pass     def built_in_io(testfile):     raw = io.FileIO(testfile, 'r')     buffer = io.BufferedReader(raw)     lines = buffer.readlines()     buffer.close()     def new_io(testfile):     raw = FileIO_new(testfile, 'r')     buffer = BufferedReader_new(raw)     lines = buffer.readlines()     buffer.close()     with with_testfile() as tf:     print(tf, tf.stat().st_size / 1024 ** 2, "MiB")     result_built_in = timeit.timeit("built_in_io(tf)", globals=globals(), number=1000)     result_new_io = timeit.timeit("new_io(tf)", globals=globals(), number=1000)     print(f"BuiltIn: {result_built_in:.3f}")     print(f"NewIO: {result_new_io:.3f}")

Output:
ramdisk/testfile.bin 4.0 MiB
BuiltIn: 3.578
NewIO: 5.005

The built-in _io is implemented in C.
If you inherit from it, then Python comes into the game.
I guess this is the cause why it's much slower.
The time spent to creatine new instances seems not to be the problem.
I changed the test code a bit, that I create the instances only once a dusing instead seek(0).
Same result, no improvement.

I guess that's why the project Borgbackup implemented the functions for file access/chunking etc. with Cython and C.
Borgbackup is a tool for backups with deduplication written in Python: https://github.com/borgbackup/borg

[wsygzyx]

Thanks for answering! Just find it weird (and interesting) somehow the performance is not the same, but still outperforms the python implementation of io (_pyio) so much. Also thanks for the tricks!