Python Forum
Performance degradation with IO class inheritance
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Performance degradation with IO class inheritance
#1
Hi all!

I recently tried to build a new IO class from python built-in IO classes. I noticed some performance degradation for these new classes. I wondering if anyone knows what might be the possible cause/fix for this.

from _io import FileIO, BufferedReader

class FileIO_new(FileIO): ()
class BufferedReader_new(BufferedReader): ()

def built_in_io():
    # a 3.5 MB file
    raw = FileIO('filename', 'r')
    buffer = BufferedReader(raw)
    lines = buffer.readlines()
    buffer.close()
def new_io():
    raw = FileIO_new('filenmae','r')
    buffer = BufferedReader_new(raw)
    lines = buffer.readlines()
    buffer.close()

>>>%timeit built_in_io()
1.79 ms ± 14.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>>>%timeit new_io()
5.25 ms ± 87 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Reply
#2
It's really slower. If you use a ramdisk, you've lesser problems with caching and buffers during testing.
But I got the similar results.

Here my testcode:
import io
import os
import shlex
import timeit
from pathlib import Path
from shutil import rmtree
from subprocess import call
from contextlib import contextmanager

RAMDISK = Path("ramdisk")
TESTFILE = RAMDISK / "testfile.bin"

def mount_ramfs():
    RAMDISK.mkdir(exist_ok=True)
    cmd = shlex.split(f"sudo mount -t ramfs ramfs {RAMDISK}")
    call(cmd)


def umount_ramfs():
    cmd = shlex.split(f"sudo umount {RAMDISK}")
    call(cmd)
    RAMDISK.rmdir()


def make_test_file():
    with TESTFILE.open("wb") as fd:
        fd.write(os.urandom(4 * 1024 ** 2))


@contextmanager
def with_testfile():
    mount_ramfs()
    make_test_file()
    try:
        yield TESTFILE
    except Exception as e:
        print(repr(e))
    umount_ramfs()


## testcode ##
class FileIO_new(io.FileIO): pass
class BufferedReader_new(io.BufferedReader): pass


def built_in_io(testfile):
    raw = io.FileIO(testfile, 'r')
    buffer = io.BufferedReader(raw)
    lines = buffer.readlines()
    buffer.close()


def new_io(testfile):
    raw = FileIO_new(testfile, 'r')
    buffer = BufferedReader_new(raw)
    lines = buffer.readlines()
    buffer.close()


with with_testfile() as tf:
    print(tf, tf.stat().st_size / 1024 ** 2, "MiB")
    result_built_in = timeit.timeit("built_in_io(tf)", globals=globals(), number=1000)
    result_new_io = timeit.timeit("new_io(tf)", globals=globals(), number=1000)
    print(f"BuiltIn: {result_built_in:.3f}")
    print(f"NewIO: {result_new_io:.3f}")
Output:
ramdisk/testfile.bin 4.0 MiB BuiltIn: 3.578 NewIO: 5.005
The built-in _io is implemented in C.
If you inherit from it, then Python comes into the game.
I guess this is the cause why it's much slower.
The time spent to creatine new instances seems not to be the problem.
I changed the test code a bit, that I create the instances only once a dusing instead seek(0).
Same result, no improvement.

I guess that's why the project Borgbackup implemented the functions for file access/chunking etc. with Cython and C.
Borgbackup is a tool for backups with deduplication written in Python: https://github.com/borgbackup/borg
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#3
Thanks for answering! Just find it weird (and interesting) somehow the performance is not the same, but still outperforms the python implementation of io (_pyio) so much. Also thanks for the tricks!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  super() and order of running method in class inheritance akbarza 7 594 Feb-04-2024, 09:35 AM
Last Post: Gribouillis
  Child class inheritance issue eakanathan 3 1,299 Apr-21-2022, 12:03 PM
Last Post: deanhystad
  Importing issues with base class for inheritance riccardoob 5 4,589 May-19-2021, 05:18 PM
Last Post: snippsat
  3D vector class with inheritance from 2D vector class buss0140 4 3,071 Dec-20-2020, 08:44 PM
Last Post: deanhystad
  Class inheritance oclmedyb 3 2,212 Dec-09-2020, 04:43 PM
Last Post: deanhystad
  Can we access instance variable of parent class in child class using inheritance akdube 3 13,885 Nov-13-2020, 03:43 AM
Last Post: SalsaBeanDip
  Performance enhancement fimmu 0 1,573 Feb-12-2020, 02:42 PM
Last Post: fimmu
  performance kerzol81 1 1,874 Oct-07-2019, 10:19 AM
Last Post: buran
  How I can do performance testing on my API a21250450 0 1,367 Jul-18-2019, 09:29 AM
Last Post: a21250450
  Python performance rvkstudent 4 2,930 Apr-25-2019, 09:29 AM
Last Post: rvkstudent

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020