Python Forum

Python 3.7.1

My program works with big data, where I don't see the point of using dynamic language capabilities (adding class members or methods during program execution). Also I try to create a good application architecture, that has multiple layers. All this makes the callstack very long, and the program runs slowly. In fact, my program is written in C++ style.

Is it possible to optimize my code, by indicating that I will abandon the dynamic capabilities of the Python? I know there is PyPy (which uses embedding of small functions, which reduces stack depth), but it requires refusal libraries written in C, which is unacceptable in the case of machine learning. What other optimization tools are available that allow you to create fast programs with good architecture?

it wouldn't be static if optimized, right?

(Nov-01-2019, 05:02 PM)Larz60+ Wrote: [ -> ]it wouldn't be static if optimized, right?

On the contrary: my code is static and therefore can be optimized.

I found a way using Cython:
inline- operator
Early binding

Are there other ways?

For example classes have dicts for their methods and attributes.
If I remind right, a dict has a size of 112 bytes (since Python 3.6).
If you have only 10-1000 objects, you just don't care.
But with one million objects, you care about memory consumption.
In this case, use __slots__ for this kind of classes.

If you have a for-loop, try to solve this not in Python.
Looping is not very efficient in Python. You can do this in cython.

Maybe you can post some example code with example data.


EDIT: Very often constructs like a list comprehension are faster than using generators. Generators are used, if you have much data to process and want to solve this in a iterative way and saving memory consumption. Maybe you can also implement a generator in Cython, which should run faster and saves memory.

(Nov-02-2019, 08:57 AM)DeaD_EyE Wrote: [ -> ]Maybe you can post some example code with example data.

Presenting the code will be difficult, because it is scattered across multiple files. But the main idea is quite simple to state: I have a set of csv- files, separated by the month. My task is to execute high-level business logic on each line from these files. The architect is represented by the following components:
1. Command - works with business logic for each line of file.
2. Storage - allows to change the way data is stored (files, database, internet, generation...), i.e. makes a call to the selected method of data storage.
3. Datа - organizes enumeration of files in the desired date range.
4. Converter - convert data from str to needed types.
5. File - actually receives the data from csv- file.

See how deep a call stack is for each line of data from a file? With such a stack, the overhead significantly exceeds the complexity of the useful work (i.e. parsing the string).