Python Forum
sysv_ipc shared memory performance
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
sysv_ipc shared memory performance
#1
I have a system that is written in C on Linux and there are 2 components. One component reads a bunch of data, serializes it into C structs then copies it into shared memory.

The second component attaches to that shared memory id and then reads the data back into C structs and compares the data against some patterns. There are multiple copies of the second component on the machine and they can all use this 'shared memory' without having to keep a local copy of the data. Through this we can do high-throughput computing on a large data set without having to have multiple copies of the data on the machine, minimizing RAM usage while persevering performance.

We're able to achieve 7.6GB/s read speeds on shared memory across multiple cores in C.

Now I would like to try to re-produce this in Python. I'm not a python expert by any means, but I've written a simple component which is below that uses sysv_ipc, attaches to the same shared memory that C uses and then just tries to loop through the data and verify that everything is parsed properly. In the dataset used by the below all the data is identical.

However, the performance I'm getting is far below what I'd expect. On the same machine that the above C program runs I only get about 19MB/s.

So my question boils down to this. What is wrong with my code? Why is the performance so slow compared to C doing exactly the same thing? Is the shared memory architecture in python not viable for high throughput processing in python?

Obviously, it should be possible to just copy the whole shared memory chunk into the local python stack, which would likely improve the performance, but that defeats the purpose of having shared memory. Suddenly there will be dozens of copies of the data laying around which will fill up RAM and cause the system to start to swap to virtual memory. So that isn't the solution I'm looking for.

FYI, this is a scaled down version of the system that only works on about 8MB/s of data. The actual full size system in C operates on gigabytes of shared memory.

Using Python 3.5.2

import sysv_ipc
import ctypes
import struct
import time

# Create shared memory object
memory = sysv_ipc.SharedMemory(123456)

current_offset = 0
n = 1

localf1 = struct.unpack('d', memory.read(8, 32))
localf2 = struct.unpack('d',memory.read(8, 40))
localf3 = struct.unpack('d',memory.read(8, 48))
localf4 = struct.unpack('d',memory.read(8, 56))

s=time.time()

while n < 120001:
  mydate = memory.read(23, current_offset)
  current_offset += 32

  myf1 = struct.unpack('d', memory.read(8, current_offset))
  current_offset += 8

  myf2 = struct.unpack('d', memory.read(8, current_offset))
  current_offset += 8

  myf3 = struct.unpack('d', memory.read(8, current_offset))
  current_offset += 8

  myf4 = struct.unpack('d', memory.read(8, current_offset))
  current_offset += 8

  if (mydate.decode('ascii') != "2019-04-01 23:59.59.999"):
      print("failure to parse date")

  if ((myf1 != localf1) or (myf2 != localf2) or (myf3 != localf3) or (myf4!= localf4)):
      print("failure to parse numbers")

  n += 1

e=time.time()
result = e-s

mbps = (current_offset / result) / 1e6

print("Processed " + str(current_offset) + " bytes of shared memory in " + str(result) + " seconds (" + str(mbps) + " MB/s).")
Reply
#2
(Apr-04-2019, 12:47 PM)comotai Wrote: What is wrong with my code? Why is the performance so slow compared to C doing exactly the same thing?

If we are talking about CPython (standard Python), implementation of Python programming language in C,
execution model of a python program (script) is significantly differed from that is being written in C language. C-program is compiled into machine code and than executed. Python program is compiled into bytecode that is further interpreted/executed by CPython engine. That is why Python program is executed slower than C one. Sometimes one might use a just-in-time (JIT) compiler that compiles bytecode into machine code and runs it (e.g. PyPy). This usually leads to increasing of execution speed. In you case, you are likely never achieve comparable to C-language execution speed using CPython (or ever PyPy, Cython, or something else). You program is already written in C, and probably optimized. If you ever used Cython, that translates Python program into C-code, that is further compiled into executable, the program wouldn't be so fast, as it would natively written in C. The only way of improving performance in your case (if program isn't too complex) is to rewrite it using assembly language (or may be apply some optimization keys at compilation time, -O2 etc.). So, why do you need to rewrite a program (originally written in C) in Python?!

You can try to achieve comparable to C (but liekly slower, than you already have, I think) performance using NumPy package.

import numpy as np
data = memoryview(b'byte')  # point to your shared memory instead... 
result = np.ndarray(buffer=data, shape=(2,), dtype=np.int16) # change shape and dtype to interpret shared memory as array properly
# result is data interpreted as two 2-byte integers
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Shared reference of variables... Denial 1 1,374 Aug-29-2020, 01:52 PM
Last Post: snippsat
  How to install and use a shared libary, via a .dll? ninjaisfast 0 1,271 Jul-09-2020, 03:23 PM
Last Post: ninjaisfast
  Performance enhancement fimmu 0 1,573 Feb-12-2020, 02:42 PM
Last Post: fimmu
  performance kerzol81 1 1,874 Oct-07-2019, 10:19 AM
Last Post: buran
  How I can do performance testing on my API a21250450 0 1,367 Jul-18-2019, 09:29 AM
Last Post: a21250450
  Python performance rvkstudent 4 2,931 Apr-25-2019, 09:29 AM
Last Post: rvkstudent
  Divisors shared the second numbers mircea_dragu 1 2,015 Feb-07-2019, 10:09 PM
Last Post: ichabod801
  running just one process shared among uses Skaperen 3 2,925 Aug-07-2018, 12:12 AM
Last Post: Skaperen
  Shared reference and equality zyo 3 3,096 Jun-30-2018, 07:10 PM
Last Post: ljmetzger
  Shared queues l00p1n6 3 2,948 May-15-2018, 01:38 PM
Last Post: DeaD_EyE

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020