sysv_ipc shared memory performance

[comotai]

I have a system that is written in C on Linux and there are 2 components. One component reads a bunch of data, serializes it into C structs then copies it into shared memory.

The second component attaches to that shared memory id and then reads the data back into C structs and compares the data against some patterns. There are multiple copies of the second component on the machine and they can all use this 'shared memory' without having to keep a local copy of the data. Through this we can do high-throughput computing on a large data set without having to have multiple copies of the data on the machine, minimizing RAM usage while persevering performance.

We're able to achieve 7.6GB/s read speeds on shared memory across multiple cores in C.

Now I would like to try to re-produce this in Python. I'm not a python expert by any means, but I've written a simple component which is below that uses sysv_ipc, attaches to the same shared memory that C uses and then just tries to loop through the data and verify that everything is parsed properly. In the dataset used by the below all the data is identical.

However, the performance I'm getting is far below what I'd expect. On the same machine that the above C program runs I only get about 19MB/s.

So my question boils down to this. What is wrong with my code? Why is the performance so slow compared to C doing exactly the same thing? Is the shared memory architecture in python not viable for high throughput processing in python?

Obviously, it should be possible to just copy the whole shared memory chunk into the local python stack, which would likely improve the performance, but that defeats the purpose of having shared memory. Suddenly there will be dozens of copies of the data laying around which will fill up RAM and cause the system to start to swap to virtual memory. So that isn't the solution I'm looking for.

FYI, this is a scaled down version of the system that only works on about 8MB/s of data. The actual full size system in C operates on gigabytes of shared memory.

Using Python 3.5.2

import sysv_ipc
import ctypes
import struct
import time

# Create shared memory object
memory = sysv_ipc.SharedMemory(123456)

current_offset = 0
n = 1

localf1 = struct.unpack('d', memory.read(8, 32))
localf2 = struct.unpack('d',memory.read(8, 40))
localf3 = struct.unpack('d',memory.read(8, 48))
localf4 = struct.unpack('d',memory.read(8, 56))

s=time.time()

while n < 120001:
  mydate = memory.read(23, current_offset)
  current_offset += 32

  myf1 = struct.unpack('d', memory.read(8, current_offset))
  current_offset += 8

  myf2 = struct.unpack('d', memory.read(8, current_offset))
  current_offset += 8

  myf3 = struct.unpack('d', memory.read(8, current_offset))
  current_offset += 8

  myf4 = struct.unpack('d', memory.read(8, current_offset))
  current_offset += 8

  if (mydate.decode('ascii') != "2019-04-01 23:59.59.999"):
      print("failure to parse date")

  if ((myf1 != localf1) or (myf2 != localf2) or (myf3 != localf3) or (myf4!= localf4)):
      print("failure to parse numbers")

  n += 1

e=time.time()
result = e-s

mbps = (current_offset / result) / 1e6

print("Processed " + str(current_offset) + " bytes of shared memory in " + str(result) + " seconds (" + str(mbps) + " MB/s).")