Help of multiple threads using cpython

[LM19952012]

I recently tried to use a lot of threads (i.e. about 100) to run the same python function. That function will call an external library written in C. However, cpython will periodically throw an error at the line of calling the external C function, which looks like the following:

Error:
Exception in thread Thread-34:
Traceback (most recent call last):
  File "/homes/liu1740/miniconda3/envs/my-rdkit-env/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/homes/liu1740/miniconda3/envs/my-rdkit-env/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/u/subspace_s4/liu1740/Research/LocalGraphClustering/localgraphclustering/ncp.py", line 50, in ncp_node_worker
    ncpdata.results.extend(ncp_experiment(ncpdata, R, func, method_stats))
  File "/u/subspace_s4/liu1740/Research/LocalGraphClustering/localgraphclustering/ncp.py", line 26, in ncp_experiment
    S = func(ncpdata.graph, R)
  File "<ipython-input-4-a67f0813539b>", line 1, in <lambda>
    funcs = {lambda G,R: lgc.spectral_clustering(G,R,alpha=alpha,rho=rho,method="acl")[0]:'acl;rho=%.0e'%(rho) for rho in rholist}
  File "/u/subspace_s4/liu1740/Research/LocalGraphClustering/localgraphclustering/spectral_clustering.py", line 96, in spectral_clustering
    rho = rho, epsilon = epsilon, method = method, ys = ys)
  File "/u/subspace_s4/liu1740/Research/LocalGraphClustering/localgraphclustering/approximate_PageRank.py", line 115, in approximate_PageRank
    alpha,rho,ref_nodes,iterations,G.lib)
  File "/u/subspace_s4/liu1740/Research/LocalGraphClustering/localgraphclustering/cpp/aclpagerank_cpp.py", line 52, in aclpagerank_cpp
    actual_length=fun(n,ai,aj,flag,alpha,eps,seedids,nseedids,maxsteps,xids,xlength,values)
ctypes.ArgumentError: argument 11: <class 'TypeError'>: 'int' object is not callable

And it can happen at any argument randomly, not just argument 11. The code works fine with few threads. My guess is something is wrong during the conversion from a python variable to a C variable, but I am not sure and don't know how to fix it.

[micseydel]

Could you provide a Minimal, Complete, and Verifiable example? Also this is one of the rare cases where OS and Python version may be very valuable.

[LM19952012]

The problem happens when I develop a python package https://github.com/MengLiuPurdue/LocalGraphClustering

After you download and install it. The problem can be easily reproduced with the following command:

import localgraphclustering as lgc
G = lgc.GraphLocal("notebooks/datasets/usroads-cc.edgelist",file_type = "edgelist", separator = " ", header = 0)
ncp_instance = lgc.NCPData(G)
gamma = 0.01/0.99
rholist = [1.0e-10]
ratio = 0.3
nthreads = 192
timeout = 1000
alpha = 1.0-1.0/(1.0+gamma)
funcs = {lambda G,R: lgc.spectral_clustering(G,R,alpha=alpha,rho=rho,method="acl")[0]:'acl;rho=%.0e'%(rho) for rho in rholist}
for func in funcs.keys():
ncp_instance.add_random_node_samples(method=func,methodname=funcs[func],ratio=ratio,nthreads=nthreads,timeout=timeout/len(funcs))

Note that, the problem happens randomly, so you might need to repeat those commands several times. The python version info is:
3.6.4 |Anaconda, Inc.| (default, Mar 13 2018, 01:15:57)
[GCC 7.2.0]
And the OS info is:
Linux 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 384
On-line CPU(s) list: 0-383
Thread(s) per core: 2
Core(s) per socket: 24
Socket(s): 8
NUMA node(s): 8
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel® Xeon® Platinum 8168 CPU @ 2.70GHz
Stepping: 4
CPU MHz: 1200.000
CPU max MHz: 2701.0000
CPU min MHz: 1200.0000
BogoMIPS: 5402.15
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 33792K

[Larz60+]

You should read and follow micseydel's referenced link

[LM19952012]

Hey, thanks for the pointer about the minimal examples. We are working to try and find a simple example to reproduce. In the meantime, are there any known bugs with multithreading the foreign function interface in ctypes (perhaps involving numpy arrays)? We only see this behavior at large numbers of cores, so it may take us a while to figure out the right set of parameters that elicit the bug in a minimal example. If there are any known bugs in this space, we'd like to avoid that.

[LM19952012]

I just created a super simple example that will give the exactly same error. The foreign function is:

void test(int* a, int* b, double* c, int na, int nb, int nc) {
for (int i = 0; i < na; i ++) {
a ++;
}
for (int i = 0; i < nb; i ++) {
b[i] ++;
}
for (int i = 0; i < nc; i ++) {
c[i] ++;
}
}

And the example is:

import numpy as np
from numpy.ctypeslib import ndpointer
import ctypes
import threading

def test(lib,a,b,c):
    fun = lib.test
    fun.restype = None
    fun.argtypes = [ndpointer(ctypes.c_int32),ndpointer(ctypes.c_int32),ndpointer(ctypes.c_double),
                   ctypes.c_int,ctypes.c_int,ctypes.c_int]
    fun(a,b,c,len(a),len(b),len(c))
    return a,b,c

def test_thread1(nthreads,lib,n):
    a = np.zeros(nthreads*10**n,dtype=np.int32)
    b = np.zeros(nthreads*10**n,dtype=np.int32)
    c = np.zeros(nthreads*10**n,dtype=np.float64)
    threads = []
    l = 10**n
    for i in range(nthreads):
        t = threading.Thread(target=test,args=(lib,a[i*l:(i+1)*l],b[i*l:(i+1)*l],c[i*l:(i+1)*l]))
        threads.append(t)
        t.start()
    for t in threads:
        t.join()
        
def test_thread2(nthreads,lib,n):
    a = np.zeros(nthreads*10**n,dtype=np.int32)
    b = np.zeros(nthreads*10**n,dtype=np.int32)
    c = np.zeros(nthreads*10**n,dtype=np.float64)
    threads = []
    a_split = np.array_split(a,nthreads)
    b_split = np.array_split(b,nthreads)
    c_split = np.array_split(c,nthreads)
    l = 10**n
    for i in range(nthreads):
        t = threading.Thread(target=test,args=(lib,a_split[i],b_split[i],c_split[i]))
        threads.append(t)
        t.start()
    for t in threads:
        t.join()

lib=ctypes.cdll.LoadLibrary("./test.so")
test_thread1(192,lib,8)
test_thread2(192,lib,8)

The error looks like this:

Error:
Exception in thread Thread-126:
Traceback (most recent call last):
  File "/homes/liu1740/miniconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/homes/liu1740/miniconda3/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-1-1710c759ecad>", line 11, in test
    fun(a,b,c,len(a),len(b),len(c))
ctypes.ArgumentError: argument 2: <class 'TypeError'>: 'numpy.ndarray' object is not callable

Again, you might need to run it several times before the error occurs. One observation that could be useful is that the error shows up much less frequently when using [i]test_thread1 comparing to test_thread2. It will be grateful if someone can help me solve this issue.

[Larz60+]

can't find ctype library test.so is this a dll?

[micseydel]

I was able to reproduce with the following additional details:

• added extern "C" at the beginning of the C code
  
• used g++ -shared -o test.so -fPIC test.c to compile the C file
  
• reduced the 8s on the last two lines to 7s because of a MemoryError
  

Error:
Exception in thread Thread-143:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "testit.py", line 11, in test
    fun(a,b,c,len(a),len(b),len(c))
ArgumentError: argument 2: <type 'exceptions.TypeError'>: 'numpy.ndarray' object is not callable

(It varies which argument number, but this is the general template)

I think I figured out what might be wrong. When you do

fun = lib.test

you're not making a copy. You're storing a reference. So in every test() call, you're mutating the restype and argtypes attributes of a single object in lots of threads. But that only needs to be done once; there might be some kind of a timing issue. Nail that down before any threading starts and I think you'll be good to go.

Generally, you should be extra careful about mutability with threads.