+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: ImportError: Can not find the shared library: libhdfs3.so (/thread-30372.html)
ImportError: Can not find the shared library: libhdfs3.so - aupres - Oct-18-2020
I am developing Hadoop File System client with python module hdfs3. My OS is CentOS 8 and IDE is eclipse. First I try to install hdfs3 with conda install command.
# conda config --add channels conda-forge
# conda config --set channel_priority strict
# conda install hdfs3
But the installation is failed. The Conda installation command can not download the package. So I use pip installation.
# pip install hdfs3
The pip installation is successful, but the python codes throw errors
from hdfs3 import HDFileSystem hdfs = HDFileSystem(host='localhost', port=8020)The errors are shown below,
Error:Traceback (most recent call last):
File "/home/jhwang/eclipse-workspace/BigDataPythonTest/com/aaa/etl/hdfs3-test.py", line 7, in <module>
hdfs = HDFileSystem(host='localhost', port=8020)
File "/usr/python/anaconda3/lib/python3.8/site-packages/hdfs3/core.py", line 88, in __init__
self.connect()
File "/usr/python/anaconda3/lib/python3.8/site-packages/hdfs3/core.py", line 106, in connect
get_lib()
File "/usr/python/anaconda3/lib/python3.8/site-packages/hdfs3/core.py", line 670, in get_lib
from .lib import _lib as l
File "/usr/python/anaconda3/lib/python3.8/site-packages/hdfs3/lib.py", line 24, in <module>
raise ImportError("Can not find the shared library: libhdfs3.so\n"
ImportError: Can not find the shared library: libhdfs3.soRE: ImportError: Can not find the shared library: libhdfs3.so - bowlofred - Oct-18-2020
https://hdfs3.readthedocs.io/en/latest/install.html shows an alternate conda command for installation (in case that was why yours failed), and it shows that if you don't use conda, you need to install the libhdfs.so file via an OS package.
Finally, there's also a note that that project isn't being developed any longer, and if you have trouble, you might want to examine PyArrow's HDFS interface.