Python Forum

I am developing Hadoop File System client with python module hdfs3. My OS is CentOS 8 and IDE is eclipse. First I try to install hdfs3 with conda install command.


# conda config --add channels conda-forge
# conda config --set channel_priority strict
# conda install hdfs3


But the installation is failed. The Conda installation command can not download the package. So I use pip installation.

# pip install hdfs3


The pip installation is successful, but the python codes throw errors

from hdfs3 import HDFileSystem
hdfs = HDFileSystem(host='localhost', port=8020)

The errors are shown below,

Error:
Traceback (most recent call last):
  File "/home/jhwang/eclipse-workspace/BigDataPythonTest/com/aaa/etl/hdfs3-test.py", line 7, in <module>
    hdfs = HDFileSystem(host='localhost', port=8020)
  File "/usr/python/anaconda3/lib/python3.8/site-packages/hdfs3/core.py", line 88, in __init__
    self.connect()
  File "/usr/python/anaconda3/lib/python3.8/site-packages/hdfs3/core.py", line 106, in connect
    get_lib()
  File "/usr/python/anaconda3/lib/python3.8/site-packages/hdfs3/core.py", line 670, in get_lib
    from .lib import _lib as l
  File "/usr/python/anaconda3/lib/python3.8/site-packages/hdfs3/lib.py", line 24, in <module>
    raise ImportError("Can not find the shared library: libhdfs3.so\n"
ImportError: Can not find the shared library: libhdfs3.so

How can I download hdfs3 and libhdfs3 modules with conda install command? Any reply will be deeply thankful.

https://hdfs3.readthedocs.io/en/latest/install.html shows an alternate conda command for installation (in case that was why yours failed), and it shows that if you don't use conda, you need to install the libhdfs.so file via an OS package.

Finally, there's also a note that that project isn't being developed any longer, and if you have trouble, you might want to examine PyArrow's HDFS interface.