pyarrow throws oserror winerror 193 1 is not a valid win32 application

pyarrow throws oserror winerror 193 1 is not a valid win32 application - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: pyarrow throws oserror winerror 193 1 is not a valid win32 application (/thread-30389.html)

pyarrow throws oserror winerror 193 1 is not a valid win32 application - aupres - Oct-19-2020

Hello! My OS is windows 10 64bit and IDE is eclipse python. I try to install pyarrow with both pip and conda. The installation of pyarrow is successful. But my codes throw the errors like below,

import pyarrow as pa
fs = pa.hdfs.connect(host='localhost', port=9000)

Error:Traceback (most recent call last):
  File "C:\eclipse-workspace\PythonFredProj\com\aaa\fred\hdfs3-test.py", line 14, in <module>
    fs = pa.hdfs.connect(host='localhost', port=9000)
  File "C:\Python-3.8.3-x64\lib\site-packages\pyarrow\hdfs.py", line 208, in connect
    fs = HadoopFileSystem(host=host, port=port, user=user,
  File "C:\Python-3.8.3-x64\lib\site-packages\pyarrow\hdfs.py", line 38, in __init__
    _maybe_set_hadoop_classpath()
  File "C:\Python-3.8.3-x64\lib\site-packages\pyarrow\hdfs.py", line 136, in _maybe_set_hadoop_classpath
    classpath = _hadoop_classpath_glob(hadoop_bin)
  File "C:\Python-3.8.3-x64\lib\site-packages\pyarrow\hdfs.py", line 163, in _hadoop_classpath_glob
    return subprocess.check_output(hadoop_classpath_args)
  File "C:\Python-3.8.3-x64\lib\subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Python-3.8.3-x64\lib\subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Python-3.8.3-x64\lib\subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Python-3.8.3-x64\lib\subprocess.py", line 1307, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
OSError: [WinError 193] %1 is not a valid win32 application

And more!

This time I execute same python codes on CentOS 8, And they show the errors like below,

Error:Traceback (most recent call last):
  File "/home/jhwang/eclipse-workspace/BigDataPythonTest/com/aaa/etl/hdfs3-test.py", line 7, in <module>
    fs = pa.hdfs.connect(host='localhost', port=9000)
  File "/usr/python/anaconda3/lib/python3.8/site-packages/pyarrow/hdfs.py", line 208, in connect
    fs = HadoopFileSystem(host=host, port=port, user=user,
  File "/usr/python/anaconda3/lib/python3.8/site-packages/pyarrow/hdfs.py", line 40, in __init__
    self._connect(host, port, user, kerb_ticket, extra_conf)
  File "pyarrow/io-hdfs.pxi", line 75, in pyarrow.lib.HadoopFileSystem._connect
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: Unable to load libhdfs: ./libhdfs.so: cannot open shared object file: No such file or directory

Both errors are caused by the same reason? I have no idea what is wrong about this simple codes.
Any idea, please. Best regards.

RE: pyarrow throws oserror winerror 193 1 is not a valid win32 application - Larz60+ - Oct-19-2020

You will probably get a quicker answer from one of the maintainers.
You can get their email addresses here: https://pypi.org/project/pyarrow/

looks like you need to install the libhdfs.so library.

RE: pyarrow throws oserror winerror 193 1 is not a valid win32 application - aupres - Oct-21-2020

Thanks for your reply. As you told, I sent the e-mail to the maintainer of pyarrow project. But I receive no reply until now. Kindly inform me of any reply.