Python Forum
OSError: Unable to load libjvm when connecting to hdfs with pyarrow 3.0.0
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
OSError: Unable to load libjvm when connecting to hdfs with pyarrow 3.0.0
#1
I try to develop hadoop file system client application with pyarrow 3 on windows 10. First, my development environment are like below,

OS : Windows 10
Language : Anaconda 2020.11
IDE : eclipse

And my environment variables are

JAVA_HOME : C:\Program Files\Java\jdk-11.0.10
HADOOP_HOME : C:\hadoop-3.3.0
ARROW_LIBHDFS_DIR : C:\hadoop-3.3.0\lib\native
CLASSPATH = 'hdfs classpath --glob'



These are my short python codes with pyarrow

from pyarrow import fs
hdfs = fs.HadoopFileSystem('localhost', port=9000)


But I can not connect to my hadoop file system. The brought error is

Error:
hdfs = fs.HadoopFileSystem('localhost', port=9000) File "pyarrow\_hdfs.pyx", line 83, in pyarrow._hdfs.HadoopFileSystem.__init__ File "pyarrow\error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status File "pyarrow\error.pxi", line 99, in pyarrow.lib.check_status OSError: Unable to load libjvm:
I think my codes have some problems with java configuration but I have no idea how to correct this error.
Kindly inform me of your advise to correct this error.
Best regards.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  OSError occurs in Linux. anna17 2 279 Mar-23-2024, 10:00 PM
Last Post: snippsat
  pyarrow error when importing pandas sravva 1 933 Jun-06-2023, 05:09 PM
Last Post: snippsat
  OSError with SMPT script Milan 0 759 Apr-28-2023, 01:34 PM
Last Post: Milan
  OSERROR When mkdir Oshadha 4 1,700 Jun-29-2022, 08:50 AM
Last Post: DeaD_EyE
  unable to load an image with C (Cython) HLD202 0 1,302 Jan-13-2022, 09:16 PM
Last Post: HLD202
  How to update values in a pyarrow table? xraphael75 1 3,664 Jan-25-2021, 02:14 PM
Last Post: xraphael75
  pyarrow throws oserror winerror 193 1 is not a valid win32 application aupres 2 3,776 Oct-21-2020, 01:04 AM
Last Post: aupres
  OSError: [Errno 26] Text file busy: '/var/tmp/tmp5qbeydbp batchenr 1 4,208 Mar-20-2020, 01:58 PM
Last Post: ibreeden
  Multiprocessing OSError 'too many open files' DreamingInsanity 3 10,336 Dec-27-2019, 04:50 PM
Last Post: DreamingInsanity
  .py to exe error "oserror winerror 193" michael1789 3 4,885 Dec-03-2019, 05:49 AM
Last Post: michael1789

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020