Python Forum
OSError: Unable to load libjvm when connecting to hdfs with pyarrow 3.0.0
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
OSError: Unable to load libjvm when connecting to hdfs with pyarrow 3.0.0
#1
I try to develop hadoop file system client application with pyarrow 3 on windows 10. First, my development environment are like below,

OS : Windows 10
Language : Anaconda 2020.11
IDE : eclipse

And my environment variables are

JAVA_HOME : C:\Program Files\Java\jdk-11.0.10
HADOOP_HOME : C:\hadoop-3.3.0
ARROW_LIBHDFS_DIR : C:\hadoop-3.3.0\lib\native
CLASSPATH = 'hdfs classpath --glob'



These are my short python codes with pyarrow

from pyarrow import fs
hdfs = fs.HadoopFileSystem('localhost', port=9000)


But I can not connect to my hadoop file system. The brought error is

Error:
hdfs = fs.HadoopFileSystem('localhost', port=9000) File "pyarrow\_hdfs.pyx", line 83, in pyarrow._hdfs.HadoopFileSystem.__init__ File "pyarrow\error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status File "pyarrow\error.pxi", line 99, in pyarrow.lib.check_status OSError: Unable to load libjvm:
I think my codes have some problems with java configuration but I have no idea how to correct this error.
Kindly inform me of your advise to correct this error.
Best regards.
Reply


Messages In This Thread
OSError: Unable to load libjvm when connecting to hdfs with pyarrow 3.0.0 - by aupres - Mar-22-2021, 10:25 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  OSError occurs in Linux. anna17 2 332 Mar-23-2024, 10:00 PM
Last Post: snippsat
  pyarrow error when importing pandas sravva 1 948 Jun-06-2023, 05:09 PM
Last Post: snippsat
  OSError with SMPT script Milan 0 797 Apr-28-2023, 01:34 PM
Last Post: Milan
  OSERROR When mkdir Oshadha 4 1,756 Jun-29-2022, 08:50 AM
Last Post: DeaD_EyE
  unable to load an image with C (Cython) HLD202 0 1,327 Jan-13-2022, 09:16 PM
Last Post: HLD202
  How to update values in a pyarrow table? xraphael75 1 3,691 Jan-25-2021, 02:14 PM
Last Post: xraphael75
  pyarrow throws oserror winerror 193 1 is not a valid win32 application aupres 2 3,811 Oct-21-2020, 01:04 AM
Last Post: aupres
  OSError: [Errno 26] Text file busy: '/var/tmp/tmp5qbeydbp batchenr 1 4,237 Mar-20-2020, 01:58 PM
Last Post: ibreeden
  Multiprocessing OSError 'too many open files' DreamingInsanity 3 10,400 Dec-27-2019, 04:50 PM
Last Post: DreamingInsanity
  .py to exe error "oserror winerror 193" michael1789 3 4,904 Dec-03-2019, 05:49 AM
Last Post: michael1789

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020