Pyarrow - parquet-cpp

[dennispoulos]

We have pyarrow 0.9 installed.
Parquet-cpp 1.4.1 is bundled with it.
We write parquet files all okay to AWS S3.

There is a Hive database with an external table overlay over the target parquet folder.
The Hive database has parquet format 1.8.1 installed.
Queries work okay.

However, we get warning messages due to the Parquet version differences.
How to remove the warning messages ?

How can we upgrade parquet-cpp to version 1.8.1 to match the hive parquet format version.
or
Is there another approach we should consider ?

After many types of searches, the solution is not found.

Any help is appreciated.
Many Thanks - DP

[Larz60+]

you should contact Joe Crobak: http://www.crobak.org
author of parquet
you should be able to upgrade with:

pip install parquet --upgrade

[dennispoulos]

Thank you Larz60+.
I reached out to Joe.
Does the Pip command upgrade only the parquet-cpp install or the parquet-me java one ?

[Larz60+]

Not sure, I am not familiar with this package, if the parquet package is dependent on parquet-me, You are at the mercy of the author. However, that being said, I would expect (never guaranteed) that the author would keep this up to date as well, otherwise his package might stop working properly, and if so (author keeps up to date) the pip upgrade would include all changes.