Jan-22-2021, 02:23 PM
(This post was last modified: Jan-22-2021, 02:23 PM by xraphael75.)
I have a python script that reads in a parquet file using pyarrow. I'm trying to loop through the table to update values in it. If I try this:
I tried using pandas, but it couldn't handle null values in the original table, and it also incorrectly translated the datatypes of the columns in the original table. Does pyarrow have a native way to edit the data?
Python 3.7.3
Debian 10
for col_name in table2.column_names: if col_name in my_columns: print('updating values in column ' + col_name) col_data = pa.Table.column(table2, col_name) row_ct = 1 for i in col_data: pa.Table.column(table2, col_name)[row_ct] = change_str(pa.StringScalar.as_py(i)) row_ct += 1I get this error:
Error:TypeError: 'pyarrow.lib.ChunkedArray' object does not support item assignment
How can I update these values? I tried using pandas, but it couldn't handle null values in the original table, and it also incorrectly translated the datatypes of the columns in the original table. Does pyarrow have a native way to edit the data?
Python 3.7.3
Debian 10