Python Forum
How to update values in a pyarrow table?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to update values in a pyarrow table?
#1
I have a python script that reads in a parquet file using pyarrow. I'm trying to loop through the table to update values in it. If I try this:

for col_name in table2.column_names:
    if col_name in my_columns:
        print('updating values in column '  + col_name)
        
        col_data = pa.Table.column(table2, col_name)
        
        row_ct = 1
        for i in col_data:
            pa.Table.column(table2, col_name)[row_ct] = change_str(pa.StringScalar.as_py(i))
            row_ct += 1
I get this error:

Error:
TypeError: 'pyarrow.lib.ChunkedArray' object does not support item assignment
How can I update these values?

I tried using pandas, but it couldn't handle null values in the original table, and it also incorrectly translated the datatypes of the columns in the original table. Does pyarrow have a native way to edit the data?

Python 3.7.3
Debian 10
Reply


Messages In This Thread
How to update values in a pyarrow table? - by xraphael75 - Jan-22-2021, 02:23 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
Question Using SQLAlchemy, prevent SQLite3 table update by multiple program instances Calab 3 704 Aug-09-2023, 05:51 PM
Last Post: Calab
  pyarrow error when importing pandas sravva 1 885 Jun-06-2023, 05:09 PM
Last Post: snippsat
Photo How to select NULL and blank values from MySQL table into csv python300 9 2,331 Dec-27-2022, 09:43 PM
Last Post: deanhystad
  store all variable values into list and insert to sql_summary table mg24 3 1,097 Sep-28-2022, 09:13 AM
Last Post: Larz60+
  Sum the values in a pandas pivot table specific columns klllmmm 1 4,545 Nov-19-2021, 04:43 PM
Last Post: klllmmm
  UPDATE SQLITE TABLE - Copy a fields content to another field. andrewarles 14 4,248 May-08-2021, 04:58 PM
Last Post: ibreeden
  OSError: Unable to load libjvm when connecting to hdfs with pyarrow 3.0.0 aupres 0 3,100 Mar-22-2021, 10:25 AM
Last Post: aupres
Question Python + Google Sheet | Best way to update specific cells in a single Update()? Vokofe 1 2,627 Dec-16-2020, 05:26 AM
Last Post: Vokofe
  pyarrow throws oserror winerror 193 1 is not a valid win32 application aupres 2 3,722 Oct-21-2020, 01:04 AM
Last Post: aupres
  Unable to Update SQLite Table sambanerjee 5 2,867 Sep-30-2020, 12:21 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020