Python Forum
csv data not written into avro file
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
csv data not written into avro file
#1
csv data not written into avro file
DataFileWriter is not appending the data in avro format

 reader = csv.DictReader(opened_in_file, dialect=args.dialect)
    header = reader.fieldnames
    avro_schema = generate_schema(args, header)
    with open(args.out_file, 'w') as opened_out_file:
        write_avro_file(header,reader)
def write_avro_file(header,reader):   
    recordwriter = io.DatumWriter(avro_schema )
 
    df_writer = datafile.DataFileWriter(
                  
                    open(OUTFILE_NAME, 'wb'),
                   
                    recordwriter,
                    
                    codec = 'deflate'
                )
    for cfr in reader :
              
        df_writer.append(cfr)
 
    
    df_writer.close()
   
Reply
#2
Is your code really formatted like that.
Better?
reader = csv.DictReader(opened_in_file, dialect=args.dialect)
header = reader.fieldnames
avro_schema = generate_schema(args, header)
with open(args.out_file, 'w') as opened_out_file:
    write_avro_file(header, reader)

def write_avro_file(header, reader):
    recordwriter = io.DatumWriter(avro_schema)
    df_writer = datafile.DataFileWriter(
        open(OUTFILE_NAME, 'wb'), recordwriter, codec='deflate')
    for cfr in reader
        df_writer.append(cfr)
    df_writer.close()
Do you call function write_avro_file?
Can you post all code with imports,and sample input always make is easier to answer.
Reply
#3
Edit - This now duplicates some of the points raised by snippsat

Hi,

Can you please add all the code that will enable us to try out your script. For example the code that imports the CSV module and defines "opened_in_file" etc.

Then we can try out your script and see what is happening.

Thank you

Bass

"The good thing about standards is that you have so many to choose from" Andy S. Tanenbaum
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  I have written a program that outputs data based on GPS signal kalle 1 1,134 Jul-22-2022, 12:10 AM
Last Post: mcmxl22
  2 or more data to be written in a row plumberpy 2 1,452 Nov-25-2021, 07:44 AM
Last Post: Gribouillis
  How to check if a file has finished being written leocsmith 2 7,694 Apr-14-2021, 04:21 PM
Last Post: perfringo
  xml file creation from an XML file template and data from an excel file naji_python 1 2,070 Dec-21-2020, 03:24 PM
Last Post: Gribouillis
  How to save CSV file data into the Azure Data Lake Storage Gen2 table? Mangesh121 0 2,079 Jun-26-2020, 11:59 AM
Last Post: Mangesh121
  Data written in loop not available afterwards Weird 7 2,890 Mar-28-2020, 10:06 PM
Last Post: Weird
  Formatting file data written to file. melblanc 2 2,254 Jan-15-2020, 03:27 PM
Last Post: melblanc

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020