Python Forum
Python script merging some columns to one column with new name
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python script merging some columns to one column with new name
#1
I have a lot of csv files with data and about 30 possible known headers, Header1 to Header30. The number of rows can be practically anything, from 1 to thousands. Not all the headers are present in each file. Seperator is a semicolon.

An example is Header1;Header3;Header8;Header10;Header11;Header17;Header18

Some of the headers have to be combined to one with a new headername.

The columns with Header1 to Header3, if one of them exists, should always be combined in column NewHeaderA devided by spaces.
The column with Header4 becomes column NewHeaderB
The column with Header5 becomes column NewHeaderC
The columns with Header6 to Header14, if one of them exists, should always be combined in column NewHeaderD devided by spaces.
And so on in more combinations for the rest of the headers.

So far I've made a script that reads a csv and writes out another file with all the same headers or part of the existing headers in a different order. After that I'm stuck. Some help would be appreciated.

Due to some system limitations, pandas or numpy are not installed. Just plain Python 3.7.6

import sys
inFile = sys.argv[1]
outFile = sys.argv[2]

# open csv file
csvfile = open(inFile, "r" )
reader = csv.DictReader(csvfile, delimiter=';')

# open output file
outfile = open(outFile, "w" )

# fieldnames=reader.fieldnames  --> uses all columns
fieldnames = ["Header1","Header4","Header5","Header8","Header19","test"] # --> uses the specific columns in the order provided, test becomes an emty column

# write the output to a file
writer = csv.DictWriter(outfile, delimiter=';', fieldnames=fieldnames, extrasaction='ignore')
headers = {} 
for n in writer.fieldnames:
    headers[n] = n
writer.writerow(headers)
for row in reader:
    writer.writerow(row)

csvfile.close()
outfile.close()
Reply


Messages In This Thread
Python script merging some columns to one column with new name - by jbveenstra - Feb-11-2020, 09:15 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Converting column of values into muliple columns of counts highland44 0 295 Feb-01-2024, 12:48 AM
Last Post: highland44
  Is there a *.bat DOS batch script to *.py Python Script converter? pstein 3 3,452 Jun-29-2023, 11:57 AM
Last Post: gologica
  J2534 Python Can Bus merging natezoom 0 755 May-01-2023, 10:37 PM
Last Post: natezoom
  Reshaping a single column in to multiple column using Python sahar 7 2,139 Jun-20-2022, 12:35 PM
Last Post: deanhystad
  df column aggregate and group by multiple columns SriRajesh 0 1,085 May-06-2022, 02:26 PM
Last Post: SriRajesh
  Transform 3 Columns into Single Column DaveG 8 1,938 Apr-04-2022, 08:42 AM
Last Post: Pedroski55
  Split single column to multiple columns SriRajesh 1 1,361 Jan-07-2022, 06:43 PM
Last Post: jefsummers
  How to remove a column or two columns in a correlation heatmap? lulu43366 3 5,340 Sep-30-2021, 03:47 PM
Last Post: lulu43366
  Merging spreadsheets with the same columns and extracting rows with matching entries johnbernard 3 10,849 Aug-19-2021, 03:08 PM
Last Post: johnbernard
  Index error - columns vs non-column Vinny 3 4,984 Aug-09-2021, 04:46 PM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020