Python Forum
Merging all file_name.log's files from directory to one and search “PerformanceINFO"
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Merging all file_name.log's files from directory to one and search “PerformanceINFO"
#1
My question is: Following question, I want to achieve it in python although I have done this in bash(i have added my code at the end) but I want that to happen in python

Step1:

I have a bunch of
Quote:.log
files in the directory

Quote:Directory : /home/vs/vs_work2/Chi_DEPB/

& merge them into one file
Quote:merged_file.txt

Quote:Now merged_file.txt looks like as follows

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

Something....
something.....

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }
Something....
something.....

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

Step2: Now, from that
Quote:merged_file.txt
look for PerformanceINFO and extract 6 lines every time they appear and put it in
Quote:Extracted.txt

Quote:Extracted.txt file looks like

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

Step3:

From
Quote:Extracted.txt
file look for
Quote:payload_sizes (bytes) & packet_datarate (bits/clk)
and merge them every time they appear and put them into
Quote:output_file.csv

Quote:Output_file.csv looks like

payload_sizes(bytes): 1,2,3,4,5,6....10,100,202,303,405,5,6....10,13,24,35,45,55,65....10
packet_datarate(bits/clk):11,22,33,44,55,.....20,11,22,33,44,55,.....20,11,22,33,44,55,....


My bash code which to be converted to python

Step1:

Quote:cat /home/vs/vs_work2/Chi_DEPB/*.log > merged_file.txt

Step2:

Quote:grep -A9 -P '^PerformanceINFO *$' merged_file.txt >> Extracted.txt

Step3:
Quote:awk -F: '/packet_datarate/ { packt=packt""gensub(/[ {}]/,"","g",$2) } /payload_sizes/ { payl=payl""gensub(/[ {}]/,"","g",$2) } END { print "payload_sizes(bytes):,"payl;print "packet_datarate(bits/clk):,"packt }' Extracted.txt >> Output_file.csv
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How to save files in a separate directory Scordomaniac 3 637 Mar-16-2022, 10:17 AM
Last Post: Gribouillis
  Search multiple CSV files for a string or strings cubangt 7 1,896 Feb-23-2022, 12:53 AM
Last Post: Pedroski55
  Help needed with merging two CSV files eyadfr 5 1,054 Dec-14-2021, 07:34 PM
Last Post: paul18fr
  Search string in mutliple .gz files SARAOOF 10 3,285 Aug-26-2021, 01:47 PM
Last Post: SARAOOF
  Sorting and Merging text-files [SOLVED] AlphaInc 10 2,331 Aug-20-2021, 05:42 PM
Last Post: snippsat
  Cloning a directory and using a .CSV file as a reference to search and replace bg25lam 2 1,286 May-31-2021, 07:00 AM
Last Post: bowlofred
  how search files Ron_Crafter 4 1,578 Apr-17-2021, 11:19 AM
Last Post: Ron_Crafter
  A better way to search the contents of a directory? alloydog 5 1,454 Apr-11-2021, 09:12 AM
Last Post: alloydog
  Rename Multiple files in directory to remove special characters nyawadasi 9 3,103 Feb-16-2021, 09:49 PM
Last Post: BashBedlam
  List of error codes to find (and count) in all files in a directory tester_V 8 2,092 Dec-11-2020, 07:07 PM
Last Post: tester_V

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020