Python Forum
Merging all file_name.log's files from directory to one and search “PerformanceINFO"
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Merging all file_name.log's files from directory to one and search “PerformanceINFO"
#1
My question is: Following question, I want to achieve it in python although I have done this in bash(i have added my code at the end) but I want that to happen in python

Step1:

I have a bunch of
Quote:.log
files in the directory

Quote:Directory : /home/vs/vs_work2/Chi_DEPB/

& merge them into one file
Quote:merged_file.txt

Quote:Now merged_file.txt looks like as follows

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

Something....
something.....

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }
Something....
something.....

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

Step2: Now, from that
Quote:merged_file.txt
look for PerformanceINFO and extract 6 lines every time they appear and put it in
Quote:Extracted.txt

Quote:Extracted.txt file looks like

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

Step3:

From
Quote:Extracted.txt
file look for
Quote:payload_sizes (bytes) & packet_datarate (bits/clk)
and merge them every time they appear and put them into
Quote:output_file.csv

Quote:Output_file.csv looks like

payload_sizes(bytes): 1,2,3,4,5,6....10,100,202,303,405,5,6....10,13,24,35,45,55,65....10
packet_datarate(bits/clk):11,22,33,44,55,.....20,11,22,33,44,55,.....20,11,22,33,44,55,....


My bash code which to be converted to python

Step1:

Quote:cat /home/vs/vs_work2/Chi_DEPB/*.log > merged_file.txt

Step2:

Quote:grep -A9 -P '^PerformanceINFO *$' merged_file.txt >> Extracted.txt

Step3:
Quote:awk -F: '/packet_datarate/ { packt=packt""gensub(/[ {}]/,"","g",$2) } /payload_sizes/ { payl=payl""gensub(/[ {}]/,"","g",$2) } END { print "payload_sizes(bytes):,"payl;print "packet_datarate(bits/clk):,"packt }' Extracted.txt >> Output_file.csv
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Cloning a directory and using a .CSV file as a reference to search and replace bg25lam 2 311 May-31-2021, 07:00 AM
Last Post: bowlofred
  how search files Ron_Crafter 4 553 Apr-17-2021, 11:19 AM
Last Post: Ron_Crafter
  A better way to search the contents of a directory? alloydog 5 562 Apr-11-2021, 09:12 AM
Last Post: alloydog
  Rename Multiple files in directory to remove special characters nyawadasi 9 796 Feb-16-2021, 09:49 PM
Last Post: BashBedlam
  List of error codes to find (and count) in all files in a directory tester_V 8 891 Dec-11-2020, 07:07 PM
Last Post: tester_V
  How to merge all the files in a directory in to one file sutra 3 628 Dec-10-2020, 12:09 AM
Last Post: sutra
  Copy files from subfolders into same name of subfolders at other directory rathoreanil 1 609 Oct-12-2020, 01:30 AM
Last Post: Larz60+
  Moving Files From Subdirectories To Another Directory Harshil 5 976 Oct-06-2020, 10:52 AM
Last Post: ndc85430
  Put all files in a directory into list. How? Denial 2 582 Sep-18-2020, 10:05 PM
Last Post: Larz60+
  Merging Excel Files JezMim 1 639 Sep-06-2020, 08:56 PM
Last Post: bowlofred

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020