Python Forum
Merging all file_name.log's files from directory to one and search “PerformanceINFO"
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Merging all file_name.log's files from directory to one and search “PerformanceINFO"
#1
My question is: Following question, I want to achieve it in python although I have done this in bash(i have added my code at the end) but I want that to happen in python

Step1:

I have a bunch of
Quote:.log
files in the directory

Quote:Directory : /home/vs/vs_work2/Chi_DEPB/

& merge them into one file
Quote:merged_file.txt

Quote:Now merged_file.txt looks like as follows

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

Something....
something.....

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }
Something....
something.....

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

Step2: Now, from that
Quote:merged_file.txt
look for PerformanceINFO and extract 6 lines every time they appear and put it in
Quote:Extracted.txt

Quote:Extracted.txt file looks like

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

PerformanceINFO
UVM_INFO_PERF ****NIB-FIB Axis Interface Per-packet Performance Report****
date : NAN,
interface_description : injection,
payload_sizes (bytes) : {312, 1100, 1132, 1404, }
packet_datarate (bits/clk): {112, 114, 113, 116, 115, 114, 114, }

Step3:

From
Quote:Extracted.txt
file look for
Quote:payload_sizes (bytes) & packet_datarate (bits/clk)
and merge them every time they appear and put them into
Quote:output_file.csv

Quote:Output_file.csv looks like

payload_sizes(bytes): 1,2,3,4,5,6....10,100,202,303,405,5,6....10,13,24,35,45,55,65....10
packet_datarate(bits/clk):11,22,33,44,55,.....20,11,22,33,44,55,.....20,11,22,33,44,55,....


My bash code which to be converted to python

Step1:

Quote:cat /home/vs/vs_work2/Chi_DEPB/*.log > merged_file.txt

Step2:

Quote:grep -A9 -P '^PerformanceINFO *$' merged_file.txt >> Extracted.txt

Step3:
Quote:awk -F: '/packet_datarate/ { packt=packt""gensub(/[ {}]/,"","g",$2) } /payload_sizes/ { payl=payl""gensub(/[ {}]/,"","g",$2) } END { print "payload_sizes(bytes):,"payl;print "packet_datarate(bits/clk):,"packt }' Extracted.txt >> Output_file.csv
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  uploading files from a ubuntu local directory to Minio storage container dchilambo 0 398 Dec-22-2023, 07:17 AM
Last Post: dchilambo
  change directory of save of python files akbarza 3 807 Jul-23-2023, 08:30 AM
Last Post: Gribouillis
  Using pyinstaller with .ui GUI files - No such file or directory error diver999 3 3,072 Jun-27-2023, 01:17 PM
Last Post: diver999
  Monitoring a Directory for new mkv and mp4 Files lastyle 3 1,568 May-07-2023, 12:33 PM
Last Post: deanhystad
  Merging multiple csv files with same X,Y,Z in each Auz_Pete 3 1,085 Feb-21-2023, 04:21 AM
Last Post: Auz_Pete
  Read directory listing of files and parse out the highest number? cubangt 5 2,251 Sep-28-2022, 10:15 PM
Last Post: Larz60+
  How to save files in a separate directory Scordomaniac 3 1,755 Mar-16-2022, 10:17 AM
Last Post: Gribouillis
  Search multiple CSV files for a string or strings cubangt 7 7,840 Feb-23-2022, 12:53 AM
Last Post: Pedroski55
  Help needed with merging two CSV files eyadfr 5 2,863 Dec-14-2021, 07:34 PM
Last Post: paul18fr
  Search string in mutliple .gz files SARAOOF 10 6,775 Aug-26-2021, 01:47 PM
Last Post: SARAOOF

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020