Python Forum
Extract parts of multiple log-files and put it in a dataframe
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Extract parts of multiple log-files and put it in a dataframe
#5
(Apr-27-2022, 08:39 AM)menator01 Wrote: I have not used glob so, don't really know about it.

Using os.listdir (Code not tested)

from os import listdir
import pandas as pd

all_files = listdir('my_dir')

items = []

for filename in all_files:
    if filename.endswith('.txt'):
        with open(filename, 'r') as input_file:
            for line in input_file:
                if ':' in line:
                    a, b = map(str.strip, line.split(':', maxsplit=1))
                    items.append(b)
df = pd.DataFrame(items, columns=['Model', 'S/N', 'Timestamp', 'SW-Version'])

print(df)

Thanks for help. When I execute your code I got this error:
Output:
Shape of passed values is (3991, 1), indices imply (3991, 4)
To fix this, I made this little extension in your code:
from os import listdir
import pandas as pd
 
all_files = listdir('D:/Data/Deep Learning/LogFiles/C1/')
 
items = []
 
for filename in all_files:
    if filename.endswith('.txt'):
        with open(filename, 'r') as input_file:
            for line in input_file:
                if ':' in line:
                    a, b = map(str.strip, line.split(':', maxsplit=1))
                    items.append(b)
        new_result = items[0:4]
df = pd.DataFrame([new_result], columns=['Model', 'S/N', 'Timestamp', 'SW-Version'])
 
print(df)
After the extension with "new_result = items[0:4] I got this output:
Output:
Model S/N Timestamp SW-Version 0 Hamilton-C1 25455 2020-09-16_21-12-40 2.2.9
But this ist the same output like for a single file. I want have a DataFrame with specific content of all files inside the folder.
How I have to change my code for this. I tried a lot, with no luck.
Reply


Messages In This Thread
RE: Extract parts of multiple log-files and put it in a dataframe - by hasiro - Apr-27-2022, 12:44 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Is it possible to extract 1 or 2 bits of data from MS project files? cubangt 8 945 Feb-16-2024, 12:02 AM
Last Post: deanhystad
  python convert multiple files to multiple lists MCL169 6 1,436 Nov-25-2023, 05:31 AM
Last Post: Iqratech
  splitting file into multiple files by searching for string AlphaInc 2 816 Jul-01-2023, 10:35 PM
Last Post: Pedroski55
Question Need help for a python script to extract information from a list of files lephunghien 6 1,033 Jun-12-2023, 05:40 PM
Last Post: snippsat
  Merging multiple csv files with same X,Y,Z in each Auz_Pete 3 1,088 Feb-21-2023, 04:21 AM
Last Post: Auz_Pete
  unittest generates multiple files for each of my test case, how do I change to 1 file zsousa 0 918 Feb-15-2023, 05:34 PM
Last Post: zsousa
  splitting a Dataframe Column in two parts nafshar 2 912 Jan-30-2023, 01:19 PM
Last Post: nafshar
  Find duplicate files in multiple directories Pavel_47 9 2,930 Dec-27-2022, 04:47 PM
Last Post: deanhystad
  extract table from multiple pages sshree43 8 5,088 Dec-12-2022, 10:34 AM
Last Post: arvin
  Save multiple Parts of Bytearray to File ? lastyle 1 909 Dec-10-2022, 08:09 AM
Last Post: Gribouillis

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020