Python Forum
help with project of reading and searching big log file
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
help with project of reading and searching big log file
#1
Hello ,
I have a log file that in the end of the day get to ~ 6GB of text
now I want to be able to cut from it a certion windows of time
for example
from 08:00:00 -- until 08:15:00

I have checked and in 15 min I have a around 1.5 milion lines (1,500,000)
when I run the code in the morning , when the log file is less then 1GB - everything is working .

when I run the code in the end of the day (when the log is more then 5GB)
It get stuck , sometime I get on my computer Memory error

and when I try to search another later window (7:00pm-7:20pm ) it can take more then 3 min before it get stuck

my question is
what can I do to make this run better ? faster ?
can pythion handale this amount of data?

this is the function
def FilterLogFile(StartDate, EndDate):

    StartDate = datetime.datetime.strptime(StartDate, '%d/%m/%Y-%H:%M:%S')
    EndDate = datetime.datetime.strptime(EndDate, '%d/%m/%Y-%H:%M:%S')
    EndDate = EndDate.strftime('%d/%m/%Y-%H:%M:%S')
    StartDate = StartDate.strftime('%d/%m/%Y-%H:%M:%S')
    StartDate = str(StartDate)
    EndDate = str(EndDate)
    print(StartDate)
    print(EndDate)
    count = 0
    StartLine = 0
    EndLine = 0
    FullLogFile = open('/home/pi/logs/java.txt', 'r')
    Lines = FullLogFile.readlines()   ###------->>>> this part take to much time when it doens't stuck "Memory Error"
    FullLogFile.close()
    for line in Lines:
        count += 1
        if StartDate in line and StartLine == 0:
            print("Start Line {}: {}".format(count, line.strip()))
            StartLine = count
        if EndDate in line and EndLine == 0:
            print("End Line {}: {}".format(count, line.strip()))
            EndLine = count
        if StartLine != 0 and EndLine != 0:
            break  ## to stop the scan when he get to the wanted end time , no need to scan after the wanted time 

    count = 0
    print('start line is %d , end line is %d' % (StartLine, EndLine))
    print('total number of line is  %d' % (EndLine - StartLine))
    with open(OutputFile, 'w') as f:
        for line in Lines:
            count += 1
            if StartLine <= count <= EndLine:
                f.write(line.strip() + "\r\n")
    
    
    return OutputFile
Thanks,
maybe to read
Reply


Messages In This Thread
help with project of reading and searching big log file - by korenron - Jun-24-2021, 09:17 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
Sad problems with reading csv file. MassiJames 3 619 Nov-16-2023, 03:41 PM
Last Post: snippsat
  Reading a file name fron a folder on my desktop Fiona 4 900 Aug-23-2023, 11:11 AM
Last Post: Axel_Erfurt
  splitting file into multiple files by searching for string AlphaInc 2 888 Jul-01-2023, 10:35 PM
Last Post: Pedroski55
  Reading data from excel file –> process it >>then write to another excel output file Jennifer_Jone 0 1,091 Mar-14-2023, 07:59 PM
Last Post: Jennifer_Jone
  Reading a file JonWayn 3 1,093 Dec-30-2022, 10:18 AM
Last Post: ibreeden
  Reading Specific Rows In a CSV File finndude 3 981 Dec-13-2022, 03:19 PM
Last Post: finndude
  Web project and running a .py file emont 0 637 Dec-11-2022, 11:15 PM
Last Post: emont
  Excel file reading problem max70990 1 891 Dec-11-2022, 07:00 PM
Last Post: deanhystad
  Replace columns indexes reading a XSLX file Larry1888 2 977 Nov-18-2022, 10:16 PM
Last Post: Pedroski55
  Failing reading a file and cannot exit it... tester_V 8 1,801 Aug-19-2022, 10:27 PM
Last Post: tester_V

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020