time difference bettwenn logs

[enkliy]

Hello,
I have a log file in txt. lets say

<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57418 -> 17.57.172.11:443 (TCP) close connection 
<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57422 -> 17.57.172.11:443 (TCP)  
<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57423 -> 17.57.172.11:443 (TCP)  
<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57424 -> 17.57.13.65:443 (TCP)  
<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57425 -> 17.57.13.65:443 (TCP)  
<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57426 -> 17.57.13.65:443 (TCP)  
<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57427 -> 17.57.172.11:443 (TCP)  
<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57428 -> 17.57.13.65:443 (TCP)  
<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57429 -> 17.57.13.65:443 (TCP)  
<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57430 -> 17.57.13.65:443 (TCP)  
<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57431 -> 96.17.179.45:443 (TCP)  
<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57432 -> 17.57.13.65:443 (TCP)  
<150>Nov  7 07:38:34 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57427 -> 17.57.172.11:443 (TCP) close connection 
<150>Nov  7 07:38:34 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57433 -> 17.188.182.68:443 (TCP)  
<150>Nov  7 07:38:35 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57434 -> 17.57.146.88:5223 (TCP)  
<150>Nov  7 07:38:37 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57435 -> 17.248.211.69:443 (TCP)  

I want to have a report/ on doing time difference between each line
So any help would be great. I have tried with pandas but that not possible...

[snippsat]

So any help would be great. I have tried with pandas but that not possible...

You should show what you have tried,can do it Pandas but also there has parse the string first.
To help first step which should be is to parse time out and make it a time format.

>>> import re
>>> from datetime import datetime
>>> 
>>> s = '<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57422 -> 17.57.172.11:443 (TCP)' 
>>> r = re.search(r'\d{2}:\d{2}:\d{2}', s)
>>> time_str = r.group()
>>> time_str
'07:38:33'
>>> datetime.strptime(time_str, '%H:%M:%S')
datetime.datetime(1900, 1, 1, 7, 38, 33)

When have time format can use timedelta between times.

[rob101]

Personally, I would not use regex in situations where the position of the data is known and unchanging; rather, I'd simply use the index positions.

As a simple demonstration:

with open("log", mode="r", encoding="UTF-8") as log:
    for entry, item in enumerate(log, 1):
        time_stamp = item[12:20]
        print(f"{entry:02d}:~", time_stamp)

To add: as working demonstration of the difference between the time entries:

from datetime import datetime

time_stack = []

with open("log", mode="r", encoding="UTF-8") as log:
    for entry, item in enumerate(log, 1):
        time_str = item[12:20]
        time_stamp = datetime.strptime(time_str, '%H:%M:%S')
        time_stack.append(time_stamp)
        if len(time_stack) > 1:
            print(f"{entry:02d}:~", time_str, (time_stack[1] - time_stack[0]))
            time_stack.pop(0)
        else:
            print(f"{entry:02d}:~ Start", time_str)

Output:
01:~ Start 07:38:33
02:~ 07:38:33 0:00:00
03:~ 07:38:33 0:00:00
04:~ 07:38:33 0:00:00
05:~ 07:38:33 0:00:00
06:~ 07:38:33 0:00:00
07:~ 07:38:33 0:00:00
08:~ 07:38:33 0:00:00
09:~ 07:38:33 0:00:00
10:~ 07:38:33 0:00:00
11:~ 07:38:33 0:00:00
12:~ 07:38:33 0:00:00
13:~ 07:38:34 0:00:01
14:~ 07:38:34 0:00:00
15:~ 07:38:35 0:00:01
16:~ 07:38:37 0:00:02

[enkliy]

Many thanks rob it was very easy and simple  solution. The approach way I really appreciate your help

[deanhystad]

It is possible with pandas, but pandas doesn't provide much help with this task. The format of the log file also makes it difficult for pandas to parse easily.

You should include the date when getting the times so periods spanning the end of day compute the correct delta. The log should also include year.

from io import StringIO
from datetime import datetime

log = StringIO(
    """<150>Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57422 -> 17.57.172.11:443 (TCP)
<150>Nov  7 07:38:43 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57423 -> 17.57.172.11:443 (TCP)
<150>Nov  7 08:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57424 -> 17.57.13.65:443 (TCP)
<150>Nov  8 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57425 -> 17.57.13.65:443 (TCP)"""
)

prev = None
for line in log:
    dt = datetime.strptime(line[5:20], "%b %d %H:%M:%S")
    if prev is not None:
        print(dt - prev)
    prev = dt

Output:
0:00:10
0:59:50
23:00:00

[Larz60+]

FIY:
A good writup on many aspects of date and time manipulation can be found datetime — Date and Time Value Manipulation

[Pedroski55]

I put the years in, just for fun:

# if you want to repeat, reset file seek to zero: log.seek(0)
log = StringIO(
    """<150>2022 Nov  7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57422 -> 17.57.172.11:443 (TCP)
<150>2023 Nov  7 07:38:43 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57423 -> 17.57.172.11:443 (TCP)
<150>2023 Nov  7 08:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57424 -> 17.57.13.65:443 (TCP)
<150>2024 Nov  8 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57425 -> 17.57.13.65:443 (TCP)"""
)

Then just do what deanhystad said:

prev = None
for line in log:
    dt = datetime.strptime(line[5:25], "%Y %b %d %H:%M:%S")
    if prev is not None:
        print(dt - prev)
    prev = dt
# need this to repeat reading the log
log.seek(0)

Output:
365 days, 0:00:10
0:59:50
366 days, 23:00:00

[enkliy]

I was testing with this code and working on displaying only values where the difference greater than 5 minutes
but output goes always ...with Start...
I imporetd also timedelta

from datetime import datetime, timedelta
 
time_stack = []
 
with open("log.txt", mode="r", encoding="UTF-8") as log:
    for entry, item in enumerate(log, 1):
        time_str = item[12:20]
        time_stamp = datetime.strptime(time_str, '%H:%M:%S')
        time_stack.append(time_stamp)
        time_diff = timedelta(minutes=5)
        if  len(time_stack) > 1:
            diffe = time_stack[1] - time_stack[0]
            if diffe >= time_diff:
               print(f"{entry:02d}:~", time_str, diffe)
               time_stack.pop(0)
            else:
                print(f"{entry:02d}:~ Start", time_str)

Larz60+ write Nov-22-2023, 12:01 PM:
Please post all code, output and errors (it it's entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.

[rob101]

What does the new log file look like?

With the one posted, there are no time difference > 2 seconds, let alone 5 minutes.

[enkliy]

This is the output that I have got at lease it had to be empty

02:~ Start 07:38:32
03:~ Start 07:38:32
04:~ Start 07:38:32
05:~ Start 07:38:32
06:~ Start 07:38:32
07:~ Start 07:38:32
08:~ Start 07:38:32
09:~ Start 07:38:33
10:~ Start 07:38:33
11:~ Start 07:38:33
12:~ Start 07:38:33
13:~ Start 07:38:33
14:~ Start 07:38:33
15:~ Start 07:38:33
16:~ Start 07:38:33
17:~ Start 07:38:33
18:~ Start 07:38:33
19:~ Start 07:38:33
20:~ Start 07:38:33
21:~ Start 07:38:34
22:~ Start 07:38:34
23:~ Start 07:38:35
24:~ Start 07:38:37
25:~ Start 07:38:41
26:~ Start 07:38:41
27:~ Start 07:38:47
28:~ Start 07:38:47