+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: time difference bettwenn logs (/thread-41154.html)
Pages:
1
2
time difference bettwenn logs - enkliy - Nov-20-2023
Hello,
I have a log file in txt. lets say
<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57418 -> 17.57.172.11:443 (TCP) close connection
<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57422 -> 17.57.172.11:443 (TCP)
<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57423 -> 17.57.172.11:443 (TCP)
<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57424 -> 17.57.13.65:443 (TCP)
<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57425 -> 17.57.13.65:443 (TCP)
<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57426 -> 17.57.13.65:443 (TCP)
<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57427 -> 17.57.172.11:443 (TCP)
<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57428 -> 17.57.13.65:443 (TCP)
<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57429 -> 17.57.13.65:443 (TCP)
<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57430 -> 17.57.13.65:443 (TCP)
<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57431 -> 96.17.179.45:443 (TCP)
<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57432 -> 17.57.13.65:443 (TCP)
<150>Nov 7 07:38:34 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57427 -> 17.57.172.11:443 (TCP) close connection
<150>Nov 7 07:38:34 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57433 -> 17.188.182.68:443 (TCP)
<150>Nov 7 07:38:35 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57434 -> 17.57.146.88:5223 (TCP)
<150>Nov 7 07:38:37 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57435 -> 17.248.211.69:443 (TCP)
I want to have a report/ on doing time difference between each line
So any help would be great. I have tried with pandas but that not possible...
RE: time difference bettwenn logs - snippsat - Nov-20-2023
(Nov-20-2023, 09:25 AM)enkliy Wrote: So any help would be great. I have tried with pandas but that not possible...You should show what you have tried,can do it Pandas but also there has parse the string first.
To help first step which should be is to parse time out and make it a time format.
>>> import re
>>> from datetime import datetime
>>>
>>> s = '<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57422 -> 17.57.172.11:443 (TCP)'
>>> r = re.search(r'\d{2}:\d{2}:\d{2}', s)
>>> time_str = r.group()
>>> time_str
'07:38:33'
>>> datetime.strptime(time_str, '%H:%M:%S')
datetime.datetime(1900, 1, 1, 7, 38, 33)When have time format can use timedelta between times.
RE: time difference bettwenn logs - rob101 - Nov-20-2023
Personally, I would not use regex in situations where the position of the data is known and unchanging; rather, I'd simply use the index positions.
As a simple demonstration:
with open("log", mode="r", encoding="UTF-8") as log:
for entry, item in enumerate(log, 1):
time_stamp = item[12:20]
print(f"{entry:02d}:~", time_stamp)To add: as working demonstration of the difference between the time entries:from datetime import datetime
time_stack = []
with open("log", mode="r", encoding="UTF-8") as log:
for entry, item in enumerate(log, 1):
time_str = item[12:20]
time_stamp = datetime.strptime(time_str, '%H:%M:%S')
time_stack.append(time_stamp)
if len(time_stack) > 1:
print(f"{entry:02d}:~", time_str, (time_stack[1] - time_stack[0]))
time_stack.pop(0)
else:
print(f"{entry:02d}:~ Start", time_str)Output:01:~ Start 07:38:33
02:~ 07:38:33 0:00:00
03:~ 07:38:33 0:00:00
04:~ 07:38:33 0:00:00
05:~ 07:38:33 0:00:00
06:~ 07:38:33 0:00:00
07:~ 07:38:33 0:00:00
08:~ 07:38:33 0:00:00
09:~ 07:38:33 0:00:00
10:~ 07:38:33 0:00:00
11:~ 07:38:33 0:00:00
12:~ 07:38:33 0:00:00
13:~ 07:38:34 0:00:01
14:~ 07:38:34 0:00:00
15:~ 07:38:35 0:00:01
16:~ 07:38:37 0:00:02RE: time difference bettwenn logs - enkliy - Nov-20-2023
Many thanks rob it was very easy and simple solution. The approach way I really appreciate your help
RE: time difference bettwenn logs - deanhystad - Nov-20-2023
It is possible with pandas, but pandas doesn't provide much help with this task. The format of the log file also makes it difficult for pandas to parse easily.
You should include the date when getting the times so periods spanning the end of day compute the correct delta. The log should also include year.
from io import StringIO
from datetime import datetime
log = StringIO(
"""<150>Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57422 -> 17.57.172.11:443 (TCP)
<150>Nov 7 07:38:43 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57423 -> 17.57.172.11:443 (TCP)
<150>Nov 7 08:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57424 -> 17.57.13.65:443 (TCP)
<150>Nov 8 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57425 -> 17.57.13.65:443 (TCP)"""
)
prev = None
for line in log:
dt = datetime.strptime(line[5:20], "%b %d %H:%M:%S")
if prev is not None:
print(dt - prev)
prev = dtOutput:0:00:10
0:59:50
23:00:00RE: time difference bettwenn logs - Larz60+ - Nov-21-2023
FIY:
A good writup on many aspects of date and time manipulation can be found datetime — Date and Time Value Manipulation
RE: time difference bettwenn logs - Pedroski55 - Nov-21-2023
I put the years in, just for fun:
# if you want to repeat, reset file seek to zero: log.seek(0)
log = StringIO(
"""<150>2022 Nov 7 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57422 -> 17.57.172.11:443 (TCP)
<150>2023 Nov 7 07:38:43 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57423 -> 17.57.172.11:443 (TCP)
<150>2023 Nov 7 08:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57424 -> 17.57.13.65:443 (TCP)
<150>2024 Nov 8 07:38:33 DrayTek: Local User (MAC=B0-8C-75-C0-FF-8F): 172.16.91.7:57425 -> 17.57.13.65:443 (TCP)"""
)Then just do what deanhystad said:prev = None
for line in log:
dt = datetime.strptime(line[5:25], "%Y %b %d %H:%M:%S")
if prev is not None:
print(dt - prev)
prev = dt
# need this to repeat reading the log
log.seek(0)Output:365 days, 0:00:10
0:59:50
366 days, 23:00:00RE: time difference bettwenn logs - enkliy - Nov-21-2023
I was testing with this code and working on displaying only values where the difference greater than 5 minutes
but output goes always ...with Start...
I imporetd also timedelta
from datetime import datetime, timedelta
time_stack = []
with open("log.txt", mode="r", encoding="UTF-8") as log:
for entry, item in enumerate(log, 1):
time_str = item[12:20]
time_stamp = datetime.strptime(time_str, '%H:%M:%S')
time_stack.append(time_stamp)
time_diff = timedelta(minutes=5)
if len(time_stack) > 1:
diffe = time_stack[1] - time_stack[0]
if diffe >= time_diff:
print(f"{entry:02d}:~", time_str, diffe)
time_stack.pop(0)
else:
print(f"{entry:02d}:~ Start", time_str)
RE: time difference bettwenn logs - rob101 - Nov-21-2023
What does the new log file look like?
With the one posted, there are no time difference > 2 seconds, let alone 5 minutes.
RE: time difference bettwenn logs - enkliy - Nov-21-2023
This is the output that I have got at lease it had to be empty
02:~ Start 07:38:32
03:~ Start 07:38:32
04:~ Start 07:38:32
05:~ Start 07:38:32
06:~ Start 07:38:32
07:~ Start 07:38:32
08:~ Start 07:38:32
09:~ Start 07:38:33
10:~ Start 07:38:33
11:~ Start 07:38:33
12:~ Start 07:38:33
13:~ Start 07:38:33
14:~ Start 07:38:33
15:~ Start 07:38:33
16:~ Start 07:38:33
17:~ Start 07:38:33
18:~ Start 07:38:33
19:~ Start 07:38:33
20:~ Start 07:38:33
21:~ Start 07:38:34
22:~ Start 07:38:34
23:~ Start 07:38:35
24:~ Start 07:38:37
25:~ Start 07:38:41
26:~ Start 07:38:41
27:~ Start 07:38:47
28:~ Start 07:38:47