Python Forum
Creating Dictionary form LOG /text file
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Creating Dictionary form LOG /text file
#1
Hi All,
I am new to Python world and because of that I am not expert to do all the stuff. I am having some issue to convert below log file content to Dictionary format. Appreciate if someone from this forum please help me to do that. Thanks in Advance.

log file content like below. This log file will have several thousand record and each line will have more key.

002d2d62-4445-a06d.log:20190208 09:19:43.115 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"12234678901","transactions":[{"TransCode":"XXX","AcctTypeCode":"DD",,"BranchNumber":"000x","Desc":"02/05 FEDEXOFFICE 00 GGGG","ProductCode":"104"}]}
002d2d63-4445-a06d.log:20190208 09:19:52.538 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"22222222222","transactions":[{"TransCode":"YYY","AcctTypeCode":"DD","BranchNumber":"000y","Desc":"02/05 FEDEXOFFICE 00 YYYYY","ProductCode":"104"}]}
Reply
#2
(Feb-12-2019, 07:51 PM)DG1234 Wrote: I am having some issue to convert below log file content to Dictionary format.

what issues do you have?
you can split each line at :. and Message is JSON
note that you have double comma on first line in the message - between "DD" and "BranchNumber". I hope it's error because it invalidate the JSON
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#3
Double comma is typo, that will not have.

Can you please give an example code, So I can use that to create Dictionary from this file.

I am very new and not familiar with all syntax and process. Please help me with a sample code.
Reply
#4
import json

log_file = ['002d2d62-4445-a06d.log:20190208 09:19:43.115 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"12234678901","transactions":[{"TransCode":"XXX","AcctTypeCode":"DD","BranchNumber":"000x","Desc":"02/05 FEDEXOFFICE 00 GGGG","ProductCode":"104"}]}',
'002d2d63-4445-a06d.log:20190208 09:19:52.538 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"22222222222","transactions":[{"TransCode":"YYY","AcctTypeCode":"DD","BranchNumber":"000y","Desc":"02/05 FEDEXOFFICE 00 YYYYY","ProductCode":"104"}]}']

for line in log_file:
    print('\n--- NEW LOG LINE ---\n')
    log, m, json_message = line.split(': ')
    log_id, log_time, *rest = log.split(' ')
    log_id, log_date = log_id.split(':')
    print(log_id)
    print(log_date)
    print(log_time)
    print(rest)
    print(m)
    message = json.loads(json_message)
    print(message["sourceSysCode"])
    print(message["sourceSysAcctNumber"])
    for transaction in message['transactions']:
        print(transaction)
Output:
--- NEW LOG LINE --- 002d2d62-4445-a06d.log 20190208 09:19:43.115 ['Debug', 'Generic_OnWorkComplete'] Message TP 12234678901 {'Desc': '02/05 FEDEXOFFICE 00 GGGG', 'BranchNumber': '000x', 'TransCode': 'XXX', 'ProductCode': '104', 'AcctTypeCode': 'DD'} --- NEW LOG LINE --- 002d2d63-4445-a06d.log 20190208 09:19:52.538 ['Debug', 'Generic_OnWorkComplete'] Message TP 22222222222 {'Desc': '02/05 FEDEXOFFICE 00 YYYYY', 'BranchNumber': '000y', 'TransCode': 'YYY', 'ProductCode': '104', 'AcctTypeCode': 'DD'}
log_file is just dummy. You need to open the log file and read file line by line, e.g.
import json
with open('your log file') as log_file:
    for line in log_file:
        # rest of the code
My code prints different parts, but you can easy construct dictionary as you like.
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#5
Thank you for you code and clarifications, I appreciate, you have explained very nicely.

My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.
Reply
#6
(Feb-12-2019, 11:08 PM)DG1234 Wrote: My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.
@buran dos that.
message = json.loads(json_message)
After json.load() it's a Python dictionary.
>>> type(message)
<class 'dict'>

>>> message
{'sourceSysAcctNumber': '22222222222',
 'sourceSysCode': 'TP',
 'transactions': [{'AcctTypeCode': 'DD',
                   'BranchNumber': '000y',
                   'Desc': '02/05 FEDEXOFFICE 00 YYYYY',
                   'ProductCode': '104',
                   'TransCode': 'YYY'}]}

>>> message['sourceSysAcctNumber']
'22222222222'

# When coming from json is also normal to have list mixed into the dictionary
>>> message['transactions'][0]['ProductCode']
'104'
Reply
#7
(Feb-12-2019, 11:08 PM)DG1234 Wrote: My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.

I understand, but it's your task/requirements. We are glad to help, but we are not going to do your work for you. Normally we won't even start helping before we see some effort from you. Last night I made an exception and show you how to parse one line into meaningful blocks of data. Now you have to construct the dict in desired format. In any case you need to provide example output, i.e. how your dictionary looks like and what data to include, e.g. do you need the log id, log date, log time, do you need to combine the date and time, do you need to parse the message or just keep the whole message as one key:value pair... Do you need to parse the message into dict (using json) or just want to keep message as json str, If you parse the message, what will happen if there are more that one transaction, is it even possible, etc.
As a more general remark, my advice would be to research parts that are not clear. You may start with our tutorial on dicts. Then put some effort to complete the task and ask questions along the way. In long run it's better for you if you understand the task and how to accomplish it. If we give you ready solution it won't help you in the future when you get yet another task that you don't know how to complete. Not to mention that satisfaction for completing the task yourself will be priceless.
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply
#8
Smile Thank you so much Buran, I appreciate your time and help. Your code really help me a lot.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Help creating shell scrip for python file marciokoko 10 1,254 Sep-16-2023, 09:46 PM
Last Post: snippsat
  Form that puts diacritics on the words in the text Melcu54 13 1,382 Aug-22-2023, 07:07 AM
Last Post: Pedroski55
Thumbs Up Need to compare the Excel file name with a directory text file. veeran1991 1 1,062 Dec-15-2022, 04:32 PM
Last Post: Larz60+
  Creating csv files from Excel file azizrasul 40 5,326 Nov-03-2022, 08:33 PM
Last Post: azizrasul
  Modify values in XML file by data from text file (without parsing) Paqqno 2 1,575 Apr-13-2022, 06:02 AM
Last Post: Paqqno
Question How do I skipkeys on json file read to python dictionary? BrandonKastning 3 1,829 Mar-08-2022, 09:34 PM
Last Post: BrandonKastning
  trying to write a dictionary in a csv file CompleteNewb 13 6,379 Mar-04-2022, 04:43 AM
Last Post: deanhystad
  Converted Pipe Delimited text file to CSV file atomxkai 4 6,841 Feb-11-2022, 12:38 AM
Last Post: atomxkai
  Creating file with images BobSmoss 1 1,350 Jan-08-2022, 08:46 PM
Last Post: snippsat
  Class-Aggregation and creating a list/dictionary IoannisDem 1 1,882 Oct-03-2021, 05:16 PM
Last Post: Yoriz

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020