Python Forum
Creating Dictionary form LOG /text file - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: General Coding Help (https://python-forum.io/forum-8.html)
+--- Thread: Creating Dictionary form LOG /text file (/thread-16053.html)



Creating Dictionary form LOG /text file - DG1234 - Feb-12-2019

Hi All,
I am new to Python world and because of that I am not expert to do all the stuff. I am having some issue to convert below log file content to Dictionary format. Appreciate if someone from this forum please help me to do that. Thanks in Advance.

log file content like below. This log file will have several thousand record and each line will have more key.

002d2d62-4445-a06d.log:20190208 09:19:43.115 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"12234678901","transactions":[{"TransCode":"XXX","AcctTypeCode":"DD",,"BranchNumber":"000x","Desc":"02/05 FEDEXOFFICE 00 GGGG","ProductCode":"104"}]}
002d2d63-4445-a06d.log:20190208 09:19:52.538 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"22222222222","transactions":[{"TransCode":"YYY","AcctTypeCode":"DD","BranchNumber":"000y","Desc":"02/05 FEDEXOFFICE 00 YYYYY","ProductCode":"104"}]}


RE: Creating Dictionary form LOG /text file - buran - Feb-12-2019

(Feb-12-2019, 07:51 PM)DG1234 Wrote: I am having some issue to convert below log file content to Dictionary format.

what issues do you have?
you can split each line at :. and Message is JSON
note that you have double comma on first line in the message - between "DD" and "BranchNumber". I hope it's error because it invalidate the JSON


RE: Creating Dictionary form LOG /text file - DG1234 - Feb-12-2019

Double comma is typo, that will not have.

Can you please give an example code, So I can use that to create Dictionary from this file.

I am very new and not familiar with all syntax and process. Please help me with a sample code.


RE: Creating Dictionary form LOG /text file - buran - Feb-12-2019

import json

log_file = ['002d2d62-4445-a06d.log:20190208 09:19:43.115 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"12234678901","transactions":[{"TransCode":"XXX","AcctTypeCode":"DD","BranchNumber":"000x","Desc":"02/05 FEDEXOFFICE 00 GGGG","ProductCode":"104"}]}',
'002d2d63-4445-a06d.log:20190208 09:19:52.538 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"22222222222","transactions":[{"TransCode":"YYY","AcctTypeCode":"DD","BranchNumber":"000y","Desc":"02/05 FEDEXOFFICE 00 YYYYY","ProductCode":"104"}]}']

for line in log_file:
    print('\n--- NEW LOG LINE ---\n')
    log, m, json_message = line.split(': ')
    log_id, log_time, *rest = log.split(' ')
    log_id, log_date = log_id.split(':')
    print(log_id)
    print(log_date)
    print(log_time)
    print(rest)
    print(m)
    message = json.loads(json_message)
    print(message["sourceSysCode"])
    print(message["sourceSysAcctNumber"])
    for transaction in message['transactions']:
        print(transaction)
Output:
--- NEW LOG LINE --- 002d2d62-4445-a06d.log 20190208 09:19:43.115 ['Debug', 'Generic_OnWorkComplete'] Message TP 12234678901 {'Desc': '02/05 FEDEXOFFICE 00 GGGG', 'BranchNumber': '000x', 'TransCode': 'XXX', 'ProductCode': '104', 'AcctTypeCode': 'DD'} --- NEW LOG LINE --- 002d2d63-4445-a06d.log 20190208 09:19:52.538 ['Debug', 'Generic_OnWorkComplete'] Message TP 22222222222 {'Desc': '02/05 FEDEXOFFICE 00 YYYYY', 'BranchNumber': '000y', 'TransCode': 'YYY', 'ProductCode': '104', 'AcctTypeCode': 'DD'}
log_file is just dummy. You need to open the log file and read file line by line, e.g.
import json
with open('your log file') as log_file:
    for line in log_file:
        # rest of the code
My code prints different parts, but you can easy construct dictionary as you like.


RE: Creating Dictionary form LOG /text file - DG1234 - Feb-12-2019

Thank you for you code and clarifications, I appreciate, you have explained very nicely.

My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.


RE: Creating Dictionary form LOG /text file - snippsat - Feb-12-2019

(Feb-12-2019, 11:08 PM)DG1234 Wrote: My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.
@buran dos that.
message = json.loads(json_message)
After json.load() it's a Python dictionary.
>>> type(message)
<class 'dict'>

>>> message
{'sourceSysAcctNumber': '22222222222',
 'sourceSysCode': 'TP',
 'transactions': [{'AcctTypeCode': 'DD',
                   'BranchNumber': '000y',
                   'Desc': '02/05 FEDEXOFFICE 00 YYYYY',
                   'ProductCode': '104',
                   'TransCode': 'YYY'}]}

>>> message['sourceSysAcctNumber']
'22222222222'

# When coming from json is also normal to have list mixed into the dictionary
>>> message['transactions'][0]['ProductCode']
'104'



RE: Creating Dictionary form LOG /text file - buran - Feb-13-2019

(Feb-12-2019, 11:08 PM)DG1234 Wrote: My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.

I understand, but it's your task/requirements. We are glad to help, but we are not going to do your work for you. Normally we won't even start helping before we see some effort from you. Last night I made an exception and show you how to parse one line into meaningful blocks of data. Now you have to construct the dict in desired format. In any case you need to provide example output, i.e. how your dictionary looks like and what data to include, e.g. do you need the log id, log date, log time, do you need to combine the date and time, do you need to parse the message or just keep the whole message as one key:value pair... Do you need to parse the message into dict (using json) or just want to keep message as json str, If you parse the message, what will happen if there are more that one transaction, is it even possible, etc.
As a more general remark, my advice would be to research parts that are not clear. You may start with our tutorial on dicts. Then put some effort to complete the task and ask questions along the way. In long run it's better for you if you understand the task and how to accomplish it. If we give you ready solution it won't help you in the future when you get yet another task that you don't know how to complete. Not to mention that satisfaction for completing the task yourself will be priceless.


RE: Creating Dictionary form LOG /text file - DG1234 - Feb-13-2019

Smile Thank you so much Buran, I appreciate your time and help. Your code really help me a lot.