Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Creating Dictionary form LOG /text file
#1
Hi All,
I am new to Python world and because of that I am not expert to do all the stuff. I am having some issue to convert below log file content to Dictionary format. Appreciate if someone from this forum please help me to do that. Thanks in Advance.

log file content like below. This log file will have several thousand record and each line will have more key.

002d2d62-4445-a06d.log:20190208 09:19:43.115 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"12234678901","transactions":[{"TransCode":"XXX","AcctTypeCode":"DD",,"BranchNumber":"000x","Desc":"02/05 FEDEXOFFICE 00 GGGG","ProductCode":"104"}]}
002d2d63-4445-a06d.log:20190208 09:19:52.538 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"22222222222","transactions":[{"TransCode":"YYY","AcctTypeCode":"DD","BranchNumber":"000y","Desc":"02/05 FEDEXOFFICE 00 YYYYY","ProductCode":"104"}]}
Quote
#2
(Feb-12-2019, 07:51 PM)DG1234 Wrote: I am having some issue to convert below log file content to Dictionary format.

what issues do you have?
you can split each line at :. and Message is JSON
note that you have double comma on first line in the message - between "DD" and "BranchNumber". I hope it's error because it invalidate the JSON
Quote
#3
Double comma is typo, that will not have.

Can you please give an example code, So I can use that to create Dictionary from this file.

I am very new and not familiar with all syntax and process. Please help me with a sample code.
Quote
#4
import json

log_file = ['002d2d62-4445-a06d.log:20190208 09:19:43.115 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"12234678901","transactions":[{"TransCode":"XXX","AcctTypeCode":"DD","BranchNumber":"000x","Desc":"02/05 FEDEXOFFICE 00 GGGG","ProductCode":"104"}]}',
'002d2d63-4445-a06d.log:20190208 09:19:52.538 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"22222222222","transactions":[{"TransCode":"YYY","AcctTypeCode":"DD","BranchNumber":"000y","Desc":"02/05 FEDEXOFFICE 00 YYYYY","ProductCode":"104"}]}']

for line in log_file:
    print('\n--- NEW LOG LINE ---\n')
    log, m, json_message = line.split(': ')
    log_id, log_time, *rest = log.split(' ')
    log_id, log_date = log_id.split(':')
    print(log_id)
    print(log_date)
    print(log_time)
    print(rest)
    print(m)
    message = json.loads(json_message)
    print(message["sourceSysCode"])
    print(message["sourceSysAcctNumber"])
    for transaction in message['transactions']:
        print(transaction)
Output:
--- NEW LOG LINE --- 002d2d62-4445-a06d.log 20190208 09:19:43.115 ['Debug', 'Generic_OnWorkComplete'] Message TP 12234678901 {'Desc': '02/05 FEDEXOFFICE 00 GGGG', 'BranchNumber': '000x', 'TransCode': 'XXX', 'ProductCode': '104', 'AcctTypeCode': 'DD'} --- NEW LOG LINE --- 002d2d63-4445-a06d.log 20190208 09:19:52.538 ['Debug', 'Generic_OnWorkComplete'] Message TP 22222222222 {'Desc': '02/05 FEDEXOFFICE 00 YYYYY', 'BranchNumber': '000y', 'TransCode': 'YYY', 'ProductCode': '104', 'AcctTypeCode': 'DD'}
log_file is just dummy. You need to open the log file and read file line by line, e.g.
import json
with open('your log file') as log_file:
    for line in log_file:
        # rest of the code
My code prints different parts, but you can easy construct dictionary as you like.
snippsat likes this post
Quote
#5
Thank you for you code and clarifications, I appreciate, you have explained very nicely.

My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.
Quote
#6
(Feb-12-2019, 11:08 PM)DG1234 Wrote: My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.
@buran dos that.
message = json.loads(json_message)
After json.load() it's a Python dictionary.
>>> type(message)
<class 'dict'>

>>> message
{'sourceSysAcctNumber': '22222222222',
 'sourceSysCode': 'TP',
 'transactions': [{'AcctTypeCode': 'DD',
                   'BranchNumber': '000y',
                   'Desc': '02/05 FEDEXOFFICE 00 YYYYY',
                   'ProductCode': '104',
                   'TransCode': 'YYY'}]}

>>> message['sourceSysAcctNumber']
'22222222222'

# When coming from json is also normal to have list mixed into the dictionary
>>> message['transactions'][0]['ProductCode']
'104'
Quote
#7
(Feb-12-2019, 11:08 PM)DG1234 Wrote: My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.

I understand, but it's your task/requirements. We are glad to help, but we are not going to do your work for you. Normally we won't even start helping before we see some effort from you. Last night I made an exception and show you how to parse one line into meaningful blocks of data. Now you have to construct the dict in desired format. In any case you need to provide example output, i.e. how your dictionary looks like and what data to include, e.g. do you need the log id, log date, log time, do you need to combine the date and time, do you need to parse the message or just keep the whole message as one key:value pair... Do you need to parse the message into dict (using json) or just want to keep message as json str, If you parse the message, what will happen if there are more that one transaction, is it even possible, etc.
As a more general remark, my advice would be to research parts that are not clear. You may start with our tutorial on dicts. Then put some effort to complete the task and ask questions along the way. In long run it's better for you if you understand the task and how to accomplish it. If we give you ready solution it won't help you in the future when you get yet another task that you don't know how to complete. Not to mention that satisfaction for completing the task yourself will be priceless.
Quote
#8
Smile Thank you so much Buran, I appreciate your time and help. Your code really help me a lot.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  parsing complex text file anna 1 91 Apr-10-2019, 09:54 PM
Last Post: Larz60+
  Putting text on images from excel/csv file sam77am 1 79 Apr-05-2019, 07:36 AM
Last Post: Larz60+
  eric6 IDE no output into text file daveq 5 158 Mar-22-2019, 11:31 PM
Last Post: daveq
  filtering and summing data in a text file apexman 3 115 Mar-18-2019, 09:25 PM
Last Post: ichabod801
  Getting the maximum point of the graph from the text file Erfans 2 153 Mar-09-2019, 11:01 PM
Last Post: Erfans
  Looking for simple help - text file stripping DerekK 1 103 Mar-08-2019, 10:06 AM
Last Post: Larz60+
  problems with writting a text file sunny 6 246 Feb-24-2019, 10:52 PM
Last Post: Larz60+
  write each line of a text file into separate text files and save with different names Shameendra 3 147 Feb-20-2019, 09:07 AM
Last Post: buran
  reading a file int a dictionary specifically sonicx05 3 131 Feb-18-2019, 05:15 PM
Last Post: woooee
  Search for a text in a html file ebolisa 4 219 Feb-09-2019, 10:46 PM
Last Post: ebolisa

Forum Jump:


Users browsing this thread: 1 Guest(s)