Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Creating Dictionary form LOG /text file
#1
Hi All,
I am new to Python world and because of that I am not expert to do all the stuff. I am having some issue to convert below log file content to Dictionary format. Appreciate if someone from this forum please help me to do that. Thanks in Advance.

log file content like below. This log file will have several thousand record and each line will have more key.

002d2d62-4445-a06d.log:20190208 09:19:43.115 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"12234678901","transactions":[{"TransCode":"XXX","AcctTypeCode":"DD",,"BranchNumber":"000x","Desc":"02/05 FEDEXOFFICE 00 GGGG","ProductCode":"104"}]}
002d2d63-4445-a06d.log:20190208 09:19:52.538 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"22222222222","transactions":[{"TransCode":"YYY","AcctTypeCode":"DD","BranchNumber":"000y","Desc":"02/05 FEDEXOFFICE 00 YYYYY","ProductCode":"104"}]}
Quote
#2
(Feb-12-2019, 07:51 PM)DG1234 Wrote: I am having some issue to convert below log file content to Dictionary format.

what issues do you have?
you can split each line at :. and Message is JSON
note that you have double comma on first line in the message - between "DD" and "BranchNumber". I hope it's error because it invalidate the JSON
Quote
#3
Double comma is typo, that will not have.

Can you please give an example code, So I can use that to create Dictionary from this file.

I am very new and not familiar with all syntax and process. Please help me with a sample code.
Quote
#4
import json

log_file = ['002d2d62-4445-a06d.log:20190208 09:19:43.115 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"12234678901","transactions":[{"TransCode":"XXX","AcctTypeCode":"DD","BranchNumber":"000x","Desc":"02/05 FEDEXOFFICE 00 GGGG","ProductCode":"104"}]}',
'002d2d63-4445-a06d.log:20190208 09:19:52.538 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"22222222222","transactions":[{"TransCode":"YYY","AcctTypeCode":"DD","BranchNumber":"000y","Desc":"02/05 FEDEXOFFICE 00 YYYYY","ProductCode":"104"}]}']

for line in log_file:
    print('\n--- NEW LOG LINE ---\n')
    log, m, json_message = line.split(': ')
    log_id, log_time, *rest = log.split(' ')
    log_id, log_date = log_id.split(':')
    print(log_id)
    print(log_date)
    print(log_time)
    print(rest)
    print(m)
    message = json.loads(json_message)
    print(message["sourceSysCode"])
    print(message["sourceSysAcctNumber"])
    for transaction in message['transactions']:
        print(transaction)
Output:
--- NEW LOG LINE --- 002d2d62-4445-a06d.log 20190208 09:19:43.115 ['Debug', 'Generic_OnWorkComplete'] Message TP 12234678901 {'Desc': '02/05 FEDEXOFFICE 00 GGGG', 'BranchNumber': '000x', 'TransCode': 'XXX', 'ProductCode': '104', 'AcctTypeCode': 'DD'} --- NEW LOG LINE --- 002d2d63-4445-a06d.log 20190208 09:19:52.538 ['Debug', 'Generic_OnWorkComplete'] Message TP 22222222222 {'Desc': '02/05 FEDEXOFFICE 00 YYYYY', 'BranchNumber': '000y', 'TransCode': 'YYY', 'ProductCode': '104', 'AcctTypeCode': 'DD'}
log_file is just dummy. You need to open the log file and read file line by line, e.g.
import json
with open('your log file') as log_file:
    for line in log_file:
        # rest of the code
My code prints different parts, but you can easy construct dictionary as you like.
snippsat likes this post
Quote
#5
Thank you for you code and clarifications, I appreciate, you have explained very nicely.

My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.
Quote
#6
(Feb-12-2019, 11:08 PM)DG1234 Wrote: My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.
@buran dos that.
message = json.loads(json_message)
After json.load() it's a Python dictionary.
>>> type(message)
<class 'dict'>

>>> message
{'sourceSysAcctNumber': '22222222222',
 'sourceSysCode': 'TP',
 'transactions': [{'AcctTypeCode': 'DD',
                   'BranchNumber': '000y',
                   'Desc': '02/05 FEDEXOFFICE 00 YYYYY',
                   'ProductCode': '104',
                   'TransCode': 'YYY'}]}

>>> message['sourceSysAcctNumber']
'22222222222'

# When coming from json is also normal to have list mixed into the dictionary
>>> message['transactions'][0]['ProductCode']
'104'
Quote
#7
(Feb-12-2019, 11:08 PM)DG1234 Wrote: My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.

I understand, but it's your task/requirements. We are glad to help, but we are not going to do your work for you. Normally we won't even start helping before we see some effort from you. Last night I made an exception and show you how to parse one line into meaningful blocks of data. Now you have to construct the dict in desired format. In any case you need to provide example output, i.e. how your dictionary looks like and what data to include, e.g. do you need the log id, log date, log time, do you need to combine the date and time, do you need to parse the message or just keep the whole message as one key:value pair... Do you need to parse the message into dict (using json) or just want to keep message as json str, If you parse the message, what will happen if there are more that one transaction, is it even possible, etc.
As a more general remark, my advice would be to research parts that are not clear. You may start with our tutorial on dicts. Then put some effort to complete the task and ask questions along the way. In long run it's better for you if you understand the task and how to accomplish it. If we give you ready solution it won't help you in the future when you get yet another task that you don't know how to complete. Not to mention that satisfaction for completing the task yourself will be priceless.
Quote
#8
Smile Thank you so much Buran, I appreciate your time and help. Your code really help me a lot.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Trying to make column based file from text file scor1pion 4 60 2 hours ago
Last Post: ichabod801
  How best to format text in a simple text file? Pedroski55 2 363 Jun-14-2019, 06:07 AM
Last Post: noisefloor
  Analyzing large text file with nltk.corpus (stopwords ) Drone4four 9 716 Jun-06-2019, 09:30 PM
Last Post: Drone4four
  Need Help Typing Text into Tough Form [xpath / selenium] digitalmatic7 0 112 Jun-05-2019, 06:46 AM
Last Post: digitalmatic7
  Save console output to text file (like a log) koticphreak 9 355 Jun-02-2019, 07:45 PM
Last Post: snippsat
  write image into string format into text file venkat18 2 238 Jun-01-2019, 06:46 AM
Last Post: venkat18
  How to remove form feed from file gurbhej_singh 5 328 May-26-2019, 05:51 AM
Last Post: gurbhej_singh
  Print to Text file in Python 3 iteachpc 2 242 May-02-2019, 04:38 AM
Last Post: perfringo
  Convert List of Dictionary to dictionary of dictionary list in python kk230689 2 230 Apr-27-2019, 03:13 AM
Last Post: perfringo
  parsing complex text file anna 1 220 Apr-10-2019, 09:54 PM
Last Post: Larz60+

Forum Jump:


Users browsing this thread: 1 Guest(s)