Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Creating Dictionary form LOG /text file
#1
Hi All,
I am new to Python world and because of that I am not expert to do all the stuff. I am having some issue to convert below log file content to Dictionary format. Appreciate if someone from this forum please help me to do that. Thanks in Advance.

log file content like below. This log file will have several thousand record and each line will have more key.

002d2d62-4445-a06d.log:20190208 09:19:43.115 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"12234678901","transactions":[{"TransCode":"XXX","AcctTypeCode":"DD",,"BranchNumber":"000x","Desc":"02/05 FEDEXOFFICE 00 GGGG","ProductCode":"104"}]}
002d2d63-4445-a06d.log:20190208 09:19:52.538 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"22222222222","transactions":[{"TransCode":"YYY","AcctTypeCode":"DD","BranchNumber":"000y","Desc":"02/05 FEDEXOFFICE 00 YYYYY","ProductCode":"104"}]}
Quote
#2
(Feb-12-2019, 07:51 PM)DG1234 Wrote: I am having some issue to convert below log file content to Dictionary format.

what issues do you have?
you can split each line at :. and Message is JSON
note that you have double comma on first line in the message - between "DD" and "BranchNumber". I hope it's error because it invalidate the JSON
Quote
#3
Double comma is typo, that will not have.

Can you please give an example code, So I can use that to create Dictionary from this file.

I am very new and not familiar with all syntax and process. Please help me with a sample code.
Quote
#4
import json

log_file = ['002d2d62-4445-a06d.log:20190208 09:19:43.115 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"12234678901","transactions":[{"TransCode":"XXX","AcctTypeCode":"DD","BranchNumber":"000x","Desc":"02/05 FEDEXOFFICE 00 GGGG","ProductCode":"104"}]}',
'002d2d63-4445-a06d.log:20190208 09:19:52.538 Debug Generic_OnWorkComplete: Message: {"sourceSysCode":"TP","sourceSysAcctNumber":"22222222222","transactions":[{"TransCode":"YYY","AcctTypeCode":"DD","BranchNumber":"000y","Desc":"02/05 FEDEXOFFICE 00 YYYYY","ProductCode":"104"}]}']

for line in log_file:
    print('\n--- NEW LOG LINE ---\n')
    log, m, json_message = line.split(': ')
    log_id, log_time, *rest = log.split(' ')
    log_id, log_date = log_id.split(':')
    print(log_id)
    print(log_date)
    print(log_time)
    print(rest)
    print(m)
    message = json.loads(json_message)
    print(message["sourceSysCode"])
    print(message["sourceSysAcctNumber"])
    for transaction in message['transactions']:
        print(transaction)
Output:
--- NEW LOG LINE --- 002d2d62-4445-a06d.log 20190208 09:19:43.115 ['Debug', 'Generic_OnWorkComplete'] Message TP 12234678901 {'Desc': '02/05 FEDEXOFFICE 00 GGGG', 'BranchNumber': '000x', 'TransCode': 'XXX', 'ProductCode': '104', 'AcctTypeCode': 'DD'} --- NEW LOG LINE --- 002d2d63-4445-a06d.log 20190208 09:19:52.538 ['Debug', 'Generic_OnWorkComplete'] Message TP 22222222222 {'Desc': '02/05 FEDEXOFFICE 00 YYYYY', 'BranchNumber': '000y', 'TransCode': 'YYY', 'ProductCode': '104', 'AcctTypeCode': 'DD'}
log_file is just dummy. You need to open the log file and read file line by line, e.g.
import json
with open('your log file') as log_file:
    for line in log_file:
        # rest of the code
My code prints different parts, but you can easy construct dictionary as you like.
snippsat likes this post
Quote
#5
Thank you for you code and clarifications, I appreciate, you have explained very nicely.

My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.
Quote
#6
(Feb-12-2019, 11:08 PM)DG1234 Wrote: My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.
@buran dos that.
message = json.loads(json_message)
After json.load() it's a Python dictionary.
>>> type(message)
<class 'dict'>

>>> message
{'sourceSysAcctNumber': '22222222222',
 'sourceSysCode': 'TP',
 'transactions': [{'AcctTypeCode': 'DD',
                   'BranchNumber': '000y',
                   'Desc': '02/05 FEDEXOFFICE 00 YYYYY',
                   'ProductCode': '104',
                   'TransCode': 'YYY'}]}

>>> message['sourceSysAcctNumber']
'22222222222'

# When coming from json is also normal to have list mixed into the dictionary
>>> message['transactions'][0]['ProductCode']
'104'
Quote
#7
(Feb-12-2019, 11:08 PM)DG1234 Wrote: My requirements how to create DICTIONARY FROM THAT TEXT/LOG FILE. We have to create data in a Dictionary form from the file.

I understand, but it's your task/requirements. We are glad to help, but we are not going to do your work for you. Normally we won't even start helping before we see some effort from you. Last night I made an exception and show you how to parse one line into meaningful blocks of data. Now you have to construct the dict in desired format. In any case you need to provide example output, i.e. how your dictionary looks like and what data to include, e.g. do you need the log id, log date, log time, do you need to combine the date and time, do you need to parse the message or just keep the whole message as one key:value pair... Do you need to parse the message into dict (using json) or just want to keep message as json str, If you parse the message, what will happen if there are more that one transaction, is it even possible, etc.
As a more general remark, my advice would be to research parts that are not clear. You may start with our tutorial on dicts. Then put some effort to complete the task and ask questions along the way. In long run it's better for you if you understand the task and how to accomplish it. If we give you ready solution it won't help you in the future when you get yet another task that you don't know how to complete. Not to mention that satisfaction for completing the task yourself will be priceless.
Quote
#8
Smile Thank you so much Buran, I appreciate your time and help. Your code really help me a lot.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  reading a file int a dictionary specifically sonicx05 3 53 Feb-18-2019, 05:15 PM
Last Post: woooee
  Search for a text in a html file ebolisa 4 133 Feb-09-2019, 10:46 PM
Last Post: ebolisa
  Add stdout to text file maxtimbo 3 148 Feb-05-2019, 12:53 AM
Last Post: maxtimbo
  error creating new object after loading pickled objects from file arogers 2 150 Feb-02-2019, 10:43 AM
Last Post: Larz60+
  get the number in the line in text file lateublegende 2 165 Jan-29-2019, 06:03 PM
Last Post: lateublegende
  Invalid Identifier In Long Text File asilverg 1 86 Jan-25-2019, 11:59 PM
Last Post: ichabod801
  Saving python cmd output into a text file with colours kapilan15 2 108 Jan-25-2019, 06:25 PM
Last Post: metulburr
  Extending my text file word count ranker and calculator Drone4four 8 416 Jan-25-2019, 08:25 AM
Last Post: steve_shambles
  modified file text getting cut off droidus 3 171 Jan-22-2019, 01:55 PM
Last Post: droidus
  How to only extract upper or lower triangular matrix into tabular form SriRajesh 3 280 Jan-04-2019, 01:21 AM
Last Post: scidam

Forum Jump:


Users browsing this thread: 1 Guest(s)