Python Forum
Decoding log files in binary using an XML file.
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Decoding log files in binary using an XML file.
#1
Good day.

I am attempting to create a new log decoder based on an existing program, the message data is compared against an XML file and binary strings are translated in to human readable format.

below is an example of one of the messages:
Output:
[01-04-2019 13:16:18.209] MESSAGE-RX [DSE->ATO] ID=102 SEQNUM=0000173551 TIME=1554124577 SRC=01 DST=00 LEN=040 VALID=1 CRC=6CC41467 MESSAGE=6600280002A5EF00010008000000005CA20F21000100005CA20F210000050D00000000006714C46C
each line corresponds to one message, below is an example of the XML file that has the protocol that the log explorer uses to decode each message:

Output:
<message id="5" minlen="30" maxlen="30" name="hel locomotive command" msgType="request"> <header> <item name="sequenceNum" len="32" /> <item name="timestamp" len="32" /> <item name="version" len="16" /> <item name="sourceId" len="16" /> <item name="destinationId" len="16" /> <item name="length" len="16" /> </header> <payload> <field><item name="messageId" len="8" descriptionLink="ATO_MESSAGE_ID" ><values><value>5</value></values></item> </field> <field><item name="targetLocmPosition" len="8" descriptionLink="LOCOMOTIVE_POSITION" ><values><range from="0" to="16" /></values></item> </field> <field name="locmCommands"> <field><item name="applyEngineRunIsolate" len="8" descriptionLink="LOCOMOTIVE_RUN_ISOLATED" ><values><range from="0" to="3" /></values></item> </field> <field><item name="applyShortHoodRedLights" len="8" descriptionLink="COMMAND_ON_OFF" ><values><range from="0" to="1" /></values></item> </field> <field><item name="applyLongHoodRedLights" len="8" descriptionLink="COMMAND_ON_OFF" ><values><range from="0" to="1" /></values></item> </field> <field><item name="applySafeToBoardLights" len="8" descriptionLink="COMMAND_ON_OFF" ><values><range from="0" to="1" /></values></item> </field> <field><item name="applySafeToAlightLights" len="8" descriptionLink="COMMAND_ON_OFF" ><values><range from="0" to="1" /></values></item> </field> <field><item name="applyBell" len="8" descriptionLink="COMMAND_ON_OFF" ><values><range from="0" to="1" /></values></item> </field> <field><item name="applyHorn" len="8" descriptionLink="COMMAND_ON_OFF" ><values><range from="0" to="1" /></values></item> </field> <field><item name="cmmBypass" len="8" descriptionLink="COMMAND_ON_OFF" ><values><range from="0" to="1" /></values></item> </field> </field> </payload> <integrity poly="0x1EDC6F41" init="0x00000000" refin="false" refout="false" xored="0x00000000"> <item name="crc" len="32" /> </integrity> </message>
Does anyone know of any libraries that I can use to make this process easier? id prefer to not hard code anything in the translation part because I dont want my work to become obsolete when a new protocol is released.

Thank you
Reply
#2
I found a library that might fit the bill, PyXML or easeXML seem like they might be able to achieve what I am looking for.

looking at the input and the structure of the message and the structure of the xml that decodes it the lengths quoted dont seem to match up:

Message:
Output:
[01-04-2019 13:16:18.209] MESSAGE-RX [DSE->ATO] ID=102 SEQNUM=0000173551 TIME=1554124577 SRC=01 DST=00 LEN=040 VALID=1 CRC=6CC41467 MESSAGE=6600280002A5EF00010008000000005CA20F21000100005CA20F210000050D00000000006714C46C
XML:
Output:
<message id="102" minlen="40" maxlen="40" name="dse internal status" msgType="request"> <payload> <field><item name="messageId" len="8" descriptionLink="DSE_MESSAGE_ID" ><values><value>102</value></values></item> </field> <field name="dseHeader"> <field><item name="length" len="16" /> </field> <field><item name="sequenceNumber" len="32" /> </field> <field><item name="destinationId" len="8" descriptionLink="DEVICE_ID" ><values><range from="0" to="3" /></values></item> </field> <field><item name="sourceId" len="8" descriptionLink="DEVICE_ID" ><values><range from="0" to="3" /></values></item> </field> <field><item name="ack" len="8" descriptionLink="BOOLEAN" ><values><range from="0" to="1" /></values></item> </field> <field><item name="protocolVersion" len="8" /> </field> <field><item name="debug" len="32" /> </field> <field><item name="timestamp" len="32" /> </field> </field> <field><item name="currentDseOperativeMode" len="8" descriptionLink="DSE_MODE" ><values><range from="0" to="2" /></values></item> </field> <field><item name="role" len="8" descriptionLink="ROLE" ><values><range from="0" to="1" /></values></item> </field> <field><item name="isInsideStopWindow" len="8" descriptionLink="BOOLEAN" ><values><range from="0" to="1" /></values></item> </field> <field><item name="liftingStrategyStatus" len="8" descriptionLink="LIFTING_STRATEGY_STATUS" ><values><range from="0" to="2" /></values></item> </field> <field><item name="dseInternalUtc" len="32" /> </field> <field><item name="dseFault" len="8" descriptionLink="DSE_FAULT" ><values><range from="0" to="8" /></values></item> </field> <field name="dseSwVersion"> <field><item name="unused" len="8" /> </field> <field><item name="majorVersion" len="8" /> </field> <field><item name="minorVersion" len="8" /> </field> <field><item name="buildVersion" len="8" /> </field> </field> <field name="dseTrackDbVersion"> <field><item name="unused" len="8" /> </field> <field><item name="majorVersion" len="8" /> </field> <field><item name="minorVersion" len="8" /> </field> <field><item name="buildVersion" len="8" /> </field> </field> </payload> <integrity poly="0x4A503DF1" init="0x00000000" refin="false" refout="false" xored="0x00000000"> <item name="crc" len="32" /> </integrity> </message>
I feel like im missing something obvious here but the message is 80 bytes and the length is listed as 40(DEC) or 64(HEX)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Hashing an address for binary file Python_help 8 1,632 Nov-04-2021, 06:23 AM
Last Post: ndc85430
  flask app decoding problem mesbah 0 1,466 Aug-01-2021, 08:32 PM
Last Post: mesbah
  Decoding a serial stream AKGentile1963 7 4,857 Mar-20-2021, 08:07 PM
Last Post: deanhystad
  Read/Write binary file deanhystad 3 2,152 Feb-01-2021, 10:29 AM
Last Post: Larz60+
  Binary File Error SalsaBeanDip 1 1,205 Oct-12-2020, 09:13 PM
Last Post: bowlofred
  Convert file of hex strings to binary file medatib531 4 9,303 Oct-09-2020, 05:42 PM
Last Post: DeaD_EyE
  Binary File Read Aussie 6 5,440 Sep-03-2020, 03:57 AM
Last Post: deanhystad
  Binary file Aussie 12 2,956 Aug-31-2020, 09:20 PM
Last Post: Aussie
  python read binary file Pyguys 4 2,438 Jul-13-2020, 02:34 AM
Last Post: Pyguys
  Failure in writing binary text to file Gigux 7 2,697 Jul-04-2020, 08:41 AM
Last Post: Gigux

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020