File transfer with xmodem CRC (1024) checksum

[Jhonbovi]

Thanks for reading my post.

Problem;
Cannot complete to receive a binary file with xmodem CRC (1024) mode.

Description;
Currently I try to download/receive a binary data file from a device (sender) via RS-232C to Raspberry Pi (receiver).
Typical filesize is 170 kByte.
Checksum type of the sender is XMODEM CRC(1024).
Therefore, I coded a python script refering sample codes on the internet as following:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

import serial import time import logging from xmodem import XMODEM1k, NAK from time import sleep logging.basicConfig(level=logging.DEBUG)   def readUntil(char = None):     def serialPortReader():         while True:             tmp = port.read(1)             if not tmp or (char and char == tmp):                 break             yield tmp     return ''.join(serialPortReader())           def getc(size, timeout=1):     return port.read(size)   def putc(data, timeout=1):     port.write(data)     sleep(0.001)   port = serial.Serial(port='/dev/ttyUSB0',parity=serial.PARITY_NONE,bytesize=serial.EIGHTBITS,stopbits=serial.STOPBITS_ONE,timeout=0,xonxoff=0,rtscts=0,dsrdtr=0,baudrate=115200)   port.write(">\r\n") time.sleep(2) port.write("dat\r\n") time.sleep(0.02) readUntil(NAK) buffer = open('test', 'wb') print XMODEM1k(getc, putc).recv(buffer,crc_mode = 1,quiet = 1,timeout = 300) readUntil() port.write("end\r\n") port.close()

There are some procedures to communicate with the sender;
1) In order to start communicate with the sender, input ">\r\n" is required.
2) after that, command "dat\r\n" is required to transfer the binary
data file from the sender.
3) Finally, command "end\r\n" to quit communication.

By executing the above script, only the first part of the binary file seems to be transfered
and the file seize is 1024 Byte which is quite equivalent to one data block size.

While executing, some error and warning messages are described in a logging.DEBUG output as following:

Error:
DEBUG:xmodem.XMODEM:recv error: putc failed, sleeping for 1
DEBUG:xmodem.XMODEM:recv: SOH
DEBUG:xmodem.XMODEM:recv: data block 1
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
WARNING:xmodem.XMODEM:recv error: expected SOH, EOT; got ''
INFO:xmodem.XMODEM:error_count reached 16, aborting.
None

Could anybody give me some idea or comments to solve this problem?

[Larz60+]

which version of xmodem are you using, should be 3.4.5
There was an issue with NAK processing (packed should be resent) that was corrected in version 0.4.1 (way back in 2015), see:
https://github.com/tehmaze/xmodem/pull/12

It's been a while since I have worked with half-duplex transfer protocol, at the packet level, but I question your code
'readUntil(NAK)', especially after the manual delay. Seems that the packet should be resentand that the delay should be handled by the hardware.
(keeping in mind I am very rusty on this).

the suggested method is:

1 2 3 4 5 6 7 8 9 10

>>> import serial >>> from xmodem import XMODEM >>> ser = serial.Serial('/dev/ttyUSB0', timeout=0) # or whatever port you need >>> def getc(size, timeout=1): ...     return ser.read(size) or None ... >>> def putc(data, timeout=1): ...     return ser.write(data)  # note that this ignores the timeout ... >>> modem = XMODEM(getc, putc)

see: https://pypi.org/project/xmodem/

is there any valid reason why you are using python 2.7?
If possible you should upgrade to 3.7.1.
2.7 is at the end of it's life cycle and soon will no longer be supported.

One final question, it looks to me like the error you post is not verbatim (but that could just be antique python, can't remember 2.7 error meggases). If not, please post entire unmodified error traceback.

[Jhonbovi]

Dear Larz60+,

tahnks for your quick and important suggestion.
Now, file transfer issue has been completely solved.

As I posted before, debuginfo suggested that sleeping time is changed to 1 s while processing.
Thus finally, sleep(0.001) was modified to sleep(1) in the putc definition section.
The modified code is as follows:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

import serial import time   from xmodem import XMODEM1k, NAK from time import sleep   def readUntil(char = None):     def serialPortReader():         while True:             tmp = port.read(1)             if not tmp or (char and char == tmp):                 break             yield tmp     return ''.join(serialPortReader())           def getc(size, timeout=1):     return port.read(size)   def putc(data, timeout=1):     port.write(data)     sleep(1)   port = serial.Serial(port='/dev/ttyUSB0',parity=serial.PARITY_NONE,bytesize=serial.EIGHTBITS,stopbits=serial.STOPBITS_ONE,timeout=0,xonxoff=0,rtscts=0,dsrdtr=0,baudrate=115200)   port.write(">\r\n") time.sleep(2) port.write("dat\r\n") time.sleep(1) readUntil(NAK) buffer = open('data', 'wb') XMODEM1k(getc, putc).recv(buffer,crc_mode = 1) readUntil() port.write("end\r\n") port.close()

And let me answer your questions.
1) xmodem version
The version of xmodem is 0.4.5 which was downloaded from:
https://pypi.org/project/xmodem/
I guess it suppose to be the latest version.

2) Reason to use python2.7
As you may already notice, I have just started to use python.
Python2.7 seems to be pre-installed version for both RaspberryPi and Fedora OS which I usually use.
Thus I recognized that python2.x series is major stream instead of python3.x series.

3) Error traceback
I posted debuginfo which was created by Logger module
instead of standard output on a terminal just in order to provide details including system information.
Now, the problem was sloved but I will attach verbatim error traceback in my next new post.

Again, I would appreciate your kind support.

Best regards.

[Larz60+]

Your quote:
Thus I recognized that python2.x series is major stream instead of python3.x series.

From python.org
The End Of Life date (EOL, sunset date) for Python 2.7 has been moved five years into the future, to 2020.

Just an FYI - Python 2.7 is nearing it's EOL.

Glad to hear your serial link is working now.