Text file read issues

[Complex_Potential]

Hello all

New to the forums, about six months experience with simple(ish) Python scripting.

I've got a bit of an issue that isn't with code specifically but seems to be to do with a set of .txt files. Basically, we have a BMS that outputs temperature readings line by line in the form of a simple .txt file and I want to read these lines and store them as a list in Python. The code for this is simple and works fine on every .txt file I've tested EXCEPT for these generated by the BMS.

On the surface the .txt output looks normal but returns blank lines when interrogated. If I copy/paste the lines into a fresh .txt file the data reads fine. The only difference I can see between the BMS generated .txt file and the copy/pasted version is the file size; the file size halves when the data is copy/pasted to a fresh file.

I'm sure there's something non standard about these BMS generated .txt files but I can't figure out what it is. Ideally I need someone more experienced to look at the files to see if they can spot what's different about them but I cannot attached files to this forum post.

Anyone seen something like this before? Any ideas on how I could get a sample of one of these .txt files across for examination?

Thanks

CP

[gontajones]

I don't know what is a BMS, but I have some points that you can investigate:

1 - What is the encoding of the BMS .txt file?
2 - Maybe when you are running your python script, the BMS file is being used by the BMS and python is getting nothing (exception).
3 - Post at least the file handling part of your code here (with tags) and use some lines of the BMS .txt file.

[buran]

maybe upload a sample txt file along with your code in python tags?

[Complex_Potential]

Thanks for the replies. I'll do my best to answer and add a bit more info. BMS stands for Building Management System btw.

Gontajones - Your comment about encoding got me on the right track. The encoding of the file is Unicode which seems to be causing the problem. When I do a save as on the file it defaults to ANSI and all is good. Unfortunately I'm not sure how to convert from Unicode to ANSI within Python.

For reference, the lines are supposed to look like this from the text file.

25 August 2017 09:15:00 23.19
25 August 2017 09:00:00 22.98
25 August 2017 08:45:00 22.76
25 August 2017 08:30:00 22.62
25 August 2017 08:15:00 22.41
25 August 2017 08:00:00 22.34

So... when working with the Unicode .txt file I've tried a few things:

with open("C:\\Users\\UKRCM003\\Desktop\\Sample datafiles\\170825  t05 L11O44S3.txt") as f:
    content = f.readlines()
for con in content:
    print(con)

This just spits out mostly blank lines with some weird characters on the first line

Output:
þ2

But, more interestingly, modifying the code to this:

with open("C:\\Users\\UKRCM003\\Desktop\\Sample datafiles\\170825  t05 L11O44S3.txt") as f:
    content = f.readlines()
for con in content:
    print(con.encode('cp1252'))

Results in this....

Output:
b'\xff\xfe2\x005\x00 \x00A\x00u\x00g\x00u\x00s\x00t\x00 \x002\x000\x001\x007\x00 \x000\x009\x00:\x001\x005\x00:\x000\x000\x00\t\x002\x003\x00.\x001\x009\x00\n'
b'\x00\n'
b'\x002\x005\x00 \x00A\x00u\x00g\x00u\x00s\x00t\x00 \x002\x000\x001\x007\x00 \x000\x009\x00:\x000\x000\x00:\x000\x000\x00\t\x002\x002\x00.\x009\x008\x00\n'
b'\x00\n'
b'\x002\x005\x00 \x00A\x00u\x00g\x00u\x00s\x00t\x00 \x002\x000\x001\x007\x00 \x000\x008\x00:\x004\x005\x00:\x000\x000\x00\t\x002\x002\x00.\x007\x006\x00\n'
b'\x00\n'
b'\x002\x005\x00 \x00A\x00u\x00g\x00u\x00s\x00t\x00 \x002\x000\x001\x007\x00 \x000\x008\x00:\x003\x000\x00:\x000\x000\x00\t\x002\x002\x00.\x006\x002\x00\n'
b'\x00\n'
b'\x002\x005\x00 \x00A\x00u\x00g\x00u\x00s\x00t\x00 \x002\x000\x001\x007\x00 \x000\x008\x00:\x001\x005\x00:\x000\x000\x00\t\x002\x002\x00.\x004\x001\x00\n'
b'\x00\n'

Still not what I need but at least it's seeing something.

Is this making more sense now?

[buran]

please, upload/attach real txt file as it comes from BMS. it looks like encoding issue.
what python version do you use?

[Complex_Potential]

Thanks buran

I'm using Python 3.5

How do I attach a file to these forums please? There does not appear to be an obvious option.

[buran]

How do I attach a file to these forums please? There does not appear to be an obvious option.

From the help -> How to attach a file

[Complex_Potential]

Thanks again. Unfortunately I dont seem to have the necessary permissions to attach files at the moment.

[snippsat]

One more post and you can upload or use eg beamit.

[Complex_Potential]

Here's me posting again....