Junk values

[neethuvp]

HI,
i'm reading some values using serial from one energy device... in my out data i'm getting some junk values like
[C]*~ '� � @�ظ��
15:49:50 ??. 0 II

i want to remove the junk values and need only the original data from device.. how to remove the junk values?

[Gribouillis]

You can remove junk values if you're able to define which part of the data is made of junk

>>> "This string contains junk values".replace("junk", "")
'This string contains  values'

For example 15:49:00 doesn't look like junk in your output. The much better solution is to understand why the device sends these values and if they have a meaning, but that may be unreachable?

[neethuvp]

cannot define the exact position of the junk values.. then how can i use it?

[Gribouillis]

How do you know these are junk values? It looks like a copyright notice to me, something like [C][email protected] 15:49:00 bacon but it may be encoded in a exotic codec.

[perfringo]

These are not 'junk values', these are 'real values' replaced by unicode special

The replacement character � (often a black diamond with a white question mark or an empty square box) is a symbol found in the Unicode standard at code point U+FFFD in the Specials table. It is used to indicate problems when a system is unable to render a stream of data to a correct symbol. It is usually seen when the data is invalid and does not match any character:

Consider a text file containing the German word "für" in the ISO-8859-1 encoding (0x66 0xFC 0x72). This file is now opened with a text editor that assumes the input is UTF-8. The first and last byte are valid UTF-8 encodings of ASCII, but the middle byte (0xFC) is not a valid byte in UTF-8. Therefore, a text editor could replace this byte with the replacement character symbol to produce a valid string of Unicode code points. The whole string now displays like this: "f�r".

[neethuvp]

how to remove these Unicode specials in data?

[perfringo]

how to remove these Unicode specials in data?

From wikipedia article: "Since the replacement is the same for all errors this makes it impossible to recover the original character."

Somehow you need to get original data with original characters and decode appropriately.

I don't see lot of data remaining if you remove specials and some other chars (ظ, ? etc)