I would see if convert files to utf-8 eg online Convert files to UTF-8.
Chardet will detect/guess on encoding.
Chardet will detect/guess on encoding.
G:\div_code\file_test λ chardetect file_1.txt file_1.txt: ascii with confidence 1.0 # The file i test in code G:\div_code\file_test λ chardetect file_2.txt file_2.txt: utf-8 with confidence 0.99 G:\div_code\file_test λ chardetect file_le.txt file_le.txt: UTF-16 with confidence 1.0An option is to read file with
try: except
if no error will use file,if decode error(UnicodeDecodeError
) will go on and try next encoding.try: with open("file_2.txt", encoding='utf-16-le') as fp: content = fp.read() print(content) except Exception as error: print(f'{error}\n') try: with open("file_2.txt", encoding='cp1252') as fp: content = fp.read() print(content) except Exception as error: print(f'{error}\n') try: with open("file_2.txt", encoding='utf-8', errors='ignore') as fp: content = fp.read() print(content) except Exception as error: print(f'{error}\n')
Output:'utf-16-le' codec can't decode byte 0xa9 in position 46: truncated data
'charmap' codec can't decode byte 0x8d in position 10: character maps to <undefined>
hello
楍牣獯景
�����
résumé