i=open(ifn)
for line in i:
    ...
    ...

i = open(ifn, errors='replace') # this will replace the character with '?' for example

(Oct-23-2018, 07:37 AM)wavic Wrote:

i = open(ifn, errors='replace') # this will replace the character with '?' for example

More here: https://docs.python.org/3/library/functions.html#open

---

You may like 'backslashreplace'. I presume

(Oct-23-2018, 08:10 AM)Larz60+ Wrote: either replace as wavic suggests, or use proper codec

You can use: https://github.com/chardet/chardet
to detect (most of the time) the proper file codec

Quote:i just want to skip the lines that are not valid UTF-8.

i = open(ifn, encoding="utf-8", errors="ignore")

class LineSkipper:
    def __init__(self, iterable):
        self.iterable = iter(iterable)

    def __iter__(self):
        return self

    def __next__(self):
        while True:
            try:
                return next(self.iterable)
            except StopIteration:
                # re-raise the stopiteration, so the caller knows we've reached the end of the iterable
                raise
            except:
                # ignore any errors reading the line and skip it entirely
                pass

with open("spam.txt") as f:
    for line in LineSkipper(f):
        print(f"{line.strip()}")