all i want to do is count the lines in each file

[Skaperen]

all i want to do is count the lines in each file but there are strange binary bytes or older ISO codes.

Output:
Traceback (most recent call last):
  File "last-edited.py", line 38, in <module>
    pf(t,n)
  File "last-edited.py", line 7, in pf
    c = len([x for x in f])
  File "last-edited.py", line 7, in <listcomp>
    c = len([x for x in f])
  File "/usr/host/bin/../../lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 93: invalid start byte

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

from sfc import * def pf(t,n):     if os.path.exists(n):         with open(n) as f:             c = len([x for x in f])         print(t,str(c).rjust(8),n) fn = '.edit_log' argv.pop(0) cd() # be in home directory with open(fn) as el:     nt = {}     for ee in el:         t,e,n = ee.strip().split()[:3]         if n in nt:             nt[n].append(t)         else:             nt[n] = [t] ns = sorted(nt.keys()) tn = {} for n in ns:     nt[n].sort() for n in ns:     t = nt[n][-1]     if t[13]=='-':         t=t[0:7]+t[8:13]+t[14:16]+t[17:]     tn[t] = n for t,n in sorted([x for x in sorted(tn.items())]):     if argv:         for a in argv:             if n.endswith(a):                 if os.path.exists(n):                     pf(t,n)                 break     else: # none requested so print all         if os.path.exists(n):             pf(t,n)

i think the only things it uses from sfc are os and cd (called with no args changes directory to user home directory.

[Larz60+]

you need to know the file encoding.
Though not foolproof, you can usually find it with chardet: https://pypi.org/project/chardet/

[Skaperen]

maybe it will be simpler to read most of the file in binary and count the b'\n' in the bytes string i get. really big files i don't need the exact number, just a general size.

[Larz60+]

There are some strange encodings, and the easiest way to deal with them is to find the proper encoding.

[Skaperen]

if it's a strange encoding i don't care if this is accurate. this is operation on my edit log which has time of edit and a file path. this script lists recently edited files with time. i'm inserting a column for number of lines. for my Python source files and most other text and source, the encoding will be ASCII or UTF-8. the file causing problem was a C file from 2006 when i was using ISO8859 for the copyright symbol. cat that line today (the Linux kernel is doing UTF-8 reasonably well) and the copyright is just a question in an inverted cell.

i've already done this and have it showing ">9999" (if the number of lines exceeds 9999) and narrowed it to 5 characters. if the file exceeds 1048575 bytes then it prints ">####". i do f.read(1048576). i could reduce that.

[perfringo]

If I need to have quick look at number of lines in file then I use wc in terminal.

1	cat my_filename | wc -l

It is easy to use in Python with subprocess. Specific example on file named shakespeare.txt; check_output returns bytes so I converted it into integer:

1 2 3 4 5

>>> import subprocess >>> source = subprocess.Popen(['cat', 'shakespeare.txt'], stdout=subprocess.PIPE)   >>> lines = int(subprocess.check_output(['wc', '-l'], stdin=source.stdout))     >>> lines 4155

[Gribouillis]

An alternative with more_itertools.ilen()

1 2 3 4 5 6 7 8 9

λ cat paillasse/sometest.py | wc -l 60 λ python ... >>> from more_itertools import ilen >>> with open('paillasse/sometest.py') as ifh: ...     print(ilen(ifh)) ... 60

[snippsat]

if it's a strange encoding i don't care if this is accurate. this is operation on my edit log which has time of edit and a file path.

Can just ignore encoding errors,there is a parameter for this errors="ignore" or errors='replace'(will be ?).
So can do a version showing this can just copy my own code from this Thread and make a little change.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

import os   def find_files(file_type, path):     os.chdir(path)     with os.scandir(path) as it:         for entry in it:             if entry.name.endswith(file_type) and entry.is_file():                 yield entry.name   def count_lines(files):     for file in files:         with open(file, encoding='utf-8', errors="ignore") as f:             for line_nr, _ in enumerate(f, -1):                 pass         yield file, line_nr + 1   if __name__ == '__main__':     path = r'E:\div_code'     file_type = '.txt'     files = find_files(file_type, path)     line_count = count_lines(files)     print(list(line_count))

Output:
[('alice_in_wonderland.txt', 3599), ('test.txt', 1807), ('W2Testfile.txt', 1396)]

Compare with wc

1 2 3 4 5

λ wc -l *.txt   1396 W2Testfile.txt   3599 alice_in_wonderland.txt   1807 test.txt   6802 total

[Pedroski55]

Hi, I don't understand lambda in this post, can you please explain?

I tried:

λ wc -l /home/pedro/summer2021/19BE/scansforOCR/*.text

in bash and just got:

pedro@pedro-HP:~$ λ wc /home/pedro/summer2021/19BE/scansforOCR/*.text
λ: command not found
pedro@pedro-HP:~$

[snippsat]

Hi, I don't understand lambda in this post, can you please explain?

You shall not use λ ,it's default sign because i use cmder

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

wc --help Usage: wc [OPTION]... [FILE]...   or:  wc [OPTION]... --files0-from=F Print newline, word, and byte counts for each FILE, and a total line if more than one FILE is specified.  A word is a non-zero-length sequence of characters delimited by white space.   With no FILE, or when FILE is -, read standard input.   The options below may be used to select which counts are printed, always in the following order: newline, word, character, byte, maximum line length.   -c, --bytes            print the byte counts   -m, --chars            print the character counts   -l, --lines            print the newline counts       --files0-from=F    read input from the files specified by                            NUL-terminated names in file F;                            If F is - then read names from standard input   -L, --max-line-length  print the maximum display width   -w, --words            print the word counts       --help     display this help and exit       --version  output version information and exit   GNU coreutils online help: <http://www.gnu.org/software/coreutils/> Report wc translation bugs to <http://translationproject.org/team/> Full documentation at: <http://www.gnu.org/software/coreutils/wc> or available locally via: info '(coreutils) wc invocation'