[SOLVED]passing a lot of information to subprocess.call()

[Barrowman]

Hi,
I have a long list of names and phone numbers in the following format:

0100 12345
Harry horse
0100 43234
Jane Jones
0121 67574
Mary Long
0123 26927
Bob Short

I call this file first.txt
What I am doing is create a new file which lists each name and records at the side of it how many times their name appears in the original list.
What I have been doing is using a simple python script to generate a file I call second.txt.
This just contains the names.
From that I run the following:

1	cat second.txt|sort|uniq>third.txt

This generates another file which has each name in alphabetical order once.
Another python script then reads through third.txt and for each line reads through second.txt and to count how many times that name appears. After each search it writes to a new file this number with the name beside it.

So what I wanted to do is to make one python script to do it all from start to finish rather than running the 3 separate commands. I tried os.system and then subprocess to incorporate the sort and uniq commands line but I cannot seem to get it to work. What ever I have tried gives syntax errors.
For example

1 2	cmd = "'cat', /home/norman/khconf/second.txt, '|sort', '|uniq', '>', /home/norman/khconf/third.txt" subprocess.call([cmd])

Error:
Traceback (most recent call last):
  File "./getnames.py", line 21, in <module>
    subprocess.call([cmd])
  File "/usr/lib/python3.5/subprocess.py", line 557, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: "'cat', /home/norman/khconf/second.txt, '|sort', '|uniq', '>', /home/norman/khconf/third.txt"

so then I tried:

1 2	cmd = "'/bin/cat', /home/norman/khconf/second.txt, '|/usr/bin/sort', '|/usr/bin/uniq', '>', /home/norman/khconf/third.txt" subprocess.call([cmd])

gave

Error:
Traceback (most recent call last):
  File "./getnames.py", line 21, in <module>
    subprocess.call([cmd])
  File "/usr/lib/python3.5/subprocess.py", line 557, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: "'/bin/cat', /home/norman/khconf/second.txt, '|/usr/bin/sort', '|/usr/bin/uniq', '>', /home/norman/khconf/third.txt"

Can someone point me in the right direction please?

[Barrowman]

This is all running under python 3.5.2
Well after a lot more googling I have found a different approach instead of subprocess.call

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

cat = subprocess.Popen(['cat', '/home/norman/khconf/second.txt'],                         stdout=subprocess.PIPE,                         )   sort = subprocess.Popen(['sort'],                         stdin=cat.stdout,                         stdout=subprocess.PIPE,                         )   uniq = subprocess.Popen(['uniq'],                         stdin=sort.stdout,                         stdout=subprocess.PIPE,                         )     end_of_pipe = uniq.stdout lastfile = docs + "/third.txt"   lastone = open(lastfile, 'w') lastone.write("Hello World") for line in end_of_pipe:     lastone.write(line)

I get no errors but third.txt only contains the "Hello World"
I modified the code from https://pymotw.com/2/subprocess/
Can anyone suggest how I can fix this please?

[anapbevi]

What if you delete the last 3 lines? Won`t work?
Im just guessing, cause I dont have the files to make a simulation...
(First day at coding...)

[wavic]

1 2	command = "cat /home/norman/khconf/second.txt | sort | uniq > /home/norman/khconf/third.txt".split() subprocess.call(command)

When you open a file with 'w' flag for writing it overwrites the previous content. Use 'a' to append.
https://docs.python.org/3/library/functions.html#open

[Barrowman]

1 2	command = "cat /home/norman/khconf/second.txt | sort | uniq > /home/norman/khconf/third.txt".split() subprocess.call(command)

the syntax is

1	subprocess.call([command])

When I run the command from the terminal it works fine without the .split

When you open a file with 'w' flag for writing it overwrite the previous content. Use 'a' to append.
https://docs.python.org/3/library/functions.html#open

Yes, but it is a file I am creating and it only opens a new file or clears the old contents.
From then on it will automatically append lines until the file is closed. This is what I want it to do.
If you look at my first post I use a python script to read first.txt and create second.txt. First.txt has 46 lines in it and second.txt has 23 lines in it. This is exactly as it should be. I am trying to get this all working as the input file will continue to grow and could end up with hundreds of lines in it

[wavic]

If the 'command' is already a list of arguments created by split() why you closing it in another []s?

Print 'end_of_pipe' to see if it contains something.

[Barrowman]

If the 'command' is already a list of arguments created by split() why you closing it in another []s?

Print 'end_of_pipe' to see if it contains something.

Sorry wavic when I do

1 2	command = "cat /home/norman/khconf/second.txt | sort | uniq > /home/norman/khconf/third.txt".split() subprocess.call(command)

I get

Error:
cat: '|': No such file or directory
cat: sort: No such file or directory
cat: '|': No such file or directory
cat: uniq: No such file or directory
cat: '>': No such file or directory

[Barrowman]

I have found out how to do it now.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

fout="/home/norman/khconf/third.txt" fo = open(fout, 'w')   cat = subprocess.Popen(['cat', 'second.txt'],                         stdout=subprocess.PIPE,                         )   sort = subprocess.Popen(['sort'],                         stdin=cat.stdout,                         stdout=subprocess.PIPE,                         ) uniq = subprocess.Popen(['uniq'],             stdin=sort.stdout,             stdout=fo             )

[wavic]

Sorry wavic when I do
Python Code: (Double-click to select all)
1
2

command = "cat /home/norman/khconf/second.txt | sort | uniq > /home/norman/khconf/third.txt".split()
subprocess.call(command)
I get

Error:
cat: '|': No such file or directory
cat: sort: No such file or directory
cat: '|': No such file or directory
cat: uniq: No such file or directory
cat: '>': No such file or directory

This is because the call method doesn't use the shell to execute a command. You have to pass shell=True as a parameter to the method. But this is not safe.

[buran]

well, now you can learn how to do it with python, without system calls and without need of 2 intermediate files...

1 2 3 4 5 6 7

from collections import Counter   with open('first.txt', 'r') as f1:     names = [ln.strip() for i, ln in enumerate(f1) if i%2] with open('fourth.txt', 'w') as f4:     for key, value in sorted(Counter(names).items()):         f4.write('{} {}\n'.format(key, value))