Python Forum
[SOLVED]passing a lot of information to subprocess.call()
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
[SOLVED]passing a lot of information to subprocess.call()
#1
Hi,
I have a long list of names and phone numbers in the following format:
Quote:0100 12345
Harry horse
0100 43234
Jane Jones
0121 67574
Mary Long
0123 26927
Bob Short
I call this file first.txt
What I am doing is create a new file which lists each name and records at the side of it how many times their name appears in the original list.
What I have been doing is using a simple python script to generate a file I call second.txt.
This just contains the names.
From that I run the following:
cat second.txt|sort|uniq>third.txt
This generates another file which has each name in alphabetical order once.
Another python script then reads through third.txt and for each line reads through second.txt and to count how many times that name appears. After each search it writes to a new file this number with the name beside it.

So what I wanted to do is to make one python script to do it all from start to finish rather than running the 3 separate commands. I tried os.system and then subprocess to incorporate the sort and uniq commands line but I cannot seem to get it to work. What ever I have tried gives syntax errors.
For example
cmd = "'cat', /home/norman/khconf/second.txt, '|sort', '|uniq', '>', /home/norman/khconf/third.txt"
subprocess.call([cmd])
Error:
Traceback (most recent call last): File "./getnames.py", line 21, in <module> subprocess.call([cmd]) File "/usr/lib/python3.5/subprocess.py", line 557, in call with Popen(*popenargs, **kwargs) as p: File "/usr/lib/python3.5/subprocess.py", line 947, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child raise child_exception_type(errno_num, err_msg) FileNotFoundError: [Errno 2] No such file or directory: "'cat', /home/norman/khconf/second.txt, '|sort', '|uniq', '>', /home/norman/khconf/third.txt"
so then I tried:
cmd = "'/bin/cat', /home/norman/khconf/second.txt, '|/usr/bin/sort', '|/usr/bin/uniq', '>', /home/norman/khconf/third.txt"
subprocess.call([cmd])
gave
Error:
Traceback (most recent call last): File "./getnames.py", line 21, in <module> subprocess.call([cmd]) File "/usr/lib/python3.5/subprocess.py", line 557, in call with Popen(*popenargs, **kwargs) as p: File "/usr/lib/python3.5/subprocess.py", line 947, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child raise child_exception_type(errno_num, err_msg) FileNotFoundError: [Errno 2] No such file or directory: "'/bin/cat', /home/norman/khconf/second.txt, '|/usr/bin/sort', '|/usr/bin/uniq', '>', /home/norman/khconf/third.txt"
Can someone point me in the right direction please?
Reply
#2
This is all running under python 3.5.2
Well after a lot more googling I have found a different approach instead of subprocess.call
cat = subprocess.Popen(['cat', '/home/norman/khconf/second.txt'], 
                        stdout=subprocess.PIPE,
                        )

sort = subprocess.Popen(['sort'],
                        stdin=cat.stdout,
                        stdout=subprocess.PIPE,
                        )

uniq = subprocess.Popen(['uniq'],
                        stdin=sort.stdout,
                        stdout=subprocess.PIPE,
                        )


end_of_pipe = uniq.stdout
lastfile = docs + "/third.txt"

lastone = open(lastfile, 'w')
lastone.write("Hello World")
for line in end_of_pipe:
    lastone.write(line)
I get no errors but third.txt only contains the "Hello World"
I modified the code from https://pymotw.com/2/subprocess/
Can anyone suggest how I can fix this please?
Reply
#3
What if you delete the last 3 lines? Won`t work?
Im just guessing, cause I dont have the files to make a simulation...
(First day at coding...)
Reply
#4
command = "cat /home/norman/khconf/second.txt | sort | uniq > /home/norman/khconf/third.txt".split()
subprocess.call(command)
When you open a file with 'w' flag for writing it overwrites the previous content. Use 'a' to append.
https://docs.python.org/3/library/functions.html#open
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#5
(Jan-26-2018, 09:44 PM)wavic Wrote:
command = "cat /home/norman/khconf/second.txt | sort | uniq > /home/norman/khconf/third.txt".split()
subprocess.call(command)
the syntax is
subprocess.call([command])
When I run the command from the terminal it works fine without the .split
Quote:When you open a file with 'w' flag for writing it overwrite the previous content. Use 'a' to append.
https://docs.python.org/3/library/functions.html#open

Yes, but it is a file I am creating and it only opens a new file or clears the old contents.
From then on it will automatically append lines until the file is closed. This is what I want it to do.
If you look at my first post I use a python script to read first.txt and create second.txt. First.txt has 46 lines in it and second.txt has 23 lines in it. This is exactly as it should be. I am trying to get this all working as the input file will continue to grow and could end up with hundreds of lines in it
Reply
#6
If the 'command' is already a list of arguments created by split() why you closing it in another []s?

Print 'end_of_pipe' to see if it contains something.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#7
(Jan-27-2018, 07:29 AM)wavic Wrote: If the 'command' is already a list of arguments created by split() why you closing it in another []s?

Print 'end_of_pipe' to see if it contains something.

Sorry wavic when I do
command = "cat /home/norman/khconf/second.txt | sort | uniq > /home/norman/khconf/third.txt".split()
subprocess.call(command)
I get
Error:
cat: '|': No such file or directory cat: sort: No such file or directory cat: '|': No such file or directory cat: uniq: No such file or directory cat: '>': No such file or directory
Reply
#8
I have found out how to do it now.
fout="/home/norman/khconf/third.txt"
fo = open(fout, 'w')

cat = subprocess.Popen(['cat', 'second.txt'], 
                        stdout=subprocess.PIPE,
                        )

sort = subprocess.Popen(['sort'],
                        stdin=cat.stdout,
                        stdout=subprocess.PIPE,
                        )
uniq = subprocess.Popen(['uniq'],
			stdin=sort.stdout,
			stdout=fo
			)
Reply
#9
(Jan-27-2018, 09:09 AM)Barrowman Wrote: Sorry wavic when I do
Python Code: (Double-click to select all)
1
2

command = "cat /home/norman/khconf/second.txt | sort | uniq > /home/norman/khconf/third.txt".split()
subprocess.call(command)
I get

Error:
cat: '|': No such file or directory
cat: sort: No such file or directory
cat: '|': No such file or directory
cat: uniq: No such file or directory
cat: '>': No such file or directory

This is because the call method doesn't use the shell to execute a command. You have to pass shell=True as a parameter to the method. But this is not safe.
"As they say in Mexico 'dosvidaniya'. That makes two vidaniyas."
https://freedns.afraid.org
Reply
#10
well, now you can learn how to do it with python, without system calls and without need of 2 intermediate files...
from collections import Counter

with open('first.txt', 'r') as f1:
    names = [ln.strip() for i, ln in enumerate(f1) if i%2]
with open('fourth.txt', 'w') as f4:
    for key, value in sorted(Counter(names).items()):
        f4.write('{} {}\n'.format(key, value))
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  continue if 'subprocess.call' failes tester_V 11 5,013 Aug-26-2021, 12:16 AM
Last Post: tester_V
  printing out the contents aftre subprocess.call() Rakshan 3 2,701 Jul-30-2021, 08:27 AM
Last Post: DeaD_EyE
  subprocess call cannot find the file specified RRR 6 16,386 Oct-15-2020, 11:29 AM
Last Post: RRR
  Why wont subprocess call work? steve_shambles 3 2,605 Apr-28-2020, 03:06 PM
Last Post: steve_shambles
  subprocess.call - Help ! sniper6 0 1,499 Nov-27-2019, 07:42 PM
Last Post: sniper6
  Mock call to subprocess.Popen failing with StopIteration tharpa 7 5,821 Nov-08-2019, 05:00 PM
Last Post: Gribouillis
  Echo call to VLC bash using subprocess.Popen on Linux LuukS001 17 9,543 Jan-07-2019, 03:58 AM
Last Post: LuukS001
  Why is subprocess.call command not working? zBernie 5 10,652 Nov-19-2018, 11:11 PM
Last Post: snippsat
  Help with subprocess..call oldcity 1 2,511 Sep-28-2018, 03:59 PM
Last Post: volcano63
  subprocess.call salmaankamil 1 2,677 Jul-27-2018, 06:50 PM
Last Post: woooee

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020