Python Forum

i have the call to main() wrapped to catch IOError (line 156 in the code below) so the output can be stopped in a POSIX/Unix/Linux command line piped to head or another program that quits reading the pipe (causes an error writin to the pipe because the other end is closed).  yet i get this:

Output:
lt1/forums /home/forums 74> python filetopystr.py 0 'http://www.google.com/' ip 80|head -80|tail -3
connecting to 'http://www.google.com/'
connected  to 'http://www.google.com/'
reading  from 'http://www.google.com/'
read 10890 bytes
' href="https://mail.google.com/mail/?tab=wm">Gmail</a> <a class=gb1 href="htt'\
'ps://drive.google.com/?tab=wo">Drive</a> <a class=gb1 style="text-decoration:'\
'none" href="https://www.google.com/intl/en/options/"><u>More</u> &raquo;</a><'\
Traceback (most recent call last):
  File "filetopystr.py", line 158, in <module>
    stdout.flush()
IOError: [Errno 32] Broken pipe
lt1/forums /home/forums 75>

i should get this:

Output:
lt1/forums /home/forums 74> python filetopystr.py 0 'http://www.google.com/' ip 80|head -80|tail -3
connecting to 'http://www.google.com/'
connected  to 'http://www.google.com/'
reading  from 'http://www.google.com/'
read 10890 bytes
' href="https://mail.google.com/mail/?tab=wm">Gmail</a> <a class=gb1 href="htt'\
'ps://drive.google.com/?tab=wo">Drive</a> <a class=gb1 style="text-decoration:'\
'none" href="https://www.google.com/intl/en/options/"><u>More</u> &raquo;</a><'\
lt1/forums /home/forums 75>

i routinely have the try/except around the call to main() to make my scripts smoothly handle being pipelined to the head command.


any ideas what i did wrong to cause IOError to not be caught?


here is filetopystr.py:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import print_function
"""
file          filetostr.py
purpose       Read a file and convert it to a string in a code snippit output
              to stdout.
email         10054452614123394844460370234029112340408691

The intent is that this command works correctly under both Python 2 and
Python 3.  Currently, not all character codespaces are supported under Python 3.
Please report other failures or code improvement to the author.
"""

__license__ = """
Copyright (C) 2016, by Phil D. Howard - all other rights reserved

Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA, OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE, OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

The author may be contacted by decoding the number
10054452614123394844460370234029112340408691
(provu igi la numeron al duuma)
"""

from sys import argv, stderr, stdout, version

def help( outfile ):
    """
function       help
purpose        Output the help contents to the open file given in the one argument.
"""
    print( '\n'
           'Usage: filetostr indent file_name [ variable_name [ code_text_width ] ]\n'
           'Usage: filetostr -h | +h | -help | +help | --help | ++help\n'
           '\n'
           'Convert the contents of a file to a Python code snippit, output to stdout,\n'
           'ready to insert or merge into Python code where it is needed.\n'
           'Note 1: The code text width accounts for the specified indent.\n'
           'Note 2: The indent is always space, tabs are not supported.\n'
           'Note 3: Not all character codespaces are supported under Python 3.\n',
           file=outfile )
    return 1


def main( args ):
    """
function       main
arguments      [ indent ] file_name [ variable_name [ line_width ] ]
purpose        Do the main work to read the specified file and output its contents as
               Python code for an assignment of a long string literal representing
               the file contents.  By default, or if specified, output help text to
               stderr.
note           Indenting uses spaces, only.  Tabs are not supported.  Continued lines
               are additionally indented to match the first line.
"""
    a = args

    opts = True
    if len(a) > 1:
        if a[1] in ('--','++'):
            opts = False
            a = a[0:1] + a[2:]

    if len(a) > 1:
        if opts and a[1].lower() in ('-h','+h','-help','+help','--help','++help'):
            return help( stderr )
        try:
            indent = ' ' * int(a[1])
        except (TypeError,ValueError):
            print( a[0].split('/')[-1] + ': invalid operand', '(indent)', file=stderr )
            return help( stderr )
    else:
        print( a[0].split('/')[-1] + ': missing operand', '(indent)', file=stderr )
        return help( stderr )

    if len(a) > 2:
        fname = a[2]
    else:
        print( a[0].split('/')[-1] + ': missing operand', '(file name)', file=stderr )
        return help( stderr )
    
    if len(a) > 3:
        vname = a[3]
    else:
        vname = 'text'

    if len(a) > 4:
        max_len = int( a[4] )
    else:
        max_len = 80
    front = vname + ' = '

    if fname.lower() in ('-','+','/stdin','/dev/stdin','c:\\stdin'):
        data = stdin.read( 0x1000000 )
        stdin.close
    elif fname[:7] in ('http://','https:/'):
        import urllib2
        try:
            print('connecting to', repr(fname), file=stderr)
            conn = urllib2.urlopen(fname)
            print('connected  to', repr(fname), file=stderr)
        except urllib2.URLError:
            print( 'Error connecting to', repr(fname), file=stderr )
            return 1
        try:
            print('reading  from', repr(fname), file=stderr)
            data = conn.read()
            print('read', repr(len(data)), 'bytes', file=stderr)
            conn.close()
        except urllib2.URLError:
            print( 'Error reading from', repr(fname), file=stderr )
    else:
        try:
            with open( fname, 'r' ) as file:
                data = file.read( 0x1000000 )
        except IOError:
            print( 'Error opening file', repr(fname), file=stderr )
            return 1

    line_num = 0
    while len(data) > 0:
        line_num += 1
        for try_len in range(max_len,0,-1):
            rdata = data[try_len:]
            rlen = len(rdata)
            line = indent + front + repr(data[:try_len])
            if len(data)>try_len:
                line += '\\'
            if len(line) <= max_len:
                break
        print( line )
        if rlen == 0:
            break
        data = rdata
        front = ''

    return 0


if __name__ == '__main__':
    try:
        result = main( argv )
    except KeyboardInterrupt:
        result = 140
        print( '' )
    except IOError:
        result = 141
    stdout.flush()
    try:
        exit( int( result ) )
    except ValueError:
        print( str( result ), file=stderr )
        exit( 1 )
    except TypeError:
        if result == None:
            exit( 0 )
        exit( 255 )

# EOF

this script is not done, yet, so you may find many things wrong.

I don't see any error that stands out,

you can try to capture all exceptions just to see what is being thrown

something like:

import sys

        try:
            ...
        except:
            print("Unexpected error:", sys.exc_info()[0])

i added that ... but IOError still fails to be caught and triggers the incomplete traceback.  could it be that i cannot do this through a function call?  it's like my except statements have no effect.

Output:
lt1/forums /home/forums 89> tail -24 filetopystr.py
if __name__ == '__main__':
    try:
        result = main( argv )
    except KeyboardInterrupt:
        result = 140
        print( '' )
    except IOError:
        print( 'it worked this time', file=stderr )
        result = 141
    except:
        print( 'wtf:', exc_info()[0], file=stderr )
        result = 141
    stdout.flush()
    try:
        exit( int( result ) )
    except ValueError:
        print( str( result ), file=stderr )
        exit( 1 )
    except TypeError:
        if result == None:
            exit( 0 )
        exit( 255 )

# EOF
lt1/forums /home/forums 90> python filetopystr.py 0 'http://www.google.com/' ip 80|head -80|tail -3
connecting to 'http://www.google.com/'
connected  to 'http://www.google.com/'
reading  from 'http://www.google.com/'
read 10871 bytes
'google.com/mail/?tab=wm">Gmail</a> <a class=gb1 href="https://drive.google.co'\
'm/?tab=wo">Drive</a> <a class=gb1 style="text-decoration:none" href="https://'\
'www.google.com/intl/en/options/"><u>More</u> &raquo;</a></nobr></div><div id='\
Traceback (most recent call last):
  File "filetopystr.py", line 162, in <module>
    stdout.flush()
IOError: [Errno 32] Broken pipe
lt1/forums /home/forums 91>

What if

sys.exit(main())

Sys.exit() can be used like this: sys.exit("No internet connection!!!") so if main() crashes I think sys.exit() is going to intercept the error message and print it out. I didn't play with it

the internet connection is working. the command pipeline has first "head -80" which quits reading STDIN after reading 80 lines (this is the pipe error i am trying to catch and ignore). then the "tail -3" gives a small peek of the output (lines 78 to 80 based on counting from 1).

---

this is how much i am getting from the working internet connection.  note that google gives slightly different content each time.  i can set up a static file in linuxhomepage.com if that would help somehow.

Output:
lt1/forums /home/forums 95> python filetopystr.py 0 'http://www.google.com/' ip 80|wc -lc          
connecting to 'http://www.google.com/'
connected  to 'http://www.google.com/'
reading  from 'http://www.google.com/'
read 10931 bytes
    143   11542
lt1/forums /home/forums 96>

"No internet connection!!!" is just an example. It's not related to your script. 
As I know urllib2 uses a socket module. A little digging tells me that socket module wouldn't print out IOError but socket.error exception.

i used socket module for my validip() function. its inet_pton() raises an exception for an invalid IP (how i evaluate an IP for now). i might use that for scraping logic. we'll see.

Quote:

Error:
Traceback (most recent call last):
  File "filetopystr.py", line 162, in <module>
    stdout.flush()
IOError: [Errno 32] Broken pipe

The line causing the error is also the only line not in a try/except. Why are you flushing the output, anyway?

(Dec-29-2016, 04:13 PM)nilamo Wrote: [ -> ]

Quote:

Error:
Traceback (most recent call last):
  File "filetopystr.py", line 162, in <module>
    stdout.flush()
IOError: [Errno 32] Broken pipe

The line causing the error is also the only line not in a try/except.  Why are you flushing the output, anyway?

it is a leftover from the template and was used to syncronize stdout and stderr.  i thought i had gotten that part of the code cleaned up.  this is an example of using the code my mind believes to exist rather than what really does.  i better clean up the template, too, before this happens on another project.

thanks, nilamo ... it would have taken me quite a while to re-read the message enough to realize how wrong the code was.