Dec-29-2016, 07:03 AM
is urllib2 a good choice for getting website contents? it is used in that code i posted regarding catchin IOError failing. it gets my website OK as well as google. but when i append the search stuff it fails on both sites (while lynx and curl work). i cross checked what urllib2 and lynx sent to a dummy local website and they both look right.
more testing:
note that my website does not have /search script so i just used the apex page for these tests.
more testing:
note that my website does not have /search script so i just used the apex page for these tests.
Output:lt1/forums /home/forums 1> python filetopystr.py 0 'http://www.google.com/' ip 80|wc -c
connecting to 'http://www.google.com/'
connected to 'http://www.google.com/'
reading from 'http://www.google.com/'
read 10872 bytes
11478
lt1/forums /home/forums 2> python filetopystr.py 0 'http://linuxhomepage.com/' ip 80|wc -c
connecting to 'http://linuxhomepage.com/'
connected to 'http://linuxhomepage.com/'
reading from 'http://linuxhomepage.com/'
read 30994 bytes
32900
lt1/forums /home/forums 3> python filetopystr.py 0 'http://ipv6.linuxhomepage.com/' ip 80|wc -c
connecting to 'http://ipv6.linuxhomepage.com/'
connected to 'http://ipv6.linuxhomepage.com/'
reading from 'http://ipv6.linuxhomepage.com/'
read 30988 bytes
32894
lt1/forums /home/forums 4> python filetopystr.py 0 'http://www.google.com/search?hl=en&as_q=what+is+my+ip+address' ip 80|wc -c
connecting to 'http://www.google.com/search?hl=en&as_q=what+is+my+ip+address'
Error connecting to 'http://www.google.com/search?hl=en&as_q=what+is+my+ip+address'
0
lt1/forums /home/forums 5> python filetopystr.py 0 'http://linuxhomepage.com/?hl=en&as_q=what+is+my+ip+address' ip 80|wc -c
connecting to 'http://linuxhomepage.com/?hl=en&as_q=what+is+my+ip+address'
connected to 'http://linuxhomepage.com/?hl=en&as_q=what+is+my+ip+address'
reading from 'http://linuxhomepage.com/?hl=en&as_q=what+is+my+ip+address'
read 30994 bytes
32900
lt1/forums /home/forums 6> python filetopystr.py 0 'http://ipv6.linuxhomepage.com/?hl=en&as_q=what+is+my+ip+address' ip 80|wc -c
connecting to 'http://ipv6.linuxhomepage.com/?hl=en&as_q=what+is+my+ip+address'
connected to 'http://ipv6.linuxhomepage.com/?hl=en&as_q=what+is+my+ip+address'
reading from 'http://ipv6.linuxhomepage.com/?hl=en&as_q=what+is+my+ip+address'
read 30988 bytes
32894
lt1/forums /home/forums 7> lynx -mime_header 'http://www.google.com/search?hl=en&as_q=what+is+my+ip+address'|wc -c
19816
lt1/forums /home/forums 8> lynx -mime_header 'http://linuxhomepage.com/?hl=en&as_q=what+is+my+ip+address'|wc -c
29505
lt1/forums /home/forums 9> lynx -mime_header 'http://ipv6.linuxhomepage.com/?hl=en&as_q=what+is+my+ip+address'|wc -c
29499
lt1/forums /home/forums 10>