urlib2 issues

[Skaperen]

here's what i got from urllib2:

Output:
GET / HTTP/1.1
Accept-Encoding: identity
Host: localhost:8001
Connection: close
User-Agent: Python-urllib/2.7

so, do you think google is rejecting a scripted search?

[wavic]

Don't know. Perhaps

I had some strange results once. I've wrote a script to use google.translate from the terminal. But during that I had no internet connection so I have used a downloaded page just to see the html. So the script was working well but when I've tried it in real it didn't print anything from similar meaning words - they appears under the translated word as synonyms.

[Skaperen]

yes, they are...


using the same user agent on lynx i got a 403 Forbidden and a message explaining that the client does not have permission.  it also says to read http://www.google.com/terms_of_service.htm but that URL gets a 404 error.

so i don't literally know, yet, if they allow a script to discover its ip address through their search service or not, probably not since humans won't see their ads this way and they are very ad  oriented.  so maybe i should just use the services with a implemented api, like the ones that provide a json response.

[wavic]

Did you try to switch the user agent? Yesterday I wrote script to scrape my government's site and it works normal. But I've changed the user agent to this: 'Mozilla/5.0 (Windows NT 6.0; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0'. I didn't try the script without it.