Python Forum
How to send unicode string encoded in utf-8 in http request in Python
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to send unicode string encoded in utf-8 in http request in Python
#1
I'm trying to send a unicode string, which contains chinese characters, from a python program to a web server.

The code for this is:
import json
import requests

    #get json
    data = json.dumps({"Input": {"Row": [{"BusinessCard": "那是一句话"}]}}, ensure_ascii=False)
    #encode json to utf-8
    encoded_data = data.encode('utf-8')

    r = requests.post("http://some-server.com/service", data=encoded_data,
                      headers={'Content-Type': 'application/json; charset=UTF-8'},
                      auth=requests.auth.HTTPBasicAuth("user", "password"))
The encoded_data variable:
b'{"Input": {"Row": [{"BusinessCard": "\xe9\x82\xa3\xe6\x98\xaf\xe4\xb8\x80\xe5\x8f\xa5\xe8\xaf\x9d"}]}}'
My problem is, if send this to the server, the server doesn't recognize the utf-8 encoded data as chinese characters. The server just sees '"\xe9\x82\xa3...' and can't handle them.

But I'm forced to encode the data to utf-8 or the post method will give me the this unicode error:
Output:
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 37-41: Body ('那是一句话') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.
If I send the request from Postman, it works fine. The postman request is similiar, there is only the header for basic authentication and the content type header.

Also, if I send from the python program a test request to a simple spring boot server with a rest controller, the controller/spring recognizes the encoded values right as chinese characters.

How can I send the json data in the body like Postman does, so that the server understands the chinese characters?

I searched a few hours but couldn't find any answer to a similar case like this.

I would be really thankful for some help.

Greetings
MaverinCode

Ok, got some help from Stack Overflow.

I set the ensure_ascii parameter to 'True' and it works.

But I would be thankful for some further explanation if someone knows more.

The documentation states: "If ensure_ascii is true (the default), the output is guaranteed to have all incoming non-ASCII characters escaped. If ensure_ascii is false, these characters will be output as-is."

Why have servers no trouble to encode escaped unicode signs which have been encoded to utf-8 but trouble with none escape unicode signs which have been also been encode to utf-8?
Reply


Messages In This Thread
How to send unicode string encoded in utf-8 in http request in Python - by MaverinCode - Apr-25-2019, 05:25 PM

Possibly Related Threads…
Thread Author Replies Views Last Post
Question Responding correctly to HTTP request Calab 2 1,157 Apr-14-2023, 04:11 PM
Last Post: Calab
Question Python request (post/get) Drunknmonkie 1 2,627 Jan-19-2023, 02:02 PM
Last Post: prvncpa
  Retrieve images base64 encoded MongoDB and Flask Nuwan16 2 3,234 Oct-13-2020, 06:25 PM
Last Post: Nuwan16
  python beginner HTTP Error 500 leofcastro 0 2,136 Jan-24-2020, 04:37 PM
Last Post: leofcastro
  How to send keys at beginning of textbox using python Prince_Bhatia 2 3,022 May-21-2018, 08:15 AM
Last Post: Prince_Bhatia
  Syntax error for HTTP request GET THX1138 1 6,345 May-12-2018, 12:02 PM
Last Post: snippsat
  HTTP Header request, how to improve efficiency andreamoro 5 5,056 May-01-2017, 03:49 PM
Last Post: micseydel

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020