Aug-23-2019, 01:36 PM
(This post was last modified: Aug-23-2019, 02:29 PM by adnanahsan.)
I tried bro, but it didn't work http://prntscr.com/owkxi3
is it possible that python may fail to correct encoding ??
you are right, thats also i wanted to figure out, i am using selenium, grabbing page vide driver.get(url) and that is returning such bad encoded content .. and this problem is only in my server ubuntu, its working fine on my windows and vmware ubuntu on windows pc . but on live server its not working.
Here is my source code ..
It works on my windows
It works on my ubuntu installed on vmware on my windows
It doesn't work on my servers
PS. it shows correct encoding on browser, but problem is on console / terminal .. i need to match encoded strings for that i need it work on file / terminal
is it possible that python may fail to correct encoding ??
(Aug-23-2019, 12:09 PM)DeaD_EyE Wrote: The encoding you're using for your string is broken.
You can fix broken encodings with ftfy.
>>> import ftfy >>> s = ' Relógio feminino dourado ' >>> s ' Relógio feminino dourado ' >>> ftfy.fix_encoding(s) ' Relógio feminino dourado ' >>>Encoding fixed...
If you try to fix broken encodings without this module, is not easy.
you are right, thats also i wanted to figure out, i am using selenium, grabbing page vide driver.get(url) and that is returning such bad encoded content .. and this problem is only in my server ubuntu, its working fine on my windows and vmware ubuntu on windows pc . but on live server its not working.
(Aug-23-2019, 12:18 PM)snippsat Wrote:(Aug-23-2019, 10:06 AM)adnanahsan Wrote: I am getting the string from scrapping webpage, its not returning valid utf-8You should try to get data with right encoding as the first step correct.
Using eg Requests in combo with BS,will in almost all case get correct encoding.
Requests will return correct site encoding.
>>> import requests >>> url = 'http://CNN.com' >>> response = requests.get(url) >>> response.encoding 'utf-8'Parsing would look like this.
import requests from bs4 import BeautifulSoup url = 'http://CNN.com' response = requests.get(url) soup = BeautifulSoup(response.content, 'lxml') print(soup.find('title').text)from bs4 import BeautifulSoup import re html = '''\ <!DOCTYPE html> <html> <head> <meta charset="UTF-8"> <title>Page Title</title> </head> <body> <h1>This is a Heading</h1> <p>Relógio feminino dourado</p> </body>''' soup = BeautifulSoup(html, 'lxml') label = soup.find('p')>>> label <p>Relógio feminino dourado</p> >>> label.text 'Relógio feminino dourado'Remember when have correct value in Python 3 it's Unicode as all string in Python 3 are of this type.
If data comes from utf-8,latin-1..ect dos not matter if look correct in Python 3,
then in and out of Python or server always useutf-8.
Here is my source code ..
It works on my windows
It works on my ubuntu installed on vmware on my windows
It doesn't work on my servers
# -*- coding: utf-8 -*-
from seleniumwire import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support import expected_conditions as ec
import time
import ftfy
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-certificate-errors-spki-list')
options.add_argument('--ignore-ssl-errors')
options.add_argument("--headless")
options.add_argument("--window-size=1920x1080")
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
chrome_path = '/var/www/chromedriver'
driver = webdriver.Chrome(chrome_path, options=options)
driver.get("http://www.correios.com.br/solucoes-empresariais/correios-facil")
driver.implicitly_wait(10)
a = driver.find_elements_by_css_selector("p")
for i in a:
s = i.text
s = ftfy.fix_encoding(s)
print(s)
driver.quit()PS. it shows correct encoding on browser, but problem is on console / terminal .. i need to match encoded strings for that i need it work on file / terminal