Weird characters scraped

[samuelbachorik]

Hi i am trying to scrape webpage but in names of products i get weird characters, i will show what is in webpage sourcode and what beautiful soup scrape for me...

This is how it looks like in web page trouught inspect:

<strong>Clamps P&S Black 100</strong>

And this is what beautifulsoup scrapes:

<strong>Clapms P&amp;S Black 100</strong>

How to get it right ?

Thank you

[SpongeB0B]

Hi @samuelbachorik,

& is not a "weird" character it's an "HTML character entity references" .. Damn that a long name...
https://en.wikipedia.org/wiki/List_of_XM...references

If you plan to use the extracted data outside HTML then you could convert those character entity references into utf-8

Cheers,

[snippsat]

When use .text to get content out of tag it will be ok.

from bs4 import BeautifulSoup

html = '<strong>Clamps P&S Black 100</strong>'
soup = BeautifulSoup(html, 'lxml')
tag = soup.select_one('strong')

>>> tag
<strong>Clamps P&amp;S Black 100</strong>
>>> tag.text
'Clamps P&S Black 100'

[DeaD_EyE]

With stdlib from Python:

import html


s = "<strong>Clamps P&amp;S Black 100</strong>"
text = html.unescape(s)

print(text)

Output:
<strong>Clamps P&S Black 100</strong>

But BeautifulSoup does it already. You should use this 3rd party library.