Normalizig scraped text

Thread Rating:

0 Vote(s) - 0 Average
1
2
3
4
5

Thread Modes

Normalizig scraped text

Larz60+
aetate et sapientia
Super Moderators

Posts: 12,036

Threads: 486

Joined: Sep 2016

Reputation: 452

Jan-06-2020, 01:45 PM

first you set all_text to unicodedata.normalize("NFC", all_text)
then immediately after, you overwrite that with whatever returns from sent_tokenize(all_text)
is that what you intended?

Find

Messages In This Thread

Normalizig scraped text - by wuggs - Jan-06-2020, 01:05 PM

RE: Normalizig scraped text - by Larz60+ - Jan-06-2020, 01:45 PM

RE: Normalizig scraped text - by wuggs - Jan-06-2020, 02:21 PM

RE: Normalizig scraped text - by Larz60+ - Jan-07-2020, 03:32 AM

Possibly Related Threads…
Thread		Author	Replies	Views	Last Post
	Weird characters scraped	samuelbachorik	3	2,104	Oct-29-2023, 02:36 PM Last Post: DeaD_EyE
	Web scraper not populating .txt with scraped data	BlackHeart	5	2,842	Apr-03-2023, 05:12 PM Last Post: snippsat
	Python Obstacles \| Krav Maga \| Wiki Scraped Content [Column Copy]	BrandonKastning	4	3,330	Jan-03-2022, 06:59 AM Last Post: BrandonKastning
	Python Obstacles \| Kapap \| Wiki Scraped Content [Column Nulling]	BrandonKastning	2	2,599	Jan-03-2022, 04:26 AM Last Post: BrandonKastning
	Any way to remove HTML tags from scraped data? (I want text only)	SeBz2020uk	1	4,564	Nov-02-2020, 08:12 PM Last Post: Larz60+
	cant loop through scraped site	matt42	3	3,397	Aug-12-2020, 06:48 AM Last Post: ndc85430
	Parsing infor from scraped files.	Larz60+	2	4,678	Apr-12-2019, 05:06 PM Last Post: Larz60+
	beautiful soup - parsing scraped code in a script	lilbigwill99	2	4,107	Mar-09-2018, 04:10 PM Last Post: lilbigwill99
	Need Tip On Cleaning My BS4 Scraped Data	digitalmatic7	2	4,074	Jan-29-2018, 08:49 PM Last Post: digitalmatic7

Users browsing this thread: 1 Guest(s)

View a Printable Version

Normalizig scraped text

User Panel Messages

Announcements