Handling Large XML Files (>10GB) in Python

Thread Rating:

0 Vote(s) - 0 Average
1
2
3
4
5

Thread Modes

Handling Large XML Files (>10GB) in Python

Clunk_Head
Splitter

Posts: 158

Threads: 27

Joined: Jan 2019

Reputation: 8

Dec-22-2019, 05:46 AM

Use lxml and iterparse. When you've finished with an element you will need to use clear to prevent a buildup in memory.
This should let you open an XML file of most any size.

Find

Messages In This Thread

Handling Large XML Files (>10GB) in Python - by onlydibs - Dec-22-2019, 04:17 AM

RE: Handling Large XML Files (>10GB) in Python - by Clunk_Head - Dec-22-2019, 05:46 AM

Possibly Related Threads…
Thread		Author	Replies	Views	Last Post
	Insert 10gb csv files into sql table via python	mg24	2	2,164	Apr-28-2023, 04:14 PM Last Post: snippsat
	python exception handling handling .... with traceback	mg24	3	1,441	Nov-09-2022, 07:29 PM Last Post: Gribouillis
	Handling pdf files with python.	fuzzin	1	1,363	Jan-19-2022, 02:24 PM Last Post: ThiefOfTime
	Handling Python Fatal Error	richajain1785	7	6,323	Oct-14-2021, 01:34 PM Last Post: Tails86
	Handling large images with OpenCV.	Apenz1	0	1,709	Jan-16-2021, 05:12 PM Last Post: Apenz1
	Handling multi-input/output audio in python	bor1904	4	3,781	Nov-04-2020, 08:25 AM Last Post: CHLOVRL
	Python Requests package: Handling xml response	soumyarani	1	2,276	Sep-14-2020, 11:41 AM Last Post: buran
	Iterate 2 large text files across lines and replace lines in second file	medatib531	13	6,360	Aug-10-2020, 11:01 PM Last Post: medatib531
	Handling Multiple USB ports in Python	samalpramod	0	4,637	Aug-01-2020, 07:40 PM Last Post: samalpramod
	Iterating Large Files	Robotguy	10	5,504	Jul-22-2020, 09:13 PM Last Post: Gribouillis

Users browsing this thread: 1 Guest(s)

View a Printable Version

Handling Large XML Files (>10GB) in Python

User Panel Messages

Announcements