Python Forum
Handling Large XML Files (>10GB) in Python
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Handling Large XML Files (>10GB) in Python
#1
Parsing Big XML Files
Need help to parse very large xml files(>10GB). Could you please help how to avoid memory issues. We can't read the entire file once. Any code example or conceptual explanation please. Thanks
Reply
#2
Use lxml and iterparse. When you've finished with an element you will need to use clear to prevent a buildup in memory.
This should let you open an XML file of most any size.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Insert 10gb csv files into sql table via python mg24 2 1,842 Apr-28-2023, 04:14 PM
Last Post: snippsat
Star python exception handling handling .... with traceback mg24 3 1,219 Nov-09-2022, 07:29 PM
Last Post: Gribouillis
  Handling pdf files with python. fuzzin 1 1,219 Jan-19-2022, 02:24 PM
Last Post: ThiefOfTime
  Handling Python Fatal Error richajain1785 7 5,764 Oct-14-2021, 01:34 PM
Last Post: Tails86
  Handling large images with OpenCV. Apenz1 0 1,597 Jan-16-2021, 05:12 PM
Last Post: Apenz1
  Handling multi-input/output audio in python bor1904 4 3,498 Nov-04-2020, 08:25 AM
Last Post: CHLOVRL
  Python Requests package: Handling xml response soumyarani 1 2,104 Sep-14-2020, 11:41 AM
Last Post: buran
  Iterate 2 large text files across lines and replace lines in second file medatib531 13 5,707 Aug-10-2020, 11:01 PM
Last Post: medatib531
  Handling Multiple USB ports in Python samalpramod 0 4,366 Aug-01-2020, 07:40 PM
Last Post: samalpramod
  Iterating Large Files Robotguy 10 5,060 Jul-22-2020, 09:13 PM
Last Post: Gribouillis

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020