Python Forum
Parsing Xml files >3gb using lxml iterparse
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Parsing Xml files >3gb using lxml iterparse
#1
I am not able to parse XML file of huge size using lxml tree. What I came to know from my research is that lxml iterparse loads the xml file until it gets tag which it is looking for. This is snippet of my code :-

for event, child in etree.iterparse(xml_file,tag='test'):
        print(sys.getsizeof(child))
It is not even reaching print statement and is getting killed. I am running this on server. Any help on this matter?
Reply
#2
Please show enough of a a code snippet so it can be run.
attach a small sample of the XML file.
Reply
#3
for event, x in etree.iterparse(file,tag='status'):
		dict.update({'script_exec_end_time':x.attrib['endtime']})
		dict.update({'script_exec_start_time':x.attrib['starttime']})
		dict.update({'script_exec_duration':str((datetime.strptime(str(x.attrib['endtime']),'%Y%m%d %H:%M:%S.%f') - datetime.strptime(str(x.attrib['starttime']),'%Y%m%d %H:%M:%S.%f')).total_seconds())})
		dict.update({'script_result':x.attrib['status']})
		x.clear()
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  opening files and output of parsing leodavinci1990 4 2,526 Oct-12-2020, 06:52 AM
Last Post: bowlofred
  Parsing Attached .MSG Files with Python3 ericl42 1 3,668 Apr-12-2019, 06:28 PM
Last Post: ericl42
  lxml - etree/lxml need help storing variable for most inserted element goeb 0 2,554 Apr-01-2019, 03:09 AM
Last Post: goeb
  parsing local xml files to csv erdem_ustunmu 8 5,126 Feb-27-2019, 12:37 PM
Last Post: erdem_ustunmu

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020