Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 XML parsing from URL
#1
Hello I started my trek into Python a few days ago.

I am receiving the following error:
Quote:Please enter an XML URL to parse: http://py4e-data.dr-chuck.net/comments_42.xml
Traceback (most recent call last):
File "/home/lamidotijjo/Documents/PythonProjects/parsexml.py", line 46, in <module>
tree = ET.parse(url_to_open)
File "/usr/lib/python3.6/xml/etree/ElementTree.py", line 1196, in parse
tree.parse(source, parser)
File "/usr/lib/python3.6/xml/etree/ElementTree.py", line 586, in parse
source = open(source, "rb")
OSError: [Errno 36] File name too long

I am currently working on the Python for Everybody specialization. I thought that my source code was fine, here it is

# Extracting Data from XML

# In this assignment you will write a Python program somewhat similar to http://www.py4e.com/code3/geoxml.py. 
# The program will prompt for a URL, read the XML data from that URL using urllib and then parse and extract 
# the comment counts from the XML data, compute the sum of the numbers in the file.

# We provide two files for this assignment. One is a sample file where we give you the sum for your testing and the 
# other is the actual data you need to process for the assignment.

# Sample data: http://py4e-data.dr-chuck.net/comments_42.xml (Sum=2553)
# Actual data: http://py4e-data.dr-chuck.net/comments_156437.xml (Sum ends with 70)
# You do not need to save these files to your folder since your program will read the data directly from the URL. 
# Note: Each student will have a distinct data url for the assignment - so only use your own data url for analysis.

# Data Format and Approach
# The data consists of a number of names and comment counts in XML as follows:

# <comment>
#   <name>Matthias</name>
#   <count>97</count>
# </comment>
# You are to look through all the <comment> tags and find the <count> values sum the numbers. The closest sample code that shows 
# how to parse XML is geoxml.py. But since the nesting of the elements in our data is different than the data we are parsing in 
# that sample code you will have to make real changes to the code.

# To make the code a little simpler, you can use an XPath selector string to look through the entire tree 
# of XML for any tag named 'count' with the following line of code:

# counts = tree.findall('.//count')
# Take a look at the Python ElementTree documentation and look for the supported XPath syntax for details. 
# You could also work from the top of the XML down to the comments node and then loop through the child nodes of the comments node.
# Sample Execution

# $ python3 solution.py
# Enter location: http://py4e-data.dr-chuck.net/comments_42.xml
# Retrieving http://py4e-data.dr-chuck.net/comments_42.xml
# Retrieved 4189 characters
# Count: 50
# Sum: 2...

import urllib.parse, urllib.request, urllib.error
import xml.etree.ElementTree as ET

url = input('Please enter an XML URL to parse: ')
url_to_open = urllib.request.urlopen(url).read()
tree = ET.parse(url_to_open)
root_node = tree.getroot()

lst = root_node.findall('comment')
for item in lst:
    print(item)



Why am I getting that error? Thanks!
Quote
#2
Your url_to_parse holds the contents of the xml file, and .parse() expects a path or an open file.
You should either pass the response object to .parse() (and not the data read from it), or use .fromstring() instead.
Quote
#3
My take on this is that you should drop urllib and ElementTree all together.
So what to use insted?
For reading url and all other HTTP work use Requests.
For parsing lxml and BeautifulSoup.
Have tutorial here.

A example,solve first task.
lxml:
from lxml import html
import requests

url = 'http://py4e-data.dr-chuck.net/comments_42.xml'
response = requests.get(url)
tree = html.fromstring(response.content)
count = tree.xpath('//count')
total =  sum(int(i.text) for i in count)
print(f'The sum of all count is doc is: {total}')
Output:
The sum of all count in doc are: 2553
BeautifulSoup:
from bs4 import BeautifulSoup
import requests

url = 'http://py4e-data.dr-chuck.net/comments_42.xml'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'lxml')
count = soup.find_all('count')
total =  sum(int(i.text) for i in count)
print(f'The sum of all count in doc are: {total}')
Output:
The sum of all count in doc are: 2553
Quote
#4
(Nov-21-2018, 07:37 AM)stranac Wrote: Your url_to_parse holds the contents of the xml file, and .parse() expects a path or an open file.
You should either pass the response object to .parse() (and not the data read from it), or use .fromstring() instead.

Despite this, I am running into a conundrum:

Quote:Please enter an XML URL to parse: http://py4e-data.dr-chuck.net/comments_42.xml
Traceback (most recent call last):
File "/home/lamidotijjo/Documents/PythonProjects/parsexml.py", line 48, in <module>
tree = ET.parse(url)
File "/usr/lib/python3.6/xml/etree/ElementTree.py", line 1196, in parse
tree.parse(source, parser)
File "/usr/lib/python3.6/xml/etree/ElementTree.py", line 586, in parse
source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: 'http://py4e-data.dr-chuck.net/comments_42.xml'

Here is the modified code:
import urllib.parse, urllib.request, urllib.error
import xml.etree.ElementTree as ET

list_of_ints = []
url = input('Please enter an XML URL to parse: ')
# url_to_open = urllib.request.urlopen(url).read()

tree = ET.parse(url)
root = tree.getroot()
lst = root.findall('comment')
for item in lst:
   print(item.find('count'))
Quote
#5
Nevermind, I figured it out. Thanks!
Quote
#6
mightyn00b - Please take a look at snippsat's post. If you follow his advise, you will be very pleased.
lxml has a lot to offer above and beyond ElementTree, and it also has all of the features of ElementTree as well.
For a few minutes of investigation and trying it out you will be surprised what you will gain.
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  Parsing infor from scraped files. Larz60+ 2 317 Apr-12-2019, 05:06 PM
Last Post: Larz60+
  Fetching and Parsing XML Data FalseFact 3 347 Apr-01-2019, 10:21 AM
Last Post: Larz60+
  Selenium Parsing (unable to Parse page after loading) oneclick 6 633 Jan-13-2019, 03:10 AM
Last Post: oneclick
  XML Parsing - Find a specific text (ElementTree) TeraX 3 632 Oct-09-2018, 09:06 AM
Last Post: TeraX
  XML parsing and generating HTML page Python 3.6 Madhuri 2 577 Aug-24-2018, 02:48 PM
Last Post: snippsat
  Problem parsing website html file thefpgarace 2 741 May-01-2018, 11:09 AM
Last Post: Standard_user
  parsing table ian 10 3,368 Apr-27-2018, 01:05 PM
Last Post: snippsat
  beautiful soup - parsing scraped code in a script lilbigwill99 2 653 Mar-09-2018, 04:10 PM
Last Post: lilbigwill99
  BeautifulSoup Parsing Error slinkplink 6 3,915 Feb-12-2018, 02:55 PM
Last Post: seco
  parsing comment tag ian 2 761 Jan-22-2018, 03:38 AM
Last Post: Larz60+

Forum Jump:


Users browsing this thread: 1 Guest(s)