Python Forum
Cant set api response as tree in Element tree - Printable Version

+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Web Scraping & Web Development (https://python-forum.io/forum-13.html)
+--- Thread: Cant set api response as tree in Element tree (/thread-11990.html)

Cant set api response as tree in Element tree - hey_arnold - Aug-03-2018

Hi All,

I can't get the response from the api to be recognised if I try and define tree. Any suggestions would be much appreciated.

import requests
import xml.etree.ElementTree as ET

def getXML():
    url="http://energywatch.natgrid.co.uk/EDP-PublicUI/PublicPI/InstantaneousFlowWebService.asmx"
    
    headers = {'content-type': 'application/soap+xml; charset=utf-8'}
    body ="""<soap12:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap12="http://www.w3.org/2003/05/soap-envelope">
                <soap12:Body>
                <GetInstantaneousFlowData xmlns="http://www.NationalGrid.com/EDP/UI/" />
                </soap12:Body>
                </soap12:Envelope>"""

    response = requests.post(url,data=body,headers=headers)
    return response.content
    
    tree = ET.XML(response) 
    root = tree.getroot()
    root.tag
    print(root.tag)

    for EDPObjectName in root.iter('EDPObjectName'):
        print(EDPObjectName.text)
I can't work out what I'm doing wrong, because If I download the XML file then run the following code, it works:

import xml.etree.ElementTree as ET
tree = ET.parse("C:\\Users\\Hey Arnold\\Documents\\Python Scripts\\new1.xml")

root = tree.getroot()
root.tag
print(root.tag)

for EDPObjectName in root.iter('EDPObjectName'):
    print(EDPObjectName.text)


RE: Cant set api response as tree in Element tree - snippsat - Aug-03-2018

All of code is inside the function then after return nothing happens with lines 17-23.
Here a fix an i use BS with lxml as parser,i never use parser in standard library.
import requests
from bs4 import BeautifulSoup

def get_xml():
    url="http://energywatch.natgrid.co.uk/EDP-PublicUI/PublicPI/InstantaneousFlowWebService.asmx"
    headers = {'content-type': 'application/soap+xml; charset=utf-8'}
    body ="""<soap12:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap12="http://www.w3.org/2003/05/soap-envelope">
                <soap12:Body>
                <GetInstantaneousFlowData xmlns="http://www.NationalGrid.com/EDP/UI/" />
                </soap12:Body>
                </soap12:Envelope>"""

    response = requests.post(url,data=body,headers=headers)
    return response.content

response = get_xml()
soup = BeautifulSoup(response, 'lxml')
for item in soup.find_all('EDPObjectName'.lower()):
    print(item.text)
Output:
ALDBROUGH
AVONMOUTH
BACTON BBL
BACTON IC
BACTON PERENCO
BACTON SEAL
BACTON SHELL
BARROW SOUTH
DYNEVOR ARMS
EASINGTON DIMLINGTON
EASINGTON LANGELED
EASINGTON ROUGH ST
.......


RE: Cant set api response as tree in Element tree - hey_arnold - Aug-03-2018

Thanks very much, but is it possible with element tree. It would be great to know what i was doing wrong.

RE: Cant set api response as tree in Element tree - hey_arnold - Aug-04-2018

OK, I think I'm close but no cigar.....yet! Does it look like I have the correct namespace root xpath etc? 

import requests
from lxml import etree
    
fromDate = "2018-07-29"

def getXML():
    url="http://energywatch.natgrid.co.uk/EDP-PublicUI/PublicPI/InstantaneousFlowWebService.asmx"
    headers = {'content-type': 'application/soap+xml; charset=utf-8'}
    body ="""<soap12:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap12="http://www.w3.org/2003/05/soap-envelope">
  <soap12:Body>
    <GetInstantaneousFlowData xmlns="http://www.NationalGrid.com/EDP/UI/" />
  </soap12:Body>
</soap12:Envelope>"""

    response = requests.post(url,data=body,headers=headers)
    return response.content

import pandas as pd
df1 = pd.DataFrame(columns=("applicable_at","name","value","created_date"))   
for pd_date in pd.date_range(fromDate, periods=1):
    day = pd_date.strftime('%Y-%m-%d')
     
    root = etree.fromstring(getXML())
    #map prefix 'd' to the default namespace URI
    ns = {'d': 'http://www.nationalgrid.com/EDP/BusinessEntities/Public'}
     
    publication_objects = root.xpath('//d:EDPObjectCollection', namespaces=ns)
     
    for obj in publication_objects:
        name = obj.find('d:EDPObjectName', ns).text
     
        for data in obj.findall('d:EnergyDataList/d:EDPEnergyDataBE', ns):
            applicable_at = pd.to_datetime(data.find('d:ApplicableAt', ns).text)    
            value = float(data.find('d:FlowRate', ns).text)
            created_date = pd.to_datetime(data.find('d:ScheduleTime', ns).text)

        
         
            df1.loc[len(df1) +1] = [applicable_at,name, value,created_date]


RE: Cant set api response as tree in Element tree - dsarin - Mar-04-2019

hey_arnold, Have you managed to find a solution?