KeyError issue - unable to find root cause

[simms7400]

Hi Folks -

I'm very new to python and I'm trying to muddle way through an issue I'm having but can't seem to figure it out.

I have the following code:

import requests
import sys,os
import datetime
import json,time
from functions import read_file,getdate
from functions import getcurrentdate,constructJson_Request
from functions import postURL,send_mail

workorder_list = read_file(str(sys.argv[1]))

currentdate = getcurrentdate()

config_filepath = os.getcwd()+"/configuration.json"
configuration = read_file(config_filepath,"configuration")

for req in workorder_list:
	send_to = str(req['emailid'])
    
	subject = "Request ID:"+req['workorderid']+" - Reminder 1"
	text = "Request ID:"+req['workorderid']+" - Reminder 1"
      
	try:
		resp=send_mail(configuration, send_to, subject, text)
		print("Mail Sent" + resp)
	except KeyError:
		print("Email Id does not exist")

But I'm getting the following error, even though I tried to code in an exception.

Schedule : Aging_Open_Requests executed, Result: Traceback (most recent call last):
File "date_diff.py", line 17, in <module>
send_to = str(req['emailid'])
KeyError: 'emailid'

Does this mean emailid does not exist in my functions.py?

Thanks for your help!!

[j.crater]

Hello, it's hard to tell what's wrong without knowing how your file looks like, or what read_file() does exactly.
Nonetheless, you can try to put a print in your for loop (print(req)) and maybe you will quickly find why (wrong format? not correctly read data?) you get the error.
Error suggest there is no 'emailid' key in the req dictionary.

[simms7400]

Hello, it's hard to tell what's wrong without knowing how your file looks like, or what read_file() does exactly.
Nonetheless, you can try to put a print in your for loop (print(req)) and maybe you will quickly find why (wrong format? not correctly read data?) you get the error.
Error suggest there is no 'emailid' key in the req dictionary.

Thank you for the reply!

So essentially, the argument is a report I run from a HelpDesk system which consists of three columns:

workorderid,emailid,last_tech_update

which the list is then passed to the python script.

[buran]

which the list is then passed to the python script

You need to pass list of dict, i.e req need to be dict. It almost certain the problem is with read_file function

[simms7400]

So i did print(req) and it's showing an invalid email address. Meaning the value for req is not a valid email address getting passed as 'emailid'. Is there a way to code in exceptions and ignore bad email addresses so the process doesn't bomb out?

---

hmmm - how do I make req a dictionary? Can someone assist?

Also, how do i show what 'emailid' is?

This doesn't work:

print(req['emailid'])

Thanks!

[buran]

show read_file function, everything else is just blind guessing. Several lines from the beginning of the file (incl. header, if there is one) would be helpful too. You can provide dummy data, but the structure should be the same as the real file

[simms7400]

Hi THere -

Will do - thank you!

The function is listed inside by function.py file:

# $Id$
#!/usr/bin/python
# -*- coding: utf-8 -*-
import smtplib
from smtplib import SMTPException
from os.path import basename
from email.mime.application import MIMEApplication
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.utils import COMMASPACE, formatdate
import json
import urllib
import requests
import datetime
import time
import os
import requests
import json
import operator
import os
import xml.etree.ElementTree as ET


# ------------------ Function to parse input from Request JSON file----------------------

def read_file(file_Path, key=None):
    with open(file_Path) as data_file:
        data = json.load(data_file)
    if key is None:
        return data
    else:
        dataObj = data[key]
        return dataObj


# ----------------------------  Construct Task Input Json  ----------------------------

# send data as jsonstring
# use json.dumps function to convert dict to jsonstring
def constructJson_Task(data, module, id):
    json_data = ''' {"task": {'''+module+''': { "id": ''' +id + ''' }, ''' + data + '''} }'''
    return json_data


# ----------------------------  Construct Worklog Input Json  ----------------------------

def constructJson_Worklog(data, module, id):
    json_data = ''' {"worklog": { '''+module+''': { "id": ''' +id + ''' }, ''' + data + '''} }'''
    return json_data


# ----------------------------  Construct Request Input Json  ----------------------------

def constructJson_Request(data):
    json_data = ''' {"operation": { "details": ''' + data + '''} }'''
    return json_data


# ----------------------------  Construct Change Input Json  ----------------------------

def constructJson_Change(data):
    json_data = ''' {"operation": { "details": ''' + data + '''} }'''
    return json_data


# ----------------------------  Construct Note Input Json  ----------------------------

def constructJson_Note(data):
    json_data = '''{ "operation": { "details": { "notes": { "note": '''+data+'''}}}}'''
    return json_data

def constructJson_shareRequest(groups=None,technicians=None,users=None,department=None,sites=None):
    tempDataObj={}
    if isinstance(groups, list):
        tempDataObj['groups']=getListAsDictionary(groups,"name")
    if isinstance(technicians, list):
        tempDataObj['technicians']=getListAsDictionary(technicians,"name")
    if isinstance(sites, list):
        tempDataObj['sites']=getListAsDictionary(sites,"name")
    if isinstance(users, list):
        tempDataObj['users']=getListAsDictionary(users,"name")
    if isinstance(department, list):
        tempDataObj['departments']=getListAsDictionary(department,"name")
    json_data = ''' {"share": ''' + json.dumps(tempDataObj) + '''}'''
    return json_data

# ------------------ Function to get the field details from the json -------------------

def getFieldDetails(responseObj, field):
    value = 'Error'
    status = responseObj['operation']['result']['status']
    if status == 'Success':
        for elem in responseObj['operation']['details']:
            if str(elem['NAME']) == field:
                value = elem['VALUE']
    return value


# ------------------ Function to send mail -------------------

# sent to and ccAdd should be commma seperated email strings
# configuration is a dictionary
def send_mail(configuration,subject,text, toAdd=None,ccAdd=None):
    server = configuration['server']
    port = configuration['port']
    username = configuration['username']
    password = configuration['password']
    fromAdd = configuration['send_from']
    
    if toAdd is None:
        send_to=configuration['toAddress']
    else:
        send_to=toAdd

    if isinstance(send_to,list):
        send_to = ",".join(send_to)
    if isinstance(ccAdd,list):
        ccAdd = ",".join(ccAdd)
    send_toList = send_to.split(',')
    if ccAdd is not None:
        ccAddList=ccAdd.split(',')
    else:
        ccAdd=''
        ccAddList=[]
    
    try:
        msg = MIMEMultipart()
        msg['From'] = fromAdd
        msg['To'] = send_to
        msg['CC'] = ccAdd
        msg['Date'] = formatdate(localtime=True)
        msg['Subject'] = subject
        msg.attach(MIMEText(text,'html'))

        send_toList = send_toList + ccAddList
        
        smtp = smtplib.SMTP(server, port)
        smtp.login(username, password) #uncomment if required
        
        smtp.sendmail(fromAdd, send_toList, msg.as_string())
        q = "Successfully sent email"
    except SMTPException as error:
        q = "Error: unable to send email :  {err}".format(err=error)
		#This message will appear in the Request History and can be modified based on requirement.
    finally:
        smtp.close()
    return q


# ------------------ Function to add approvers to a stage -------------------

def create_approval_JSON(operationName, operationJson,send_immediately="false"):
    operationJson["send_immediately"]=send_immediately
    resultjson = {}
    resultjson['operation'] = []
    resultjson['result'] = 'success'
    message = 'Sample Python script for ' + operationName
    resultjson['message'] = message
    operationJson['OPERATIONNAME'] = operationName
    resultjson['operation'].append(operationJson)
    return resultjson


def add_approval_stage(operationJson, stage, approversList):
    approvalArray = {stage: []}
    for approver in approversList:
        approvalArray[stage].append(approver)    
    operationJson['INPUT_DATA'].append(approvalArray)
    return operationJson

# ------------------ Function to get current date -------------------

def getcurrentdate(date_format="%Y-%m-%d"):
    now = datetime.datetime.now()
    currentdate = now.strftime(date_format)
    currentdate = datetime.datetime.strptime(currentdate, date_format)
    return currentdate


# ------------------ Function to get date -------------------

def getdate(date_param, date_format="%Y-%m-%d"):
    date = datetime.datetime.fromtimestamp(int(date_param)/1000)
    date = date.strftime(date_format)
    date = datetime.datetime.strptime(date, date_format)
    return date


# ------------- Function to add Parameter in CI request -------------

# Need to call like below.
# criteria = addParameter("CI Type","Windows Workstation","START WITH")

def addParameter(name,value,compOperator=None,relOperator=None):
    if compOperator:
        name = " compOperator=\'" + compOperator + "\'>"+ name
    else:
        name = ">" + name
    parameter = ''' <parameter>
                <name''' + name + '''</name>
                <value>''' + value + '''</value>
            </parameter>'''
    if relOperator:
        parameter = parameter + "<reloperator>" + relOperator +"</reloperator>"
    return parameter

# ------------------ Function to add CI -------------------

def addCI_inputData(parameter):
    inputdata = '''<API version='1.0' locale='en'> <records> <record>''' + parameter + '''</record> </records> </API>'''
    return inputdata

# ------------- Function to add Names in CI request -------------

# Need to call like below..
# returnparameters = addNames_CI(["CI Name", "CI Type"])

def addNames_CI(list):
    names = ""
    for item in list:
        names = names + "<name>" + item + "</name>"
    return names

# ------------------ Function to get CI -------------------

'''

Fetch CI details using Criteria.

# criteria = addParameter("CI Type","Windows Workstation","START WITH")
    # Refer above function syntax
# returnparameters = addNames_CI(["CI Name", "CI Type"])
    # Refer above function syntax
# sortparameters = addNames_CI(["CI Name","Site"])
    # Refer above function syntax
# sortOrder = "desc"
# startindex = "1"
# limit = "50"

'''

def getCI_inputData(criteria, returnparameters, sortparameters, sortOrder, startindex, limit):
    inputdata = ''' <API version='1.0' locale='en'> 
                <criterias> <criteria>''' + criteria + '''</criteria> </criterias> 
                <returnparameters>''' + returnparameters + '''</returnparameters> 
                <sortparameters sortOrder=\'''' + sortOrder + '''\'>''' + sortparameters + '''</sortparameters> 
                <range>
                    <startindex>''' + startindex + '''</startindex>
                    <limit>''' + limit + '''</limit>
                </range>
                </API>'''
    return inputdata

#---------------------------- Get Method -----------------------------------

def getURL(url, TechnicianKey, operationName=None, json_data=None, params_data=None):
    data = {
        'input_data': json_data,
        'TECHNICIAN_KEY': TechnicianKey,
        'format': 'json',
        'OPERATION_NAME': operationName,
        }
    params = {
        'input_data': params_data,
        'TECHNICIAN_KEY': TechnicianKey,
        'format': 'json',
        }
    with requests.Session() as s:
        return s.get(url,data= data,params= params)

# ----------------------------  Construct Attachment Input Json  ----------------------------

def constructJson_Attachment(module, id):
    json_data = {"attachment": {module: { "id": str(id) } } }
    return json.dumps(json_data)


# ----------------------------  Delete All Attachments ----------------------------
def deleteAttachments(configuration,module,id):
    url = configuration['url'] + "/api/v3/attachments"

    json_data = constructJson_Attachment(module,id)

    r = getURL(url, configuration['technicianKey'], None,None, json_data)
    attachmentresponse = r.json()   

    try:
        if("attachments" in attachmentresponse ):
            attachmentData = attachmentresponse['attachments']
            for attachment in attachmentData:
                url = configuration['url'] + "/api/v3/attachments/" + attachment['id'] 
                deleteURL(url,configuration['technicianKey'])
            print("Successfully Deleted all attachments!")
    except:
        print("Deleting attachment failed!")
        


# ----------------------------  Delete Method  ----------------------------

def deleteURL(url, TechnicianKey):
    data = {
        'TECHNICIAN_KEY': TechnicianKey,
        'format': 'json',
        }
    with requests.Session() as s:
        return s.delete(url,data=data)



#----------------------------- To bundle together attachments ------------------------


# fileNamesList = ['test1.txt', 'test2.txt']
# multiple_files = constructFiles_Attachment(fileNamesList)
# querystring = constructJson_Attachment(module,id)
# return postURL(url,TechnicianKey, params_data = querystring,files=multiple_files)

def constructFiles_Attachment(fileNamesList):
	multiple_files = []
	temp = 0
	for fileName in fileNamesList:
			if(os.path.isfile(fileName)):
				file = open(fileName, "rb")
				fileNameArr = fileName.split("\\")
				fileName = fileNameArr[len(fileNameArr)-1]
				obj =(fileName, file, 'text/plain')
				multiple_files.append(("file"+str(temp), obj))
				temp = temp+1
			else:
				print(fileName + " file not present")
	return multiple_files

# ----------------------------  Post Method  ----------------------------


# Function to send post http connection to the given url

# url  : Api url
# TechnicianKey 
# operationName : Specifies the operation name like ADD_REQUEST. Not mandatory
# json_data : Specifies the data in the request body. Not mandatory
# params_data : Invoke API call with input data in the request url params. Not mandatory
# files  : Related to attachment file data. Not mandatory

# Invoke be like
#response = postURL(baseURL, TechnicianKey, params_data=json_data)

def postURL(url, TechnicianKey, operationName=None, json_data=None, params_data=None, files=None):
    data = {
        'INPUT_DATA': json_data,
        'TECHNICIAN_KEY': TechnicianKey,
        'format': 'json',
        'OPERATION_NAME': operationName,
        }
    params = {
        'INPUT_DATA': params_data,
        'TECHNICIAN_KEY': TechnicianKey,
        'format': 'json',
        }
    with requests.Session() as s:
        return s.post(url,data=data, params=params, files=files,verify=False)

# ----------------------------  Put Method  ----------------------------


# Function to send post http connection to the given url

# url  : Api url
# TechnicianKey 
# operationName : Specifies the operation name like ADD_REQUEST. Not mandatory
# json_data : Specifies the data in the request body. Not mandatory
# params_data : Invoke API call with input data in the request url params. Not mandatory
# files  : Related to attachment file data. Not mandatory

# Invoke be like
#response = putURL(baseURL, TechnicianKey, params_data=json_data)

def putURL(url, TechnicianKey, operationName=None, json_data=None, params_data=None, files=None):
    data = {
        'INPUT_DATA': json_data,
        'TECHNICIAN_KEY': TechnicianKey,
        'format': 'json',
        'OPERATION_NAME': operationName,
        }
    params = {
        'INPUT_DATA': params_data,
        'TECHNICIAN_KEY': TechnicianKey,
        'format': 'json',
        }
    with requests.Session() as s:
        return s.put(url,data=data, params=params, files=files,verify=False)


 # ----------------------------  checkConditions Method  ----------------------------

# Function to check if all the conditions are satisfied

# Call like below:
#   cond_filepath = os.getcwd()+"/conditions.json"
#   conditions = read_file(cond_filepath,"conditions")

# Sample conditions will be : 
#   {"conditions":[{"name":"PRIORITY","value":"High","connector":"and"},{"name":"SLA","value":"High SLA","connector":"and"},{"name":"DESCRIPTION","value":"how","connector":"and"},{"name":"Approval_Code","value":"ADVC","connector":""}]}

# Invoke condition check will be 
#   result = checkConditions(requestObj, conditions)


def checkConditions(requestObj, conditions):
    # Define bitwise operators
    operators = {
        "and": operator.and_,
        "or": operator.or_
    }

    # Initialize Values
    connector = "and"
    finalResult = True

    # loop through each condition and update finalResult
    for cond in conditions:
        name = cond["name"]
        value = cond["value"]
        conn = cond["connector"]
        if isinstance(requestObj[name], list):
            requestObj[name] = map(str.lower,requestObj[name])
        else:
            requestObj[name] = requestObj[name].lower()
        result = (value.lower() in requestObj[name])
        finalResult = operators.get(connector)(finalResult,result)
        connector = conn
    # return final Result
    return finalResult

# ----------------------------  Handling Responses ----------------------------

# support below url calls and return true/false boolean value.
# sdpapi
# api/v3
# api/cmdb 

def getResponseStatus(url,response):
    responseobj=response.json()
    returnString = ""
    if "api/v3" in url:
        status=responseobj['response_status']['status']
        if status == 'success':
            returnString = 'Success : '
        else :
            returnString = 'Failed : '
        return returnString + responseobj['response_status']['messages'][0]['message']
    elif "sdpapi" in url:
        status=responseobj['operation']['result']['status']
        if status == 'Success':
            returnString = 'Success : '
        else :
            returnString = 'Failed : '
        return returnString + responseobj['operation']['result']['message']
    elif "api/cmdb" in url:
        status=responseobj['API']['response']['operation']['result']['status']
        if status == 'Success':
            returnString = 'Success : '
        else :
            returnString = 'Failed : '
        return returnString + responseobj['API']['response']['operation']['result']['message']
    else :
        return responseobj


#-------------Method for reading Configurations from xml file ----------------

# Root tag named 'configuration is ignored'

def readXMLFile(xmlFile):
    tree=ET.parse(xmlFile)
    root=tree.getroot()    
    configuration={}
    for element in root.iter():
        if element.tag=='configuration':
            continue
        configuration[element.tag]=element.text           
    return configuration

#------------ Constructing the Json Object for updating the request.---------

# data is Json that will have the Field Name/Value that needs to be updated in the request
# data = {"LEVEL":"TIER 1","PRIORITY":"High","IMPACT":"High"}
# Need to pass the data to actionPlugin_constructReqJSON to vet the full json

def actionPlugin_UpdateRequest(data,OperationName="EDIT_REQUEST",module=None,additionalParams=""):
	if module is not None:
		temp={}
		temp[module]=json.loads(data)
		tempString=json.dumps(temp)
		data=tempString
	json_data = '''{
		"INPUT_DATA": [''' + data +	'''],
		"OPERATIONNAME": "''' + OperationName + '''",
        ''' + additionalParams + '''
	},'''
	return json_data

#------------ Constructing the Json Object Add Notes.---------

# notestext in string and ispublic is optional 
# Need to pass the data to actionPlugin_constructReqJSON to vet the full json

def actionPlugin_AddNote(notestext, ispublic="false"):
	json_data = '''{
		"INPUT_DATA": [{
			"notes": {
				"notestext": ''' + notestext + ''',
                "ispublic": ''' + str(ispublic) + '''
			}
		}],
		"OPERATIONNAME": "ADD_NOTE"
	},'''
	return json_data

#------------ Constructing the Json for default return functionality.---------

# Use a combination of above two functions to construct the data parameter
# data = actionPlugin_AddNote("Ticket has been created in JIRA and information populated in SDP")

def actionPlugin_constructReqJSON(data, message="Request Updated Successfully"):
	json_data = '''{
			"message":"''' +message + '''",
			"result":"success",
			"operation":[''' + data + ''']
			}'''  
	return json_data

# ------------------ Function to get date as string -------------------

def getdatestring(unixtime, date_format='%d %b %Y, %H:%M:%S'):
    time = int(unixtime)
    if time > 0:
        date = datetime.datetime.fromtimestamp(time/1000)
        date = date.strftime(date_format)
    else: 
        date = 'N/A'
    return date

# -----------Function to replace placeholders from a mapping Dictionary------

# output_data is a string
# mappingDict is a dictionary. the key is replaced by value in output_data
# use convertdates option to convert date fields to string

def handlePlaceHolder (mappingDict, output_data, convertdates=False):
    dateFieldList=['CREATEDTIME', 'DUEBYTIME', 'RESPONDEDTIME', 'RESOLVEDTIME', 'COMPLETEDTIME']
    for elem in mappingDict.keys():
        if(str('$'+elem+'$') in output_data):
            if (convertdates == True) and (elem in dateFieldList):
                output_data=output_data.replace('$'+elem+'$',str(getdatestring(mappingDict[elem])))
            else:
                output_data=output_data.replace('$'+elem+'$',str(mappingDict[elem]))
        if 'resource' in mappingDict:
            for resources in mappingDict['resource'].keys():
                for questions in mappingDict['resource' ][resources].keys():
                    if str('$' + questions + '$') in output_data:
                        if mappingDict['resource'][resources][questions]:
                            answer = mappingDict['resource'][resources][questions]
                        else:										
                            answer = 'No Answer'
                        output_data = output_data.replace('$'+ questions + '$', str(answer))
    return output_data

# --------- Function to Get all Tasks ------------------------

# serverURL = 'http://localhost'
# reqID = 40
# technicialKey = 'KEY'

def getAllTasks(serverURL, reqID, technicianKey):
    r = getURL(serverURL + '/api/v3/request/' + reqID
            + '/tasks', technicianKey)
    taskData = r.json()
    if r.status_code == 200:
        
        if taskData['list_info']['has_more_rows'] is True:
            row_count = taskData['list_info']['total_count']
            jsonData = {
                "list_info": {
                    "row_count": row_count,
                    "fields_required": "[get_all]"
                }
            }
            
            lr = getURL(serverURL + '/api/v3/request/' + reqID + '/tasks', technicianKey, params_data=jsonData)

            if lr.status_code == 200:
                taskData = lr.json()
    return taskData


#-------------- Function to get Requester Email ---------------------

def getReqeusterMail(workorderid,appurl,apiKey):

    url = appurl
    TechnicianKey= apiKey
    OperationName='GET_REQUEST_FIELDS'
    REQUESTEREMAIL='[email protected]'

    #Preparing the API call and submitting it.
    apprUrl = url + "/sdpapi/request/" + str(workorderid)
    r = postURL(apprUrl,TechnicianKey,OperationName)

    if(r.status_code == 200):

        responseObj=r.json()
        status=responseObj['operation']['result']['status']
        message=responseObj['operation']['result']['message']

        if status == 'Success':
            for elem in responseObj['operation']['details']:
                if(str(elem['NAME']) == 'REQUESTEREMAIL'):
                    REQUESTEREMAIL=elem['VALUE']
        else:
            print(status)
            print(message)
    else:
        print('Error: ' + str(r.status_code))
                
    return REQUESTEREMAIL

#----------- Function to change status using action plugin ---------------
# Will close the status if status s not passed

def updateReqStatus_JSON(status):
    inputJson={}
    inputJson["STATUS"]=status
    updatejson = actionPlugin_UpdateRequest(json.dumps(inputJson))
    returnJson = actionPlugin_constructReqJSON(updatejson,"Successfully Updated Request")
    print(returnJson)
    return returnJson

#----------- Function to change status using API call -------------------
# Will close the status if status s not passed

def updateReqStatus_API(url,technicianKey,status):
    json_data = constructJson_Request('{"technician":"'+status+'"}')
    r = postURL(url, technicianKey,"EDIT_REQUEST", json_data)
    return getResponseStatus(url,r)
def getValueFromDiffJSON(diffJSON,key):
    if key in diffJSON:
        return diffJSON[key]['OLD']
    else:
        print("No value present in Diff JSON")

Let me know - thank you!

[buran]

So essentially, the argument is a report I run from a HelpDesk system which consists of three columns:

Quote:
workorderid,emailid,last_tech_update

well, that is strange....
you claim the file consists of three columns, which suggests csv file based on the header you show.
at the same time you try to load it as json file in the read_file function. I would expect you will get JSONDecoderError long before getting to KeyError.
What is the file actual file format? That is why I ask to see sample of the file. Help us to help you, don't make us ask same thing over and over again

[simms7400]

HI Braun -

Well the file can be downloaded as html, pdf or csv. However, whent he automation runs it, I'm not sure the technical name and in what format. I've attached a picture of what the file looks like when run the query runs.



The entire output of that report is passed as an argument to the python script.
Please let me know if that works.

[buran]

So I assume you download it as csv and then pass the path/filename as argument
read_file function will not work in this case because it expects json, not csv file
so best is to use csv module

import requests
import sys,os
import datetime
import json,time
from functions import read_file,getdate
from functions import getcurrentdate,constructJson_Request
from functions import postURL,send_mail
import csv
 
currentdate = getcurrentdate()
 
config_filepath = os.getcwd()+"/configuration.json"
configuration = read_file(config_filepath,"configuration")

with open(sys.argv[1]) as workorder_file:
    rdr = csv.DictReader(workorder_file)
    for req in rdr:
        send_to = req['emailid']
        subject = "Request ID:" + req['workorderid'] + " - Reminder 1"
        text = "Request ID:" + req['workorderid'] + " - Reminder 1"
        resp = send_mail(configuration, send_to, subject, text)
        print("Mail Sent" + resp)