Posts: 6
Threads: 5
Joined: Oct 2021
Hello to All! I hope someone could help figure out how to get this one working. I have a AWS Lambda function that is triggered by a json fill upload to an s3 bucket. The lambda function job is to take and read the json fill and insert the data in the json fill in to dynamoDB table. This is my code for the lambda function:
import pandas as pd
import json
#from io import BytesIO
import boto3
import ast
s3_client = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('tbjson2dataframe')
def lambda_handler(event, context):
bucket = event['Records'][0]['s3']['bucket']['name']
json_file_name = event['Records'][0]['s3']['object']['key']
print(bucket)
print(json_file_name)
print(str(event))
json_object = s3_client.get_object(Bucket=bucket,Key=json_file_name)
jsonFileReader = json_object['Body'].read().decode("utf-8")
#now changing json string to dictionary
jsonFileReader = ast.literal_eval(jsonFileReader)
#df = pd.read_json(BytesIO(b'{"col": [1, 2, 3]}'))
#df = pd.DataFrame(jsonFileReader)
table.put_item(Item=jsonFileReader)and this this the error message I'm getting:
Error: [ERROR] ParamValidationError: Parameter validation failed:
Invalid type for parameter Item, value: [{'Industry name ': 'Industry name ', 'Number of firms ': 'Number of firms ', 'Inventory/Sa les ': 'Inventory/Sa les ', 'Inventory/Enterprise Value ': 'Inventory/Enterprise Value ', 'Number of Days Sales in Inventory ': 'Number of Days Sales in Inventory '}, {'Industry name ': 'Advertising ', 'Number of firms ': '65 ', 'Inventory/Sa les ': '4.53% ', 'Inventory/Enterprise Value ': '2.37% ', 'Number of Days Sales in Inventory ': '16.52 '}, {'Industry name ': 'Aerospace/Defense ', 'Number of firms ': '95 ', 'Inventory/Sa les ': '20.22% ', 'Inventory/Enterprise Value ': '13.26% ', 'Number of Days Sales in Inventory ': '73.82 '}, {'Industry name ': 'Air Transport ', 'Number of firms ': '25 ', 'Inventory/Sa les ': '2.74% ', 'Inventory/Enterprise Value ': '2.59% ', 'Number of Days Sales in Inventory ': '9.98 '}, {'Industry name ': 'Apparel ', 'Number of firms ': '70 ', 'Inventory/Sa les ': '18.00% ', 'Inventory/Enterprise Value ': '8.09% ', 'Number of Days Sales in Inventory ': '65.72 '}, {'Industry name ': 'Auto & Truck ', 'Number of firms ': '26 ', 'Inventory/Sa les ': '8.08% ', 'Inventory/Enterprise Value ': '8.97% ', 'Number of Days Sales in Inventory ': '29.49 '}, {'Industry name ': 'Auto Parts ', 'Number of firms ': '75 ', 'Inventory/Sa les ': '8.06% ', 'Inventory/Enterprise Value ': '9.34% ', 'Number of Days Sales in Inventory ': '29.40 '}, {'Industry name ': 'Bank ', 'Number of firms ': '7 ', 'Inventory/Sa les ': '4.31% ', 'Inventory/Enterprise Value ': '0.83% ', 'Number of Days Sales in Inventory ': '15.71 '}, {'Industry name ': 'Banks (Regional) ', 'Number of firms ': '721 ', 'Inventory/Sa les ': '6.12% ', 'Inventory/Enterprise Value ': '1.21% ', 'Number of Days Sales in Inventory ': '22.34 '}, {'Industry name ': 'Beverage ', 'Number of firms ': '47 ', 'Inventory/Sa les ': '6.10% ', 'Inventory/Enterprise Value ': '2.02% ', 'Number of Days Sales in Inventory ': '22.28 '}, {'Industry name ': 'Beverage (Alcoholic) ', 'Number of firms ': '19 ', 'Inventory/Sa les ': '33.20% ', 'Inventory/Enterprise Value ': '7.07% ', 'Number of Days Sales in Inventory ': '121.18 '}, {'Industry name ': 'Biotechnology ', 'Number of firms ': '349 ', 'Inventory/Sa les ': '12.11% ', 'Inventory/Enterprise Value ': '1.13% ', 'Number of Days Sales in Inventory ': '44.20 '}, {'Industry name ': 'Broadcasting ', 'Number of firms ': '30 ', 'Inventory/Sa les ': '1.38% ', 'Inventory/Enterprise Value ': '0.33% ', 'Number of Days Sales in Inventory ': '5.04 '}, {'Industry name ': 'Brokerage & Investment Banking ', 'Number of firms ': '49 ', 'Inventory/Sa les ': '0.04% ', 'Inventory/Enterprise Value ': '0.01% ', 'Number of Days Sales in Inventory ': '0.15 '}, {'Industry name ': 'Building Materials ', 'Number of firms ': '37 ', 'Inventory/Sa les ': '11.46% ', 'Inventory/Enterprise Value ': '8.34% ', 'Number of Days Sales in Inventory ': '41.84 '}, {'Industry name ': 'Business & Consumer Services ', 'Number of firms ': '179 ', 'Inventory/Sa les ': '1.96% ', 'Inventory/Enterprise Value ': '1.16% ', 'Number of Days Sales in Inventory ': '7.16 '}, {'Industry name ': 'Cable TV ', 'Number of firms ': '16 ', 'Inventory/Sa les ': '0.62% ', 'Inventory/Enterprise Value ': '0.23% ', 'Number of Days Sales in Inventory ': '2.26 '}, {'Industry name ': 'Chemical (Basic) ', 'Number of firms ': '47 ', 'Inventory/Sa les ': '14.53% ', 'Inventory/Enterprise Value ': '8.83% ', 'Number of Days Sales in Inventory ': '53.03 '}, {'Industry name ': 'Chemical (Diversified) ', 'Number of firms ': '10 ', 'Inventory/Sa les ': '16.57% ', 'Inventory/Enterprise Value ': '10.98% ', 'Number of Days Sales in Inventory ': '60.50 '}, {'Industry name ': 'Chemical (Specialty) ', 'Number of firms ': '100 ', 'Inventory/Sa les ': '13.41% ', 'Inventory/Enterprise Value ': '6.18% ', 'Number of Days Sales in Inventory ': '48.95 '}, {'Industry name ': 'Coal & Related Energy ', 'Number of firms ': '45 ', 'Inventory/Sa les ': '9.68% ', 'Inventory/Enterprise Value ': '5.92% ', 'Number of Days Sales in Inventory ': '35.32 '}, {'Industry name ': 'Computer Services ', 'Number of firms ': '129 ', 'Inventory/Sa les ': '5.47% ', 'Inventory/Enterprise Value ': '4.84% ', 'Number of Days Sales in Inventory ': '19.96 '}, {'Industry name ': 'Computer Software ', 'Number of firms ': '273 ', 'Inventory/Sa les ': '1.84% ', 'Inventory/Enterprise Value ': '0.40% ', 'Number of Days Sales in Inventory ': '6.72 '}, {'Industry name ': 'Computers/Peripherals ', 'Number of firms ': '66 ', 'Inventory/Sa les ': '4.01% ', 'Inventory/Enterprise Value ': '2.00% ', 'Number of Days Sales in Inventory ': '14.65 '}, {'Industry name ': 'Construction ', 'Number of firms ': '18 ', 'Inventory/Sa les ': '13.32% ', 'Inventory/Enterprise Value ': '4.20% ', 'Number of Days Sales in Inventory ': '48.60 '}, {'Industry name ': 'Diversified ', 'Number of firms ': '20 ', 'Inventory/Sa les ': '11.84% ', 'Inventory/Enterprise Value ': '2.91% ', 'Number of Days Sales in Inventory ': '43.21 '}, {'Industry name ': 'Educational Services ', 'Number of firms ': '40 ', 'Inventory/Sa les ': '1.07% ', 'Inventory/Enterprise Value ': '0.85% ', 'Number of Days Sales in Inventory ': '3.92 '}, {'Industry name ': 'Electrical Equipment ', 'Number of firms ': '135 ', 'Inventory/Sa les ': '10.41% ', 'Inventory/Enterprise Value ': '4.91% ', 'Number of Days Sales in Inventory ': '37.99 '}, {'Industry name ': 'Electronics ', 'Number of firms ': '191 ', 'Inventory/Sa les ': '15.23% ', 'Inventory/Enterprise Value ': '9.50% ', 'Number of Days Sales in Inventory ': '55.60 '}, {'Industry name ': 'Electronics (Consumer & Office) ', 'Number of firms ': '26 ', 'Inventory/Sa les ': '8.03% ', 'Inventory/Enterprise Value ': '7.90% ', 'Number of Days Sales in Inventory ': '29.29 '}, {'Industry name ': 'Engineering ', 'Number of firms ': '56 ', 'Inventory/Sa les ': '0.99% ', 'Inventory/Enterprise Value ': '1.66% ', 'Number of Days Sales in Inventory ': '3.63 '}, {'Industry name ': 'Entertainment ', 'Number of firms ': '85 ', 'Inventory/Sa les ': '3.66% ', 'Inventory/Enterprise Value ': '1.22% ', 'Number of Days Sales in Inventory ': '13.35 '}, {'Industry name ': 'Environmental & Waste Services ', 'Number of firms ': '108 ', 'Inventory/Sa les ': '1.46% ', 'Inventory/Enterprise Value ': '0.67% ', 'Number of Days Sales in Inventory ': '5.33 '}], type: <class 'list'>, valid types: <class 'dict'>
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 30, in lambda_handler
table.put_item(Item=jsonFileReader)
File "/var/runtime/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/var/runtime/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(*args, **params)
File "/var/runtime/botocore/client.py", line 391, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 691, in _make_api_call
request_dict = self._convert_to_request_dict(
File "/var/runtime/botocore/client.py", line 739, in _convert_to_request_dict
request_dict = self._serializer.serialize_to_request(
File "/var/runtime/botocore/validate.py", line 360, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
The json fill is the result of extracting it from the attached PDF fill.
Posts: 584
Threads: 1
Joined: Aug 2019
(Dec-25-2022, 03:43 AM)gdbengo Wrote: Error: Traceback (most recent call last): File "/var/task/lambda_function.py", line 30, in lambda_handler
table.put_item(Item=jsonFileReader)
So the error occurs in line 30.
The error is:
Error: type: <class 'list'>, valid types: <class 'dict'>
The data starts with:
Error: Invalid type for parameter Item, value: [{'Industry name ' ...
The data starts with a sqare bracket, so it is a list. Then there is a curly brace, so it is a list of dictionaries.
To provide a dictionary to the function, line 30 must be changed to:
table.put_item(Item=jsonFileReader[0]) This will only handle the first dictionary.
If you want to handle all the data, you must iterate over the list.
for mydict in jsonFileReader:
table.put_item(Item=mydict)I cannot test this myself, so I am curious if my analysis is right.
Posts: 6
Threads: 5
Joined: Oct 2021
(Dec-25-2022, 03:43 AM)gdbengo Wrote: Hello to All! I hope someone could help figure out how to get this one working. I have a AWS Lambda function that is triggered by a json fill upload to an s3 bucket. The lambda function job is to take and read the json fill and insert the data in the json fill in to dynamoDB table. This is my code for the lambda function:
import pandas as pd
import json
#from io import BytesIO
import boto3
import ast
s3_client = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('tbjson2dataframe')
def lambda_handler(event, context):
bucket = event['Records'][0]['s3']['bucket']['name']
json_file_name = event['Records'][0]['s3']['object']['key']
print(bucket)
print(json_file_name)
print(str(event))
json_object = s3_client.get_object(Bucket=bucket,Key=json_file_name)
jsonFileReader = json_object['Body'].read().decode("utf-8")
#now changing json string to dictionary
jsonFileReader = ast.literal_eval(jsonFileReader)
#df = pd.read_json(BytesIO(b'{"col": [1, 2, 3]}'))
#df = pd.DataFrame(jsonFileReader)
table.put_item(Item=jsonFileReader)and this this the error message I'm getting:
Error: [ERROR] ParamValidationError: Parameter validation failed:
Invalid type for parameter Item, value: [{'Industry name ': 'Industry name ', 'Number of firms ': 'Number of firms ', 'Inventory/Sa les ': 'Inventory/Sa les ', 'Inventory/Enterprise Value ': 'Inventory/Enterprise Value ', 'Number of Days Sales in Inventory ': 'Number of Days Sales in Inventory '}, {'Industry name ': 'Advertising ', 'Number of firms ': '65 ', 'Inventory/Sa les ': '4.53% ', 'Inventory/Enterprise Value ': '2.37% ', 'Number of Days Sales in Inventory ': '16.52 '}, {'Industry name ': 'Aerospace/Defense ', 'Number of firms ': '95 ', 'Inventory/Sa les ': '20.22% ', 'Inventory/Enterprise Value ': '13.26% ', 'Number of Days Sales in Inventory ': '73.82 '}, {'Industry name ': 'Air Transport ', 'Number of firms ': '25 ', 'Inventory/Sa les ': '2.74% ', 'Inventory/Enterprise Value ': '2.59% ', 'Number of Days Sales in Inventory ': '9.98 '}, {'Industry name ': 'Apparel ', 'Number of firms ': '70 ', 'Inventory/Sa les ': '18.00% ', 'Inventory/Enterprise Value ': '8.09% ', 'Number of Days Sales in Inventory ': '65.72 '}, {'Industry name ': 'Auto & Truck ', 'Number of firms ': '26 ', 'Inventory/Sa les ': '8.08% ', 'Inventory/Enterprise Value ': '8.97% ', 'Number of Days Sales in Inventory ': '29.49 '}, {'Industry name ': 'Auto Parts ', 'Number of firms ': '75 ', 'Inventory/Sa les ': '8.06% ', 'Inventory/Enterprise Value ': '9.34% ', 'Number of Days Sales in Inventory ': '29.40 '}, {'Industry name ': 'Bank ', 'Number of firms ': '7 ', 'Inventory/Sa les ': '4.31% ', 'Inventory/Enterprise Value ': '0.83% ', 'Number of Days Sales in Inventory ': '15.71 '}, {'Industry name ': 'Banks (Regional) ', 'Number of firms ': '721 ', 'Inventory/Sa les ': '6.12% ', 'Inventory/Enterprise Value ': '1.21% ', 'Number of Days Sales in Inventory ': '22.34 '}, {'Industry name ': 'Beverage ', 'Number of firms ': '47 ', 'Inventory/Sa les ': '6.10% ', 'Inventory/Enterprise Value ': '2.02% ', 'Number of Days Sales in Inventory ': '22.28 '}, {'Industry name ': 'Beverage (Alcoholic) ', 'Number of firms ': '19 ', 'Inventory/Sa les ': '33.20% ', 'Inventory/Enterprise Value ': '7.07% ', 'Number of Days Sales in Inventory ': '121.18 '}, {'Industry name ': 'Biotechnology ', 'Number of firms ': '349 ', 'Inventory/Sa les ': '12.11% ', 'Inventory/Enterprise Value ': '1.13% ', 'Number of Days Sales in Inventory ': '44.20 '}, {'Industry name ': 'Broadcasting ', 'Number of firms ': '30 ', 'Inventory/Sa les ': '1.38% ', 'Inventory/Enterprise Value ': '0.33% ', 'Number of Days Sales in Inventory ': '5.04 '}, {'Industry name ': 'Brokerage & Investment Banking ', 'Number of firms ': '49 ', 'Inventory/Sa les ': '0.04% ', 'Inventory/Enterprise Value ': '0.01% ', 'Number of Days Sales in Inventory ': '0.15 '}, {'Industry name ': 'Building Materials ', 'Number of firms ': '37 ', 'Inventory/Sa les ': '11.46% ', 'Inventory/Enterprise Value ': '8.34% ', 'Number of Days Sales in Inventory ': '41.84 '}, {'Industry name ': 'Business & Consumer Services ', 'Number of firms ': '179 ', 'Inventory/Sa les ': '1.96% ', 'Inventory/Enterprise Value ': '1.16% ', 'Number of Days Sales in Inventory ': '7.16 '}, {'Industry name ': 'Cable TV ', 'Number of firms ': '16 ', 'Inventory/Sa les ': '0.62% ', 'Inventory/Enterprise Value ': '0.23% ', 'Number of Days Sales in Inventory ': '2.26 '}, {'Industry name ': 'Chemical (Basic) ', 'Number of firms ': '47 ', 'Inventory/Sa les ': '14.53% ', 'Inventory/Enterprise Value ': '8.83% ', 'Number of Days Sales in Inventory ': '53.03 '}, {'Industry name ': 'Chemical (Diversified) ', 'Number of firms ': '10 ', 'Inventory/Sa les ': '16.57% ', 'Inventory/Enterprise Value ': '10.98% ', 'Number of Days Sales in Inventory ': '60.50 '}, {'Industry name ': 'Chemical (Specialty) ', 'Number of firms ': '100 ', 'Inventory/Sa les ': '13.41% ', 'Inventory/Enterprise Value ': '6.18% ', 'Number of Days Sales in Inventory ': '48.95 '}, {'Industry name ': 'Coal & Related Energy ', 'Number of firms ': '45 ', 'Inventory/Sa les ': '9.68% ', 'Inventory/Enterprise Value ': '5.92% ', 'Number of Days Sales in Inventory ': '35.32 '}, {'Industry name ': 'Computer Services ', 'Number of firms ': '129 ', 'Inventory/Sa les ': '5.47% ', 'Inventory/Enterprise Value ': '4.84% ', 'Number of Days Sales in Inventory ': '19.96 '}, {'Industry name ': 'Computer Software ', 'Number of firms ': '273 ', 'Inventory/Sa les ': '1.84% ', 'Inventory/Enterprise Value ': '0.40% ', 'Number of Days Sales in Inventory ': '6.72 '}, {'Industry name ': 'Computers/Peripherals ', 'Number of firms ': '66 ', 'Inventory/Sa les ': '4.01% ', 'Inventory/Enterprise Value ': '2.00% ', 'Number of Days Sales in Inventory ': '14.65 '}, {'Industry name ': 'Construction ', 'Number of firms ': '18 ', 'Inventory/Sa les ': '13.32% ', 'Inventory/Enterprise Value ': '4.20% ', 'Number of Days Sales in Inventory ': '48.60 '}, {'Industry name ': 'Diversified ', 'Number of firms ': '20 ', 'Inventory/Sa les ': '11.84% ', 'Inventory/Enterprise Value ': '2.91% ', 'Number of Days Sales in Inventory ': '43.21 '}, {'Industry name ': 'Educational Services ', 'Number of firms ': '40 ', 'Inventory/Sa les ': '1.07% ', 'Inventory/Enterprise Value ': '0.85% ', 'Number of Days Sales in Inventory ': '3.92 '}, {'Industry name ': 'Electrical Equipment ', 'Number of firms ': '135 ', 'Inventory/Sa les ': '10.41% ', 'Inventory/Enterprise Value ': '4.91% ', 'Number of Days Sales in Inventory ': '37.99 '}, {'Industry name ': 'Electronics ', 'Number of firms ': '191 ', 'Inventory/Sa les ': '15.23% ', 'Inventory/Enterprise Value ': '9.50% ', 'Number of Days Sales in Inventory ': '55.60 '}, {'Industry name ': 'Electronics (Consumer & Office) ', 'Number of firms ': '26 ', 'Inventory/Sa les ': '8.03% ', 'Inventory/Enterprise Value ': '7.90% ', 'Number of Days Sales in Inventory ': '29.29 '}, {'Industry name ': 'Engineering ', 'Number of firms ': '56 ', 'Inventory/Sa les ': '0.99% ', 'Inventory/Enterprise Value ': '1.66% ', 'Number of Days Sales in Inventory ': '3.63 '}, {'Industry name ': 'Entertainment ', 'Number of firms ': '85 ', 'Inventory/Sa les ': '3.66% ', 'Inventory/Enterprise Value ': '1.22% ', 'Number of Days Sales in Inventory ': '13.35 '}, {'Industry name ': 'Environmental & Waste Services ', 'Number of firms ': '108 ', 'Inventory/Sa les ': '1.46% ', 'Inventory/Enterprise Value ': '0.67% ', 'Number of Days Sales in Inventory ': '5.33 '}], type: <class 'list'>, valid types: <class 'dict'>
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 30, in lambda_handler
table.put_item(Item=jsonFileReader)
File "/var/runtime/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/var/runtime/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(*args, **params)
File "/var/runtime/botocore/client.py", line 391, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 691, in _make_api_call
request_dict = self._convert_to_request_dict(
File "/var/runtime/botocore/client.py", line 739, in _convert_to_request_dict
request_dict = self._serializer.serialize_to_request(
File "/var/runtime/botocore/validate.py", line 360, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
The json fill is the result of extracting it from the attached PDF fill.
Hello, thank you so much for having taken the time and effort to assist me. I think your reply put me on the right path. This is how my code looks now:
import pandas as pd
import json
import boto3
import ast
s3_client = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('dataframe2dynamodb_tb')
def lambda_handler(event, context):
bucket = event['Records'][0]['s3']['bucket']['name']
json_file_name = event['Records'][0]['s3']['object']['key']
print(bucket)
print(json_file_name)
print(str(event))
json_object = s3_client.get_object(Bucket=bucket,Key=json_file_name)
jsonFileReader = json_object['Body'].read().decode("utf-8")
#jsonFileReader = json_object['Body'].read()
#jsonDict = json.loads(jsonFileReader)
#now changing json string to dictionary
jsonFileReader = ast.literal_eval(jsonFileReader)
#df = pd.DataFrame(jsonFileReader)
with table.batch_writer() as batch:
for mydict in jsonFileReader:
print(mydict)
batch.put_item(Item=mydict)
return "Upload to DynamoDB is a success!"This is the error message I'm now getting:
Error: [ERROR] ClientError: An error occurred (ValidationException) when calling the BatchWriteItem operation: The provided key element does not match the schema
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 34, in lambda_handler
batch.put_item(Item=mydict)
File "/var/runtime/boto3/dynamodb/table.py", line 101, in put_item
self._add_request_and_process({'PutRequest': {'Item': Item}})
File "/var/runtime/boto3/dynamodb/table.py", line 110, in _add_request_and_process
self._flush_if_needed()
File "/var/runtime/boto3/dynamodb/table.py", line 131, in _flush_if_needed
self._flush()
File "/var/runtime/boto3/dynamodb/table.py", line 136, in _flush
response = self._client.batch_write_item(
File "/var/runtime/botocore/client.py", line 391, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 719, in _make_api_call
raise error_class(parsed_response, operation_name)
Posts: 584
Threads: 1
Joined: Aug 2019
(Dec-25-2022, 03:04 PM)gdbengo Wrote: Error: [ERROR] ClientError: An error occurred (ValidationException) when calling the BatchWriteItem operation: The provided key element does not match the schema
I am afraid this is not a Python error so I will not be able to help you. I think there is a mismatch betrween the table and the provided data.
You have the output of line 33 ( print(mydict)). Compare this with how the table 'tbjson2dataframe' is defined.
|
