+- Python Forum (https://python-forum.io)
+-- Forum: Python Coding (https://python-forum.io/forum-7.html)
+--- Forum: Homework (https://python-forum.io/forum-9.html)
+--- Thread: Read data from a CSV file in S3 bucket and store it in a dictionary in python (/thread-26805.html)
Read data from a CSV file in S3 bucket and store it in a dictionary in python - Rupini - May-14-2020
I am trying to read a csv file from S3 bucket and store its content into a dictionary. Sample csv file data. I want to use my first row as key and subsequent rows as value
sample data:
name,origin,dest
xxx,uk,france
yyyy,norway,finland
zzzz,denmark,canada
I am using the below code which is storing the entire row in a dictionary. But I want to loop through each row and store each field in a row as key value pair.
Current output: {'content': 'xxx,uk,france'}
Required output: {'name':'xxx','origin':'uk','dest':'france'}
import boto3
s3 = boto3.client('s3')
obj = s3.get_object(Bucket = 'bucket_name', Key = 'logs/log.csv')
lines = obj['Body'].read().decode("utf-8").replace("'", '"')
lines = lines.splitlines()
if (isinstance(lines, str)):
lines = (lines)
docData = {}
for line in lines:
docData['content'] = str(line)
print(docData)
RE: Read data from a CSV file in S3 bucket and store it in a dictionary in python - snippsat - May-14-2020
If use csv module and DictReader.
It will give you that structure bye default.
import csv
with open("log.csv") as f:
records = csv.DictReader(f)
for row in records:
print(row)Output:{'name': 'xxx', 'origin': 'uk', 'dest': 'france'}
{'name': 'yyyy', 'origin': 'norway', 'dest': 'finland'}
{'name': 'zzzz', 'origin': 'denmark', 'dest': 'canada'}RE: Read data from a CSV file in S3 bucket and store it in a dictionary in python - Rupini - May-14-2020
Hi @snippsat, Thank you for your reply. I won't be able to use CSV module in this case as my file can be either csv or text file. My S3 bucket will include network log files (can be .csv or .log depending on the source) which I am trying to read. Is there other ways for me to achieve the same without CSV module?.
RE: Read data from a CSV file in S3 bucket and store it in a dictionary in python - snippsat - May-15-2020
(May-14-2020, 11:52 AM)Rupini Wrote: is there other ways for me to achieve the same without CSV module?Yes you can write a own
csv.DictReader implementation.Also if you look at csv module at top there is a link to Source code: Lib/csv.py .
So there can look at how they have written it,the important part start at line 119.
Here is start with some good hints,as this is homework there missing a little part.
lst = []
with open("log.csv") as f:
header = next(f)
header = header.strip().split(',')
for row in f:
row = row.strip().split(',')
print(row)Look at what have now.['xxx', 'uk', 'france']
['yyyy', 'norway', 'finland']
['zzzz', 'denmark', 'canada']
>>> header
['name', 'origin', 'dest']
>>> row
['zzzz', 'denmark', 'canada']
# Now can test line 119
>>> dict(zip(header, row))
{'name': 'zzzz', 'origin': 'denmark', 'dest': 'canada'}