Python Forum
selecting a particular column in csv file shows error
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
selecting a particular column in csv file shows error
#1
I am selecting a particular column from twitter username database which is in CSV file.
I tried the following with a simple csv file which is made by me. The code runs fine. But when I am reading the file with huge data it gives me error.

import csv

filename = 'twitter-gender-classifier.csv'
# filename = 'test.csv'

with open(filename) as csvfile:
    readCSV = csv.reader(csvfile, delimiter=',')
    data2 = []
    for row in readCSV:
        data = []
        data.append(row[14]) # appending names
        data.append(row[5])  # appending gender
        data2.append(data)

    print(data2)
The same code works fine with test.py file

with twitter-gender-classifier.py file it gives me the error
Error:
C:\Users\Dileep.Kumar\AppData\Local\Programs\Python\Python36\python.exe C:/Users/Dileep.Kumar/PycharmProjects/Twitter_Gender_Classification/test2.py Traceback (most recent call last): File "C:/Users/Dileep.Kumar/PycharmProjects/Twitter_Gender_Classification/test2.py", line 9, in <module> for row in readCSV: File "C:\Users\Dileep.Kumar\AppData\Local\Programs\Python\Python36\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1009: character maps to <undefined>
Reply
#2
Looks like the file is using an encoding different what your python uses by default.
You'll need to specify the encoding when opening the file: https://docs.python.org/3/library/functions.html#open
Reply
#3
I have checked the encoding technqiue and it is utf8, so I have modified the code line as
with open(filename, encoding='utf8') as csvfile:
then it returns another error
Error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 927: invalid start byte
Reply
#4
Judging by that error, it isn't utf8.
Reply
#5
when I check the file it gave me this

[Image: dda5b509725975e23d83cfae88c6ae2c.html]

Please help.
Reply
#6
Can you attach the file itself?
Would be much more useful than a broken link to download an image that shows something.

Also, how was the file created?
Reply
#7
Issue is resolved.
A little modification helped by ignoring the encoding errors

with open(filename, encoding='utf8', errors='ignore') as csvfile:
Reply
#8
Take a look at: https://pypi.python.org/pypi/chardet
this might be useful
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Help copying a column from a csv to another file with some extras g0nz0uk 3 403 Feb-01-2024, 03:12 PM
Last Post: DeaD_EyE
  How to read csv file update matplotlib column chart regularly SamLiu 2 1,015 Jan-21-2023, 11:33 PM
Last Post: SamLiu
  Read xml column inside csv file with Python estertabita 2 1,327 Jul-26-2022, 06:09 PM
Last Post: Larz60+
  ModuleNotFound but pip shows module installed biscotty666 2 1,511 Jul-14-2022, 05:17 PM
Last Post: Axel_Erfurt
  Os command output in variable shows wrong value paulo79 2 1,467 Apr-09-2022, 03:48 PM
Last Post: ndc85430
  How to split file by same values from column from imported CSV file? Paqqno 5 2,704 Mar-24-2022, 05:25 PM
Last Post: Paqqno
  Appending Excel column value as CSV file name sh1704 0 1,267 Feb-06-2022, 10:32 PM
Last Post: sh1704
  Create zip file from the BLOB column in ORACLE DB nnsatpute 2 1,878 Dec-31-2021, 11:00 AM
Last Post: ibreeden
  split txt file data on the first column value shantanu97 2 2,376 Dec-29-2021, 05:03 PM
Last Post: DeaD_EyE
  Index error - columns vs non-column Vinny 3 4,848 Aug-09-2021, 04:46 PM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020