DataFrame.astype('category') duplicates column

[garikhgh0]

Hi. I have a problem on convesion of object type into category.
My data shape is (1000000, 6)[Date, object,object, object, int64, column_1]
when using the below code, it duplicates last column, the column_1.

1	df.column_1 = df.column_1.astype('category')

before conversion it is in object type, after conversion it shows category but already duplicated.

one more point. the label of the duplicated column contains whitespace in the end of it.


thanks in advance

[scidam]

before conversion it is in object type, after conversion it shows category but already duplicated.

You can get unique values of the categorical column as follows:

1 2	df.column_1 = df.column_1.astype('category') df.column_1.cat.categories #unique categories

the label of the duplicated column contains whitespace in the end of it.

Didn't understand, but if want to remove duplicates from the original data frame, you can use drop_duplicates method.
e.g.

1 2	df = df.drop_duplicates(['column_1'])  # or append .reset_index(drop=True) if needed # removes rows with duplicated values in column_1

[garikhgh0]

thanks a lot. I would also mention that, when converting objects itno category, the Dtaframe.pivot_table does not work correctly. creates duplictaes

[scidam]

the Dtaframe.pivot_table does not work correctly

Was trying to reproduce, but couldn't find the error:

1 2 3

import pandas as pd data = pd.DataFrame({'x': pd.np.random.randint(0,100,1000), 'y': pd.np.random.choice(['a', 'b', 'c'], 1000)}) pd.pivot_table(data, aggfunc=pd.np.sum, values='x', columns=['y'])

Output:
y      a      b      c
x  16924  16650  16377

1 2 3

# change column type data.y = data.y.astype('category') pd.pivot_table(data, aggfunc=pd.np.sum, values='x', columns=['y'])

# the result is the same...

Output:
y      a      b      c
x  16924  16650  16377