Bottom Page

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
 Problem with date type (object to datetime)
#1
Hi all,
so I have this issue with the convertion to date format.
here is how my csv file looks like:csv

and after setting headers and selecting some columns I got this: df

but when I want to convert to datetime64[ns] I got this error: error

Ialso tried this:
new_data['Date_Time']=pd.to_datetime(new_data['Date_Time'])
new_data.dtypes
but almost the same error:
~/anaconda3/lib/python3.7/site-packages/dateutil/parser/_parser.py in _build_naive(self, res, default)
   1225                 repl['day'] = monthrange(cyear, cmonth)[1]
   1226 
-> 1227         naive = default.replace(**repl)
   1228 
   1229         if res.weekday is not None and not res.day:

ValueError: year 1552231082 is out of range
Thks for your help.
B. regards
Karlito
Quote
#2
While using read_csv one can pass something like

df = pd.read_csv('my.csv', headers=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')})
and format as datetime while reading without need to convert it later (if passing headers name then converters key must be corresponding to that header name)
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Quote
#3
(Oct-15-2019, 01:48 PM)perfringo Wrote: While using read_csv one can pass something like

df = pd.read_csv('my.csv', headers=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')})
and format as datetime while reading without need to convert it later (if passing headers name then converters key must be corresponding to that header name)

Thks Perfringo,

but I got this new error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-76-5dc85874d75f> in <module>
      7 
      8 #data = pd.read_csv('data.csv', low_memory = False)
----> 9 data = pd.read_csv('data.csv', header=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')})
     10 data.head()

~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, doublequote, delim_whitespace, low_memory, memory_map, float_precision)
    676                     skip_blank_lines=skip_blank_lines)
    677 
--> 678         return _read(filepath_or_buffer, kwds)
    679 
    680     parser_f.__name__ = name

~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
    444 
    445     try:
--> 446         data = parser.read(nrows)
    447     finally:
    448         parser.close()

~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in read(self, nrows)
   1034                 raise ValueError('skipfooter not supported for iteration')
   1035 
-> 1036         ret = self._engine.read(nrows)
   1037 
   1038         # May alter columns / col_dict

~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in read(self, nrows)
   1846     def read(self, nrows=None):
   1847         try:
-> 1848             data = self._reader.read(nrows)
   1849         except StopIteration:
   1850             if self._first_chunk:

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader.read()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._read_low_memory()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._read_rows()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_column_data()

pandas/_libs/parsers.pyx in pandas._libs.parsers._apply_converter()

<ipython-input-76-5dc85874d75f> in <lambda>(x)
      7 
      8 #data = pd.read_csv('data.csv', low_memory = False)
----> 9 data = pd.read_csv('data.csv', header=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')})
     10 data.head()

AttributeError: type object 'datetime.datetime' has no attribute 'datetime'
Quote
#4
It might be related how you import: import datetime or from datetime import datetime
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Quote
#5
(Oct-16-2019, 06:22 AM)perfringo Wrote: It might be related how you import: import datetime or from datetime import datetime
Thks for answering! :)

I tried 3 cases :) ... first
just import datetime as
and from datetime import datetime
also both together
Quote
#6
You should try to make .strptime to work and then use it in lambda. Something like:

>>> datetime.datetime.strptime('31.12.2018 23:54', '%d.%m.%Y %H:%M')      # whichever version of it works in your settings                      
datetime.datetime(2018, 12, 31, 23, 54)
I personally use startup.py file which loads before interactive session starts and in this file I use 'import datetime'
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Quote
#7
edited: didnt see you reply before posting this :)

Ok this is what I've done so far and its working.

import pandas as pd
import datetime as dt
#from datetime import datetime 
import matplotlib.pyplot as plt
import numpy as np
import random, string

data = pd.read_csv('dec18_oct19.csv', sep = ';', header = None, low_memory = False)
data.head()
have this dataframe -> df
I don't really know what happened so that data[0].dtype became datetime64 type because since yesterday it was an object type !!!! and the reason of my post. lol

and then

#Because I just want to work with specific columns
dat = data[[0, 5, 6, 7, 8, 9, 10, 14, 15, 16]].copy(deep=False)
#dat[0] = pd.to_datetime(data[0], errors = 'coerce')
dat[5] = pd.to_numeric(dat[5], errors = 'coerce')
dat.head()
result -> slicing

now I can set the headers

dat.columns = [
'Date_Time',
'Netzspannung1', 
'Netzspannung2', 
'Netzspannung3', 
'Phasenstrom (Wirkstrom)1', 
'Phasenstrom (Wirkstrom)2', 
'Phasenstrom (Wirkstrom)3',
'Netzstrom1', 
'Netzstrom2', 
'Netzstrom3']
result -> final df

Is there any ideas to improve this. Thks
Quote

Top Page

Possibly Related Threads...
Thread Author Replies Views Last Post
  replace nan values by mean group by date.year, date.month wissam1974 4 509 Jul-06-2019, 02:00 AM
Last Post: wissam1974
  Pandas and Date: problem with operator.How to resolve frame 6 1,184 May-13-2019, 05:50 PM
Last Post: frame
  TensorFlow problem in Object Detection yksingh1097 1 714 Oct-03-2018, 09:57 PM
Last Post: Larz60+
  Finding date count from a list of date range in pandas trillerducas72 0 670 May-24-2018, 02:30 AM
Last Post: trillerducas72

Forum Jump:


Users browsing this thread: 1 Guest(s)