Python Forum
Problem with date type (object to datetime)
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Problem with date type (object to datetime)
#1
Hi all,
so I have this issue with the convertion to date format.
here is how my csv file looks like:csv

and after setting headers and selecting some columns I got this: df

but when I want to convert to datetime64[ns] I got this error: error

Ialso tried this:
new_data['Date_Time']=pd.to_datetime(new_data['Date_Time'])
new_data.dtypes
but almost the same error:
~/anaconda3/lib/python3.7/site-packages/dateutil/parser/_parser.py in _build_naive(self, res, default)
   1225                 repl['day'] = monthrange(cyear, cmonth)[1]
   1226 
-> 1227         naive = default.replace(**repl)
   1228 
   1229         if res.weekday is not None and not res.day:

ValueError: year 1552231082 is out of range
Thks for your help.
B. regards
Karlito
Reply
#2
While using read_csv one can pass something like

df = pd.read_csv('my.csv', headers=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')})
and format as datetime while reading without need to convert it later (if passing headers name then converters key must be corresponding to that header name)
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#3
(Oct-15-2019, 01:48 PM)perfringo Wrote: While using read_csv one can pass something like

df = pd.read_csv('my.csv', headers=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')})
and format as datetime while reading without need to convert it later (if passing headers name then converters key must be corresponding to that header name)

Thks Perfringo,

but I got this new error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-76-5dc85874d75f> in <module>
      7 
      8 #data = pd.read_csv('data.csv', low_memory = False)
----> 9 data = pd.read_csv('data.csv', header=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')})
     10 data.head()

~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, doublequote, delim_whitespace, low_memory, memory_map, float_precision)
    676                     skip_blank_lines=skip_blank_lines)
    677 
--> 678         return _read(filepath_or_buffer, kwds)
    679 
    680     parser_f.__name__ = name

~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
    444 
    445     try:
--> 446         data = parser.read(nrows)
    447     finally:
    448         parser.close()

~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in read(self, nrows)
   1034                 raise ValueError('skipfooter not supported for iteration')
   1035 
-> 1036         ret = self._engine.read(nrows)
   1037 
   1038         # May alter columns / col_dict

~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in read(self, nrows)
   1846     def read(self, nrows=None):
   1847         try:
-> 1848             data = self._reader.read(nrows)
   1849         except StopIteration:
   1850             if self._first_chunk:

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader.read()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._read_low_memory()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._read_rows()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_column_data()

pandas/_libs/parsers.pyx in pandas._libs.parsers._apply_converter()

<ipython-input-76-5dc85874d75f> in <lambda>(x)
      7 
      8 #data = pd.read_csv('data.csv', low_memory = False)
----> 9 data = pd.read_csv('data.csv', header=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')})
     10 data.head()

AttributeError: type object 'datetime.datetime' has no attribute 'datetime'
Reply
#4
It might be related how you import: import datetime or from datetime import datetime
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#5
(Oct-16-2019, 06:22 AM)perfringo Wrote: It might be related how you import: import datetime or from datetime import datetime
Thks for answering! :)

I tried 3 cases :) ... first
just import datetime as
and from datetime import datetime
also both together
Reply
#6
You should try to make .strptime to work and then use it in lambda. Something like:

>>> datetime.datetime.strptime('31.12.2018 23:54', '%d.%m.%Y %H:%M')      # whichever version of it works in your settings                      
datetime.datetime(2018, 12, 31, 23, 54)
I personally use startup.py file which loads before interactive session starts and in this file I use 'import datetime'
I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy

Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame.
Reply
#7
edited: didnt see you reply before posting this :)

Ok this is what I've done so far and its working.

import pandas as pd
import datetime as dt
#from datetime import datetime 
import matplotlib.pyplot as plt
import numpy as np
import random, string

data = pd.read_csv('dec18_oct19.csv', sep = ';', header = None, low_memory = False)
data.head()
have this dataframe -> df
I don't really know what happened so that data[0].dtype became datetime64 type because since yesterday it was an object type !!!! and the reason of my post. lol

and then

#Because I just want to work with specific columns
dat = data[[0, 5, 6, 7, 8, 9, 10, 14, 15, 16]].copy(deep=False)
#dat[0] = pd.to_datetime(data[0], errors = 'coerce')
dat[5] = pd.to_numeric(dat[5], errors = 'coerce')
dat.head()
result -> slicing

now I can set the headers

dat.columns = [
'Date_Time',
'Netzspannung1', 
'Netzspannung2', 
'Netzspannung3', 
'Phasenstrom (Wirkstrom)1', 
'Phasenstrom (Wirkstrom)2', 
'Phasenstrom (Wirkstrom)3',
'Netzstrom1', 
'Netzstrom2', 
'Netzstrom3']
result -> final df

Is there any ideas to improve this. Thks
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  [Numpy] Load date/time from .txt to 'datetime64' type. water 4 405 Mar-01-2024, 11:16 PM
Last Post: Gribouillis
  pymongo diff type problem to find images on two drives darter 0 1,271 Mar-11-2021, 04:52 AM
Last Post: darter
  replace nan values by mean group by date.year, date.month wissam1974 5 8,327 Feb-19-2020, 06:25 PM
Last Post: AnkitGupta
  Pandas and Date: problem with operator.How to resolve frame 6 4,271 May-13-2019, 05:50 PM
Last Post: frame
  TensorFlow problem in Object Detection yksingh1097 1 2,764 Oct-03-2018, 09:57 PM
Last Post: Larz60+
  Finding date count from a list of date range in pandas trillerducas72 0 2,720 May-24-2018, 02:30 AM
Last Post: trillerducas72

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020