Problem with date type (object to datetime) - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: Data Science (https://python-forum.io/forum-44.html) +--- Thread: Problem with date type (object to datetime) (/thread-21808.html) |
Problem with date type (object to datetime) - karlito - Oct-15-2019 Hi all, so I have this issue with the convertion to date format. here is how my csv file looks like:csv and after setting headers and selecting some columns I got this: df but when I want to convert to datetime64[ns] I got this error: error Ialso tried this: new_data['Date_Time']=pd.to_datetime(new_data['Date_Time']) new_data.dtypesbut almost the same error: ~/anaconda3/lib/python3.7/site-packages/dateutil/parser/_parser.py in _build_naive(self, res, default) 1225 repl['day'] = monthrange(cyear, cmonth)[1] 1226 -> 1227 naive = default.replace(**repl) 1228 1229 if res.weekday is not None and not res.day: ValueError: year 1552231082 is out of rangeThks for your help. B. regards Karlito RE: Problem with date type (object to datetime) - perfringo - Oct-15-2019 While using read_csv one can pass something like df = pd.read_csv('my.csv', headers=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')})and format as datetime while reading without need to convert it later (if passing headers name then converters key must be corresponding to that header name) RE: Problem with date type (object to datetime) - karlito - Oct-16-2019 (Oct-15-2019, 01:48 PM)perfringo Wrote: While using read_csv one can pass something like Thks Perfringo, but I got this new error: --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-76-5dc85874d75f> in <module> 7 8 #data = pd.read_csv('data.csv', low_memory = False) ----> 9 data = pd.read_csv('data.csv', header=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')}) 10 data.head() ~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, doublequote, delim_whitespace, low_memory, memory_map, float_precision) 676 skip_blank_lines=skip_blank_lines) 677 --> 678 return _read(filepath_or_buffer, kwds) 679 680 parser_f.__name__ = name ~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds) 444 445 try: --> 446 data = parser.read(nrows) 447 finally: 448 parser.close() ~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in read(self, nrows) 1034 raise ValueError('skipfooter not supported for iteration') 1035 -> 1036 ret = self._engine.read(nrows) 1037 1038 # May alter columns / col_dict ~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in read(self, nrows) 1846 def read(self, nrows=None): 1847 try: -> 1848 data = self._reader.read(nrows) 1849 except StopIteration: 1850 if self._first_chunk: pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader.read() pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._read_low_memory() pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._read_rows() pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_column_data() pandas/_libs/parsers.pyx in pandas._libs.parsers._apply_converter() <ipython-input-76-5dc85874d75f> in <lambda>(x) 7 8 #data = pd.read_csv('data.csv', low_memory = False) ----> 9 data = pd.read_csv('data.csv', header=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')}) 10 data.head() AttributeError: type object 'datetime.datetime' has no attribute 'datetime' RE: Problem with date type (object to datetime) - perfringo - Oct-16-2019 It might be related how you import: import datetime or from datetime import datetime RE: Problem with date type (object to datetime) - karlito - Oct-16-2019 (Oct-16-2019, 06:22 AM)perfringo Wrote: It might be related how you import: import datetime or from datetime import datetimeThks for answering! :) I tried 3 cases :) ... first just import datetime as and from datetime import datetime also both together RE: Problem with date type (object to datetime) - perfringo - Oct-16-2019 You should try to make .strptime to work and then use it in lambda. Something like: >>> datetime.datetime.strptime('31.12.2018 23:54', '%d.%m.%Y %H:%M') # whichever version of it works in your settings datetime.datetime(2018, 12, 31, 23, 54)I personally use startup.py file which loads before interactive session starts and in this file I use 'import datetime' RE: Problem with date type (object to datetime) - karlito - Oct-16-2019 edited: didnt see you reply before posting this :) Ok this is what I've done so far and its working. import pandas as pd import datetime as dt #from datetime import datetime import matplotlib.pyplot as plt import numpy as np import random, string data = pd.read_csv('dec18_oct19.csv', sep = ';', header = None, low_memory = False) data.head()have this dataframe -> df I don't really know what happened so that data[0].dtype became datetime64 type because since yesterday it was an object type !!!! and the reason of my post. lol and then #Because I just want to work with specific columns dat = data[[0, 5, 6, 7, 8, 9, 10, 14, 15, 16]].copy(deep=False) #dat[0] = pd.to_datetime(data[0], errors = 'coerce') dat[5] = pd.to_numeric(dat[5], errors = 'coerce') dat.head()result -> slicing now I can set the headers dat.columns = [ 'Date_Time', 'Netzspannung1', 'Netzspannung2', 'Netzspannung3', 'Phasenstrom (Wirkstrom)1', 'Phasenstrom (Wirkstrom)2', 'Phasenstrom (Wirkstrom)3', 'Netzstrom1', 'Netzstrom2', 'Netzstrom3']result -> final df Is there any ideas to improve this. Thks |