Posts: 26 Threads: 9 Joined: Oct 2019 Reputation: 0 Likes received: 0 Hi all, so I have this issue with the convertion to date format. here is how my csv file looks like: csv and after setting headers and selecting some columns I got this: df but when I want to convert to datetime64[ns] I got this error: error Ialso tried this: new_data['Date_Time']=pd.to_datetime(new_data['Date_Time'])
new_data.dtypes but almost the same error: ~/anaconda3/lib/python3.7/site-packages/dateutil/parser/_parser.py in _build_naive(self, res, default)
1225 repl['day'] = monthrange(cyear, cmonth)[1]
1226
-> 1227 naive = default.replace(**repl)
1228
1229 if res.weekday is not None and not res.day:
ValueError: year 1552231082 is out of range Thks for your help. B. regards Karlito Posts: 1,123 Threads: 5 Joined: Jun 2018 Reputation: 116 Likes received: 255 Oct-15-2019, 01:48 PM (This post was last modified: Oct-15-2019, 01:48 PM by perfringo. Edited 1 time in total.) While using read_csv one can pass something like
df = pd.read_csv('my.csv', headers=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')})
and format as datetime while reading without need to convert it later (if passing headers name then converters key must be corresponding to that header name) I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame. Posts: 26 Threads: 9 Joined: Oct 2019 Reputation: 0 Likes received: 0 (Oct-15-2019, 01:48 PM)perfringo Wrote: While using read_csv one can pass something like
df = pd.read_csv('my.csv', headers=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')})
and format as datetime while reading without need to convert it later (if passing headers name then converters key must be corresponding to that header name) Thks Perfringo, but I got this new error: ---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-76-5dc85874d75f> in <module>
7
8 #data = pd.read_csv('data.csv', low_memory = False)
----> 9 data = pd.read_csv('data.csv', header=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')})
10 data.head()
~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, doublequote, delim_whitespace, low_memory, memory_map, float_precision)
676 skip_blank_lines=skip_blank_lines)
677
--> 678 return _read(filepath_or_buffer, kwds)
679
680 parser_f.__name__ = name
~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
444
445 try:
--> 446 data = parser.read(nrows)
447 finally:
448 parser.close()
~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in read(self, nrows)
1034 raise ValueError('skipfooter not supported for iteration')
1035
-> 1036 ret = self._engine.read(nrows)
1037
1038 # May alter columns / col_dict
~/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in read(self, nrows)
1846 def read(self, nrows=None):
1847 try:
-> 1848 data = self._reader.read(nrows)
1849 except StopIteration:
1850 if self._first_chunk:
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader.read()
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._read_low_memory()
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._read_rows()
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_column_data()
pandas/_libs/parsers.pyx in pandas._libs.parsers._apply_converter()
<ipython-input-76-5dc85874d75f> in <lambda>(x)
7
8 #data = pd.read_csv('data.csv', low_memory = False)
----> 9 data = pd.read_csv('data.csv', header=None, converters={0: lambda x: datetime.datetime.strptime(x, '%d.%m.%Y %H:%M')})
10 data.head()
AttributeError: type object 'datetime.datetime' has no attribute 'datetime' Posts: 1,123 Threads: 5 Joined: Jun 2018 Reputation: 116 Likes received: 255 It might be related how you import: import datetime or from datetime import datetime I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame. Posts: 26 Threads: 9 Joined: Oct 2019 Reputation: 0 Likes received: 0 (Oct-16-2019, 06:22 AM)perfringo Wrote: It might be related how you import: import datetime or from datetime import datetime Thks for answering! :) I tried 3 cases :) ... first just import datetime as and from datetime import datetime also both together Posts: 1,123 Threads: 5 Joined: Jun 2018 Reputation: 116 Likes received: 255 You should try to make .strptime to work and then use it in lambda. Something like:
>>> datetime.datetime.strptime('31.12.2018 23:54', '%d.%m.%Y %H:%M') # whichever version of it works in your settings
datetime.datetime(2018, 12, 31, 23, 54)
I personally use startup.py file which loads before interactive session starts and in this file I use 'import datetime' I'm not 'in'-sane. Indeed, I am so far 'out' of sane that you appear a tiny blip on the distant coast of sanity. Bucky Katt, Get Fuzzy Da Bishop: There's a dead bishop on the landing. I don't know who keeps bringing them in here. ....but society is to blame. Posts: 26 Threads: 9 Joined: Oct 2019 Reputation: 0 Likes received: 0 Oct-16-2019, 08:07 AM (This post was last modified: Oct-16-2019, 08:07 AM by karlito. Edited 1 time in total. Edit Reason: ok ) edited: didnt see you reply before posting this :) Ok this is what I've done so far and its working. import pandas as pd
import datetime as dt
#from datetime import datetime
import matplotlib.pyplot as plt
import numpy as np
import random, string
data = pd.read_csv('dec18_oct19.csv', sep = ';', header = None, low_memory = False)
data.head() have this dataframe -> df I don't really know what happened so that data[0].dtype became datetime64 type because since yesterday it was an object type !!!! and the reason of my post. lol and then
#Because I just want to work with specific columns
dat = data[[0, 5, 6, 7, 8, 9, 10, 14, 15, 16]].copy(deep=False)
#dat[0] = pd.to_datetime(data[0], errors = 'coerce')
dat[5] = pd.to_numeric(dat[5], errors = 'coerce')
dat.head()
result -> slicing now I can set the headers dat.columns = [
'Date_Time',
'Netzspannung1',
'Netzspannung2',
'Netzspannung3',
'Phasenstrom (Wirkstrom)1',
'Phasenstrom (Wirkstrom)2',
'Phasenstrom (Wirkstrom)3',
'Netzstrom1',
'Netzstrom2',
'Netzstrom3']
result -> final df Is there any ideas to improve this. Thks |