Select data from between two timestamps

[Makada]

Hi,

I managed to set the x-axis (time) for a fixed range, so it wont autoscale.
I post it for future reference
("import pandas as pd")
Example below:

1	plt.xlim([pd.to_datetime('2020-01-28 23:50:00'), pd.to_datetime('220-01-29 00:10:00')])

[Makada]

Hi,

I am currently trying to plot multiple figures with data.
I have managed to plot (as a start) 2 plots, but i noticed an error in my code.
As your can see in the picture, with code below, i am getting 2 plots ok, but the first plot isnt displaying the date range in the code (but the whole available data) and every update there is a duplicate legend.
While the second plot is doing all fine.
What have i coded wrong?

Thanks.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57

import pandas as pd import numpy as np import datetime as dt import csv from pandas import read_csv from datetime import datetime import matplotlib import matplotlib.pyplot as plt  import time   starttime=time.time() plt.ion() fig=plt.figure()   ax1=fig.add_subplot(211) ax2=fig.add_subplot(212) ax=plt.gca()   while(True):       df = pd.read_csv("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie .dat",skiprows=1, parse_dates=[0], dtype='unicode')       df=pd.DataFrame(df, columns=['TIMESTAMP','WS_kph_S_WVT','WS_kph_Max'])       df['TIMESTAMP']=pd.to_datetime(df.TIMESTAMP, errors='coerce')     df['WS_kph_S_WVT']=pd.to_numeric(df.WS_kph_S_WVT, errors='coerce')     df['WS_kph_Max']=pd.to_numeric(df.WS_kph_Max, errors='coerce')       df=df.dropna()#to remove Nan values       df.info()       start_date = pd.to_datetime('2020-01-19 21:00:00', dayfirst=True)     end_date = pd.to_datetime('2020-12-02 00:00:00', dayfirst=True)     mask=(df['TIMESTAMP'] > start_date) & (df['TIMESTAMP'] <= end_date)     df.loc[mask]     df=df.loc[mask]         ax.clear()     df.plot(x='TIMESTAMP',y='WS_kph_S_WVT',color='k', kind='line', linewidth=0.5, ax=ax1, label='my lines')     plt.xlim([pd.to_datetime('2020-02-02 00:00:00'), pd.to_datetime('2020-02-03 00:00:00')])       plt.draw()       ax.clear()     df.plot(x='TIMESTAMP',y='WS_kph_Max',color='r', kind='line', linewidth=0.5, ax=ax2, label='my lines')     plt.xlim([pd.to_datetime('2020-02-02 00:00:00'), pd.to_datetime('2020-02-03 00:00:00')])           plt.draw()     plt.pause(60 - time.time() % 60)   plt.show() print(df) print(df.dtypes)

[Makada]

Hi,

I found out it seems to work with  "ax1.clear()" and  "ax2.clear()".

Just have another question:

With the code in my last post, how is the graph plotted?
Is it plot a new figure every update or does it write the plot on top of the previous Plot (and causes to use more memory every plot)?

Thanks.

[Makada]

Hi,

I got an error today with the following message:

Error:
Traceback (most recent call last):
  File "C:\Users\Makada\Desktop\python\read and graph from cr1000 file updating live 2 dataframess multi plot.py", line 26, in <module>
    df = pd.read_csv("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie .dat",skiprows=1, parse_dates=[0], dtype='unicode')
  File "C:\Python\lib\site-packages\pandas\io\parsers.py", line 685, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "C:\Python\lib\site-packages\pandas\io\parsers.py", line 463, in _read
    data = parser.read(nrows)
  File "C:\Python\lib\site-packages\pandas\io\parsers.py", line 1154, in read
    ret = self._engine.read(nrows)
  File "C:\Python\lib\site-packages\pandas\io\parsers.py", line 2059, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 881, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 896, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas/_libs/parsers.pyx", line 950, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 937, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas/_libs/parsers.pyx", line 2132, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: out of memory
>>> 

Is it caused by overwriting the plots every minute?
I dont know if the plots are cleared before plotting new data and if this error iscrelated?

Thanks.

[Larz60+]

whatever was done used up all available memory (C code where automatic paging doesn't occur)
This is one that I'd inform the pandas group about.

[Makada]

Hi ,

Thanks for your response.
How do i inform the pandas group about it?

Thanks.

[Larz60+]

wasn't easy to find:
see: https://pandas.pydata.org/pandas-docs/st...t-requests

[Makada]

Thanks, ill have a look

[Makada]

I just noticed theres an update of pandas, version 1.0.1
I am using version 0.25.3
So i first upgrade to the latest version and see if it will solve the error.


Thanks.

Heres the changelog:

Version 1.0

What’s new in 1.0.0 (January 29, 2020)
New Deprecation Policy
Enhancements
Experimental new featuresOther enhancements
Backwards incompatible API changes
Deprecations
Removal of prior version deprecations/changesPerformance improvements
Bug fixes
Contributors
What’s new in 1.0.1 (February 5, 2020)
Fixed regressions
Deprecations
Bug fixes
Contributors