Select data from between two timestamps

[Makada]

Hi,

I managed to set the x-axis (time) for a fixed range, so it wont autoscale.
I post it for future reference
("import pandas as pd")
Example below:

plt.xlim([pd.to_datetime('2020-01-28 23:50:00'), pd.to_datetime('220-01-29 00:10:00')])

[Makada]

Hi,

I am currently trying to plot multiple figures with data.
I have managed to plot (as a start) 2 plots, but i noticed an error in my code.
As your can see in the picture, with code below, i am getting 2 plots ok, but the first plot isnt displaying the date range in the code (but the whole available data) and every update there is a duplicate legend.
While the second plot is doing all fine.
What have i coded wrong?

Thanks.

import pandas as pd
import numpy as np
import datetime as dt
import csv
from pandas import read_csv
from datetime import datetime
import matplotlib
import matplotlib.pyplot as plt  
import time

starttime=time.time()
plt.ion()
fig=plt.figure()

ax1=fig.add_subplot(211)
ax2=fig.add_subplot(212)
ax=plt.gca()

while(True):

    df = pd.read_csv("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie .dat",skiprows=1, parse_dates=[0], dtype='unicode')

    df=pd.DataFrame(df, columns=['TIMESTAMP','WS_kph_S_WVT','WS_kph_Max'])

    df['TIMESTAMP']=pd.to_datetime(df.TIMESTAMP, errors='coerce')
    df['WS_kph_S_WVT']=pd.to_numeric(df.WS_kph_S_WVT, errors='coerce')
    df['WS_kph_Max']=pd.to_numeric(df.WS_kph_Max, errors='coerce')

    df=df.dropna()#to remove Nan values

    df.info()

    start_date = pd.to_datetime('2020-01-19 21:00:00', dayfirst=True)
    end_date = pd.to_datetime('2020-12-02 00:00:00', dayfirst=True)
    mask=(df['TIMESTAMP'] > start_date) & (df['TIMESTAMP'] <= end_date)
    df.loc[mask]
    df=df.loc[mask]


    ax.clear()
    df.plot(x='TIMESTAMP',y='WS_kph_S_WVT',color='k', kind='line', linewidth=0.5, ax=ax1, label='my lines')
    plt.xlim([pd.to_datetime('2020-02-02 00:00:00'), pd.to_datetime('2020-02-03 00:00:00')])

    plt.draw()

    ax.clear()
    df.plot(x='TIMESTAMP',y='WS_kph_Max',color='r', kind='line', linewidth=0.5, ax=ax2, label='my lines')
    plt.xlim([pd.to_datetime('2020-02-02 00:00:00'), pd.to_datetime('2020-02-03 00:00:00')])



    plt.draw()
    plt.pause(60 - time.time() % 60)

plt.show()
print(df)
print(df.dtypes)

[Makada]

Hi,

I found out it seems to work with  "ax1.clear()" and  "ax2.clear()".

Just have another question:

With the code in my last post, how is the graph plotted?
Is it plot a new figure every update or does it write the plot on top of the previous Plot (and causes to use more memory every plot)?

Thanks.

[Makada]

Hi,

I got an error today with the following message:

Error:
Traceback (most recent call last):
  File "C:\Users\Makada\Desktop\python\read and graph from cr1000 file updating live 2 dataframess multi plot.py", line 26, in <module>
    df = pd.read_csv("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie .dat",skiprows=1, parse_dates=[0], dtype='unicode')
  File "C:\Python\lib\site-packages\pandas\io\parsers.py", line 685, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "C:\Python\lib\site-packages\pandas\io\parsers.py", line 463, in _read
    data = parser.read(nrows)
  File "C:\Python\lib\site-packages\pandas\io\parsers.py", line 1154, in read
    ret = self._engine.read(nrows)
  File "C:\Python\lib\site-packages\pandas\io\parsers.py", line 2059, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 881, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 896, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas/_libs/parsers.pyx", line 950, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 937, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas/_libs/parsers.pyx", line 2132, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: out of memory
>>> 

Is it caused by overwriting the plots every minute?
I dont know if the plots are cleared before plotting new data and if this error iscrelated?

Thanks.

[Larz60+]

whatever was done used up all available memory (C code where automatic paging doesn't occur)
This is one that I'd inform the pandas group about.

[Makada]

Hi ,

Thanks for your response.
How do i inform the pandas group about it?

Thanks.

[Larz60+]

wasn't easy to find:
see: https://pandas.pydata.org/pandas-docs/st...t-requests

[Makada]

Thanks, ill have a look

[Makada]

I just noticed theres an update of pandas, version 1.0.1
I am using version 0.25.3
So i first upgrade to the latest version and see if it will solve the error.


Thanks.

Heres the changelog:

Version 1.0

What’s new in 1.0.0 (January 29, 2020)
New Deprecation Policy
Enhancements
Experimental new featuresOther enhancements
Backwards incompatible API changes
Deprecations
Removal of prior version deprecations/changesPerformance improvements
Bug fixes
Contributors
What’s new in 1.0.1 (February 5, 2020)
Fixed regressions
Deprecations
Bug fixes
Contributors