Python Forum
Select data from between two timestamps
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Select data from between two timestamps
#11
Hi,

I managed to set the x-axis (time) for a fixed range, so it wont autoscale.
I post it for future reference Smile
("import pandas as pd")
Example below:


plt.xlim([pd.to_datetime('2020-01-28 23:50:00'), pd.to_datetime('220-01-29 00:10:00')])
Reply
#12
Hi,

I am currently trying to plot multiple figures with data.
I have managed to plot (as a start) 2 plots, but i noticed an error in my code.
As your can see in the picture, with code below, i am getting 2 plots ok, but the first plot isnt displaying the date range in the code (but the whole available data) and every update there is a duplicate legend.
While the second plot is doing all fine.
What have i coded wrong?

Thanks.


import pandas as pd
import numpy as np
import datetime as dt
import csv
from pandas import read_csv
from datetime import datetime
import matplotlib
import matplotlib.pyplot as plt  
import time

starttime=time.time()
plt.ion()
fig=plt.figure()

ax1=fig.add_subplot(211)
ax2=fig.add_subplot(212)
ax=plt.gca()

while(True):

    df = pd.read_csv("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie .dat",skiprows=1, parse_dates=[0], dtype='unicode')

    df=pd.DataFrame(df, columns=['TIMESTAMP','WS_kph_S_WVT','WS_kph_Max'])

    df['TIMESTAMP']=pd.to_datetime(df.TIMESTAMP, errors='coerce')
    df['WS_kph_S_WVT']=pd.to_numeric(df.WS_kph_S_WVT, errors='coerce')
    df['WS_kph_Max']=pd.to_numeric(df.WS_kph_Max, errors='coerce')

    df=df.dropna()#to remove Nan values

    df.info()

    start_date = pd.to_datetime('2020-01-19 21:00:00', dayfirst=True)
    end_date = pd.to_datetime('2020-12-02 00:00:00', dayfirst=True)
    mask=(df['TIMESTAMP'] > start_date) & (df['TIMESTAMP'] <= end_date)
    df.loc[mask]
    df=df.loc[mask]


    ax.clear()
    df.plot(x='TIMESTAMP',y='WS_kph_S_WVT',color='k', kind='line', linewidth=0.5, ax=ax1, label='my lines')
    plt.xlim([pd.to_datetime('2020-02-02 00:00:00'), pd.to_datetime('2020-02-03 00:00:00')])

    plt.draw()

    ax.clear()
    df.plot(x='TIMESTAMP',y='WS_kph_Max',color='r', kind='line', linewidth=0.5, ax=ax2, label='my lines')
    plt.xlim([pd.to_datetime('2020-02-02 00:00:00'), pd.to_datetime('2020-02-03 00:00:00')])



    plt.draw()
    plt.pause(60 - time.time() % 60)

plt.show()
print(df)
print(df.dtypes)


[Image: Screenshot-20200204-090906.png]
Reply
#13
Hi,

I found out it seems to work with  "ax1.clear()" and  "ax2.clear()".

Just have another question:

With the code in my last post, how is the graph plotted?
Is it plot a new figure every update or does it write the plot on top of the previous Plot (and causes to use more memory every plot)?

Thanks.
Reply
#14
Hi,

I got an error today with the following message:

Error:
Traceback (most recent call last): File "C:\Users\Makada\Desktop\python\read and graph from cr1000 file updating live 2 dataframess multi plot.py", line 26, in <module> df = pd.read_csv("C:\\Users\\Makada\\Desktop\\CR1000_Table1 - kopie .dat",skiprows=1, parse_dates=[0], dtype='unicode') File "C:\Python\lib\site-packages\pandas\io\parsers.py", line 685, in parser_f return _read(filepath_or_buffer, kwds) File "C:\Python\lib\site-packages\pandas\io\parsers.py", line 463, in _read data = parser.read(nrows) File "C:\Python\lib\site-packages\pandas\io\parsers.py", line 1154, in read ret = self._engine.read(nrows) File "C:\Python\lib\site-packages\pandas\io\parsers.py", line 2059, in read data = self._reader.read(nrows) File "pandas/_libs/parsers.pyx", line 881, in pandas._libs.parsers.TextReader.read File "pandas/_libs/parsers.pyx", line 896, in pandas._libs.parsers.TextReader._read_low_memory File "pandas/_libs/parsers.pyx", line 950, in pandas._libs.parsers.TextReader._read_rows File "pandas/_libs/parsers.pyx", line 937, in pandas._libs.parsers.TextReader._tokenize_rows File "pandas/_libs/parsers.pyx", line 2132, in pandas._libs.parsers.raise_parser_error pandas.errors.ParserError: Error tokenizing data. C error: out of memory >>>
Is it caused by overwriting the plots every minute?
I dont know if the plots are cleared before plotting new data and if this error iscrelated?

Thanks.
Reply
#15
whatever was done used up all available memory (C code where automatic paging doesn't occur)
This is one that I'd inform the pandas group about.
Reply
#16
Hi ,

Thanks for your response.
How do i inform the pandas group about it?

Thanks.
Reply
#17
wasn't easy to find:
see: https://pandas.pydata.org/pandas-docs/st...t-requests
Reply
#18
Thanks, ill have a look Smile
Reply
#19
I just noticed theres an update of pandas, version 1.0.1
I am using version 0.25.3
So i first upgrade to the latest version and see if it will solve the error.


Thanks.

Heres the changelog:

Version 1.0

What’s new in 1.0.0 (January 29, 2020)
New Deprecation Policy
Enhancements
Experimental new featuresOther enhancements
Backwards incompatible API changes
Deprecations
Removal of prior version deprecations/changesPerformance improvements
Bug fixes
Contributors
What’s new in 1.0.1 (February 5, 2020)
Fixed regressions
Deprecations
Bug fixes
Contributors
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Select column between to dates CSV data PythonJD 0 400 Apr-14-2020, 12:22 PM
Last Post: PythonJD
  select data based on indice Staph 4 631 Jul-15-2019, 02:05 AM
Last Post: scidam
  how to select particular rows data from a array raady07 3 2,057 Mar-06-2017, 02:21 AM
Last Post: raady07

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020