Dickey Fuller failure

[Led_Zeppelin]

When I run the Dickey-Fuller test as shown here:

# dickey-fuller test
dft = adfuller (image_data_org['sensor_04'], autolag='AIC')
# dickey fuller test output
dfo = pd.Series(dft[0:4], index=['Test Statistic','p-value','Lags Used','Observations Used'])
for key,value in dft[4].items():
    dfo['Critical Value (%s)'%key] = value
    
print(dfo)

I get a successful run on most sensors. But on sensor_03 it fails. The error is now shown:

Output:
alueError                                Traceback (most recent call last)
Input In [61], in <cell line: 2>()
      1 # dickey-fuller test
----> 2 dft = adfuller (image_data_org['sensor_04'], autolag='AIC')
      3 # dickey fuller test output
      4 dfo = pd.Series(dft[0:4], index=['Test Statistic','p-value','Lags Used','Observations Used'])

File ~\miniconda3\envs\pump-failure-pred\lib\site-packages\statsmodels\tsa\stattools.py:260, in adfuller(x, maxlag, regression, autolag, store, regresults)
    165 def adfuller(
    166     x,
    167     maxlag: int | None = None,
   (...)
    171     regresults=False,
    172 ):
    173     """
    174     Augmented Dickey-Fuller unit root test.
    175 
   (...)
    258     See example notebook
    259     """
--> 260     x = array_like(x, "x")
    261     maxlag = int_like(maxlag, "maxlag", optional=True)
    262     regression = rename_trend(regression)

File ~\miniconda3\envs\pump-failure-pred\lib\site-packages\statsmodels\tools\validation\validation.py:147, in array_like(obj, name, dtype, ndim, maxdim, shape, order, contiguous, optional)
    145     if arr.ndim != ndim:
    146         msg = "{0} is required to have ndim {1} but has ndim {2}"
--> 147         raise ValueError(msg.format(name, ndim, arr.ndim))
    148 if shape is not None:
    149     for actual, req in zip(arr.shape, shape):

ValueError: x is required to have ndim 1 but has ndim 2

I have no idea what this error means or how to correct it.
Please explain this error.

Thanks in advance.

Respectfully,

LZ

[deanhystad]

Do you mean "sensor_04"? The error message says and the posted code say "sensor_04".

If it works for some sensors and not for others, I would examine how this differs for different sensors.

image_data_org['sensor_0X']

If Dickey-Fuller works for sensor_2 and fails for sensor_4, the value in image_data_org[] will differ. It sounds like it will differ structurally. The error message says the program failed because it expects image_data_org['sensor_04'] to be a 1D array but it is really a 2D array.

To debug, fund two examples; one that runs the Dickey-Fuller test and one that fails, Look at the shape of the array you are passing to the test. How do they differ. If you still have problems after that, report back with what you discovered.

[Led_Zeppelin]

I see your point. But please remember that all of the data came from Kaggle. So, it should be okay. Others are using this data.

All I did was use it and like they did I put through standard scaler. Please elaborate more on the method to find this error.
A reference would be fine.

I am just not sure. I am not looking forward to looking through 220320 data points to find one incorrect one. I did replace all missing data
with the median of that specific sensor.

Wha else can I do.

Respectfully,

LZ

[deanhystad]

You really need to stop saying things like "So it should be ok" followed by "All I did". I assume every line of code I write contains a bug and that everything is suspect until ruled out. You could have messed up getting the data. You could have messed up processing the data. You could have a messed up a cut and paste somewhere that ended up replacing some of your sensor data. Until you look you do not know.

You do not need to look through the data. You only need to look at the type, and if both are arrays of the same type, then look at the shape. My guess is your problem sensors might be arrays of strings or some other iterable type.

[Led_Zeppelin]

Now this is highly unusual. I tried to recreate this error and I did after reading your post.

Then I quit for the day and came back the next day.

I isolated the error. Running the Dickey-Fuller test on only that sensor, sensor_04. No matter what I did, and I did move sensor_04 Dickey-Fuller around, I could not recreate the error.

I am not the only one who uses this computer my default web browser has been Edge. By that I mean when I run jupyter notebook it uses Edge.

But I suddenly noticed it was Firefox connected to the jupyter notebook the second day. So, I ran the whole enchilada
again and no error. The exact same code that caused the error under Edge shows no error under Firefox.

I am not complaining, but it started when I could not recreate the error after recreating several times the day before.

I guess the only way to be sure is to run the program again and see if the error manifests itself.

How does one select the web browser to run with jupyter notebook. I thought that jupyter notebook only uses the default
browser. Is there another way?

I want to run jupyter notebook and select which browser it connects. I do not want it selected for me.

R,

LZ