Getting an error while trying to process data, low memory when memory is not low?

[bkeith12]

Im using a program called PING Mapper to process side scan sonar. I usually work in MATLAB so I don't have a ton of experience in Python, this PING Mapper app is pretty good and certainly good enough for what I want to do. I have a nice GUI in matlab and want to run this python code in my app in matlab to process data.

Any way, I followed the install instructions for PING Mapper and all seems to be good. I can open a .DAT file and export jpg images of the sonar, in PING Mapper there is an option for georectifying the data, meaning it can be thrown in GIS or mapped in MATLAB. This is where the problem lies, when I select this option and click run it goes through the usual process and I can monitor what is happening in cmd. It will georectify one sonar file and then I get the error below. My Python skills arent great but what I think its saying is that there isnt enough allocated memory for the process to continue.

My computer has 64gb of ram, when I run task manager at the same time as the code I never go above 25% RAM usage. Honestly im not sure what to try next or how to continue, I need to figure this out soon and Im hopeing some one can help me with the error below:

Here is what cmd is telling me when the program stops:

!!!!!!!!!!!!!!!!!
An Error Occured.

When an error occurs an error log file is created, here is what it says:

Error Thrown:
joblib.externals.loky.process_executor._RemoteTraceback:
"""

Error:
Traceback (most recent call last):
  File "C:\Users\name\miniconda3\envs\ping\Lib\site-packages\joblib\externals\loky\process_executor.py", line 463, in _process_worker
    r = call_item()
        ^^^^^^^^^^^
  File "C:\Users\name\miniconda3\envs\ping\Lib\site-packages\joblib\externals\loky\process_executor.py", line 291, in __call__
    return self.fn(*self.args, **self.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\name\miniconda3\envs\ping\Lib\site-packages\joblib\parallel.py", line 598, in __call__
    return [func(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\name\PythonPackages\PINGMapper\src\class_rectObj.py", line 1402, in _rectSonParallel
    out = warp(img.T,
          ^^^^^^^^^^^
  File "C:\Users\name\miniconda3\envs\ping\Lib\site-packages\skimage\transform\_warps.py", line 1030, in warp
    coords = warp_coords(coord_map, output_shape)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\name\miniconda3\envs\ping\Lib\site-packages\skimage\transform\_warps.py", line 688, in warp_coords
    tf_coords = coord_map(tf_coords)
                ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\name\miniconda3\envs\ping\Lib\site-packages\skimage\transform\_warps.py", line 1022, in coord_map
    return inverse_map(*args, **map_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\name\PythonPackages\PINGMapper\src\funcs_common.py", line 397, in __call__
    affines = np.array(
              ^^^^^^^^^
numpy._core._exceptions._ArrayMemoryError: Unable to allocate 1.92 GiB for an array with shape (28649892, 3, 3) and data type float64
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\name\PythonPackages\PINGMapper\gui_main.py", line 334, in <module>
    rectify_master_func(**params)
  File "C:\Users\name\PythonPackages\PINGMapper\src\main_rectify.py", line 514, in rectify_master_func
    Parallel(n_jobs= np.min([len(chunks), threadCnt]), verbose=10)(delayed(son._rectSonParallel)(i, filter, cog, wgs=False) for i in chunks)
  File "C:\Users\name\miniconda3\envs\ping\Lib\site-packages\joblib\parallel.py", line 2007, in __call__
    return output if self.return_generator else list(output)
                                                ^^^^^^^^^^^^
  File "C:\Users\name\miniconda3\envs\ping\Lib\site-packages\joblib\parallel.py", line 1650, in _get_outputs
    yield from self._retrieve()
  File "C:\Users\name\miniconda3\envs\ping\Lib\site-packages\joblib\parallel.py", line 1754, in _retrieve
    self._raise_error_fast()
  File "C:\Users\name\miniconda3\envs\ping\Lib\site-packages\joblib\parallel.py", line 1789, in _raise_error_fast
    error_job.get_result(self.timeout)
  File "C:\Users\name\miniconda3\envs\ping\Lib\site-packages\joblib\parallel.py", line 745, in get_result
    return self._return_or_raise()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\name\miniconda3\envs\ping\Lib\site-packages\joblib\parallel.py", line 763, in _return_or_raise
    raise self._result
numpy._core._exceptions._ArrayMemoryError: Unable to allocate 1.92 GiB for an array with shape (28649892, 3, 3) and data type float64

EDIT: It might have something to do with my data? When installing PING Mapper, it also installs some example data and when I process that it works fine, this example data is bigger than my data set so at this point I have no idea what the issue might be. Thanks!

Larz60+ write Dec-21-2024, 07:41 AM:
Please post all code, output and errors (it it's entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button. Tags have been added fro you this time. Please use on future posts.