How can I solve this file handling issue?

[stevendaprano]

I've re-written your code, removing some unneeded imports and redundant code, and putting in some timing code to show you where the time is actually being spent:

# Untested.
import datetime
import os
import struct

import pigpio
import spidev

# We only have SPI bus 0 available to us on the Pi
bus = 0
#Device is the chip select pin. Set to 0 or 1, depending on the connections
device = 0

# Enable SPI
spi = spidev.SpiDev()
# Open a connection to a specific bus and device (chip select pin)
spi.open(bus, device)
# Set SPI speed and mode
spi.max_speed_hz = 4000000
spi.mode = 0

pi = pigpio.pi()
pi.set_mode(25, pigpio.INPUT)

def output_file_path():
    return os.path.join(os.path.dirname(__file__),
               datetime.datetime.now().strftime("%dT%H.%M.%S") + ".csv")

input("Press Enter to start the process ")
print("SM1 Process started...")
spi.xfer2([0x01])
while True:
    if pi.wait_for_edge(25, pigpio.RISING_EDGE, 5.0):
        print("Detected")
        data = [0]*2048
        
        with open(output_file_path(), 'w') as f:
            for x in range(392):
                t1 = datetime.datetime.now()
                spi.xfer2(data)
                print("time taken reading data:", datetime.datetime.now() - t1)
                t1 = datetime.datetime.now()
                for y in range(0, 2048, 4):
                    value=data[y]<<24 | data[y+1]<<16 | data[y+2]<<8 | data[y+3]
                    f.write(str(value) + '\n')
                print("time taken writing data:", datetime.datetime.now() - t1)
        break

My prediction is that writing will not take anywhere near as much time as you think.

If that code works as you expect, then try changing these three lines:

for y in range(0, 2048, 4):
    value=data[y]<<24 | data[y+1]<<16 | data[y+2]<<8 | data[y+3]
    f.write(str(value) + '\n')

into this:

values = struct.unpack(">" +"I"*512, bytes(data))
f.write('\n'.join([str(n) for n in values]))

and see if it is faster. (Disclaimer: I have not run this code, it may contain typos or other errors.)