Python Forum

Hi everyone,

I'm quite new to coding and I currently have a fairly simple python script that I run that queries an API based on an input file I provide it and then sends me back a new file (GIS image). If the script is interrupted, it creates a log file, so that when I start it again, it knows where to begin (rather than reprocessing all 1000 input lines) I run this thousands of times for the project I'm working on. The issue is that my internet connection isn't great, so the script often dies and then I need to manually restart it, which is especially an issue when I'm sleeping as I can't manually restart it. I'm looking for a way to restart the script (including specifying the input file). Basically to run the script it's just "py .\script.py .\filename.csv 1", which I would have already used in command line to start the script.

I've looked into using a "while" loop but that doesn't quite work (at least how I've done it), since I only want the script to run again after it crashes (which it needs to do to produce the log it will use when it starts up again). Any ideas or suggestions greatly appreciated!

Here is the script below for reference:

import requests
import csv
import sys
import os
import time
import json

server="https://website.com/"

if len(sys.argv) == 1:
	print("ERROR: Need a .csv file\neg. python coverage.py mydata.csv #group")
	quit()

if not os.path.exists("calculations"+"_"+sys.argv[2]):
		os.makedirs("calculations"+"_"+sys.argv[2])

# Open CSV file
csvfile = csv.DictReader(open(sys.argv[1]))


log = open("log.csv",'a', newline='')
wrlog = csv.writer(log, dialect='excel')

errors = open("errors.csv",'a', newline='')
wrerror = csv.writer(errors, dialect='excel')

with open("log.csv", newline='') as f:
    #reader = csv.reader(f)
    done = list(f)
done_clean=[]
for place in done:
    done_clean.append(site.rstrip('\r\n'))


f = open(sys.argv[1])
numlines = len(f.readlines())
sites_error=[]
number_sites_done=0

n=0
for row in csvfile:
    if row['nam'] not in sites_done_clean:
        #time.sleep(6) # Pause script. Important otherwise server will ban you.
        start_time = time.time() # Stopwatch start
        r = requests.post(server+"/API/area", data=row)
        j=json.loads(r.text)
        if 'error' in j:
            print('Error : '+str(row['nam']))
            print(r.text)
            wrerror.writerow([row['nam'],r.text])
        else:
            if 'tiff' in j:
            	#print (j['tiff'])
                r = requests.get(j['tiff'])
                if len(r.content)>2500000:
                    fn="calculations_"+row['group']+os.sep+str(row['nam'])+".tiff"
                    filename = open(fn,"wb")
                    filename.write(r.content)
                    filename.close()
                    wrprocessed.writerow([row['nam']])
                    print("  "+str(n+1)+"/"+str(numlines-number_done)+" Place --> "+str(row['nam']))
                else:
                    print('File size error: '+str(row['nam'])+"  File Size: "+str(len(r.content)))
                    wrerror.writerow([row['nam'],"File size error. Size: "+str(len(r.content))])
        elapsed = round(time.time() - start_time,1) # Stopwatch
        n=n+1
    else:
        number_done+=1