Python Forum
Optimising processing speed
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Optimising processing speed
#1
Hi
I have an HP computer with 24 cores and 48 processors. Part of my work entails creating large tile packages. When I use the ArcGIS tools in Desktop 10.4.1 and Pro only 3 - 4% of the CPU is being used. I've tried writing a python script using multiprocessing, but the CPU % remains at 3 - 4%, with only one 1 processor being used at a time. My script is below, is there any way of adapting it to make full use of my computers processing capacity?

import arcpy, os, multiprocessing, shutil
from arcpy import env
from arcpy.sa import *
from multiprocessing import Process, Queue, Pool

# The amount of cores used for processing (rule of thumb: number of cores minus 2).
cores = 22

# Set environment settings
arcpy.env.workspace = "D:/4234_TRJE18_Land/N500"
mxd = arcpy.mapping.MapDocument("D:/4234_TRJE18_Land/N500/4234_TRJE18_Land_2018_OH_N500.mxd")
arcpy.env.overwriteOutput = True

def main():

    delInterimFiles = True
    print str(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")) + " starting the processing object for " + str(mxd) + " Building tpk using multiprocessors"
    pool = multiprocessing.Pool(processes=cores)
    arcpy.CreateMapTilePackage_management("D:/4234_TRJE18_Land/N500/4234_TRJE18_Land_2018_OH_N500.mxd", "ONLINE", "D:/4234_TRJE18_Land/N500/4234_TRJE18_Land_2018_OH_N500_tmp.tpk", "MIXED", "15")
    print str(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")) + " finished processing" + str(mxd) + " Building pyramids using multiprocessors"
    pool.close()
    pool.join()

if __name__ == '__main__':
   main()
Reply
#2
You just make a set of threads which works asynchronously, but not in parallel (not in Python - because of GIL).
Try some library, it's not easy, If you can do it in numpy, would be the best, if not, try this:
https://lwn.net/Articles/756192/
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020