May-23-2018, 03:04 AM
Hi
I had a python script which will search for directories containing the specific name and then search for the error files in those directories.
I want to run this script as a process, so that I can run 10 processes parallel at a time.
How can I acheive this ? with multi processing or multi threading?
Suggest me the best way and how to call this python script
I had a python script which will search for directories containing the specific name and then search for the error files in those directories.
I want to run this script as a process, so that I can run 10 processes parallel at a time.
How can I acheive this ? with multi processing or multi threading?
Suggest me the best way and how to call this python script
import connect_to_hbase import glob import os import csv import numpy as np connection= connect_to_hbase.conn table = connection.table(connect_to_hbase.table_name_Source) row_key = '\x00\x00\x00\x01' res = list() for row_key, data in table.scan(columns=['DETAILS:APP_ID']): result=data.values() for i in result: res.append(i) x = np.array(res) z=np.unique(x) print(z) patterns = str(z) table = connect_to_hbase.conn.table(connect_to_hbase.table_name_Target) base_path = '/ai2/data/dev/admin/inf/*{}*' for pattern in patterns: search_path = base_path.format(pattern) for f in glob.glob(search_path): print("-----------------------") print ("The directory path is:") print f print("List of files in the directory are:") os.chdir('/ai2/data/dev/admin/inf/') os.chdir(f) cwd = os.getcwd() for subdir, dirs, files in os.walk(cwd, topdown=True): for file23 in glob.glob('*.err'): print file23
Output:Connected to Hbase
['ACM' 'ACX' 'AW' 'BC' 'BLS' 'CA' 'CLP' 'CMU' 'CR' 'CSE' 'CTD' 'DHD' 'DMS'
'DRM' 'GSK' 'IPT' 'XU0']
-----------------------
The directory path is:
/ai2/data/dev/admin/inf/inf_IPT_pvt
List of files in the directory are:
run_ingest_IPT_daily_1246.err
-----------------------
The directory path is:
/ai2/data/dev/admin/inf/inf_CLP_pvt
List of files in the directory are:
run_ingest_CLP_daily_1240.err
-----------------------
The directory path is:
/ai2/data/dev/admin/inf/inf_IPT_pvt
List of files in the directory are:
run_ingest_IPT_daily_1246.err
-----------------------
The directory path is:
/ai2/data/dev/admin/inf/inf_CTD_pvt
List of files in the directory are:
run_ingest_CTD_daily_1250.err
-----------------------
The directory path is:
/ai2/data/dev/admin/inf/inf_IPT_pvt
List of files in the directory are:
run_ingest_IPT_daily_1246.err
-----------------------
The directory path is:
/ai2/data/dev/admin/inf/inf_XU0_pvt
List of files in the directory are:
t_itm.err
-----------------------
The directory path is:
/ai2/data/dev/admin/inf/inf_ACX_pvt
List of files in the directory are:
accountshighfocus.err
acm_access_log.err
-----------------------
The directory path is:
/ai2/data/dev/admin/inf/inf_XU0_pvt
List of files in the directory are:
t_itm.err
-----------------------
The directory path is:
/ai2/data/dev/admin/inf/inf_CMU_pvt
List of files in the directory are:
run_ingest_CMU_daily_1247.err
-----------------------
Thanks