Python Forum
python script for inserting rows into hbase table
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
python script for inserting rows into hbase table
#3
I am not expecting anyone to write a code for me.
I need a suggestion.
I had a script for inserting data into hbase table.
import glob
import connect_to_hbase
import os
import csv
import testingsun1
#Changing to the directory path to search for the directory name containing "GSK"

table = connect_to_hbase.conn.table(connect_to_hbase.table_name_Target)
base_path = '/ai2/data/dev/admin/inf/*{}*'
patterns = testingsun1.z

#x = table.row(row.serialnum) 
for pattern in patterns:
	
	search_path =  base_path.format(pattern)
	for f in glob.glob(search_path):
		print("-----------------------")
		print ("The directory path is:")
		print f
		print("List of files in the directory are:")
		os.chdir('/ai2/data/dev/admin/inf/')
    		os.chdir(f)
    		cwd = os.getcwd()
		for subdir, dirs, files in os.walk(cwd, topdown=True):
				for file23 in glob.glob('*.err'):
					print file23
					with open(file23) as f:
						strings = f.read()
						search_word = 'checksum'
						search_word1 = 'Threshold'				
						if(search_word1 in strings):
							with open('/ai2/data/dev/serial/sys/sys_ath_pvt/main/qv_thresholdhistory.csv','r') as csvfile:
								readCSV = csv.reader(csvfile, delimiter=',')
								next(readCSV, None)
								for row in readCSV:
									#print row[2]
									#print os.path.splitext(file23)[0]
									if(os.path.splitext(file23)[0] in row[2]):
										if row[5] and row[9]:
											if(int(row[5]) - int(row[9]) >= 0):
												a = row[3].split("|")
												b = row[4].split("|")
												recom=b[1]
												app_id=row[0]
												entity=row[2]
												existing=a[1]
												data_control=search_word1
												store=recom,app_id,entity,existing,data_control
												print(store)
												table.put('row-key', {'APP_VALUES:SEGMENT':'','APP_VALUES:API_KEY':'','APP_VALUES:RECOMMENDED_VALUE':store[0], 'APP_VALUES:APPLICATION_NAME':store[1],'APP_VALUES:ENTITY_NAME':store[2],'APP_VALUES:EXISTING_VALUE':store[3],'APP_VALUES:DATA_CONTROL':store[4],'APP_VALUES:APPLICATION_ID':store[1]})
											
									
						else:
							print ("word not found")
In the above code print (store) is printing all the values which are to be inserted into the hbase table.
But table.put statement is inserting only the last output value of the print (store) statement.

Print store output:
Output:
The directory path is: /ai2/data/dev/admin/inf/inf_DRM_pvt List of files in the directory are: call_report_y9clookup.err ('0', 'drm', 'call_report_y9clookup', '300', 'Threshold') ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_GSK_pvt List of files in the directory are: gs_attribute.err word not found ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_IPT_pvt List of files in the directory are: run_ingest_IPT_daily_1246.err word not found ----------------------- The directory path is: /ai2/data/dev/admin/inf/inf_XU0_pvt List of files in the directory are: t_itm.err ('12', 'xu0', 't_itm', '500', 'Threshold')
Data inserted in the table:
Output:
1.8.7-p357 :030 > scan 'TMP_ATLAS_DSCVRY_APPS' ROW COLUMN+CELL row-key column=APP_VALUES:API_KEY, timestamp=1526959583897, value= row-key column=APP_VALUES:APPLICATION_ID, timestamp=1526959583897, value=xu0 row-key column=APP_VALUES:APPLICATION_NAME, timestamp=1526959583897, value=xu0 row-key column=APP_VALUES:DATA_CONTROL, timestamp=1526959583897, value=Threshold row-key column=APP_VALUES:ENTITY_NAME, timestamp=1526959583897, value=t_itm row-key column=APP_VALUES:EXISTING_VALUE, timestamp=1526959583897, value=500 row-key column=APP_VALUES:RECOMMENDED_VALUE, timestamp=1526959583897, value=12 row-key column=APP_VALUES:SEGMENT, timestamp=1526959583897, value= 1 row(s) in 0.3440 seconds
Where I am doing wrong?
Please suggest.

Thanks
Reply


Messages In This Thread
RE: python script for inserting rows into hbase table - by lravikumarvsp - May-22-2018, 03:28 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  Is there a *.bat DOS batch script to *.py Python Script converter? pstein 3 3,598 Jun-29-2023, 11:57 AM
Last Post: gologica
  The code I have written removes the desired number of rows, but wrong rows Jdesi1983 0 1,681 Dec-08-2021, 04:42 AM
Last Post: Jdesi1983
  Dynamically Add rows to table TommyAutomagically 1 2,189 Nov-04-2021, 10:59 PM
Last Post: TommyAutomagically
  Slittping table into Multiple tables by rows drunkenneo 1 2,145 Oct-06-2021, 03:17 PM
Last Post: snippsat
  Pandas DataFrame combine rows by column value, where Date Rows are NULL rhat398 0 2,181 May-04-2021, 10:51 PM
Last Post: rhat398
  Indexing [::-1] to Reverse ALL 2D Array Rows, ALL 3D, 4D Array Columns & Rows Python Jeremy7 8 7,457 Mar-02-2021, 01:54 AM
Last Post: Jeremy7
  Python script to summarize excel tables, then output a composite table? i'm a total n surfer349 1 2,452 Feb-05-2021, 04:37 PM
Last Post: nilamo
  write mariadb table rows query to each file? shams 1 1,983 Feb-02-2021, 04:10 PM
Last Post: buran
  Inserting multiple rows in a single request. swaroop 2 3,009 Jan-11-2021, 01:34 PM
Last Post: swaroop
  Fetch Oracle DB rows & print it in HTML file with table's col headers in table format tssr_2001 1 3,067 Sep-04-2020, 01:39 PM
Last Post: ibreeden

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020