Python Forum
Convert String of an int array to a Numpy array of ints
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Convert String of an int array to a Numpy array of ints
#1
Hi

I am sending a Numpy int array over RabbitMQ and it seem to be receiving correctly.

This is the output RabbitMQ's publish that is sent:

>py msg_send_test.py
Output:
[x] Sent '0,1617731222,1617731222810382600,4,[ 48 251 9 205]'
This is the output on the receiving end from RabbitMQ's consumer:
>py msg_receive_test.py
Output:
[*] Waiting for messages. To exit press CTRL+C [x] message '0,1617732445,1617732445003573700,4,[209 227 205 72]' [x] message type <class 'str'> [x] parts ['0', '1617732445', '1617732445003573700', '4', '[209 227 205 72]'] [x] parts len 5 [x] imageStr [209 227 205 72] [x] imageStr type <class 'str'>
This is the section of code to produce the output from the RabbitMQ consumer:
    message = body.decode()
    parts = message.split(',')
    print("[x] message      %r" % message)
    print("[x] message type %s" % type(message))
    print("[x] parts        %s" % parts)
    print("[x] parts len    %s" % len(parts))
    imageStr = str(parts[4])
    print("[x] imageStr     %s" % imageStr)
    print("[x] imageStr type %s" % type(imageStr))
What I would like to do is convert the imageStr into a Numpy array of ints. The whole array is a string, not an array of individual strings.

Any suggestions would be appreciated.

Thanks...

When I try using Numpy's fromstring method:
    imageArr = np.fromstring(imageStr, dtype=float, sep=' ')
    print("[x] imageArr type %s" % type(imageArr))
    print("[x] ", imageArr)
it gives the following output:
Output:
[x] imageStr type <class 'str'> [x] imageArr type <class 'numpy.ndarray'> [x] []
Reply
#2
I'm trying this out. In the output from fromstring, I added the following
    imageStr = str(parts[4])
    imageStr = imageStr.replace('[','')
    imageStr = imageStr.replace(']','')
    imageArr = np.fromstring(imageStr, dtype=int, sep=' ')
    print("[x] imageArr type %s" % type(imageArr))
    print("[x] imageArr ", imageArr)
And this is now the output:
Output:
[*] Waiting for messages. To exit press CTRL+C [x] message '0,1617735054,1617735054620256300,4,[104 126 180 87]' [x] message type <class 'str'> [x] parts ['0', '1617735054', '1617735054620256300', '4', '[104 126 180 87]'] [x] parts len 5 [x] imageStr 104 126 180 87 [x] imageStr type <class 'str'> [x] imageArr type <class 'numpy.ndarray'> [x] imageArr [104 126 180 87]
Looks like the '[]' had to be removed from the string, and Numpy puts them back when converting to the int array.
Reply
#3
I have found that this only works up to some limits.

If the array becomes too large, it only handles the first three elements, I've been able to get it to work up to a 1000 element array. It does not work correctly at 10000 elements. Any idea why the conversion doesn't work?

Thanks...
Reply
#4
In sending a message (Numpy int array) using RabbitMQ, this is how it is setup and sent

testImage = np.random.randint(0, 255, 10*1000)  # grey scale images
strImage = np.array_str(testImage)
message = <string of meta data> + strImage
channel.basic_publish(exchange='', routing_key='task_queue', body=message, properties=pika.BasicProperties(delivery_mode=2,  # make message persistent
    ))
This is the output of the message sent:
Output:
[x] Message params '0,1617801844,1617801844400951300,10000,[113 75 36 ... 167 157 131]'
In modifying the received message (from rabbitMQ) to get a Numpy int array I've tried:
    message = body.decode()
    parts = message.split(',')
    print("[x] message      %r" % message)
    print("[x] message type %s" % type(message))
    print("[x] parts        %s" % parts)
    print("[x] parts len    %s" % len(parts))
    
This is the output for the above code:
Output:
[x] message '0,1617801844,1617801844400951300,10000,[113 75 36 ... 167 157 131]' [x] message type <class 'str'> [x] parts ['0', '1617801844', '1617801844400951300', '10000', '[113 75 36 ... 167 157 131]'] [x] parts len 5
I then start working of the message part that is the Numpy array:
    imagePart = parts[4]
    print()
    print("[x] imagePart ", imagePart)
    print(repr(imagePart))
    print(re.sub("\s+", ",", imagePart.strip()))
    print(type(re.sub("\s+", ",", imagePart.strip())))
    print(np.array(re.sub("\s+", ",", imagePart.strip())))
    print(np.array(re.sub("\s+", ",", imagePart.strip())).size)
    print(type(np.array(re.sub("\s+", ",", imagePart.strip()))))
    print(repr(np.array(re.sub("\s+", ",", imagePart.strip()))))
    print()
    
The output for the above code block is (I'm not sure why it thinks it's of dtype 'U27'):
Output:
[x] imagePart [113 75 36 ... 167 157 131] '[113 75 36 ... 167 157 131]' [113,75,36,...,167,157,131] <class 'str'> [113,75,36,...,167,157,131] 1 <class 'numpy.ndarray'> array('[113,75,36,...,167,157,131]', dtype='<U27')
I then try a different approach:
    npPart = np.asarray(imagePart, dtype=np.str)
    npSplit = np.char.split(npPart)
    npSplitPart = np.array(npSplit, dtype=object)
    print(npSplitPart)
    print(npSplitPart.dtype)
    print("[x] npSplitPart type ", type(npSplitPart))
    print("[x] npSplitPart shape ", npSplitPart.shape)
    print("[x] npSplitPart size ", npSplitPart.size)
    print(repr(npSplitPart))
    print(np.array(npSplitPart))
This is the output for the above code block:
Output:
['[113', '75', '36', '...', '167', '157', '131]'] object [x] npSplitPart type <class 'numpy.ndarray'> [x] npSplitPart shape () [x] npSplitPart size 1 array(list(['[113', '75', '36', '...', '167', '157', '131]']), dtype=object) ['[113', '75', '36', '...', '167', '157', '131]']
What I would like is to get back to the original Numpy int array. Any idea what I am doing wrong or missing?

Thanks...
Reply
#5
I think it would be easier to do something like this:
import numpy as np
import json

testImage = np.random.randint(0, 255, 10*1000)
metadata = ""
data = {"image": testImage, "metadata": metadata}
string_data = json.dumps(data)  # a string format of the data

# Receiving end
data = json.loads(string_data)
metadata = data["metadata"]
image = data["image"]
Reply
#6
Thanks SheeppOSU,

I had tried something like that (if '{}' create a list as that is what I did. I read that lists were serializable out of the box.

#sending
listImage = testImage.tolist()

msgList = [imageID, timeStampSeconds, timeStampNano, int(testImage.size)]
msgList += listImage

json.dumps(msgList)

#receiving
message = json.loads(body)
recvBody = list(message)
arr = np.array(msgList[4:len(msgList)]) # metadata is len 4
Thanks again...
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Where does the array printout come from? Mark17 2 113 Aug-02-2021, 05:42 PM
Last Post: Mark17
  Problem with Json Array kwekey 2 119 Aug-02-2021, 05:11 PM
Last Post: kwekey
  Convert each element of a list to a string for processing tester_V 6 566 Jun-16-2021, 02:11 AM
Last Post: tester_V
Photo How to process the array correctly Fly_jia 4 430 Jun-09-2021, 01:47 AM
Last Post: Fly_jia
Question convert unlabeled list of tuples to json (string) masterAndreas 4 1,217 Apr-27-2021, 10:35 AM
Last Post: masterAndreas
  Compare each element of an array in a logic statement without using a for loop leocsmith 3 664 Apr-01-2021, 07:57 PM
Last Post: deanhystad
  Loop different actions for an array Tibovdv 4 623 Mar-25-2021, 06:46 PM
Last Post: jefsummers
  Indexing [::-1] to Reverse ALL 2D Array Rows, ALL 3D, 4D Array Columns & Rows Python Jeremy7 8 1,441 Mar-02-2021, 01:54 AM
Last Post: Jeremy7
  All Array Index JohnnyCoffee 3 532 Feb-26-2021, 06:46 AM
Last Post: ndc85430
Shocked An array "mystery": The same array, the same operations but different outcomes mewss 3 463 Feb-17-2021, 06:34 PM
Last Post: mewss

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020