Python Forum
Pickle a file over a socket in 1024 segments
Thread Rating:
  • 1 Vote(s) - 2 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Pickle a file over a socket in 1024 segments
#1
Hello,

I really need some help. I have to create a client and server python program for school. The client is supposed to get files from the server using specific commands. I got everything working, but technically I am supposed to pickle the file and then send it across the network. I can not figure out how to send and then receive a pickled object over the network. I must send it in byte sizes of 1024. Any help would be appreciated.
Reply
#2
You can use on sender side io.BytesIO, which behaves like a file.
Then just read with the given buffer size of 1024 for example and
if the length is > 0, then send it. Otherwise break out of the while True loop.

import io

# your code
# sock = socket.socket()
# sock.connect(('ip', port))

data_to_send = b'Foo' * 2000 # we must send bytes
print('Data to send:', len(data_to_send))

data = io.BytesIO(data_to_send)
while True:
    chunk = data.read(1024)
    if not chunk:
        break
    sock.send(chunk)
What you do before and after this loop, is your responsibility. Maybe you've a protocol to tell the receiver side how much data you are going to send. Maybe you can use a sentinel to tell the receiver side, when the transfer has been finished. Socket programming is not easy.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#3
(Aug-07-2017, 10:36 AM)DeaD_EyE Wrote: You can use on sender side io.BytesIO, which behaves like a file.
Then just read with the given buffer size of 1024 for example and
if the length is > 0, then send it. Otherwise break out of the while True loop.

import io

# your code
# sock = socket.socket()
# sock.connect(('ip', port))

data_to_send = b'Foo' * 2000 # we must send bytes
print('Data to send:', len(data_to_send))

data = io.BytesIO(data_to_send)
while True:
    chunk = data.read(1024)
    if not chunk:
        break
    sock.send(chunk)
What you do before and after this loop, is your responsibility. Maybe you've a protocol to tell the receiver side how much data you are going to send. Maybe you can use a sentinel to tell the receiver side, when the transfer has been finished. Socket programming is not easy.

Thank you for your response!

I guess I should clarify a little bit. I have already successfully been able to transfer a file across the socket connection by reading the the file in bytes and then transferring in way such as you mentioned above.

However, per the instructions I'm supposed to pickle the file and then send it. I have tried opening a file, pickling it, and then sending it in chucks, but it is not working the same. I can not find a single simple example of what I'm trying to do, though I know it's possible. Any thoughts?
Reply
#4
You should show some code.  A pickled file is still just a file, so if it works before pickling, it should work after.
Reply
#5
The file is pickled already?
It's not clear to me.
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#6
So a snippet my attempted pickling server code:
 
with open(file_requested, 'rb') as temp:
     f = temp.read()
     f = dumps(f)

     # DOESN'T WORK - CAN'T READ 
     segment = f.read(BUFFER)

     self.connection_socket.send(segment)
     print('Sending file....')

     # Run while segment has binary data
     while segment:

          # DOESN'T WORK - CAN'T READ
          segment = f.read(BUFFER)
          self.connection_socket.send(segment)
A snippet my attempted pickling client code:
f = open('new_' + requested_file, 'wb')
segment = client_socket.recv(BUFFER)

# Keep track of the current size of the file as it's downloaded
current_file_size = len(segment)

# Start creating the file with the current segment
f.write(segment)

     # Run until the file is downloaded
     while current_file_size < file_size:
          segment = client_socket.recv(BUFFER)
          current_file_size += len(segment)
          f.write(segment)
# THEN UNPICKLE FILE
So, I know that I can read the parts of the pickled file as it gives me an error. Which means I have to break the file up some other way. Then, I know that my client needs to read each segment differently (maybe as a byte string) and then somehow I need to unpickle the completed file.

I'm new to Python and network programming, so it's a double whammy. Thanks for any help.

Another thought. Let's assume the following:
  • I have successfully grabbed a file (image, pdf, etc) of 10000 bytes from a directory
  • I then pickled the file (we will call it pickledFile)

Can someone give me basic code to send the pickled file (in chunks of 1024 bytes) over a socket connection and then basic code to receive the pickled file (in chunks of 1024 bytes) and then unpickle it?

Then I can see how it's done and modify my code to work.

i think some of the confusion for me is that I was told it is easier to pickle file and then send it as opposed to other options. Though in my original code I just read the file as binary and then sent it in chucks and had no problem.

Thanks.
Reply
#7
Here a minimal implementation. I have done it to wake up.
Later i realized that my variable name for received was wrong..

server.py
import socket
import os
import struct
import hashlib


def handle(client, addr):
    buffsize = 1024
    file = '/home/deadeye/foster_the_people_20110408_128.mp3'
    print('File size:', os.path.getsize(file))
    fsize = struct.pack('!I', os.path.getsize(file))
    print('Len of file size struct:', len(fsize))
    client.send(fsize)
    with open(file, 'rb') as fd:
        while True:
            chunk = fd.read(buffsize)
            if not chunk:
                break
            client.send(chunk)
        fd.seek(0)
        hash = hashlib.sha512()
        while True:
            chunk = fd.read(buffsize)
            if not chunk:
                break
            hash.update(chunk)
        client.send(hash.digest())
    client.close()

addr = ('', 8080)
sock = socket.socket()
sock.bind(addr)
sock.listen(5)

# while True loop normally
# to handle incomming connections 
client, addr = sock.accept()
print('Got connection from:', addr)
handle(client, addr)
sock.close()
client.py
mport socket
import os
import struct
import hashlib

addr = ('localhost', 8080)
sock = socket.socket()
sock.connect(addr)
print('Connected to', addr)

received = 0
chunks = []
while received< 4:
    data = sock.recv(4 - received)
    received += len(data)
    chunks.append(data)
print('Received len of file size struct', len(b''.join(chunks)))
fsize = struct.unpack('!I', b''.join(chunks))[0]
print('Filesize:', fsize)

buffer = 1024
received = 0
chunks = []
while received < fsize:
    data = sock.recv(min(fsize - received, buffer))
    received += len(data)
    chunks.append(data)
file = b''.join(chunks)
print('Received file')
print('Expected size:', fsize)
print('Received size:', len(file))

received = 0
chunks = []
while received < 64:
    data = sock.recv(64 - received)
    received += len(data)
    chunks.append(data)
sha512 = b''.join(chunks)
print('Received Hash', len(sha512), sha512)
sock.close()

hash_ok = hashlib.sha512(file).digest() == sha512
print('Hash is ok') if hash_ok else print('Hash is not ok')
The problem with sockets are, that the call recv(size) does not guarantee that you really receiving this given data size. So you have to send the file size you want to send. With struct.pack and struct.unpack you can send the given file size. Afterwards I'm sending a hash, to check if the transfer was really right.

I did not any optimization. On client side you can write the data to disk instead of appending the chunks into a list. The check afterwards can be done with the same procedure as shown on server side. But this is optional to check the data integrity.

If you want to have the song, write a PN :-D
Almost dead, but too lazy to die: https://sourceserver.info
All humans together. We don't need politicians!
Reply
#8
I finally got it. Thanks for all the help.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  how to send an image from server to client using PICKLE module dafdaf 1 3,066 Jun-02-2020, 01:08 PM
Last Post: nuffink
  File transfer with xmodem CRC (1024) checksum Jhonbovi 3 8,211 Nov-08-2018, 09:01 AM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020