refused connection

[Gribouillis]

Are you 100% positive that the problem I'm facing has to do with cv2 ?

No, not at all. You need to investigate why the loop never exits. First thing it is a while True loop, so how do you expect it to exit? Also the s.recv(4096) is potentially blocking if there is nothing to read on the socket. The best thing to do is to add print or logging statements in the loop until you find which statements are actually executed.