![]() |
websockets and pymongo exception - Printable Version +- Python Forum (https://python-forum.io) +-- Forum: Python Coding (https://python-forum.io/forum-7.html) +--- Forum: General Coding Help (https://python-forum.io/forum-8.html) +--- Thread: websockets and pymongo exception (/thread-38391.html) |
websockets and pymongo exception - sankar2000 - Oct-07-2022 Hello everyone. I am building a websocket client app that receives all the data and inserts it into a MongoDB database. The program runs, but after a while, some exceptions are rising. import websocket import _thread import time import json import asyncio import os import pwd import socket import traceback as tb import pymongo from datetime import datetime import threading import ssl httpthreat = mongo['httpthreat'] collection1 = httpthreat['collection1'] websocket_link = "wss://somewebservice.com/service" def get_data(HOST,PORT,TIMEOUT=5,RETRY=3): for itry in range(1,RETRY+1): try: sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.settimeout(TIMEOUT) sock.connect((HOST, PORT)) # processing some data # processing some data # processing some data return str(data,'utf-8') except Exception as e: traceback_str = ''.join(tb.format_exception(None, e, e.__traceback__)) print("get_data exception") time.sleep(TIMEOUT) print(traceback_str) return None def process_data(IP,PORT): try: data = get_data(IP,int(PORT)) collection1.insert_one({"ip":IP,"port":PORT,"data":data,"date":datetime.now()}) # <==== this line throws exceptions # i even tried this, but no use: ''' for i in range(5): try: collection1.insert_one({"ip":IP,"port":PORT,"data":data,"date":datetime.now()}) # <==== this line throws exceptions anyway break except pymongo.errors.AutoReconnect: print("pymongo.errors.AutoReconnect exception ["+str(i)+"]") time.sleep(pow(2, i)) ''' except Exception as e: traceback_str = ''.join(tb.format_exception(None, e, e.__traceback__)) print("process_data exception") print(traceback_str) def on_message(ws, message): message_json = json.loads(message) if(message_json['protocol'] == "http"): try: threads = [threading.Thread(target=process_data, args=(message_json['ip'], message_json['port']))] for thread in threads: thread.start() #for thread in threads: #thread.join() # waits for thread to complete its task <==== if i uncomment this, program slows except Exception as e: print("on_message exception") traceback_str = ''.join(tb.format_exception(None, e, e.__traceback__)) print(traceback_str) def on_error(ws, error): print(error) def on_close(ws, close_status_code, close_msg): print("### closed ###") time.sleep(1) websocket.enableTrace(True) ws = websocket.WebSocketApp(websocket_link, on_open=on_open, on_message=on_message, on_error=on_error, on_close=on_close) ws.daemon = True threading.Thread(target=ws.run_forever(ping_interval=70, ping_timeout=10,sslopt={"cert_reqs": ssl.CERT_NONE})) def on_open(ws): print("Opened connection") if __name__ == "__main__": while True: try: print("Starting") websocket.enableTrace(True) ws = websocket.WebSocketApp(websocket_link, on_open=on_open, on_message=on_message, on_error=on_error, on_close=on_close) ws.daemon = True threading.Thread(target=ws.run_forever(ping_interval=70, ping_timeout=10,sslopt={"cert_reqs": ssl.CERT_NONE})) except Exception as e: traceback_str = ''.join(tb.format_exception(None, e, e.__traceback__)) print("main exception") print(traceback_str) print("Restarting ...") time.sleep(1) I am usining websocket library and python3 Errors that i am getting : Error1 (occurance interval 10 minutes): I don't understand why this occurs at this interval, if i load the socket in browser, this never happenError2 (occurance interval about 10 minutes): When this happens, it triggers restarting of the websocket.WebSocketAppI am also getting a lot of 'pymongo' errors Error3: After a while i get a lot of pymongo.errors.AutoReconnect then program does not write to the database anymore. and I don't understand how come a program can run a while then start to throw exception like crazy, especially when you connect it to a MongoDB database with pymongo. I have read about pymongo and it is not fork safe, but thread safe yes.If i load the websocket in browser, it is simply working with no problems at all. After this script is completed i would like to post it here as it might help others. Any hints please? |