Jun-27-2024, 05:56 PM
I need some high level advice on how to code this, thinking asyncio, threading, and multiprocessing...
I'm implementing a stock algo to trade in real time. For this, I am using polygon.io as my market data provider, and Tradestation.com as my broker. Each has an API.
Basically the algo needs to take in real time market data from the Polygon API, crunch the numbers, make trade decisions, and use the Tradestation API to send those trade decisions for execution.
If I was doing this for just one stock simultaneously, the answer would be pretty simple just use asyncio. Each stock symbol is completely independent of the others so I was going to write one python script, and just run multiple instances of it with different symbols. Thus, the OS could also schedule each instance on it's own core for efficiency. The problem is that the costs for more than one data stream from Polygon get really expensive... one data stream means this all needs to be done in a single python script. Again, if it was a single stock symbol... no problem. But if I'm running 10 or 20 or 30 symbols at the same time, at some point, it WILL get bogged down on a single core....
So this all starts with the Polygon API which provides market data. This portion is quite simple - it's just a single websockets stream that send me data in a firehose. The more symbols... the more data. I can subscribe to as many as I want in a single stream, and as long as I can handle it on the receiving end there is no limit from the Polygon end how much they will send. So I was thinking of using multiprocessing to have this run on it's own core... and all this would need to do is separate out the symbols and decode the raw json data into a format that my algo can use for input.
Next comes the algo data crunching and order sending via the Tradestation API. Here is where we break it out by stock symbol, as each one is independent. I was thinking of putting each symbol on it's own thread so that multiprocessing would then run each on different cores. So first the algo needs to run to crunch the raw data to make either a buy or sell decision - and this would happen once per second. It then sends that off to Tradestation via their REST API for execution. The actual execution can be rather complicated - because I lose too much on slippage on market orders I need to do creative things liked pegged orders, IOC slicing, timed limit orders, etc and dealing with partial fills that may result so there will be a lot of back and forth between my server and Tradestation. This will all be I/O limited, not CPU limited hence I was thinking of implementing this part with asyncio.
So in summary my plan is to create one thread for the market data from Polygon, and to create another thread for each stock symbol and have all these threads run in parallel using multiprocessing. Then, on each stock thread, use an asyncio implementation on that thread to communicate with the Tradestation API.
However, this begs the question, since I'm using threads already, am I just better off sticking with groups of threads ran concurrently for each stock symbol to handle the asynchronism in the I/O operations, or am I better off using asyncio on a single thread for each stock symbol?
Additionally, is my big picture plan here of using multiprocessing with one core handling the market data and a separate core for each symbol solid? Any advice on that? Thanks.
I'm implementing a stock algo to trade in real time. For this, I am using polygon.io as my market data provider, and Tradestation.com as my broker. Each has an API.
Basically the algo needs to take in real time market data from the Polygon API, crunch the numbers, make trade decisions, and use the Tradestation API to send those trade decisions for execution.
If I was doing this for just one stock simultaneously, the answer would be pretty simple just use asyncio. Each stock symbol is completely independent of the others so I was going to write one python script, and just run multiple instances of it with different symbols. Thus, the OS could also schedule each instance on it's own core for efficiency. The problem is that the costs for more than one data stream from Polygon get really expensive... one data stream means this all needs to be done in a single python script. Again, if it was a single stock symbol... no problem. But if I'm running 10 or 20 or 30 symbols at the same time, at some point, it WILL get bogged down on a single core....
So this all starts with the Polygon API which provides market data. This portion is quite simple - it's just a single websockets stream that send me data in a firehose. The more symbols... the more data. I can subscribe to as many as I want in a single stream, and as long as I can handle it on the receiving end there is no limit from the Polygon end how much they will send. So I was thinking of using multiprocessing to have this run on it's own core... and all this would need to do is separate out the symbols and decode the raw json data into a format that my algo can use for input.
Next comes the algo data crunching and order sending via the Tradestation API. Here is where we break it out by stock symbol, as each one is independent. I was thinking of putting each symbol on it's own thread so that multiprocessing would then run each on different cores. So first the algo needs to run to crunch the raw data to make either a buy or sell decision - and this would happen once per second. It then sends that off to Tradestation via their REST API for execution. The actual execution can be rather complicated - because I lose too much on slippage on market orders I need to do creative things liked pegged orders, IOC slicing, timed limit orders, etc and dealing with partial fills that may result so there will be a lot of back and forth between my server and Tradestation. This will all be I/O limited, not CPU limited hence I was thinking of implementing this part with asyncio.
So in summary my plan is to create one thread for the market data from Polygon, and to create another thread for each stock symbol and have all these threads run in parallel using multiprocessing. Then, on each stock thread, use an asyncio implementation on that thread to communicate with the Tradestation API.
However, this begs the question, since I'm using threads already, am I just better off sticking with groups of threads ran concurrently for each stock symbol to handle the asynchronism in the I/O operations, or am I better off using asyncio on a single thread for each stock symbol?
Additionally, is my big picture plan here of using multiprocessing with one core handling the market data and a separate core for each symbol solid? Any advice on that? Thanks.