Python Forum
Is this even possible? Advanced ML
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Is this even possible? Advanced ML
#1
Lets see if I can describe this in words... I guess this is best described as a time series, so I have this "event" that randomly occurs. I can easily create a feature that outputs a "1" when event is true, else "0". I want to tell the algorithm(s) to actually "predict" when true"1", but taking into account the past (n) "0" rows as well... So I don't want it making predictions AT ALL on the "0" rows, but I can't just combine them without losing the information gained from the series aspect. My first thought was to create an "if" statement that pushes the "1" + past(n) rows to a new DF which is fed through the algo, but it still treats each row as a prediction. I want to specifically state that all rows count towards the single prediction at the end... in a way that keeps the sequential aspect of the data in tact. ANY HELP at all would be sooooooo appreciated!!! Even terms to search to help me find the answer...
Reply
#2
It would be helpful (for better understanding) if you itemize tasks as they occur:
  • wait for event (interrupt driven so no polling)
  • event occurs
  • ...
  • etc.
Quote:Devilish states: 'so I have this "event" that randomly occurs'
Is this a software event? what is its nature, where does it come from.
Reply
#3
(Jan-22-2019, 08:51 PM)Larz60+ Wrote: It would be helpful (for better understanding) if you itemize tasks as they occur:
  • wait for event (interrupt driven so no polling)
  • event occurs
  • ...
  • etc.
Quote:Devilish states: 'so I have this "event" that randomly occurs'
Is this a software event? what is its nature, where does it come from.

I've discovered a relationship between 2 features... when the difference between the 2 > (x) there is a direction change in the main data. I want to explore the point where that range expansion happens and the sequence leading up to that point in hopes to find some key relationships in the other features that could indicate either a temporary halt in momentum v.s. a true shift in momentum and direction...

an quick example of the data
MainData is the main feature I am analyzing/predicting, Feat1-4 are unimportant for this, THIS1 > 6 is the trigger event

MainData, Feat1, Feat2, Feat3, Feat4, THIS1

100, 3, 8, 9, 0, 3
104 6, 9, 5, 3, 4 < ..............After trigger, it now analyzes the 2 previous
106 4, 2, 4, 1, 3 < ..............and compares/trains with all other *sequences
108 5, 8, 6, 3, 7 * ...The Event Triggers
105 ....... ................. 2 ...............It's no longer analyzing or predicting
####UNKNOWN PERIOD OF TIME####
104 ............................. 5 <
102 ............................. 3 <
101 ............................... 8* ..... Trigger causes it to read previous 2 rows as sequence
Reply
#4
I understand (I think) basically what you are trying to analyze, it's sort of the same type of situation encountered with metal stress analysis where you are measuring values up to the point where the metal breaks, and then the after affects while readings are still available.

I can also make another case used where a mercury lamp is used for spectral calibration of scanning monochrometers, for which I have written code (think back over 30 years now) where you have a very strong triplet (see: http://www.astrosurf.com/buil/us/spe2/ca...burb_1.gif ) and then a series of Hg spectral lines before and after this 'major' triplet. The readings taken prior to, during and after seeing the first signals seems like a similar type of problem.
Calibration procedures can be found by googling: 'python code for mercury lamp calibration of scanning monochromator'
Knowing exactly what you are trying to measure would be helpful in creating a mental image to help with analysis procedures.

**EDIT 8:42 P.M. EST **
see: https://scholar.google.com/scholar?q=app...i=scholart
Reply
#5
(Jan-23-2019, 01:43 AM)Larz60+ Wrote: Knowing exactly what you are trying to measure would be helpful in creating a mental image to help with analysis procedures.
I am a futures trader, I have spent the past year hand coding all my own custom studies that look for bid, ask, patterns within each bar. I trade with a bar type called Flex-Renko, where the bar that forms on a direction turn is larger than the others. As it is, my custom studies only start popping off near/during a direction change, but I have so many each with unique information, I am wanting to look at these longer bars and 3-5 bars back to see where the correlations/patterns are. Feeding in this data straight, there is no way of knowing if the current or next bar will wind up being that larger bar, so when that bar actually closes... I then want it to feed in the data, but in a way that BOTH keeps the data in a 3-5 row sequence, but ALSO treats each sequence as it's own single prediction... meaning even though there is 3-5 rows of data, I only want 1 prediction at the end, based on all 5 rows. Everything I've tried still yields a prediction on each row individually

to add... one of my studies monitors for this Larger range, this is what I would use in Python...it basically works like...
normally outputs a "0"
if bar closes with H-L > range.. outputs a 1 ALSO changes the previous (n) bars to also be a 1...
so if I just straight fed it through, the previous bars would have already been processed before they were marked. I only want it analyzing when column > 0, which means I need a way of stalling the data a little... which I know I can just add a DF2 = DF1['column' > 0]... and have then when the new "1"s appear it appends and then pushes through... but it still views each line as it's own line
Reply
#6
If you have lots of history data, this is the perfect type of data for machine learning software
Googling: 'machine learning for option trading Flex-Renko analysis python'
brings up a bunch of stuff, you've been thinking about this for a while, so you're probably closer to an answer than I could be without studying for a long time. However, reading some of the other work being done just might trigger the answer your looking for.
There's a book which I think is still on sale (EPUB or pdf) for $5 at Packt publishing named 'Machine Learning Algorithms' by Giuseppe Bonaccorso that might be of great help.
If you have enough historical data, you'd be amazed at what comes out when applied to current data.
Reply
#7
I'm probably making things WAY more complicated than it needs to be! I did think of a few things to look into, maybe adding another dimension based on time, or moving windows, I still don't fully understand how batch size affects how algorithms learns. Hell 14 months ago, I had never programmed and now I've custom coded all my concept studies in C++ and have made huge leaps in Python... I really appreciate your trying to help!!!
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020