Detecting Buttons - Suggestions?

[RockBlok]

Good morning,

I'm looking for some advice on tutorials and self study. I'm working on automating some of my workflow because I use a lot of different programs (some with apis for python, some without) and it can be frustrating to perform a lot of repetitive tasks.

One approach that I'm considering is having a program flip through all the windows, programs, and internet browsers I have open and then click appropriate buttons based on visual cues.

My question is whether there is a tutorial to help with this sort of thing. I'd essentially like to find a button on a screen and click the button. The button doesn't have a predefined location so the program would have to locate it (across multiple monitors?), apply a coordinate to it, and then use the python mouse click function.

An internet search suggested that having the program take a screenshot of my computer's desktop and then doing an image comparison for the button would be a viable option, but it also said that if the image isn't an exact match, then it wouldn't be able to locate and click the button. Which led me to read about the MSE and SSIM for image comparisons.

Am I going down the right path here, or is there a better way to do this?

[Axel_Erfurt]

One approach that I'm considering is having a program flip through all the windows, programs, and internet browsers I have open and then click appropriate buttons based on visual cues.

Something like that would contradict all security aspects of an operating system.

[RockBlok]

Hmm, I didn't even think about security.

I know there are different "user" levels in windows. Is it possible to run a program at a specific user level or such?

[RockBlok]

I found a video that shows some of the basics and figured I'd share it. The video seems oriented toward people who play games, but I think the explanations are straightforward.

It seems there are built in image match "confidence levels" checks for these python modules so you can choose to not have an exact match. I'll give this a try this weekend.