[Tkinter] Tkinter & Application flow issues - using multi-threaded for GUI

[caarsonr]

Hi everyone,
I’m working on a project that involves creating a GUI region selector in Tkinter. The goal is to allow the user to draw a rectangle to select a screen region, display a red outline of the selected area, and have the main program continue running after the selection is made.

What My Code Does:
select_area(coord_queue) (from area_selector.py):

Opens a semi-transparent overlay window that allows the user to select a region by clicking and dragging.
After the selection is completed, the coordinates are passed to a queue.Queue().
create_outline_window(root, coords) (from area_selector.py):

Creates a borderless window at the selected coordinates.
Draws a red rectangle outline with a transparent center.
Adds a "Close" button to close the outline window.
Main Program Flow (operate.py):

Uses start_selection(coord_queue) to run select_area() in a separate thread.
Waits for the coordinates and then opens the red outline window.

Problem
When I call create_outline_window():

The program stalls after the selection window is closed, and the red outline is not persistant and must be exited to allow the program flow to keep going
I believe the issue lies in how I’m handling threads and the mainloop() calls. I may be mixing too many Tkinter event loops (root.mainloop() vs Toplevel()), causing the GUI to become unresponsive.

Main Code Block in operate.py:

if define_region:
    import tkinter as tk
    import queue
    import threading
    from operate.utils.area_selector import start_selection, create_outline_window

    print("Region definition mode activated.")

    coord_queue = queue.Queue()

    # Start the region selection process (runs in a separate thread)
    start_selection(coord_queue)

    # Function to periodically check the queue for selected coordinates
    def check_queue():
        if not coord_queue.empty():
            region = coord_queue.get()
            print(f"Selected region: {region}")

            # Open the red outline in a separate thread
            threading.Thread(target=create_outline_window, args=(root, region), daemon=True).start()

            # Close the region selection root
            root.quit()
        else:
            root.after(100, check_queue)

    # Create a Tkinter root for managing the selection
    root = tk.Tk()
    root.withdraw()
    check_queue()  # Start queue checking
    root.mainloop()  # Run the event loop

area_selector.py; Selection Overlay

def select_area(coord_queue):
    root = tk.Tk()
    root.withdraw()
    overlay = tk.Toplevel(root)
    overlay.attributes('-topmost', True)
    overlay.attributes('-alpha', 0.3)
    overlay.overrideredirect(True)
    overlay.geometry(f"{root.winfo_screenwidth()}x{root.winfo_screenheight()}+0+0")

    # Red rectangle drawing logic
    canvas = tk.Canvas(overlay, cursor="cross", bd=0, highlightthickness=0)
    canvas.pack(fill=tk.BOTH, expand=True)
    rect = None

    def on_press(event):
        nonlocal rect
        rect = canvas.create_rectangle(event.x, event.y, event.x, event.y, outline='red', width=4)

    def on_drag(event):
        if rect:
            canvas.coords(rect, canvas.coords(rect)[0], canvas.coords(rect)[1], event.x, event.y)

    def on_release(event):
        coords = canvas.coords(rect)
        overlay.destroy()
        coord_queue.put(coords)
        root.quit()

    canvas.bind("<ButtonPress-1>", on_press)
    canvas.bind("<B1-Motion>", on_drag)
    canvas.bind("<ButtonRelease-1>", on_release)

    root.mainloop()

I'm thinking the problem is:
Multiple Tkinter root windows?

I’m creating multiple Tk() and Toplevel() windows. Could these interfere with each other?
Mainloop Conflict:

I’m calling mainloop() in different threads and potentially mixing GUI contexts. Maybe I need a more synchronized approach?
Improper Thread Handling:

The selection overlay and outline window both use threading.Thread(). Perhaps I’m not handling thread termination properly?
The expected behavior is that it would keep the red outline there and move onto the next lines of code since it's multi-threaded. But it does not do that. Does anyone know how to resolve this?


Here's a video




The program stalls after the selection is made. The red outline is not persistent and must be exited to allow the program flow to keep going. I thought threading it would eliminate this flow issue and allow it to keep going, but it does not.

Here are the main files:

  operate.py (Size: 6.06 KB / Downloads: 50)

  area_selector.py (Size: 4.42 KB / Downloads: 48)

deanhystad write Jan-05-2025, 08:11 PM:
Please post all code, output and errors (it it's entirety) between their respective tags. Refer to BBCode help topic on how to post. Use the "Preview Post" button to make sure the code is presented as you expect before hitting the "Post Reply/Thread" button.

[deanhystad]

Multiple Tk windows are not allowed. Tk() doesn't just create a window, it initializes tkinter. If you want to create multiple top level windows, use TopLevel() to create the additional windows.

I don't see any reason for using threads. Remove the mainloop() from select_area.

[caarsonr]

Multiple Tk windows are not allowed. Tk() doesn't just create a window, it initializes tkinter. If you want to create multiple top level windows, use TopLevel() to create the additional windows.

I don't see any reason for using threads. Remove the mainloop() from select_area.

Hey! Thanks for your response!
Just to clarify I want to display a red outline and continue the program's flow while keeping the red outline visible.

Your suggestion to use Toplevel() and not spawn multiple Tk() windows is valid; however, simply removing threads and using one mainloop() might cause the program to pause after the region selection is made.

Is there a way I can keep the red outline while continuing the program flow?
I am using Toplevel() but I am struggling to ensure both non-blocking flow and red outline persistence.

[caarsonr]

I've reworked both operate.py and area_selector.py not to use threading.

If it works correctly it should print("Check passed") when you run area_selector.py and have the outlined border present.

area_selector.py

import tkinter as tk
import queue

def select_area(coord_queue):
    """
    Opens a selection overlay to allow the user to select a region on the screen.
    Once the region is selected, the coordinates are passed to the queue.
    """
    coords = []

    # Create the Tkinter root for the selection overlay
    root = tk.Tk()
    root.withdraw()  # Hide the root window

    # Create overlay window for region selection
    overlay = tk.Toplevel(root)
    overlay.title("Selection Overlay")
    overlay.attributes('-topmost', True)  # Always on top
    overlay.attributes('-alpha', 0.3)  # Semi-transparent background (30% opacity)
    overlay.overrideredirect(True)  # Remove window decorations (borderless)
    
    sw = root.winfo_screenwidth()
    sh = root.winfo_screenheight()
    overlay.geometry(f"{sw}x{sh}+0+0")

    canvas = tk.Canvas(overlay, cursor="cross", bd=0, highlightthickness=0)
    canvas.pack(fill=tk.BOTH, expand=True)

    rect = None

    def on_press(event):
        nonlocal rect
        rect = canvas.create_rectangle(event.x, event.y, event.x, event.y, outline='red', width=4)

    def on_drag(event):
        if rect:
            x1, y1, _, _ = canvas.coords(rect)
            canvas.coords(rect, x1, y1, event.x, event.y)

    def on_release(event):
        nonlocal coords
        if rect:
            coords = canvas.coords(rect)
            overlay.destroy()  # Destroy the selection overlay
            coord_queue.put(coords)  # Put the coordinates in the queue
            root.quit()  # Exit the Tkinter mainloop

    canvas.bind("<ButtonPress-1>", on_press)
    canvas.bind("<B1-Motion>", on_drag)
    canvas.bind("<ButtonRelease-1>", on_release)

    root.mainloop()  # Main GUI loop

def create_outline_window(coords):
    """
    Creates an outline window at the specified coordinates.
    """
    import tkinter as tk

    x1, y1, x2, y2 = map(int, coords)
    width = abs(x2 - x1)
    height = abs(y2 - y1)
    x = min(x1, x2)
    y = min(y1, y2)

    outline_window = tk.Toplevel()  # Independent window
    outline_window.geometry(f"{width}x{height}+{x}+{y}")
    outline_window.overrideredirect(True)  # No border
    outline_window.attributes('-topmost', True)  # Always on top
    outline_window.attributes('-transparentcolor', outline_window['bg'])  # Transparent

    outline_canvas = tk.Canvas(outline_window, bd=0, highlightthickness=0)
    outline_canvas.pack(fill=tk.BOTH, expand=True)

    outline_canvas.create_rectangle(0, 0, width, height, outline='red', width=4)

    # Close button (optional)
    close_btn = tk.Button(outline_window, text="Close", command=outline_window.destroy, bg='white', relief='flat')
    close_btn.place(relx=0.5, rely=0.5, anchor='center')

    outline_window.mainloop()  # Keeps the window persistent

# Add a testable `main` block
if __name__ == "__main__":
    # Create a queue to store the selected coordinates
    coord_queue = queue.Queue()

    print("Starting region selection test...")

    # Start the region selection
    select_area(coord_queue)

    # Get the selected coordinates
    coords = coord_queue.get()
    print(f"Selected region coordinates: {coords}")

    # Display the red outline window
    create_outline_window(coords)

    print("Check passed")

    # Run the Tkinter main event loop (to keep the red outline open)
    tk.mainloop()

operate.py:

import sys
import os
import time
import asyncio
from prompt_toolkit import prompt
from operate.exceptions import ModelNotRecognizedException
import platform

# from operate.models.prompts import USER_QUESTION, get_system_prompt
from operate.models.prompts import (
    USER_QUESTION,
    get_system_prompt,
)
from operate.config import Config
from operate.utils.style import (
    ANSI_GREEN,
    ANSI_RESET,
    ANSI_YELLOW,
    ANSI_RED,
    ANSI_BRIGHT_MAGENTA,
    ANSI_BLUE,
    style,
)
from operate.utils.operating_system import OperatingSystem
from operate.models.apis import get_next_action

# Load configuration
config = Config()
operating_system = OperatingSystem()


def main(model, terminal_prompt, voice_mode=False, verbose_mode=False, define_region=False):
    """
    Main function for the Self-Operating Computer.

    Parameters:
    - model: The model used for generating responses.
    - terminal_prompt: A string representing the prompt provided in the terminal.
    - voice_mode: A boolean indicating whether to enable voice mode.

    Returns:
    None
    """

    mic = None
    # Initialize `WhisperMic`, if `voice_mode` is True

    config.verbose = verbose_mode
    config.validation(model, voice_mode)

    if define_region:
        import tkinter as tk
        import queue
        from operate.utils.area_selector import select_area, create_outline_window

        print("Region definition mode activated.")

        # Create a queue to store the selected coordinates
        coord_queue = queue.Queue()

        # Call `select_area` directly (this will block until the selection is made)
        select_area(coord_queue)  # Directly opens the selection overlay and waits

        # Get the selected region
        region = coord_queue.get()
        print(f"Selected region: {region}")

        # Create the red outline
        create_outline_window(region)

        print("Region defined successfully. Continuing...")


    if voice_mode:
        try:
            from whisper_mic import WhisperMic

            # Initialize WhisperMic if import is successful
            mic = WhisperMic()
        except ImportError:
            print(
                "Voice mode requires the 'whisper_mic' module. Please install it using 'pip install -r requirements-audio.txt'"
            )
            sys.exit(1)
    if terminal_prompt:  # Skip objective prompt if it was given as an argument
        objective = terminal_prompt
    elif voice_mode:
        print(
            f"{ANSI_GREEN}[Self-Operating Computer]{ANSI_RESET} Listening for your command... (speak now)"
        )
        try:
            objective = mic.listen()
        except Exception as e:
            print(f"{ANSI_RED}Error in capturing voice input: {e}{ANSI_RESET}")
            return  # Exit if voice input fails
    else:
        print(
            f"[{ANSI_GREEN}Self-Operating Computer {ANSI_RESET}|{ANSI_BRIGHT_MAGENTA} {model}{ANSI_RESET}]\n{USER_QUESTION}"
        )
        print(f"{ANSI_YELLOW}[User]{ANSI_RESET}")
        objective = prompt(style=style)

    system_prompt = get_system_prompt(model, objective)
    system_message = {"role": "system", "content": system_prompt}
    messages = [system_message]

    loop_count = 0

    session_id = None

    while True:
        if config.verbose:
            print("[Self Operating Computer] loop_count", loop_count)
        try:
            operations, session_id = asyncio.run(
                get_next_action(model, messages, objective, session_id)
            )

            stop = operate(operations, model)
            if stop:
                break

            loop_count += 1
            if loop_count > 10:
                break
        except ModelNotRecognizedException as e:
            print(
                f"{ANSI_GREEN}[Self-Operating Computer]{ANSI_RED}[Error] -> {e} {ANSI_RESET}"
            )
            break
        except Exception as e:
            print(
                f"{ANSI_GREEN}[Self-Operating Computer]{ANSI_RED}[Error] -> {e} {ANSI_RESET}"
            )
            break


import time

def operate(operations, model, region=None):
    # Function to check if a point is within the defined region
    def is_within_region(x, y, region):
        """Check if a point is within the defined region."""
        x1, y1, x2, y2 = region
        return x1 <= x <= x2 and y1 <= y <= y2

    if config.verbose:
        print("[Self Operating Computer][operate] Starting operations")

    for operation in operations:
        if config.verbose:
            print("[Self Operating Computer][operate] Processing operation", operation)
        
        time.sleep(1)  # Delay for demonstration purposes
        operate_type = operation.get("operation").lower()
        operate_detail = ""
        
        if config.verbose:
            print("[Self Operating Computer][operate] Operation type:", operate_type)

        # Check if operation coordinates are within the defined region, if region is specified
        if region:
            x = operation.get("x", 0)
            y = operation.get("y", 0)
            if not is_within_region(x, y, region):
                print(f"Operation at ({x}, {y}) is outside the defined region and will be skipped.")
                continue  # Skip this operation as it's outside the defined region

        if operate_type == "press" or operate_type == "hotkey":
            keys = operation.get("keys")
            operate_detail = f"keys: {keys}"
            operating_system.press(keys)
        elif operate_type == "write":
            content = operation.get("content")
            operate_detail = f"content: '{content}'"
            operating_system.write(content)
        elif operate_type == "click":
            x = operation.get("x")
            y = operation.get("y")
            operate_detail = f"click at ({x}, {y})"
            operating_system.mouse(x, y)
        elif operate_type == "done":
            summary = operation.get("summary")
            print(f"[{ANSI_GREEN}Self-Operating Computer {ANSI_RESET}|{ANSI_BRIGHT_MAGENTA} {model}{ANSI_RESET}] Objective Complete: {ANSI_BLUE}{summary}{ANSI_RESET}\n")
            return True  # Stop operation when done
        else:
            print(f"[{ANSI_GREEN}Self-Operating Computer{ANSI_RED} Error: unknown operation response{ANSI_RESET} {operation}")
            return True  # Stop operation on error

        # Print operation details
        print(f"[{ANSI_GREEN}Self-Operating Computer{ANSI_RESET} | {ANSI_BRIGHT_MAGENTA}{model}{ANSI_RESET}] Thought: {operate_thought}, Action: {ANSI_BLUE}{operate_type}{ANSI_RESET} {operate_detail}\n")

    return False  # Continue if not done

[deanhystad]

I don’t see why not. Isn’t the red outline a mostly transparent window? Tkinter can have many windows open simultaneously