The Curious Case of NSPanel's Nonactivating Style Mask Flag

While working on a behavior that required dynamically switching between the nonactivating panel behavior and one that is more similar to the regular window behavior, I discovered a subtle but significant issue in AppKit: changing the NSWindowStyleMaskNonactivatingPanel flag after an NSPanel has been initialized doesn't fully update the window's activation behavior the way the API contract seems to suggest.

This creates a perplexing situation where changing a window's style mask – something the API suggests should be fully supported – results in inconsistent window behavior that can be difficult to diagnose, and is certainly hard to reason about without fully understanding the window behavior on macOS, including WindowServer window tags.

Understanding Nonactivating Panels

The nonactivating panel behavior, which is canonically only valid for subclasses of NSPanel, not NSWindow, allows the window that declares itself as a nonactivating panel to be sent and handle key events without activating the owning application. In other words, you will be able to type into a nonactivating panel and do everything you’d be able to do with a key window in an active application, but the application itself will not become “active.” Among other things, this will cause it to not acquire the menu bar.

The nonactivating panel behavior is useful for utility panels, inspector windows, and other elements that shouldn't disrupt the user's current context. This can be achieved by either overriding _isNonactivatingPanel instance method and returning YES, or setting the NSWindowStyleMaskNonactivatingPanel style mask flag.

A Deeper Dive

Before a key event on macOS is routed to the client process and ultimately lands in NSApplication, it is dequeued from the hardware and is routed by WindowServer. The server ultimately maintains and owns key aspects of the truth about the window positioning, order, which process currently shows the menu bar, and the active application state. Each window, in addition to things like geometry, has a set of tags, which determine its behavior.

When you click on a window, what happens in most cases – unless you perform a nonactivating click by holding Command, for example – is that the application will be made active. That means that it will start receiving key events by default. However, one of the aforementioned window tags, kCGSPreventsActivationTagBit, dictates that performing a canonically activating action on a window that has this tag will not activate the application that owns it.

How does the application get the key events in this state then? When a window is nonactivating,

1. The window will be drawn as key by the application (the process managed by AppKit), but
2. The application will not be considered active and will not own the menu bar. In other words, the previously active application will still be active.

How is it not the case that the application can receive the key events then? While a nonactivating panel is key, the application will “steal key focus” from the true active application, through a process referred to as key focus theft, managed by the CoreProcesses subsystem of the WindowServer (abbreviated as CPS). The client process will make a call to -[NSApplication _stealKeyFocusWithOptions:], which will ultimately funnel down to CPSStealKeyFocusReturningID and IPC over to the server-side code that will alter the focus stack.

Likewise, when it’s time to restore the normal behavior, -[NSApplication _releaseKeyFocus] will be called and will funnel down to CPSReleaseKeyFocusWithID. Note the use of ID in both routines, which helps maintain the correctness of the focus stack.

Reproducing the Bug

I created a minimal reproduction case to demonstrate the issue. The full code is available at https://github.com/philptr/NonactivatingPanelBug.

The sample app has:

• A panel that starts as nonactivating
• A button to toggle the nonactivating state
• A text field for testing keyboard input

When you run this application and:

1. Launch the app (which creates a nonactivating panel),
2. Switch to another application,
3. Click the "Toggle Nonactivating Panel Bit" button to make it a regular window,
4. Click on the panel;

You'll observe the following unexpected behavior:

• The panel's appearance changes correctly (gaining a title bar and close button);
• The panel shows visual indication that it has keyboard focus, i.e. will draw as key;
• BUT typing into the text field won’t work!

The Root Cause

After investigation, I determined that the issue stems from how NSPanel manages window server tags:

1. During initialization, NSPanel calls an internal method -_setPreventsActivation: to set the kCGSPreventsActivationTagBit window tag when the nonactivating style mask bit is present.
2. When you change the style mask later during the lifetime of the panel instance via -setStyleMask:, it doesn't call -_setPreventsActivation: again to update this window tag.

Based on the way things work under the hood that we’ve established above, we can gain pretty good insight into what is actually going on. The bug occurs due to a mismatch:

• The framework thinks the panel is a regular window (because the style mask no longer includes NSWindowStyleMaskNonactivatingPanel).
• The WindowServer still treats it as nonactivating (because the window tag remains set).

Since the panel doesn’t have the NSWindowStyleMaskNonactivatingPanel bit in its style mask anymore, its -_isNonactivatingPanel returns NO, and the application will not steal key focus for it when it is supposed to become key. However, mousing down on the window will also not make the application frontmost, because we’ve never removed the kCGSAvoidsActivationTagBit. As a result, the window will appear key, since that is the behavior managed by AppKit, but will not be receiving key events.

Reverse Engineering the Behavior

Discovering the root cause required investigating how AppKit interacts with the WindowServer. Here's the outline of a process one could use to reverse engineer this or similar behavior:

Symbolic Breakpoints are Your Friends

You could use LLDB to trace method calls when setting the style mask. With the application running:

(lldb) breakpoint set -n "-[NSWindow setStyleMask:]"
(lldb) breakpoint set -n "-[NSWindow _setPreventsActivation:]"

This revealed that -_setPreventsActivation: is called during panel initialization but not when the style mask is changed. In our sample application, this is tested by clicking the button that toggles the nonactivating panel style mask bit.

Inspecting Window Server Tags

To verify my hypothesis about window server tags, you can forward declare the (private) CGSGetWindowTags routine:

CG_EXTERN CGError CGSGetWindowTags(CGSConnectionID cid, CGWindowID wid, const CGSWindowTagBit tags[2], size_t maxTagSize);

By calling this function before and after changing the style mask, one may be able to confirm that the kCGSPreventsActivationTagBit tag (value 1 << 16) indeed remains set even after removing the nonactivating style mask.

Looking at Disassembly

Finally, to confirm the hypothesis that the bug indeed stems from AppKit simply not calling the relevant method that sets the window tag on the path of changing the style mask during the lifetime of the window, you can use a tool like Hopper to find its callers, which will reveal that the only time the method is called during the lifetime of an NSPanel instance is indeed during the _panelInitCommonCode C helper routine called during the initialization.

Workaround

To work around the issue, we can implement a workaround by manually setting the prevents-activation state after changing the style mask. Please note that this requires calling into NSWindow SPI. The least invasive version of the workaround involves something like the following:

extension NSWindow {
    func updateActivationBehavior() {
        // Access private method via performSelector to update activation state.
        perform(Selector(("_setPreventsActivation:")), 
                with: NSNumber(value: styleMask.contains(.nonactivatingPanel)))
    }
}

Then you may call this method after changing the style mask:

panel.styleMask = newStyleMask
panel.updateActivationBehavior()

Some Implications

The proper fix would be for -setStyleMask: to call -_setPreventsActivation: whenever the nonactivating panel bit changes, ensuring the window tags stay in sync with the style mask.

When working with windows and panels whose behavior needs to change at runtime, be aware that some style mask changes may require additional synchronization with window server state.

The bug has been reported as FB16484811. The full code of the sample application is available at https://github.com/philptr/NonactivatingPanelBug.