Limits

swing-ui-agent is early (0.1.0-SNAPSHOT). This page lists what it covers today and the known gaps. A gap shows up as a clear error, usually CAPABILITY_GAP or a notRecorded note on a step, never as a step that silently did something else.

Platforms and Java

• Operating systems: macOS, Linux (X11) and Windows, each tested in CI on every change (Linux under a virtual display). Other systems fail with a CAPABILITY_GAP error.
• Real desktops: Linux and Windows are tested on GitHub's runners and in virtual machines so far, not yet on many real desktops: GNOME or KDE, Wayland through XWayland, display scaling. Reports from yours are welcome.
• In front or behind: on all three the app can be driven while other windows are in front. On macOS, mode=background also keeps it out of the Dock.
• Java: one jar, built for Java 8 so it also attaches to legacy apps still on it. CI tests Java 8, 11, 17, 21 and 25 on macOS, and Java 8 and 25 on Linux and Windows. On Java 9 and newer the app needs no --add-opens flags.
• Toolkits: Swing only. No JavaFX, no SWT.
• Distribution: not on Maven Central yet. Build and install it from source.

Driving

• Screenshots paint the app's own components into an image. Anything in a window of its own, such as an open popup menu, a combo dropdown or a tooltip, is not in a window's image; capture that window instead.
• Drag and drop: drag posts a mouse gesture inside one window and never starts a native drag. To deliver files or text, use drop.
• The macOS screen menu bar has no input path, so menu invokes its item directly. The step is marked via:api.
• Mouse input: a component counts as taking mouse input when it has mouse listeners or a popup menu. One that only enables mouse events internally is seen as transparent, so a click on it lands on what is below.
• No tool of its own: focusFileChooserName is a step without a tool. Run it through batch or do_action. (hover, resize_column, resize_window and close_window are tools now.)

Recording and export

• Not exportable: a row matched on more than two columns, a loginAs that is not the first step, and a point on anything but a plain left click (that one is exported as clickAt). The step runs and its result says notRecorded; pick another locator, or write that line by hand.
• Starting screen: without a plugin, a reset only closes modal dialogs, so the app may be on any screen. Each test reaches its own starting screen through the menu bar or a shortcut, and the exports warn when a recording starts any other way.

The palette and recording by showing it

• Small dialogs: the palette follows an open modal dialog. One too small to hold it (a short confirmation) gets it in a small window of its own just under the dialog. That window never takes the keyboard from the dialog, so a step's sentence cannot be typed in the Steps list while the palette is there.
• The agent's question bar: it stays in the window of what the agent asks about, even when another window is in front.
• Column checks: checks on a whole table column are shown greyed, because no expect kind exists for them yet.
• Not recorded as steps of their own yet: clicks on spinner arrows, and drags. Re-do those with the agent, or write them by hand.
• Lists and trees: a click on a list entry or a tree row is recorded by its text, also a pick from a suggestion popup such as Order Desk's Customer field. Two entries that read the same cannot be told apart, and a tree's expand arrow has no step of its own, so those clicks are not recorded: the palette says so (✗ not recorded) and the agent finds them under notRecorded. To expand or collapse a row while recording, double-click it.
• A tab's close button is painted by the look and feel and has no element of its own, so it can be neither found nor recorded. Close the tab as a user would with the menu or shortcut: in Order Desk File ▸ Close tab, or Mod+W (⌘W on macOS, Ctrl+W elsewhere).
• Password fields: never recorded, by design. What a person types into one is not stored, and the agent never reads one.
• Replay all begins with a reset, as the test will. Without a plugin that only closes dialogs, so there is no defined start to return to: the first step has to get there.
• Windows and Linux: the palette's automated tests run there, but it has not been tried by hand yet.

What's next

• Roadmap
• Security