Skip to content
swing-ui-agent
A coding agent records a test in Order Desk and says each step in plain words: it opens a new order for Ada Shop Ltd, adds A4 paper and gel pens, sets the quantities and adds six checks. It saves them as a recorded test, a file with nothing to compile, which then replays every step and passes each check. In CI the same test runs on every push, under a virtual display.

Record Swing UI tests. A coding agent does the work

AI-assisted testing of existing Java Swing desktop applications. Show the test in the app, or tell the agent in words. You get a recorded test that replays from the app, the agent or CI. One jar, no changes to the app.
Get startedDownload the demo appSee the demoGitHub

Java 8 to 25 · macOS, Linux and Windows · Apache-2.0 (Limits)

What it does

Section titled “What it does”

It connects three things that usually happen apart:

  • A person shows how the app should behave, in the app itself, or describes it in words.
  • A coding agent turns that into a named, repeatable test.
  • Developers run the same test locally and in CI, and move to an ordinary Java test when a check needs more.
  1. Recording in Order Desk, in Check mode: a click on a cart line opened the Check menu with the checks it offers, such as Product is Gel pen, black in this row

    Records UI tests, from what you show or what you say

    Press ● Record in the palette and use the app; Check adds a check of what you see. Or tell a coding agent the test in words and it drives the live app. Either way the agent names each step in plain words and saves a recorded test: a JSON file in uitest/ with nothing to compile.

    Record a test by showing it · Record together with your agent

  2. An animation: the recorded test cart-totals replays step by step in Order Desk; the Steps list ticks each step and the five total checks turn green

    Replays them, and lets you read and fix them

    The steps are sentences you can read, rename, replay from any point or redo. Press ▶ Run in the app, ask the agent, or run the whole uitest/ folder from Maven and in CI under a virtual display. A failing step says what it found.

    Run and edit a recorded test · Run in CI

  3. The agent asks in the app which table holds the cart: the products table and the cart are boxed and numbered, each with a check mark to confirm it, and a bar with the question and None of these

    Asks in the app when it is not sure

    When a step could mean two things, the agent boxes the candidates in the app, numbered, and you confirm one or say none of these. The answer goes into the recording, so the test is right the first time.

    Record together with your agent

  4. A JUnit 5 UiTest class that reads the new order's number from the status bar and asserts its row in the Orders table

    Writes a Java test when you need more

    A recorded test holds exact values and checks what is on screen. When a test must read a value that changes from run to run, such as the new order's number, or check a database, files or logs, the agent writes a JUnit 5 @UiTest class instead, with the same driving API. The two kinds do not mix.

    Recorded tests or Java tests · One order, two files

  5. How it fits together: a person at the app or a coding agent, over the palette or the CLI and MCP on loopback, reach the agent jar inside the app's JVM; it writes recorded tests to uitest, which the palette, the agent, Maven and CI replay

    Drives the real app, from a shell or over MCP

    One jar attached with -javaagent: clicks, types and reads the live UI from inside the app's JVM: no changes to the app, and input goes into the app's own event queue, so your mouse and keyboard stay yours. A coding agent calls it one tool per step, from a shell (java -jar … <tool>) or over MCP on loopback. Skills teach it the workflow.

    Let your agent drive the app · CLI or MCP · Skills

Try it in two minutes

Section titled “Try it in two minutes”

Download Order Desk, a small Swing order-entry app with the agent built in. Nothing to build.

  1. Download it for Mac with Apple Silicon, Mac with Intel, Windows or Linux and any Java 8+, and open it (first launch).

  2. Run a recorded test: press Load recorded test… in the bar at the bottom, pick place-order, press ▶ Run. Order Desk comes with three; the app takes the order by itself and checks every total.

    Order Desk right after it starts: the Orders list, and at the bottom the palette's bar with Record and Load recorded test

  3. Connect your agent while the app runs:

    Terminal window
    claude mcp add --transport http order-desk http://127.0.0.1:8770/mcp

  4. Ask it: "Create a new order for Ada Shop Ltd with 3 packs of A4 paper and 10 black gel pens, check out, and tell me the order number." If it asks for a PO number, give it any.

  5. Ask again. Now it knows the screens and sends the whole order in one batch call, in about half a second. Ask it to save the steps as a recorded test, and what it learned to its project notes.

Then take the tour, or open Order Desk's source in IntelliJ and record into the project: in IntelliJ, with Claude Code.

Good to know

Section titled “Good to know”
  • No changes to the app. Add -javaagent:. Java 8 to 25, macOS, Linux and Windows. Install
  • Your mouse and keyboard stay yours. Input goes into the app's own event queue; other windows can be in front.
  • Waits for the app, not a timer. Every step lets the app settle; checks retry until they hold. Waiting
  • Finds what a person sees. Labels, button text, column headers and cell text, not pixel positions. Targets
  • Exact values, your data. A recorded test checks what is on screen, such as the cart's Total €31.32. It creates the data it checks or uses data you know exists.
  • Built for agents. An element tree instead of screenshots, errors that list the near misses, and three skills that teach the workflow. Skills
  • Local only. It listens on 127.0.0.1, refuses any other address, and no tool runs commands, loads a class or evaluates code. Security
  • Your app's words (optional). A small plugin adds reset, login and named steps such as "Add Gel pen, black to the cart". Plugin vocabulary
  • Early. 0.1.0-SNAPSHOT, not on Maven Central yet; expect changes before 1.0. A recorded test file carries its format version, and the agent refuses one it cannot read instead of guessing. Limits · Roadmap · Recorded test format

See it run

Section titled “See it run”

The demo playing above is Order Desk's recorded test uitest/cart-totals.json: it fills Ada's cart and checks its totals. Below, Order Desk's two Java tests run from IntelliJ against the same app. All of them were recorded by an agent and run in CI.

UI tests
Order Desk
UI tests running against Order Desk, slowed down: the real run takes 4.1 s. Each action gets a ripple, each assert a box that turns green when it passes.

Order Desk's CI runs its recorded tests on every change, on Linux under a virtual display (the run):

Order Desk's CI log: cart-totals passed with 11 steps, cart-with-steps with 7, place-order with 17; Tests run: 3, Failures: 0
Tour of Order DeskStart it with the agent and see every screen.
One order, two filesplace-order.json and PlaceOrderUiIT side by side: what each can check.
PlaceOrderUiIT, step by stepThe Java test, with its ten screenshots.

Apache-2.0 · A Kormcrete project · For coding agents:llms.txt, llms-full.txt