This is the full developer documentation for swing-ui-agent # Record Swing UI tests. A coding agent does the work > AI-assisted testing of existing Java Swing desktop applications. Show the test in the app, or tell the agent in words. You get a recorded test that replays from the app, the agent or CI. One jar, no changes to the app. ## What it does [Section titled “What it does”](#what-it-does) It connects three things that usually happen apart: * **A person shows** how the app should behave, in the app itself, or describes it in words. * **A coding agent turns that** into a named, repeatable test. * **Developers run the same test** locally and in CI, and move to an ordinary Java test when a check needs more. 1.  ### Records UI tests, from what you show or what you say Press **● Record** in the palette and use the app; **Check** adds a check of what you see. Or tell a coding agent the test in words and it drives the live app. Either way the agent names each step in plain words and saves a **recorded test**: a JSON file in `uitest/` with nothing to compile. [Record a test by showing it](/swing-ui-agent/start/show/) · [Record together with your agent](/swing-ui-agent/start/record-together/) 2.  ### Replays them, and lets you read and fix them The steps are sentences you can read, rename, replay from any point or redo. Press **▶ Run** in the app, ask the agent, or run the whole `uitest/` folder from Maven and in CI under a virtual display. A failing step says what it found. [Run and edit a recorded test](/swing-ui-agent/start/recorded-tests/) · [Run in CI](/swing-ui-agent/start/ci/) 3.  ### Asks in the app when it is not sure When a step could mean two things, the agent boxes the candidates in the app, numbered, and you confirm one or say none of these. The answer goes into the recording, so the test is right the first time. [Record together with your agent](/swing-ui-agent/start/record-together/) 4.  ### Writes a Java test when you need more A recorded test holds exact values and checks what is on screen. When a test must read a value that changes from run to run, such as the new order's number, or check a database, files or logs, the agent writes a JUnit 5 `@UiTest` class instead, with the same driving API. The two kinds do not mix. [Recorded tests or Java tests](/swing-ui-agent/start/choose-test-kind/) · [One order, two files](/swing-ui-agent/demo/two-files/) 5.  ### Drives the real app, from a shell or over MCP One jar attached with `-javaagent:` clicks, types and reads the live UI from inside the app's JVM: no changes to the app, and input goes into the app's own event queue, so your mouse and keyboard stay yours. A coding agent calls it one tool per step, from a shell (`java -jar … `) or over MCP on loopback. Skills teach it the workflow. [Let your agent drive the app](/swing-ui-agent/start/drive/) · [CLI or MCP](/swing-ui-agent/agents/cli-or-mcp/) · [Skills](/swing-ui-agent/agents/skills/) ## Try it in two minutes [Section titled “Try it in two minutes”](#try-it-in-two-minutes) Download Order Desk, a small Swing order-entry app with the agent built in. Nothing to build. 1. **Download** it for [Mac with Apple Silicon](https://github.com/kormcrete/swing-order-desk-demo/releases/latest/download/kormcrete-order-desk-macos-arm64.dmg), [Mac with Intel](https://github.com/kormcrete/swing-order-desk-demo/releases/latest/download/kormcrete-order-desk-macos-x64.dmg), [Windows](https://github.com/kormcrete/swing-order-desk-demo/releases/latest/download/kormcrete-order-desk-windows-x64.zip) or [Linux and any Java 8+](https://github.com/kormcrete/swing-order-desk-demo/releases/latest/download/kormcrete-order-desk-java.zip), and open it ([first launch](/swing-ui-agent/demo/tour/#download-and-run-it)). 2. **Run a recorded test:** press **Load recorded test…** in the bar at the bottom, pick `place-order`, press **▶ Run**. Order Desk comes with three; the app takes the order by itself and checks every total.  3. **Connect your agent** while the app runs: ```sh claude mcp add --transport http order-desk http://127.0.0.1:8770/mcp ``` 4. **Ask it:** *"Create a new order for Ada Shop Ltd with 3 packs of A4 paper and 10 black gel pens, check out, and tell me the order number."* If it asks for a PO number, give it any. 5. **Ask again.** Now it knows the screens and sends the whole order in one `batch` call, in about half a second. Ask it to [save the steps as a recorded test](/swing-ui-agent/start/record-together/), and what it learned to its project notes. Then [take the tour](/swing-ui-agent/demo/tour/), or open Order Desk's source in IntelliJ and record into the project: [in IntelliJ](/swing-ui-agent/start/intellij/), [with Claude Code](/swing-ui-agent/start/claude-code/). ## Good to know [Section titled “Good to know”](#good-to-know) * **No changes to the app.** Add `-javaagent:`. Java 8 to 25, macOS, Linux and Windows. [Install](/swing-ui-agent/start/install/) * **Your mouse and keyboard stay yours.** Input goes into the app's own event queue; other windows can be in front. * **Waits for the app, not a timer.** Every step lets the app settle; checks retry until they hold. [Waiting](/swing-ui-agent/guides/waiting/) * **Finds what a person sees.** Labels, button text, column headers and cell text, not pixel positions. [Targets](/swing-ui-agent/guides/targets/) * **Exact values, your data.** A recorded test checks what is on screen, such as the cart's **Total** `€31.32`. It creates the data it checks or uses data you know exists. * **Built for agents.** An element tree instead of screenshots, errors that list the near misses, and three skills that teach the workflow. [Skills](/swing-ui-agent/agents/skills/) * **Local only.** It listens on `127.0.0.1`, refuses any other address, and no tool runs commands, loads a class or evaluates code. [Security](/swing-ui-agent/agents/security/) * **Your app's words (optional).** A small plugin adds reset, login and named steps such as "Add Gel pen, black to the cart". [Plugin vocabulary](/swing-ui-agent/plugins/vocabulary/) * **Early.** `0.1.0-SNAPSHOT`, not on Maven Central yet; expect changes before 1.0. A recorded test file carries its format version, and the agent refuses one it cannot read instead of guessing. [Limits](/swing-ui-agent/project/limits/) · [Roadmap](/swing-ui-agent/project/roadmap/) · [Recorded test format](/swing-ui-agent/reference/recorded-test-format/) ## See it run [Section titled “See it run”](#see-it-run) The demo playing above is Order Desk's recorded test `uitest/cart-totals.json`: it fills Ada's cart and checks its totals. Below, Order Desk's two Java tests run from IntelliJ against the same app. All of them were recorded by an agent and run in CI. UI tests Test Results CartTotalsUiIT totalsTheCart() PlaceOrderUiIT placesAnOrderForAda() Order Desk ![]() **UI tests** running against Order Desk, slowed down: the real run takes 4.1 s. Each action gets a ripple, each assert a box that turns green when it passes. Order Desk's CI runs its recorded tests on every change, on Linux under a virtual display ([the run](https://github.com/kormcrete/swing-order-desk-demo/actions/runs/39214091688)):  [Tour of Order Desk](/swing-ui-agent/demo/tour/)Start it with the agent and see every screen. [One order, two files](/swing-ui-agent/demo/two-files/)place-order.json and PlaceOrderUiIT side by side: what each can check. [PlaceOrderUiIT, step by step](/swing-ui-agent/demo/recorded-test/)The Java test, with its ten screenshots. # CLI or MCP? > Both give an agent the same tools. Pick by the agent you use. The agent in your app serves one set of tools. A coding agent reaches them in one of two ways: * **the command line:** the agent jar, one shell call per tool; * **MCP:** streamable HTTP at `http://127.0.0.1:8765/mcp`. The tools, their arguments and their answers are the same either way. ## Use the CLI when your agent has a shell [Section titled “Use the CLI when your agent has a shell”](#use-the-cli-when-your-agent-has-a-shell) This is the main path for coding agents. ```sh UI="java -jar swing-ui-agent-0.1.0-SNAPSHOT-agent.jar --port 8765" $UI session $UI snapshot '{"window":"main","depth":3,"compact":true}' $UI menu '{"path":["File","New order"]}' ``` * **Nothing to set up.** No client configuration: the agent runs the jar like any other command. * **Nothing to reconnect.** Each call connects, runs one tool and exits. Restart or rebuild the app, and the next call just works. * **Plain answers.** One JSON answer on stdout, an error on stderr, and an [exit code](/swing-ui-agent/reference/cli/) the agent can branch on: `3` means no app with the agent is running. * **Files instead of images.** `--out orders.png screenshot` writes the PNG to a file, so the agent opens it only when it needs to. * **The skills assume it.** The three [skills](/swing-ui-agent/agents/skills/) show every step as a CLI call. Recording and recorded tests are tools like the rest: `recording` (start, stop, list, edit, replay, export, load) and `run_recorded_test`. From a shell, `run_recorded_test` exits `4` when the recorded test fails (see [Run and edit a recorded test](/swing-ui-agent/start/recorded-tests/)). Each call starts a small JVM. Driving a whole task is many calls, so use `batch` when the next steps are already known: several steps, one call. ## Use MCP when your agent speaks it [Section titled “Use MCP when your agent speaks it”](#use-mcp-when-your-agent-speaks-it) Point any MCP client that supports streamable HTTP at `http://127.0.0.1:8765/mcp`. The client lists the tools itself and calls them as tool calls. Setup for a few clients is in [MCP clients](/swing-ui-agent/agents/mcp-clients/). * **No shell needed.** For agents and chat clients without one. * **Recording and running.** `recording` and `run_recorded_test` are listed like every other tool; a failed run is a normal answer with `passed: false`, not a transport error. * **Images inline.** `screenshot` returns the PNG as image content. * **Instructions on connect.** The client receives a short description of how to read and act on the app. The endpoint is stateless: each request is one JSON-RPC message, answered in the response. There is no MCP session to keep alive. ## Both at once [Section titled “Both at once”](#both-at-once) The CLI, MCP clients and running tests all talk to the same agent, so they can be mixed. One owner acts at a time: while a test run or a script holds the [lease](/swing-ui-agent/guides/leases/), others can still read, but not act. ## What's next [Section titled “What's next”](#whats-next) * [Let your agent drive the app](/swing-ui-agent/start/drive/): a worked session on Order Desk. * [Skills](/swing-ui-agent/agents/skills/): teach your coding agent the workflow. * [Tools](/swing-ui-agent/reference/tools/): every tool and its arguments. # MCP clients > Connect any MCP client that speaks streamable HTTP. The agent serves its tools over MCP's streamable HTTP transport at `http://127.0.0.1:8765/mcp`. Any MCP client that supports this transport can use them, with no adapter in between. ## Before you connect [Section titled “Before you connect”](#before-you-connect) 1. Start your app with the agent (see [Installation](/swing-ui-agent/start/install/)). 2. Check that it answers: ```sh java -jar swing-ui-agent-0.1.0-SNAPSHOT-agent.jar session ``` Exit code `0` and a JSON answer mean the agent is up. With `port=` set, pass the same `--port`, and use that port in the URL below. ## Connect a client [Section titled “Connect a client”](#connect-a-client) In Claude Code, add the server once per project: ```sh claude mcp add --transport http swing-ui http://127.0.0.1:8765/mcp ``` Other clients take the same URL in their settings. Where a client asks for a JSON entry, it usually looks like this (the exact keys depend on the client): ```json { "mcpServers": { "swing-ui": { "type": "http", "url": "http://127.0.0.1:8765/mcp" } } } ``` The client then lists the tools (`snapshot`, `click`, `expect`, `recording`, `export_recorded_test`, `run_recorded_test`, `export_test` and the rest) and receives short server instructions on how to read and act on the app. ## Recording and recorded tests over MCP [Section titled “Recording and recorded tests over MCP”](#recording-and-recorded-tests-over-mcp) * **Record:** call `recording` with `{"op":"start","reset":true}`, act with the usual tools, then `{"op":"stop"}` and `export_recorded_test`. The person at the app sees the steps in the palette and can change them. See [Record together with your agent](/swing-ui-agent/start/record-together/). * **Run:** `run_recorded_test` with a `name`, a `path`, or the file's JSON as `test`. The answer carries `passed`, `ran`, `total` and every step or check. A client has no exit code, so read `passed`. * **Images:** a failing step's picture is written to a file under `evidence`; the answer names it. ## What the endpoint does and does not do [Section titled “What the endpoint does and does not do”](#what-the-endpoint-does-and-does-not-do) * **Stateless.** Each POST carries one JSON-RPC message and gets its answer in the response. There is no MCP session to keep alive, and no server-sent stream (a GET answers 405). * **Tools only.** The server offers tools. No prompts or resources. * **Local only.** It listens on loopback and refuses requests from web pages (see [Security](/swing-ui-agent/agents/security/)). A client must run on the same machine as the app. * **The CLI next to it.** Even with an MCP client set up, the agent jar's command line still works; the skills use it. See [CLI or MCP?](/swing-ui-agent/agents/cli-or-mcp/). ## When it does not connect [Section titled “When it does not connect”](#when-it-does-not-connect) * **Nothing listens.** The app is not running with the agent, or it uses another port. `java -jar --port session` exits with `3` in that case. * **HTTP 403.** The request carried an `Origin` header for a host other than localhost, as a web page would. MCP clients normally send none. * **The client wants SSE.** This server speaks streamable HTTP only, not the older HTTP+SSE transport. Pick the "http" or "streamable HTTP" transport in the client. ## What's next [Section titled “What's next”](#whats-next) * [Tools](/swing-ui-agent/reference/tools/): every tool and its arguments. * [Skills](/swing-ui-agent/agents/skills/): they work for MCP clients too; read the CLI calls as tool calls. # Security > What the agent can and cannot do, and where to run it. The agent gives whoever can reach its port the same power over your app as a person at the keyboard. This page says who can reach it, what they can do, and where to run it. ## Who can reach it [Section titled “Who can reach it”](#who-can-reach-it) * **This machine only.** The agent listens on a loopback address: `127.0.0.1` by default, or `::1` or `localhost`. It refuses any other `bind` value at startup and starts nothing. * **No web pages.** A request carrying an `Origin` header from anything other than localhost is refused (HTTP 403). So is a body that is not `application/json` (HTTP 415), which a browser cannot send to another site without a preflight that the agent never approves. A page open in your browser cannot drive the app. * **One endpoint.** `/mcp` on the one port, POST only. Its threads are daemons, so it never keeps the app's JVM alive. ## What a caller can do [Section titled “What a caller can do”](#what-a-caller-can-do) * **What a user can do, through the UI.** Click, type, press keys, pick from menus, answer dialogs, choose files in the app's file chooser, read what is on screen, take screenshots of the app's windows. * **Write files.** `export_test` writes a test class under the tests folder (`-Dswingui.testsDir`, else `src/test/java`), and `export_recorded_test` a recorded test file under the recorded tests folder; neither overwrites another existing file. Screenshot steps and a failing `run_recorded_test` write PNGs under the screenshot folder or a path the caller gives. Exported tests are code: review them like any other change. * **Read a recorded test file.** `run_recorded_test` and `recording load` take a `path` that is absolute or relative to the app's working directory. A recorded test is a plain JSON file of UI steps; it holds no code, and running it does only what its steps say. * **Read the clipboard, and set it.** `read_clipboard` and `set_clipboard` work with the system clipboard as the app sees it. * **Call a plugin's hooks.** `plugin_call` reaches only the methods of the one interface a plugin declares as its hooks. Without a plugin there is nothing to call. * **Run a plugin's domain steps.** A `do` step, in `do_action`, a batch or a recorded test, runs code of the plugin inside the app, just as `plugin_call` does. A caller can pass only the step's name and its values, and only the steps a plugin offers. The `vocabulary` tool only reads. A recorded test with `do` steps names the plugins it needs and is refused when one is not ready. ## What a caller cannot do [Section titled “What a caller cannot do”](#what-a-caller-cannot-do) * **Run commands or evaluate code.** No tool starts a process, loads a class or evaluates an expression. * **Read a password.** A password field's content is never put into a snapshot, never read and never checked. A person typing into one while recording with the palette is not recorded either. * **See your credentials.** Login aliases and secrets live in `.swing-ui-agent/ui-users.properties` on the machine that runs the tests. Error messages list alias names, never their passwords or secrets. Keep the file out of version control. * **Take your mouse or keyboard.** Input is posted into the app's own event queue. The real pointer and keyboard stay yours, and the app can be driven while other windows are in front. ## One owner at a time [Section titled “One owner at a time”](#one-owner-at-a-time) A test run, a script or an agent can take a **lease**: then only calls that carry its token can act, and everyone else can only read. See [Leases and several clients](/swing-ui-agent/guides/leases/). ## Where to run it [Section titled “Where to run it”](#where-to-run-it) * **Development and test machines, and CI.** Start the app with the agent only where you mean to drive it. * **Test data.** The agent clicks what it is told to click. Point the app at a test database or a test account, never at production data. * **The person's choice.** The palette is on by default and is for a person at the screen. Start with `palette=off` on CI and shared machines (`mode=background` turns it off too): then no palette appears and nothing listens to input. The `recording` tool and recorded tests still work without it. ## What's next [Section titled “What's next”](#whats-next) * [Agent options](/swing-ui-agent/reference/agent-options/): `bind`, `port`, `palette`. * [Limits](/swing-ui-agent/project/limits/) # Skills > Three agent skills that teach a coding agent the workflow. A coding agent can find out on its own how to drive a Swing app, but it gets there faster and wastes fewer calls when it is told. The repository's [`skills/`](https://github.com/kormcrete/swing-ui-agent/tree/main/skills) folder holds three agent skills. Each is a folder with a `SKILL.md`, written for the CLI. The same tools also work over MCP. ## The three skills [Section titled “The three skills”](#the-three-skills) * **`swing-ui-record-test`:** record a test and save it as a recorded test file or a Java `@UiTest` class. Which kind to pick is on [Recorded tests or Java tests](/swing-ui-agent/start/choose-test-kind/). It covers: * the reset that starts a recording, and reaching the test's own starting screen first; * acting, then checking with `expect` so the check is recorded; * recording together with a person: the agent and the person share one recording and one undo history, and the agent only acts on what the person asked for ([how](/swing-ui-agent/start/record-together/)); * fixing steps, naming them in sentences (a sentence per step, per check, and per group; [how](/swing-ui-agent/guides/sentences/)), and the exports and their warnings; * proving the test passes when the app starts on another screen; * working with a person at the app: asking with `inspect highlight` and `pick`, and saving what they recorded with the palette. * **`swing-ui-targeting`:** turn words into a locator, so "the Customer field", "order ORD-1640" or "Checkout…" becomes a target the driver can reach. It also says what to report when a word does not resolve, and what needs a plugin. `references/recipes.md` lists common jobs as live steps next to the Java each one exports to. * **`swing-ui-driver`:** hand the clicking to a cheaper sub-agent. The main session keeps the decisions; the driver walks a checklist with scoped reads and batched steps, and reports pass, fail or gap per item. The file doubles as the sub-agent's definition. ## Install them [Section titled “Install them”](#install-them) 1. Copy the three folders into your agent's skills directory. For Claude Code, that is `.claude/skills/` in your project (or `~/.claude/skills/` for every project): ```sh cp -r swing-ui-agent/skills/swing-ui-* my-app/.claude/skills/ ``` 2. For the driver as a sub-agent, also make it an agent definition: copy `swing-ui-driver/SKILL.md` into your agent's sub-agent folder (for Claude Code, `.claude/agents/swing-ui-driver.md`). Add your platform's frontmatter, for example a smaller model and the shell tool. 3. If your agent reads no skill folders, put the text of `swing-ui-record-test` and `swing-ui-targeting` into its instruction file. The skills show the agent jar as `swing-ui-agent-0.1.0-SNAPSHOT-agent.jar` on port 8765. If yours lives elsewhere or listens on another port, say so once in your project's instructions. ## Use them [Section titled “Use them”](#use-them) Ask in your own words. For example: * "Record a UI test that places an order for Ada Shop Ltd with 3 × A4 paper and checks the total." * "Check by hand that Checkout… stays disabled when the cart has more Staplers than are in stock." * "Use the driver to walk the checkout for Café Lumen and tell me which steps fail." With the palette on (the default), you can also show the agent instead of describing it: see [Record a test by showing it](/swing-ui-agent/start/show/). ## The docs in one file [Section titled “The docs in one file”](#the-docs-in-one-file) The whole site is also plain text for agents: [`llms.txt`](https://kormcrete.github.io/swing-ui-agent/llms.txt) lists the pages, [`llms-full.txt`](https://kormcrete.github.io/swing-ui-agent/llms-full.txt) holds all of them, and [`llms-small.txt`](https://kormcrete.github.io/swing-ui-agent/llms-small.txt) a shorter version. Point your agent at one of them when it should know the tool beyond what the skills teach. ## What's next [Section titled “What's next”](#whats-next) * [Let your agent drive the app](/swing-ui-agent/start/drive/) * [Record a test with the agent](/swing-ui-agent/start/record/) * [CLI or MCP?](/swing-ui-agent/agents/cli-or-mcp/) # PlaceOrderUiIT, step by step > The Java test an agent recorded on Order Desk, step by step with the screenshots it takes. `PlaceOrderUiIT` places an order for Ada Shop Ltd, from File ▸ New order through the four checkout steps to the placed order in the Orders table. A coding agent recorded it through the swing-ui-agent command line; a person reviewed it and made three small edits. It takes a 2x screenshot after every step, and CI keeps them. Source: [`PlaceOrderUiIT.java`](https://github.com/kormcrete/swing-order-desk-demo/blob/main/src/test/java/io/github/kormcrete/orderdesk/PlaceOrderUiIT.java) in the Order Desk repository. ## How it was made [Section titled “How it was made”](#how-it-was-made) 1. The agent started from `session reset`, so the recording began at a defined start. 2. It drove Order Desk through the CLI, one tool call per step: the menu, the Customer field, the products table, the cart, the checkout dialog. After each step it checked the result with `expect` and took a `screenshot` with scale 2. 3. `export_test` wrote the recording as a JUnit 5 class. Each recorded check became an `assertThat`, each screenshot a `saveScreenshot(…, 2)`. 4. A person reviewed the class like any other code. The review edits: * **The order number is read, not hard-coded.** The app gives each new order the next number, so the test reads it from the status bar ("Order ORD-1640 placed") with `UiAssertions.eventually`, until that order's tab is selected. A second run against the same app passes too. * **The account terms are matched with `contains`.** The label also shows what Ada owes, which grows with every order, so the check matches only the part before it. * **Long lines are wrapped** at 120 characters. The same order as a recorded test, and what each kind can check, is on [One order, two files](/swing-ui-agent/demo/two-files/). See [Record a test with the agent](/swing-ui-agent/start/record/) for the same workflow on your app. ## The test, step by step [Section titled “The test, step by step”](#the-test-step-by-step) The class is a plain JUnit 5 test. `@UiTest` connects it to the agent and hands it a `Client`: ```java @UiTest class PlaceOrderUiIT { @Test void placesAnOrderForAda(Client ui) { ``` ### 1. A new order [Section titled “1. A new order”](#1-a-new-order) The test starts from wherever the app is, so it gets to its own starting screen through the menu bar. ```java ui.menu("File", "New order"); assertThat(ui.tab("New order")).isSelected(); ui.inWindow("main").saveScreenshot("PlaceOrderUiIT/01-new-order.png", 2); ```  ### 2. The customer [Section titled “2. The customer”](#2-the-customer) It types "Ada" into the Customer field and presses Enter. The tab is renamed after the customer, and the account terms appear. ```java ui.tab("New order").field("Customer").type("Ada"); ui.press("Enter"); assertThat(ui.tab("New order · Ada Shop Ltd")).isSelected(); assertThat(ui.tab("New order · Ada Shop Ltd") .label("Account · Invoice 30 days · Credit limit €2,000.00").contains()).isVisible(); ui.inWindow("main").saveScreenshot("PlaceOrderUiIT/02-customer.png", 2); ```  ### 3. The cart [Section titled “3. The cart”](#3-the-cart) Two products go into the cart, found by their Product column. The quantities are typed into the spinner cells in the Qty column. The checks read the totals as the cart shows them: subtotal, volume discount, VAT and total. ```java ui.tab("New order · Ada Shop Ltd").table().row("Product", "A4 paper, 500 sheets").cell("SKU").click(); ui.tab("New order · Ada Shop Ltd").button("Add to cart").click(); ui.tab("New order · Ada Shop Ltd").table().row("Product", "Gel pen, black").cell("SKU").click(); ui.tab("New order · Ada Shop Ltd").button("Add to cart").click(); ui.tab("New order · Ada Shop Ltd").table().row("Product", "A4 paper, 500 sheets").cell("Qty") .type("3", null, Boolean.TRUE); ui.tab("New order · Ada Shop Ltd").table().row("Product", "Gel pen, black").cell("Qty") .type("10", null, Boolean.TRUE); assertThat(ui.tab("New order · Ada Shop Ltd").label("Subtotal")).hasText("€26.70"); assertThat(ui.tab("New order · Ada Shop Ltd").label("Volume discount")).hasText("−€0.60"); assertThat(ui.tab("New order · Ada Shop Ltd").label("VAT (20%)")).hasText("€5.22"); assertThat(ui.tab("New order · Ada Shop Ltd").label("Total")).hasText("€31.32"); ui.inWindow("main").saveScreenshot("PlaceOrderUiIT/03-cart.png", 2); ```  ### 4. Address [Section titled “4. Address”](#4-address) **Checkout…** opens the checkout dialog. The address comes from Ada's account; the test checks the postal code and takes a screenshot of the dialog only. ```java ui.tab("New order · Ada Shop Ltd").button("Checkout…").click(); assertThat(ui.dialog().field("Postal code")).hasValue("10119"); ui.inWindow("dialog").saveScreenshot("PlaceOrderUiIT/04-address.png", 2); ```  ### 5. Shipping [Section titled “5. Shipping”](#5-shipping) ```java ui.dialog().button("Next").click(); assertThat(ui.dialog().radio("Standard · 3–5 business days")).isChecked(); ui.inWindow("dialog").saveScreenshot("PlaceOrderUiIT/05-shipping.png", 2); ```  ### 6. Payment [Section titled “6. Payment”](#6-payment) Ada pays on invoice and needs a PO number. Until there is one, the dialog says so and Next is disabled: the test checks both. ```java ui.dialog().button("Next").click(); assertThat(ui.dialog().label("Enter the PO number")).isVisible(); assertThat(ui.dialog().button("Next")).isDisabled(); ui.inWindow("dialog").saveScreenshot("PlaceOrderUiIT/06-payment.png", 2); ```  ### 7. Review [Section titled “7. Review”](#7-review) ```java ui.dialog().field("PO number").type("PO-7781"); ui.dialog().button("Next").click(); assertThat(ui.dialog().label("Total")).hasText("€37.20"); ui.inWindow("dialog").saveScreenshot("PlaceOrderUiIT/07-review.png", 2); ```  ### 8. Confirm [Section titled “8. Confirm”](#8-confirm) **Place order** asks to confirm. The test takes a screenshot of the question, then answers it with its **Place order** button. ```java ui.dialog().button("Place order").click(); ui.inWindow("dialog").saveScreenshot("PlaceOrderUiIT/08-confirm.png", 2); ui.answer("Place order"); ```  ### 9. The placed order [Section titled “9. The placed order”](#9-the-placed-order) The order is placed in the background, then its tab opens. This is the first review edit: the test reads the new order number from the status bar and waits until that order's tab is selected. ```java // the number is the app's next one: wait for the status bar to name the order whose tab is selected, so a // rerun against a running app passes too (the last run's "Order … placed" shows until this one replaces it) String order = eventually("the status bar naming the new order", Duration.ofMillis(8000L), () -> { String placed = ui.main().label("placed").contains().text(); // "Order ORD-1640 placed" String number = placed.substring("Order ".length(), placed.indexOf(" placed")); return ui.tab(number).states().contains("selected") ? number : null; }, number -> number != null); assertThat(ui.tab(order)).isSelected(); ui.inWindow("main").saveScreenshot("PlaceOrderUiIT/09-placed.png", 2); ```  ### 10. In the Orders table [Section titled “10. In the Orders table”](#10-in-the-orders-table) Back on the Orders tab, the new order is listed as Placed with its total. ```java ui.tab("Orders").click(); assertThat(ui.tab("Orders").table().cell("Status")).hasCell(row("Order", order), "Status", "Placed"); assertThat(ui.tab("Orders").table().cell("Status")).hasCell(row("Order", order), "Total", "€37.20"); ui.inWindow("main").saveScreenshot("PlaceOrderUiIT/10-orders-after.png", 2); ```  ## Run it [Section titled “Run it”](#run-it) * **By hand:** start Order Desk with the agent on port 8770, then `mvn -B -Pui verify`, as in [Run the tests](/swing-ui-agent/start/run/), or from IntelliJ as in [Record and run a UI test in IntelliJ](/swing-ui-agent/start/intellij/). The screenshots land in `target/ui-screenshots/PlaceOrderUiIT/`, `01-new-order.png` to `10-orders-after.png`. * **In CI:** the Order Desk [build workflow](https://github.com/kormcrete/swing-order-desk-demo/blob/main/.github/workflows/build.yml) runs it on every push, on Linux with Java 8, and keeps the `ui-screenshots` artifact. See [Run in CI](/swing-ui-agent/start/ci/). Each run places a real order on invoice, so Ada owes €37.20 more after it. Her credit limit is €2,000.00: restart Order Desk before about 40 runs, or the credit limit warning appears and the test fails. ## What's next [Section titled “What's next”](#whats-next) * [Tour of Order Desk](/swing-ui-agent/demo/tour/): more tests to record. * [Record a test with the agent](/swing-ui-agent/start/record/) and [Run the tests](/swing-ui-agent/start/run/) on your own app. # Tour of Order Desk > The demo app every example on this site uses, how to start it with the agent, and what to try. Order Desk is a small Swing app where a shop employee takes an order for a customer: pick the customer, add products to the cart, check out and place the order. It is the app every example on this site drives. It has the parts real Swing business apps have, so a coding agent has real work to do. All customers, products and orders are invented, and everything lives in memory: each start begins with the same data. Source: [kormcrete/swing-order-desk-demo](https://github.com/kormcrete/swing-order-desk-demo) (Java 8, FlatLaf, MigLayout, Apache-2.0).  ## Download and run it [Section titled “Download and run it”](#download-and-run-it) Order Desk is published ready to run, with the agent built in: it starts with the agent on port 8770. The Mac and Windows downloads include Java (Zulu JRE 8); the Java zip runs on your own Java 8 or newer, on any computer (Linux too). | Your computer | Download | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | Mac with Apple Silicon (M1 and later) | [kormcrete-order-desk-macos-arm64.dmg](https://github.com/kormcrete/swing-order-desk-demo/releases/latest/download/kormcrete-order-desk-macos-arm64.dmg) | | Mac with Intel | [kormcrete-order-desk-macos-x64.dmg](https://github.com/kormcrete/swing-order-desk-demo/releases/latest/download/kormcrete-order-desk-macos-x64.dmg) | | Windows (x64) | [kormcrete-order-desk-windows-x64.zip](https://github.com/kormcrete/swing-order-desk-demo/releases/latest/download/kormcrete-order-desk-windows-x64.zip) | | Linux, or any computer with Java 8 or newer | [kormcrete-order-desk-java.zip](https://github.com/kormcrete/swing-order-desk-demo/releases/latest/download/kormcrete-order-desk-java.zip) | The downloads are not signed yet, so the first launch takes one extra step: * **macOS:** open the `.dmg` and drag **Kormcrete Order Desk** to Applications. The app is not notarized by Apple yet, so macOS refuses to open it ("Apple could not verify…"). Clear the download mark once, in Terminal, then open it: ```bash xattr -dr com.apple.quarantine "/Applications/Kormcrete Order Desk.app" ``` * **Windows:** unzip, open the **Kormcrete Order Desk** folder and run **Kormcrete Order Desk.exe**. If SmartScreen stops it, click **More info → Run anyway**. * **Java zip:** unzip, then run `order-desk` in the **Kormcrete Order Desk** folder (`order-desk.cmd` on Windows). It uses `JAVA_HOME` when set, else the `java` on your PATH, and prints the agent's lines in the terminal. **Run a recorded test right away:** in the bar at the bottom of the window (the palette), press **Load recorded test…**. Order Desk's three recorded tests are listed: `place-order`, `cart-totals` and `cart-with-steps`. Pick one and press **▶ Run**, with **Step by step** if you want to watch. They run one after another on the same app, each from the Orders list. They come with the app and stay as they are: a test you record or change goes into a folder of your own (Export… asks for one). More: [Recorded tests](/swing-ui-agent/start/recorded-tests/). Then connect your coding agent while the app runs: ```bash claude mcp add --transport http order-desk http://127.0.0.1:8770/mcp ``` Other clients: [MCP clients](/swing-ui-agent/agents/mcp-clients/). If your agent cannot connect, another app may be using port 8770; the agent's log says so: `~/Library/Logs/Kormcrete Order Desk/agent.log` (macOS), `%LOCALAPPDATA%\Kormcrete Order Desk\agent.log` (Windows), the terminal (Java zip). All releases: [swing-order-desk-demo releases](https://github.com/kormcrete/swing-order-desk-demo/releases). ## Build it from source [Section titled “Build it from source”](#build-it-from-source) To record tests into the project, or change the app, build it yourself. swing-ui-agent is not on Maven Central yet, so you build both projects. 1. Clone both repositories side by side and install swing-ui-agent: ```bash git clone https://github.com/kormcrete/swing-ui-agent.git git clone https://github.com/kormcrete/swing-order-desk-demo.git (cd swing-ui-agent && mvn -B install -DskipTests) ``` 2. Build Order Desk: first its plugin (the app's own steps, which its tests use), then the app. This writes `target/order-desk.jar` and copies the agent jar to `target/swing-ui-agent-agent.jar`: ```bash cd swing-order-desk-demo mvn -B -f order-desk-plugin/pom.xml install mvn -B package ``` 3. Start it with the agent: ```bash java -javaagent:target/swing-ui-agent-agent.jar=port=8770,plugins=order-desk-plugin/target/classes \ -Dorderdesk.today=2026-10-12 -jar target/order-desk.jar ``` * `-javaagent:…` lets a coding agent drive the app on port 8770. Without it Order Desk is a normal app. * `plugins=order-desk-plugin/target/classes` loads the plugin; the recorded test `cart-with-steps` needs it. * `-Dorderdesk.today=2026-10-12` fixes the date used for new orders and the card expiry check, so every run looks the same. Leave it out to use today's date. 4. Check that the agent answers: ```bash java -jar target/swing-ui-agent-agent.jar --port 8770 session ``` ## The screens [Section titled “The screens”](#the-screens) ### Orders [Section titled “Orders”](#orders) The first tab lists the orders, newest first. Search by order number or customer, filter by status (All, Placed, Shipped, Cancelled), sort by any column, and right-click an order to open it, copy its number or cancel it. Only placed orders can be cancelled. ### New order [Section titled “New order”](#new-order) **New order** (or File ▸ New order) opens a New order tab. The ▾ next to it starts one for a recent customer. Type in the Customer field to pick a customer from its popup; the account terms appear next to it. Pick a category in the tree, select products and press **Add to cart**. In the cart, the Qty cells are spinners, and the totals update as you go: 10 or more of one product take 5% off that line.  ### Checkout [Section titled “Checkout”](#checkout) **Checkout…** opens a modal dialog with four steps: Address, Shipping, Payment and Review. Next stays disabled while a step is not complete, and the reason is shown next to it. **Address:** the customer's address on file, editable.  **Shipping:** Standard or Express, and a delivery note.  **Payment:** invoice with a PO number, or card.  **Review:** everything once more, with Edit links.  **Place order** asks to confirm, then places the order in the background ("Placing order…") and opens the new order's tab.  ### The rules [Section titled “The rules”](#the-rules) Each customer runs into a different rule, so there is always something to check. * **Credit limit:** an invoice order that takes a customer over their limit asks first. Birch & Co with an Espresso machine shows it.  * **Address errors:** Elm Street Studio has no postal code on file, so the field is marked in red and Next stays disabled until it is filled in.  * **Express:** it delivers to Germany and Austria only, so Café Lumen in Amsterdam cannot pick it.  * **Card only:** Café Lumen, Durand Bureau and Fjord Supplies pay by card, so there is no invoice choice.  * **Stock:** a cart line that asks for more than is in stock is outlined in red and Checkout… stays disabled. The Desk lamp is out of stock and cannot be added.  ### An order's tab [Section titled “An order's tab”](#an-orders-tab) A placed order opens in its own tab: number, status, its key facts, **Save receipt…**, **Cancel order**, and Items, Delivery and Invoice tabs. Save receipt… writes a plain-text receipt through a file chooser.   ### Closing and themes [Section titled “Closing and themes”](#closing-and-themes) * File ▸ Close tab, ⌘W (Ctrl+W elsewhere) and File ▸ Exit ask first while a New order tab has products in its cart. * View ▸ Theme switches between Light and Dark.    ## The Swing parts it has [Section titled “The Swing parts it has”](#the-swing-parts-it-has) Each part has a guide that shows how the agent and the tests handle it: * Closable tabs, labels, buttons and text fields: [Targets and locators](/swing-ui-agent/guides/targets/) * Sortable, filtered tables, and cart cells edited with a spinner: [Tables](/swing-ui-agent/guides/tables/) * The category tree: [Trees](/swing-ui-agent/guides/trees/) * The menu bar with accelerators, the ▾ menu next to New order, the Customer field's popup and right-click menus: [Menus and popups](/swing-ui-agent/guides/menus/) * The checkout wizard, confirm dialogs and the file chooser: [Dialogs and file choosers](/swing-ui-agent/guides/dialogs/) * Placing an order in the background, and buttons that enable when a rule is met: [Waiting](/swing-ui-agent/guides/waiting/) * Light and dark themes, and screenshots of a window or a dialog: [Screenshots](/swing-ui-agent/guides/screenshots/) ## Things to try [Section titled “Things to try”](#things-to-try) Ask your agent to do one of these, or [record it as a test](/swing-ui-agent/start/record-together/): * ▾ next to New order, then Ada Shop Ltd: a New order tab with Ada's account terms next to the Customer field. * Add 3 × A4 paper and 10 × Gel pen, black: the 5% volume discount appears and the cart total is €31.32. * Checkout… with Standard shipping and a PO number: the review shows €37.20, and Place order makes order ORD-1640. * Birch & Co with an Espresso machine on invoice: the credit limit warning, then Pay by card instead. * Elm Street Studio at checkout: the postal code is missing, so Next stays disabled until it is filled in. * Café Lumen at checkout: no Express to the Netherlands, and Café Lumen pays by card only. * 5 Staplers when 4 are in stock: the cart says so and Checkout… stays disabled; the Desk lamp cannot be added. * The Orders tab: search for "birch", filter by status, right-click an order to cancel it. * An order's tab: Save receipt… writes a plain-text receipt through a file chooser. * File ▸ Close tab, ⌘W (Ctrl+W elsewhere) and File ▸ Exit, which ask first while a cart has products. * View ▸ Theme ▸ Dark and back. ## What's next [Section titled “What's next”](#whats-next) * [Record together with your agent](/swing-ui-agent/start/record-together/): say the test in chat; the agent records it on Order Desk and saves a recorded test. * [Record a test by showing it](/swing-ui-agent/start/show/) * [Let your agent drive the app](/swing-ui-agent/start/drive/) * [Recorded tests or Java tests](/swing-ui-agent/start/choose-test-kind/) * [PlaceOrderUiIT, step by step](/swing-ui-agent/demo/recorded-test/): a Java test the agent recorded on Order Desk. * [Record a test with the agent](/swing-ui-agent/start/record/) # One order, two files > The same Order Desk order as a recorded test and as a Java test, what each can check, and where a technical check would go. Order Desk has one scenario in two files. A new order for Ada Shop Ltd goes through checkout and is placed. * **`uitest/place-order.json`** is a recorded test. A person can read it, replay it from the palette or the agent, and run it in Maven and CI. There is nothing to compile. * **`PlaceOrderUiIT`** is a Java test: a JUnit 5 class with `@UiTest`. Both were recorded by an agent and reviewed by a person. Both start at a reset and both end with the order placed. They differ in what they may check at the end. ## What the recorded test checks [Section titled “What the recorded test checks”](#what-the-recorded-test-checks) A recorded test holds exact values only: what a person could read off the screen when it was recorded. At the end of `place-order`, the agent confirms the order and checks the order's tab that opens, then the Invoice tab: ```json { "action" : "answer", "say" : "Confirm the order", "button" : "Place order", "checks" : [ { "say" : "The new order is Placed", "target" : { "role" : "label", "name" : "Placed" }, "text" : "Placed" } ] }, { "action" : "click", "say" : "Open the Invoice tab", "target" : { "role" : "tab", "name" : "Invoice" }, "checks" : [ { "say" : "The payment line shows the PO number PO-7781", "target" : { "role" : "label", "name" : "Invoice (30 days) · PO PO-7781" }, "text" : "Invoice (30 days) · PO PO-7781" }, { "say" : "The invoice total is €37.20", "target" : { "role" : "label", "name" : "Total" }, "text" : "€37.20" } ] } ``` The file holds no date, so `place-order` passes on any day. The invoice total, €37.20, is what the order adds up to. There is no order number in the file. The app gives each new order the next number, `ORD-1640` on a fresh start and a different one after that, so the recorded test cannot know it. It checks what is predictable: **Placed**, the PO line and the invoice total. A recorded test never reads a value from the app to use later; it has no parameters or captures. That is on purpose. It keeps the file readable by anyone who can use the app, and it is why the two kinds do not mix. ## What the Java test checks [Section titled “What the Java test checks”](#what-the-java-test-checks) `PlaceOrderUiIT` runs the same scenario and ends differently. It reads the new order's number from the status bar, then looks for that order in the Orders table: ```java // the number is the app's next one: wait for the status bar to name the order whose tab is selected, so a // rerun against a running app passes too (the last run's "Order … placed" shows until this one replaces it) String order = eventually("the status bar naming the new order", Duration.ofMillis(8000L), () -> { String placed = ui.main().label("placed").contains().text(); // "Order ORD-1640 placed" String number = placed.substring("Order ".length(), placed.indexOf(" placed")); return ui.tab(number).states().contains("selected") ? number : null; }, number -> number != null); assertThat(ui.tab(order)).isSelected(); ui.inWindow("main").saveScreenshot("PlaceOrderUiIT/09-placed.png", 2); ``` ```java ui.tab("Orders").click(); assertThat(ui.tab("Orders").table().cell("Status")).hasCell(row("Order", order), "Status", "Placed"); assertThat(ui.tab("Orders").table().cell("Status")).hasCell(row("Order", order), "Total", "€37.20"); ui.inWindow("main").saveScreenshot("PlaceOrderUiIT/10-orders-after.png", 2); ``` That is Java: a variable (`order`), a wait for a value that changes, and a lookup with it. A recorded test cannot do that. The Java test is also where a 2x screenshot is taken after every step. ## Where a technical check would go [Section titled “Where a technical check would go”](#where-a-technical-check-would-go) Order Desk keeps its data in memory, so it has no database to look into. In a real app, "the order is placed" may also mean a row in a database, a file on disk or a line in a log. The screen does not show those, and a recorded test cannot read them. That check is Java: after the UI steps, the test reads the new order's number as above and asks the database for it. The UI steps can come from a recording; the technical check is code that you or the agent write next to it. ## Which one to write [Section titled “Which one to write”](#which-one-to-write) * Start with a recorded test. It covers what a person can see, it is cheap to fix, and it replays from the palette. * Write a Java test when the test must read a value that changes per run, or check something the screen does not show. [Recorded tests or Java tests](/swing-ui-agent/start/choose-test-kind/) has the longer comparison. ## Run both [Section titled “Run both”](#run-both) With Order Desk running with the agent, as in [Run the tests](/swing-ui-agent/start/run/): * **The recorded test:** press **Load recorded test…** in the palette and **▶ Run**, or ask the agent to `run_recorded_test`. * **Both in Maven:** `RecordedUiIT` replays every file in `uitest/`, and `PlaceOrderUiIT` runs next to it, in `mvn -B -Pui verify`. ## What's next [Section titled “What's next”](#whats-next) * [`PlaceOrderUiIT`, step by step](/swing-ui-agent/demo/recorded-test/): the Java test with its ten screenshots. * [Run and edit a recorded test](/swing-ui-agent/start/recorded-tests/) # Dialogs and file choosers > Confirmations, wizards, option panes and file choosers. A modal dialog stops the app until someone answers it, and it stops a careless test the same way. swing-ui-agent never waits on an open dialog: the step that opened it returns, describes it, and leaves the answer to the next step. This page shows confirmations, a wizard dialog and a file chooser on Order Desk. The examples run from the Order Desk folder, with `UI="java -jar target/swing-ui-agent-agent.jar"`. ## A step that opens a dialog [Section titled “A step that opens a dialog”](#a-step-that-opens-a-dialog) Order Desk asks before it cancels an order. The step that asks comes back with the dialog under `pending`: ```sh $UI context_menu '{"target":{"scope":["tab:Orders"],"role":"table","row":{"Order":"ORD-1637"}},"path":["Cancel orders…"]}' ``` ```json {"passed":true,"action":"contextMenu(table [tab:Orders] row={Order=ORD-1637} > [Cancel orders…])","message":"","ms":39, "pending":{"title":"Cancel order","message":"Cancel order ORD-1637? Stock is restored and invoice totals are taken off what the customer owes.","buttons":["Cancel order","Keep"],"fields":[]}} ``` (Trimmed.) `pending` names the title, the message, the buttons and the fields. While the dialog shows, a step aimed at the window behind it is refused: ```json {"passed":false,"action":"click(button 'New order')","reason":"NOT_ACTIONABLE","message":"NOT_ACTIONABLE: blocked by dialog Cancel order"} ``` ## Answer it [Section titled “Answer it”](#answer-it) `answer` clicks the button with that text, or with `"escape"` presses Escape and requires the dialog to close: ```sh $UI answer '{"button":"Keep"}' $UI answer '{"button":"escape"}' ``` In Java: `ui.answer("Keep")`. `ui.modalShowing()` says whether a modal dialog shows, and `ui.describeModal()` returns its description. Use `answer` for option panes (`JOptionPane`). Their buttons can carry a name the look and feel set, such as `OptionPane.button`, with their text only in `value`, so `{"role":"button","name":"Keep"}` does not find them; `answer` goes by the text. Answer deliberately. A dialog that would send, delete or overwrite something is the test's decision, not something to click away. ## Work inside a dialog [Section titled “Work inside a dialog”](#work-inside-a-dialog) Target a dialog's elements with `"window":"dialog"` (the showing modal dialog), `"window":"dialog: