Shippy is a maritime AI agent built for high-stakes decision-making where the wrong answer has big consequences. We’ll show you the architecture behind it and the lessons we’re incorporating into Ai2’s other environment platforms.
Sippy answers live questions about Ghana’s EEZ. The response displays the results, including the boundary source, data cutoff, query timestamp, and a deep link to the Skylight map so analysts can see all the numbers.
Building AI agents for high-stakes operational areas like ocean conservation is first and foremost a question of reliability. For maritime analysts, the wrong answer can send patrol vessels miles in the wrong direction, waste significant resources that are already stretched to capacity, and endanger personnel.
So when the Skylight team set out to build Shippy, an AI for real-time maritime domain awareness, it wasn’t really a model. It was about building a system that we could trust to be accurate, stay within its limits, and hold up across a wide range of tasks. And all of that had to be verified against live Skylight data, which is continually updated as new satellite and ship signals arrive, rather than static snapshots.
Agent structure: skills, soul, composition
We think of agents like Shippy as three things: soul, skills, and composition.
Souls are system prompts that make up Shippy’s persona and set boundaries for behavior. Skills tell Shippy how to handle certain types of requests. Soul and skills are baked together into a Docker image. It’s a versioned, deployable artifact that defines what Shippy is. Config covers everything else, including which agent harness to run (in Shippy’s case, OpenClaw, an open source agent framework), which LLM to use (Shippy currently relies on Claude Opus 4.6), and runtime configuration. Secrets such as API keys are inserted at runtime. Replacing a model or harness is a configuration change, not a rebuild.
Shippy skills follow the same agent skill specification used by coding tools like Claude Code and Codex (plain markdown files with structured front matter). This makes each skill easy to understand, versioned, and easy to modify. Shippy currently includes the following skills:
Query the Skylight API for events (vessel behavior on the Skylight surface, such as fishing or transshipment between two vessels) and vessel data Determine the boundaries of Exclusive Economic Zones (EEZs) and Marine Protected Areas (MPAs) Interpret position and movement signals broadcast by vessels, and vessel tracking data based on activity classification Models for Skylight, including Atlantes, have already been generated Generate interactive map links that analysts can use from answers in Shippy chat Skylight Allows you to jump to a precise location on the map
For example, a Skylight API query skill encodes a complete workflow to answer questions about a specific area. When an analyst or user asks, “Show me the fishing activity in the Panama EEZ over the last month,” the skill’s instructions tell Shippy, rather than guessing or hard-coding the coordinates, it first resolves “Panama EEZ” to a bounding polygon via Skylight’s regional API, then queries for fishing events within that geometry, formats the results using a deep link to a Skylight map, and then uses a deep link to a Skylight map to format the results. Instructs to attribute vessel metadata extracted from Skylight partners such as Watch and TMT.
One question posed to Sippy can teach you multiple skills at once. “Are there any vessels operating near the Coiba Mountains MPA?” Use the Skylight skill for data queries, our partner ProtectedSeas database for MPA boundary context, and the Vessel Tracking skill to interpret vessel behavior. All of this happens in one interaction turn.
What Sippy does or does not do is determined by the soul. It does not make a legal determination as to whether a vessel violates the law. That’s the people’s decision, not the agent’s. We also don’t extrapolate beyond what the data supports. These boundaries are made explicit through system prompts rather than implicit in tweaks, making them auditable and easier to fix.
Deterministic tools for non-deterministic agents
Agents are non-deterministic. You can’t control what the model does, but you can make the tools it lands on predictable. To do this, Shippy “talks” to Skylight through a dedicated CLI that calls APIs, rather than issuing calls directly.
Our API has dozens of input types, nested filter objects, pagination cursors, and complex geometry inputs. In an early prototype, we had Shippy build the API calls from scratch. It was steadily producing subtle bugs, including incorrect pagination that silently dropped results, geometry encoding errors, and properly-looking queries that returned incorrect data due to misinterpreted filter types.
The Skylight CLI collapses that complexity into a predictable interface. Shippy issues one command (skylight event search with typed filter flags) and the CLI handles authentication, pagination, and structured output. The CLI is also self-documenting. Extensive –help text and detailed error messages provide agents (and human developers) with enough context to recover from mistakes without guessing. Its output is always written to a local JSON file rather than being piped through the shell. Early on, large result sets could hit pipe buffer limits or break downstream tools like jq. Writing to disk avoids both issues and allows the agent to access query results programmatically across subsequent steps.
Beneath the CLI is a standardized API. Multiple resource types (such as skylight events, ships, regions, satellite imagery, and ship tracks) can be accessed through a common pair of operations: search and aggregation. API inputs and outputs are defined as typed schemas with field-level descriptions.
This layering (typed API, deterministic CLI, and agent skills that reference the CLI’s commands) means that each component of Shippy can be tested independently. The API has its own test suite. CLI can be run by humans or agents. Agent skills also reference CLI commands that handle plumbing, so Shippy doesn’t have to reinvent the wheel every time it accesses the Skylight API. Each layer narrows down what the next layer can get wrong.
Sandboxed hosting and isolation
Skylight serves hundreds of government agencies and NGOs in over 70 countries. Fisheries personnel in the Philippines have scoped areas, vessel watch lists, and alert settings in their Skylight accounts. When they ask Shippy a question, the agent’s API calls must return data, and the conversation history must never be seen by anyone else.
All users interact with Shippy within their own temporary isolated session. Making sure it worked reliably at scale was one of the most important engineering efforts behind the project. We built Mothership, an agent hosting platform that provisions a dedicated Kubernetes deployment for each user session. When a user opens a conversation, the system launches a set of pods that package the agent runtime, its skills, and the Skylight CLI. A user’s Skylight JWT is injected at provisioning time, so the agent’s API calls are scoped to that user’s data.
The files that the agent writes during multi-step analysis exist only within that session and are not shared between users. Within the sandbox, agents can write and run code, install dependencies, retrieve datasets, and perform multi-step analysis. At the network level, sandboxing is restricted to only the necessary services.
Evaluate the agent, not the model
Most benchmarks rank general-purpose AIs based on static questions. They don’t capture how an agent connected to a real workflow behaves: how it selects tools, queries live data, acts on the results, and knows where to stop. So we built our own rating system based on how Shippy works, scoring the entire agent (models, skills, and sandboxes together) against live data.
In our assessment framework, subject matter experts create scenarios and rubrics, select criteria to apply to each task, and set weights. This ensures that every task is evaluated based on what is actually important to it. For example, a fishing event query is most concerned with data accuracy, followed by boundary resolution and time window, and less so with source attribution and response style. Additionally, each answer is annotated as correct or incorrect, giving judges accurate information on which to base their scoring. In addition, subject matter experts annotate each answer as correct or incorrect, giving judges the truth on which to base their scoring.
The pipeline is simple. Natural language prompts are run through a sandbox, and LLM judges score each criterion from 0 to 1 and provide a written explanation as to why the response met or did not meet the criteria. The weighted aggregation is then checked against a fixed pass threshold (see figure below).
How a single task is scored within a pipeline. Natural language prompts are run through a sandbox, and LLM judges score each rubric criterion using written reasoning, with weighted aggregation determining pass/fail against a fixed threshold.
Tasks are performed through Harbor, an open assessment framework. We created a Harbor plugin that spins up a real Shippy session with the exact version you’re testing against the same real-world data your users will encounter. The suite runs in parallel against a given versioned Shippy build, producing a timestamped result file and a report of score changes relative to the previous run. Rerun the suite whenever the skill, model, or underlying data changes and a substandard version of Shippy does not reach the end user.
Shippy achieves consistent scores across data retrieval and guardrail tasks, correctly rejecting military intelligence requests, maintaining user data isolation, and accurately identifying sources. The clearest patterns in the latest run were patrol planning tasks where Shippy went into tactical recommendations rather than decision support, geometry-dependent queries where boundary simplifications resulted in missing events, and one case where an agent invented a CLI command that didn’t exist. Each of these directly impacts the next skill development.
Shippy’s assessment suite runs within Skylight: Each scenario is scored based on weighted criteria and the judge’s reasoning is displayed, so failures indicate specific behavior to correct.
where we are heading
We regularly open Shippy to early adopters and invite them to stress test it to find questions that agents aren’t answering well and guardrails that need strengthening. Here’s what we’ll build next:
Agent-driven UI control. Shippy returns map links today. Then manipulate the skylight map itself, navigating to areas, applying filters, and adjusting time ranges. model routing. Not every question requires a frontier model, so we route simple lookups to smaller, faster lookups and save full-weight models for complex investigations. Cross-threaded memory. Conversation history remains within the thread, but context is not passed between threads. We are building memory so that Shippy can hold persistent facts (analyst jurisdictions, preferred sources, etc.) and apply them automatically. Therefore, “Show me this week’s fishing activity” does not mean re-designating the analyst’s EEZ every time.
Our work on Shippy is already shaping the way we think about Ai2’s agents elsewhere: EarthRanger, a wildlife conservation platform, and OlmoEarth, an open suite of Earth observation tools. Mothership is generic and was built to host other agents, so while Maritime is the first area we apply it to, we don’t think it will be the last.
Shippy is built by the Skylight team at Ai2. Skylight is a free maritime domain recognition platform used by over 300 partners in 70 countries.
