What is new

Postman, the API development platform used by a large global developer community, has detailed the engineering behind Agent Mode, an AI-native way to work across API testing, documentation, discovery and implementation inside its product. The company and Amazon Web Services jointly published an account of what it actually took to run this agent in production on Amazon Bedrock, AWS's managed service for accessing foundation models. The interesting part is not a new model or a flashy feature, but a set of architectural lessons about what breaks when you attach an AI agent to a mature, interface-heavy product rather than a clean demo.

Postman says its team initially expected model quality and prompt design to be the hardest problems. Instead, the harder work turned out to be making an 11-year-old product legible to an agent that reasons over data instead of clicking through screens, menus and tabs the way a human user would.

How it works

An AI agent, in general, is a system that combines a language model with a set of 'tools' - defined actions it can call, such as fetching a record or modifying a setting - and loops between reasoning and acting until a task is done. The more tools you give an agent, the more choices it has to make correctly at each step, which is exactly where Postman ran into trouble.

Early versions of Agent Mode used many small, precise tools, for example one tool to open a request and another to update a single field. According to Postman's testing, once the visible toolset grew past roughly 40 tools, the agent started making more selection errors: calling tools that did not exist, passing wrong arguments even with valid schemas, or picking a tool that sounded right but was wrong for the context. Larger models reduced this problem but did not eliminate it.

The fix was dynamic tool selection. Agent Mode now holds a catalog of more than 170 tools but uses a vector database of tool embeddings - a way of representing each tool's meaning as a point in mathematical space so that similar or relevant tools can be found by proximity - to narrow that list down to around 15 tools relevant to the current request. Those are handed to a context-isolated sub-agent, so the model making the final decision only ever sees the small, relevant subset.

For data-heavy parts of the product, such as the API Catalog, Postman took a different route: instead of building a tool for every possible question, it gave the agent schema-aware access to a query engine. The agent can generate SQL-style queries directly against structured tables holding information like service uptime, test results and response times, because it understands the underlying schema. The article shows an example query joining data and filtering on latency and error rate. This shifts the engineering effort from building one tool per anticipated question to modelling the data well once, letting the agent compose a much wider range of queries than any team could pre-build.

The bigger surprise, Postman says, was that missing or poorly shaped context caused more failures than missing tools. Context here means the agent's understanding of where a user is in the product, which objects are active, and what has already happened in the session. Simply feeding the agent the same data structures the interface uses for rendering did not work, because those structures were built for displaying information, not for reasoning. Postman built dedicated 'context handlers' that distil each type of object - a request, a collection, a test result - into only what the agent needs to know. The system distinguishes background context, which is broad, shallow and gathered automatically, from selected context, which is deep and chosen deliberately by the user for a specific entity.

Agent Mode combines three layers: client-side tools that perform real actions in the Postman app, generic system instructions governing how the agent behaves, and a retrieval-augmented knowledge base built from Postman's Learning Center documentation, which is pulled in at runtime based on the query and the user's current context rather than stuffed into every prompt.

Why it matters

For practitioners building production agents, this is a useful counterpoint to the common assumption that scaling an agent mainly means picking a bigger or better model. Postman's experience suggests that tool catalog design, context engineering and infrastructure plumbing can matter more than model choice once an agent moves from prototype to a product used by a large, active user base.

  • Agent Mode requires user approval before any action that changes application state.
  • Amazon Bedrock Guardrails is used to redact personally identifiable information before it reaches the underlying model; enterprise admins can enable this in settings.
  • Postman routes requests across supported Anthropic Claude models on Bedrock, choosing faster models for high-volume interactions and larger models for complex reasoning, largely as a configuration change.
  • Cross-Region inference profiles let Postman choose between a geographic profile, which keeps processing within a defined region such as the US or EU, and a global profile, which routes among more regions for extra throughput during traffic spikes.
  • Postman has configured zero data retention for supported models, meaning Bedrock does not retain those prompts and completions; availability depends on the specific model.
  • Prompt caching reuses stable parts of the prompt, such as system instructions and core tool definitions, with a one-hour cache for near-fixed content and a five-minute cache for more variable context, reducing repeated processing cost and latency.

These four Bedrock capabilities - model flexibility, cross-Region inference, data-retention controls and prompt caching - address practical, unglamorous problems: bursty developer traffic, enterprise data-residency requirements, and the cost of resending large amounts of stable context on every single turn of a conversation.

Limits and open questions

The post is written jointly by Postman and AWS and functions partly as a case study for Bedrock, so its lessons are presented as generalizable best practices rather than independently verified benchmarks. It gives concrete internal findings, such as the roughly 40-tool threshold where errors increased and the narrowing from 170-plus tools down to about 15 per task, but it does not quantify overall accuracy, latency or cost improvements from the redesigned architecture. Postman also says its production implementation is proprietary and is not available as a public sample repository, so outside developers cannot inspect or replicate the exact system, only the patterns described. The company notes it is still working on decoupling tools from interface state (for example, allowing requests to be sent without an open tab) and is exploring a filesystem-backed approach to context handling, suggesting these are ongoing engineering efforts rather than a finished design.

Sources