Quick Buzz Feed
Agents are becoming the new developer platform, using semantic search with data from tools like Git, Slack, and Jira for context. Things to consider are setting guardrails to block or allow things, and using logs, metrics, and traces to understand agent behavior.
Whitney Lee and Viktor Farcic discussed at KubeCon & CloudNativeCon Europe how AI agents are transforming developer platforms. These agents take developer input, combine it with system context, and use a model to respond or execute tools. This marks a significant shift in providing developers with advanced tooling.
Semantic search is crucial for agents to efficiently locate relevant information. This information can be drawn from diverse sources such as Git repositories (code and manifests), pull request discussions, Slack threads, Jira tickets, wiki pages, and Zoom transcripts. Farcic emphasized that these varied sources capture the 'how we like to do things' knowledge base.
Companies are developing their own agents with a strong focus on security. Guardrails are essential to control agent behavior, either by blocking certain actions or requiring specific approvals for sensitive operations. The unpredictability of LLM outputs necessitates careful control over what an agent can do.
Agent traces provide vital visibility into how developers interact with the platform. They detail the models and tools used by an agent, token costs, and the execution path for tasks. Farcic highlighted that logs, metrics, and traces, especially OpenTelemetry's semantic conventions for GenAI, are key to understanding and debugging agent actions, allowing data to be sent to various observability platforms.
Viktor Farcic clarified that semantic search quality depends less on the embedding model or vector database and more on input quality and chunking. He advised against relying solely on outdated documentation, recommending continuous ingestion of fresh, well-chunked data with metadata. He also stressed that knowledge should be indexed, while dynamic state should be fetched live via tools, and emphasized measuring retrieval accuracy through traces.
Whitney Lee explained that individual traces aid debugging, while aggregated traces reveal platform usage patterns and developer needs, fostering a feedback loop for platform improvement. Farcic added that traces are the only way to understand an agent's non-deterministic execution paths, offering insights into tool choices, cost, and providing an audit trail for changes made in production environments.