Amazon Web Services has released UAEF, a free open-source tool for testing AI agents across several software frameworks. It scores agents on 36 measures and flags regressions when prompts or models change.
Amazon Web Services has made its Universal Agent Evaluation Framework (UAEF) available as open-source software. It assesses tool use, answer quality, safety, speed, long conversations and collaboration between agents. Some scores are calculated directly; others use an AI model on Amazon Bedrock as a judge.
Agent frameworks such as LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore record an agent’s steps differently. UAEF converts those records into a shared format, addressing the need to rebuild tests when teams switch frameworks.
For teams updating an AI agent, UAEF can compare saved test runs and flag any score that falls by more than 5%, grading the regression by severity. The software is free, but AI-based judging and results storage still depend on AWS services.
A shared testing format could reduce the work needed to compare agents or switch development frameworks.
AWS-dependent judging and storage mean open-source availability does not remove cloud dependencies.
Reported use by 12 organisations does not yet establish that UAEF improves agent reliability at scale.
Written automatically by the Made in 10 Newsroom from the source above.
Databricks says Genie Ontology achieved a 91% pass rate on workspace tasks, compared with 45% and 58% for 2 frontier AI agents. The test used a fictional business whose reporting rules were scattered across tables, notebooks and documents.
BloombergNEF research finds AI data centres plan 181GW of on-site gas-fired power across the US, UK and mainland Europe. The proposals aim to bypass electricity grid connection delays, with 90% of the planned gas-fired power stations in the US.
Databricks has launched an updated Genie assistant and AI agents that continuously analyse company data. A separate Ontology feature maps information across an organisation so staff can retrieve it and run analyses.