Quick Buzz Feed
The Technology Test and Evaluation Division at NIST is launching a new program to provide researchers with a sequestered testbed environment for the evaluation of AI model performance.
NIST's Technology Test and Evaluation Division is initiating the Artificial Intelligence Technology Evaluation (AITE) program. This program offers a sequestered testbed environment for evaluating AI model performance across various tasks, datasets, modalities, and domains. AITE focuses on rigorous, objective assessment by testing AI models on blind data, mitigating the risk of train/test data contamination. Initial tasks concentrate on image analysis using large vision language models (VLMs) within the contexts of quantum science, genomics, and public safety, with plans for additional tasks to be introduced over time. NIST provides the infrastructure for common data, metrics, and scoring to help developers understand their models' performance.
AITE invites engagement from participants through two distinct tracks, each offering unique benefits. 'Data providers' are encouraged to submit original, inaccessible datasets from their domain along with a relevant task to be performed on that data, in return for receiving detailed measurements of top models on their specific task and data. 'Model providers' can submit their AI models to be tested against these diverse datasets and tasks. This allows model providers to understand their models' performance across a growing number of scenarios and compare their models against others using standardized metrics, all while ensuring that evaluation data is not used for model training.
Participation in the AITE program is open to anyone willing to engage as either a data or model provider, provided they adhere to the AITE Participation Agreement and rules. Individuals interested in participating or who have questions about the program can contact NIST via email at aite-poc[at]list[dot]nist[dot]gov. Further details, including an evaluation overview and specifications for the initial three tasks, are available on the AITE evaluation website, accessible via a 'Learn More' button provided in the announcement.