“It is clear over the last three years that transparency is on the decline while capability is going through the roof,” said Stanford professor Percy Liang, a researcher behind the Foundation Model Transparency Index. “We view this as highly problematic because we’ve seen in other areas like social media that when transparency goes down, bad things can happen as a result.”
Foundation models are AI systems trained on massive datasets that can perform a variety of tasks from writing to coding. Companies that develop foundation models are driving the surge in generative AI, which since the launch of Microsoft-backed OpenAI’s hit product ChatGPT has captivated businesses of all sizes.
In a world increasingly relying on these models for decision-making and automation, understanding their limitations and biases is crucial, the report’s authors say.
The index graded 10 popular models on 100 different transparency indicators, such as training data and how much compute was used. All models scored “unimpressively”: even the most transparent model, Meta’s Llama 2, received a score of 53 out of 100. Amazon’s Titan model ranked the lowest, at 11 out of 100. OpenAI’s GPT-4 model received a score of 47 out of 100.
The index’s authors hope that the report will encourage companies to increase their foundation model transparency, and also serve as a starting point for governments grappling with how to regulate the burgeoning field.
Discover the stories of your interest
The index is a project from the Stanford Institute for Human-Centered Artificial Intelligence’s Center for Research on Foundation Models.
