Newsora
The Information11/16 23:01 · 1m ago

The Controversial AI Interpretability Research that Caught Sam Altman’s Eye

During OpenAI’s talks in recent months with AI startup Goodfire over a possible acquisition or partnership—discussions my colleagues and I reported this morning—OpenAI CEO Sam Altman was particularly interested in the work of Nick Cammarata, a onetime OpenAI researcher now at Goodfire who has made waves recently with new claims about how AI thinks.Cammarata published an essay last month and a follow-up report last week that have caused quite a stir among AI researchers—with some applauding it as a potential breakthrough in our ability to understand AI models and others dismissing it as hype. The reaction goes to show that some areas of AI research are still so immature that scientists don’t even agree on what counts as good research—a fact that hasn’t prevented billion-dollar valuations for startups based on unproven AI technology. Goodfire’s research specialty is known as mechanistic interpretability, or mech interp for short, which seeks to understand the inner workings of AI models rather than just their outward behavior. AI models are made of trillions of numbers, making them “black boxes” that a person can’t understand the way a programmer can simply read the lines of code in traditional software. As a result, it’s tricky to know how an AI model will act in new situations and to understand mistakes when they occur.

Aggregator site: only the publisher's summary is shown here. Read the full story at the source.
Read full story →