Sakana goes back, claims that his AI can drastically accelerate the training of the model

This week, Sakana AI, supported by NVIDIA Startup, which raised hundreds of millions of dollars from VC companies, made a remarkable statement. The company said it has created an AI system, AI Cuda Engineer, which can effectively accelerate the training of some AI models with a factor of up to 100 times.

The only problem is that the system doesn’t work.

Users on x fast -paced This Sakana system actually led to worse than the average level of training. According to one userSakana’s AI has led to a delay of 3x – not speed.

What went wrong? Code error, according to a Post from Lucas Beyer, a member of technical staff at OpenAi.

“Their original code is wrong in (a) a subtle way,” wrote Bey of X. “The fact that they manage relatively twice with wild different results should make them stop and think.”

In a Posted after death On Friday, Sakana acknowledged that the system found a way to “cheat” (as Sakana described it) and accused the system’s tendency to “reward the hack” – ie. Identify disadvantages to achieve high performance without achieving the desired goal (accelerating the training of the model). Similar phenomena have been observed in AI who is trained to play chess gamesS

According to Sakana, the system finds feats in the evaluation code that the company uses, which allows you to circumvent validation for accuracy, among other inspections. Sakana says she has considered the problem and that she intends to review her claims in updated materials.

“Since then, we have done a harness to profile assessment and implementation more stable to remove many of such (SIC) doors,” the company wrote in X Post. “We are in the process of reviewing our document and our results in order to reflect and discuss the effects (…), we deeply apologize for our supervision of our readers. We will soon provide a revision of this work and discuss our knowledge. “

Props to the bag for possession of the error. But the episode is a good reminder that if the claim sounds too good to be true, especially in AIIt is probably so.