Synthetic data has its limits – why human-sourced data can help prevent AI model collapse

God, how quickly things turn around in the tech world. Just two years ago, AI was being hailed as “the next transformational technology to rule them all.” Now, instead of reaching Skynet levels and taking over the world, AI is ironically demeaning.

Once the harbinger of a new era of intelligence, AI is now tripping over its own code, struggling to live up to the brilliance it promised. But why exactly? The simple fact is that we are depriving AI of the one thing that makes it truly smart: human-generated data.

To feed these data-hungry models, researchers and organizations are increasingly turning to synthetic data. While this practice has long been a staple in AI developmentwe’re now moving into dangerous territory by over-relying on it, causing the AI ​​models to gradually deteriorate. And this is not just a minor concern ChatGPT achieving lower results — the consequences are much more dangerous.

When AI models are trained on outputs generated by previous iterations, they tend to propagate errors and introduce noise, resulting in a drop in output quality. This recursive process turns the familiar “garbage in, garbage out” cycle into a self-perpetuating problem, greatly reducing system efficiency. As AI moves away from human understanding and accuracy, it not only undermines productivity, but also raises critical concerns about the long-term viability of relying on self-generated data for continued AI development.

But this isn’t just a deterioration of technology; it is a degradation of the reality, identity and authenticity of data — posing serious risks to humanity and society. The ripple effects can be profound, leading to an increase in critical errors. As these models lose accuracy and reliability, the consequences can be dire—think medical misdiagnosis, financial loss, and even life-threatening accidents.

Another important conclusion is that the development of AI can completely stop and leave AI systems they cannot accept new data and are essentially “stuck in time”. This stagnation would not only impede progress, but also trap AI in a cycle of diminishing returns, with potentially catastrophic consequences for technology and society.

But in practice, what can businesses do to ensure the safety of their customers and users? Before we answer that question, we need to understand how it all works.

When a model crashes, reliability goes out the window

The more AI-generated content spreads online, the faster it will permeate datasets and subsequently the models themselves. And it’s happening at an accelerated pace, making it increasingly difficult for developers to filter out anything that isn’t clean, human-created training data. The fact is that the use of synthetic content in learning can cause a detrimental phenomenon known as “model collapse” or “model disorder of autophagy (MAD)’.

Model collapse is a degenerative process in which AI systems progressively lose their understanding of the true underlying data distribution they are designed to model. This often happens when AI is trained recursively on content it generates, leading to a number of problems:

• Loss of hue: Models begin to forget outliers or underrepresented information, which is critical to the overall understanding of any data set.
• Reduced diversity: There is a noticeable decline in the variety and quality of results produced by the models.
• Amplification of deviations: Existing biases, particularly against marginalized groups, can be exacerbated because the model ignores nuanced data that could moderate those biases.
• Generating meaningless results: Over time, models may begin to produce results that are completely unrelated or meaningless.

Case in point: a study published in Nature highlighted the rapid degeneracy of language models trained recursively on AI-generated text. By the ninth iteration, these models were found to produce completely irrelevant and meaningless content, demonstrating the rapid decline in data quality and model utility.

Future-proofing AI: steps businesses can take today

Enterprise organizations are in a unique position to shape the future of AI responsibly, and there are clear, actionable steps they can take to keep AI systems accurate and reliable:

• Invest in data sourcing tools: Tools that track where each piece of data comes from and how it changes over time give companies confidence in their AI input. With clear visibility into data provenance, organizations can avoid feeding models with unreliable or biased information.
• Deploy AI-powered filters to detect synthetic content: Advanced filters can capture Generated by AI or low-quality content before inserting it into training datasets. These filters help ensure that models learn from authentic, human-generated information, rather than synthetic data that lacks real-world complexity.
• Partner with trusted data providers: Strong relationships with verified data providers give organizations a steady supply of authentic, high-quality data. This means AI models get real, nuanced information that reflects actual scenarios, increasing both performance and relevance.
• Promoting digital literacy and awareness: By educating teams and customers about the importance of data authenticity, organizations can help people recognize AI-generated content and understand the risks of synthetic data. Building awareness about the responsible use of data fosters a culture that values ​​accuracy and integrity in AI development.

The future of AI depends on responsible action. Enterprises have a real opportunity to support AI based on accuracy and integrity. By choosing real, human-sourced data over shortcuts, prioritizing tools that capture and filter low-quality content, and fostering awareness of digital authenticity, organizations can steer AI down a safer, smarter path. Let’s focus on building a future where AI is powerful and truly beneficial to society.

Rick Song is the CEO and co-founder of Persona.