How Yelp reviews competing LLMS for correctness, relevance and tone to develop a user -friendly AI assistant

View app Yelp For decades, it has provided useful information to dinners and other users. He experimented with machine learning from his early years. During the recent explosion in AI technology, it still encountered obstacles as it worked to use modern large language models to power some features.

Yelp realized that customers, especially those who only occasionally use the app, have problems Connecting with its AI featuresLike his AI assistant.

“One of the obvious lessons we have seen is that it is very easy to build something that looks cool, but it is very difficult to build something that looks cool and is very useful,” Craig Saldana, Chief Product Officer at Yelp, told Venturebeat in an interview.

It was certainly not easy. After launching Assistant Yelp, his AI-powered service assistant, in April 2024, on a wider flow of customers, Yelp saw data on the use of his AI tools, which are actually beginning to decrease.

“The one who surprised us was when we started it as a beta of consumers – several users and people who are very familiar with the app – (and they) liked it. We received such a strong signal that this would be successful and then we spread it to everyone, (s) the performance just fell, “Saldana said. “It took us a long time to understand why.”

It turned out that Yelp’s more negligible users, those who occasionally visit the site or application to find a new tailor or plumber did not expect to talk to AI representative immediately.

From simple to more engaged features of AI

Most people know Yelp as a website and app to look for reviews for restaurants and menu photos. I use Yelp to find food photos at new eating facilities and see if others share my feelings for a particularly delicate dish. This is also a place that tells me, if a cafe I plan to use as a workspace for the day, there are WiFi, plugs and seating places, a rarity in Manhattan.

Saldana recalled that Yelp invests in AI “over the bigger part of a decade.”

“Back, when, I would say in the 2013-2014 timeline, we were in a very different generation AI, so our focus was on building our own models to do things like understanding requests. Part of the work of making a meaningful relationship is to help people improve their own intention to search, “he said.

But as AI continues to develop, the needs of Yelp. It invests in AI to recognize the food of photos presented by users to identify popular dishes, and then launches new ways to connect with traders and services and services and Help guide the users’ searches on the platformS

AI Assistant helps Yelp users find the right “professional” to work with. People can touch the chatbox and use prompted or introduce the task they need. The assistant then asks subsequent questions to narrow the potential service providers before making a message to Pro that they may want to bid for the job.

Saldana said professionals are encouraged to respond to users themselves, although he admits that larger brands often have call centers that process messages generated by the Yelp AI assistant.

In addition to AI Assistant, Yelp launches insights and Review accents. LLMS analyzes the mood of users and reviewer, which Yelp collects in mood results. Yelp uses detailed GPT-4o prompted To generate a set of data for a list of topics. Then, it is finely tuned with the GPT-4O-Mini.

The “Review” feature emphasizes information from reviews also uses LLM prompt to generate a data set. However, it is based on the GPT-4, with a fine setting of GPT-3.5 Turbo. Yelp said it would update the feature with GPT-4O and O1.

Yelp joined many other companies Using LLMS to improve usefulness of reviews by adding better search features based on customer comments. For example, Amazon launches rufusAI -driven assistant that helps people find recommended items.

Large models and performance needs

For many of its new AI features, including AI Assistant, Yelp turned to Openai’s GPT-4O and other models, but Saldana noted that no matter the model, Yelp data is the secret sauce for its assistants. Yelp did not want to lock in one model and there was an open mind about which LLMS would provide the best service for its customers.

“We use models from Openai, Anthropic and other AWS Beslrock models,” Saldana said.

Saldana explained that Yelp creates a section for testing the work of models in the correctness, relevance, consciousness, customer safety and correspondence. He said that “this is really the most hot models” that performed themselves best. The company manages a small pilot with each model before taking into account the cost of iteration and the latency of the answers.

Teaching users

Yelp also has made a coherent effort to train both everyday and energy users in order to feel comfortable with the new AI features. Saldana said that one of the first things they understood, especially with AI’s assistant, was that the tone should feel. Could not react too quickly or too slowly; Cannot be too encouraging or too rough.

“We make a lot of effort to help people feel comfortable, especially with this first answer. It took us almost four months to fix this second piece. And as soon as we did it, it was very obvious and you can see this hockey stick in the engagement, “Saldana said.

Part of this process included training of AI assistant to use certain words and to sound positive. After all this fine setting, the baking said that they finally see larger use numbers for Yelp’s AI features.