Great news for xAI: Grok is now very good at answering questions about Baldur’s Gate

Different AI labs have different priorities. For example, OpenAI has traditionally focused on consumer users, while its competitor Anthropic tends to target enterprises. We recently discovered that Elon Musk’s xAI technology is specifically focused on video game walkthroughs.

On Friday, Business Insider’s Grace Kay reported A detailed and far-reaching report on xAIa recent artificial intelligence startup Acquired by SpaceXwith a special focus on how Musk makes life difficult for employees. But this particular story stood out:

In one case last year, the release of a model was delayed for several days because Musk was unhappy with the way a chatbot answered detailed questions about the video game “Baldur’s Gate,” according to people familiar with the matter. They said senior engineers were pulled from other projects to improve responses before launch.

Of course, you can imagine the frustration of any respected, experienced engineer who comes to work thinking he’ll address the fundamental problems of knowledge and machine intelligence, but goes out of his way to help a 54-year-old man beat his video game. But this anecdote raises a more pressing question: Did Musk end up getting the gaming skills he wanted?

To answer that question, we have resident RPG enthusiasts Ram Iyer Put together a set of five trivia questions about Baldur’s Gate, which we confront with xAI and the three main models in a sort of semi-benchmark that I’ve decided to call Baldorpinch.

In the interest of journalistic transparency, I’ve posted all the chat transcripts, so you can see them here: Your puppy, ChatGPT, Claudeand twin.

First, the good news: Grok actually provides very good information. Her responses were a bit dense with player-specific jargon — “scum save” instead of save and “DPS” instead of damage — but the answers were helpful and informed, provided you know what you’re talking about. Grok also really likes tables com. theorycraftWhich is what you would expect.

There are a lot of Baldur’s Gate guides, and the models were generally based on the same models, so the biggest differences were in style. ChatGPT prefers bulleted lists and sentence fragments, while Gemini likes that broad Important words.

The biggest surprise was Claude, who was particularly interested in giving me information that would spoil my experience in the game. When I asked about good pieces of music for parties, she concluded the directions by saying, “Don’t push too much and just play what sounds fun to you.” Thanks, Claude!

It’s important to keep in mind that this is a subject area we’re familiar with (thanks Business Insider reports) that xAI focused specifically on achieving parity. So we shouldn’t read too much into the fact that after the announced race, Grok’s advice turned out to be the same as other models. However, it’s good to know that xAI can make it work if you try.