absolutely

An engineer toying around with ChatGPT found OpenAI’s apparently world-leading LLM getting a little bolshy about how it would do at chess. In fact, ChatGPT itself asked Citrix engineer Robert Caruso to set it up against a basic chess program to see “how quickly” it would win: and then proceeded to get battered by an Atari 2600.

First things first: chess engines are now unquestionably superior to human players, and an off-the-shelf program like Stockfish will handily trounce the best in the world. There are also AI-based chess engines from the likes of DeepMind. And ChatGPT 4o, the latest model, may be a leader in LLMs—but it is not a chess engine.

Nevertheless, you might expect something a little more impressive than this. Talking to ChatGPT about the history of AI in chess “led to it volunteering to play Atari Chess,” said Caruso on LinkedIn. “It wanted to find out how quickly it could beat a game that only thinks 1-2 moves ahead on a 1.19 MHz CPU.”

And?

“ChatGPT got absolutely wrecked on the beginner level,” says Caruso. “Despite being given a baseline board layout to identify pieces, ChatGPT confused rooks for bishops, missed pawn forks, and repeatedly lost track of where pieces were—first blaming the Atari icons as too abstract to recognize, then faring no better even after switching to standard chess notation. It made enough blunders to get laughed out of a 3rd grade chess club.”

Video Chess is as basic as chess software comes, which is entirely a function of its era: the major challenge for the programmers was creating a working engine within 4KB (which was still double the standard 2KB for other VCS games). It essentially brute forces the best move in a given position, but lacks an overall strategy and doesn’t think ahead.

A decent human player, in other words, should have a pretty easy time conquering Video Chess. But for 90 minutes Caruso “had to stop [ChatGPT] from making awful moves and correct its board awareness multiple times per turn. It kept promising it would improve ‘if we just started over.’ Eventually, even ChatGPT knew it was beat—and conceded with its head hung low.”

(Image credit: Brian Mitchell via Getty.)

ChatGPT itself asked for the game of chess against an Atari, “which it proclaimed it would easily win,” after a conversation about Stockfish and AlphaZero. The LLM was apparently “curious how quickly it could win” and, because Caruso had told it he was a weak player, “offered to teach me strategy along the way.”

The story isn’t entirely one-sided. Caruso says that when ChatGPT had an accurate sense of the board it offered him some “solid guidance” and at times was “genuinely impressive.” But at others, and this will be familiar to anyone who’s spent much time fooling around with ChatGPT, “it made absurd suggestions… or tried to move pieces that had already been captured, even during turns when it otherwise had an accurate view of the board.”

Naturally the AI evangelists will be out in force to say this is meaningless, it’s not what LLMs are designed to do, and so on. But this does raise wider questions about the technology and particularly its understanding of context (or lack thereof). “Its inability to retain a basic board state from turn to turn was very disappointing,” says Caruso. “Is that really any different from forgetting other crucial context in a conversation?”

In a nod to Atari’s once-famous marketing slogan, Caruso signs off: “Have you played Atari today? ChatGPT wishes it hadn’t.”

Source link

Ho, Lethal Companions! Put down your airhorns, let fall your precious armfuls of plastic fish, and prick up your freakin’ ears. Something is coming to 2023’s breakout horror multiplayer game. Something that will make the music boxes and springhead marionettes look like child’s playthings! I mean, like the child’s playthings they already look like, but without the parts that make them horrifying. That something is… to be announced, but I considered the below teaser text pithy enough to be worth a shout regardless.

“The new creature is technically maybe the most powerful creature in the game, and the power easily goes to its head,” explains appallingly young developer Zeekerss on social media. “Sometimes it does things that make absolutely no sense, just because it can.”

There is an infinity of enigma couched within the words “sometimes it does things that make absolutely no sense”. There are countless things in Lethal Company that defy sense-making, not least the game’s players, who repeatedly land on emaciated moons populated by giant spiders and ghost girls to harvest rubber duckies and V-type engines, which they then feed to a mouth in a wall. I look forward to experiencing how Zeekerss plans to raise the stakes.

In general, I like the idea of an entity deliberately deciding to do things that seem nonsensical. When we encounter this in other games, we tend to call it an error, depending on our flexible understanding of where the border between sense and nonsense lies. I’ve always enjoyed reading forum thread attempts to reach across the line and redefine “purely” errant behaviour as plausible. For example, weaving a little mythology around the fact that horses in Skyrim are able to witness crimes. It’s because every horse in Tamriel is descended from Horseplay, God of Tattletaling.

Lethal Company’s thunder has been stolen by, amongst others, the recent R.E.P.O., with its “Jim Henson does Nier Automata” characters. Still, it remains a grand and very sociable anti-capitalist chiller. I’m deeply thankful to Zeekerss for engineering a genre of extraction game in which the extraction process is fundamentally cursed. On which note, this news article is now over – please gather up your airhorns and plastic fish and get back to the ship, pronto. Does anybody hear giggling?

Source link

ChatGPT asked to play an Atari 2600 at chess then ‘got absolutely wrecked on the beginner level’

Lethal Company’s next monster “does things that make absolutely no sense, just because it can”