Trapped

San Francisco-based Anthropic just dropped the fourth generation of its Claude AI models, and the results are… complicated. While Google pushes context windows past a million tokens and OpenAI builds multimodal systems that see, hear, and speak, Anthropic stuck with the same 200,000-token limit and text-only approach. It’s now the odd one out among major AI companies.

The timing feels deliberate—Google announced Gemini this week too, and OpenAI unveiled a new coding agent based on its proprietary Codex model. Claude’s answer? Hybrid models that shift between reasoning and non-reasoning modes depending on what you throw at them—delivering what OpenAI expects to bring whenever they release GPT-5.

But here’s something for API users to seriously consider: Anthropic is charging premium prices for that upgrade.

Image: t3.gg

The chatbot app, however, remains the same at $20 with Claude Max priced at $200 a month, with 20x higher usage limits.

We put the new models through their paces across creative writing, coding, math, and reasoning tasks. The results tell an interesting story with marginal improvements in some areas, surprising improvement in others, and a clear shift in Anthropic’s priorities away from general use toward developer-focused features.

Here is how both Claude Sonnet 4 and Claude Opus 4 performed in our different tests. (You can check them out, including our prompts and results, in our Github repository.)

Creative writing

Creative writing capabilities determine whether AI models can produce engaging narratives, maintain consistent tone, and integrate factual elements naturally. These skills matter for content creators, marketers, and anyone needing AI assistance with storytelling or persuasive writing.

As of now, there is no model that can beat Claude in this subjective test (not considering Longwriter, of course). So it makes no sense to compare Claude against third-party options. For this task we decided to put Sonnet and Opus face-to-face.

We asked the models to write a short story about a person who travels back in time to prevent a catastrophe but ends up realizing that their actions from the past actually were part of the events that made existence lean towards that specific future. The prompt added some details to consider and gave models enough liberty and creativity to set up a story as they see fit.

Claude Sonnet 4 produced vivid prose with the best atmospheric details and psychological nuance. The model crafted immersive descriptions and provided a compelling story, though the ending was not exactly as asked—but it fit the narrative and the expected result.

Overall, Sonnet’s narrative construction balanced action, introspection, and philosophical insights about historical inevitability.

Score: 9/10—definitely better than Claude 3.7 Sonnet

Claude Opus 4 grounded its speculative fiction in credible historical contexts, referencing indigenous worldviews and pre-colonial Tupi society with careful attention to cultural limitations. The model integrated source material naturally and provided a longer story than Sonnet, without being able to match its poetic flair, sadly.

It also showed an interesting thing: The narrative started a lot more vividly and was more immersive than what Sonnet provided, but somewhere around the middle, it shifted to rush the plot twist, making the whole result boring and predictable.

Score: 8/10

Sonnet 4 is the winner for creative writing, though the margin remained narrow. Writers, beware: Unlike with previous models, it appears that Anthropic hasn’t prioritized creative writing improvements, focusing development efforts elsewhere.

All the stories are available here.

Coding

Coding evaluation measures whether AI can generate functional, maintainable software that follows best practices. This capability affects developers using AI for code generation, debugging, and architectural decisions.

Gemini 2.5 Pro is considered the king of AI-powered coding, so we tested it against Claude Opus 4 with extended thinking.

We zero-shot our instructions for a game—a robot that must avoid journalists in its way to merge with a computer and achieve AGI—and used one additional iteration to fix bugs and clarify different aspects of the game.

Claude Opus created a top-down stealth game with sophisticated mechanics, including dynamic sound waves, investigative AI states, and vision cone occlusion. The implementation featured rich gameplay elements: journalists responded to sounds through heardSound flags, obstacles blocked line-of-sight calculations, and procedural generation created unique levels each playthrough.

Score: 8/10

Google’s Gemini produced a side-scrolling platformer with cleaner architecture using ES6 classes and named constants.

The game was not functional after two iterations, but the implementation separated concerns effectively: level.init() handled terrain generation, the Journalist class encapsulated patrol logic, and constants like PLAYER_JUMP_POWER enabled easy tuning. While gameplay remained simpler than Claude’s version, the maintainable structure and consistent coding standards earned particularly high marks for readability and maintainability.

Verdict: Claude won: It delivered superior gameplay functionality that users would prefer.

However, developers might prefer Gemini despite all this, as it created cleaner code that can be improved more easily.

Our prompt and codes are available here. And you can click here to play the game generated with Claude.

Mathematical reasoning

Mathematical problem-solving tests AI models’ ability to handle complex calculations, show reasoning steps, and arrive at correct answers. This matters for educational applications, scientific research, and any domain requiring precise computational thinking.

We compared Claude and OpenAI’s latest reasoning model, o3, asking the models to solve a problem that appeared on the FrontierMath benchmark—designed specifically to be hard for models to solve:

“Construct a degree 19 polynomial p(x) ∈ C[x] such that X := {p(x) = p(y)} ⊂ P1 × P1 has at least 3 (but not all linear) irreducible components over C. Choose p(x) to be odd, monic, have real coefficients and linear coefficient -19 and calculate p(19).”

Claude Opus 4 displayed its complete reasoning process when tackling difficult mathematical challenges. The transparency allowed evaluators to trace logic paths and identify where calculations went wrong. Despite showing all the work, the model failed to achieve perfect accuracy.

OpenAI’s o3 model achieved 100% accuracy on identical mathematical tasks, marking the first time any model solved the test problems completely. However, o3 truncated its reasoning display, showing only final answers without intermediate steps. This approach prevented error analysis and made it impossible for users to verify the logic or learn from the solution process.

Verdict: OpenAI o3 won the mathematical reasoning category through perfect accuracy, though Claude’s transparent approach offered educational advantages. For example, researchers can have an easier time catching failures while analyzing the full Chain of Thought, instead of having to either fully trust the model or solve the problem manually to corroborate results.

You can check Claude 4’s Chain of Thought here.

Non-mathematical reasoning and communication

For this evaluation, we wanted to test the models’ ability to understand complexities, craft nuanced messages, and balance interests. These skills prove essential for business strategy, public relations, and any scenario requiring sophisticated human communication.

We provided Claude, Grok, and ChatGPT instructions to craft a single communication strategy that simultaneously addresses five different stakeholder groups about a critical situation at a large medical center. Each group has vastly different perspectives, emotional states, information needs, and communication preferences.

Claude demonstrated exceptional strategic thinking through a three-pillar messaging framework for a hospital ransomware crisis: Patient Safety First, Active Response, and Stronger Future. The response included specific resource allocations of $2.3 million emergency funding, detailed timelines for each stakeholder group, and culturally sensitive adaptations for multilingual populations. Individual board member concerns received tailored attention while maintaining message consistency. The model provided a good set of opening statements to grab an idea of how to approach each audience.

ChatGPT was also good at the task, but not at the same level of detail and practicality. While providing solid frameworks with clear core principles, GPT4.1 relied more on tone variation than substantive content adaptation. The responses were extensive and detailed, anticipating questions and moods, and how our actions may impact those being addressed. However, it lacked specific resource allocations, detailed deliverables, and other details that Claude provided.

Verdict: Claude wins

You can check the results and Chain of Thought for each model, here.

Needle in the haystack

Context retrieval capabilities determine how effectively AI models can locate specific information within lengthy documents or conversations. This skill proves critical for legal research, document analysis, academic literature reviews, and any scenario requiring precise information extraction from large text volumes.

We tested Claude’s ability to identify specific information buried within progressively larger context windows using the standard “needle in a haystack” methodology. This evaluation involved placing a targeted piece of information at various positions within documents of different lengths and measuring retrieval accuracy.

Claude Sonnet 4 and Opus 4 successfully identified the needle when embedded within an 85,000 token haystack. The models demonstrated reliable retrieval capabilities across different placement positions within this context range, maintaining accuracy whether the target information appeared at the beginning, middle, or end of the document. Response quality remained consistent, with the model providing precise citations and relevant context around the retrieved information.

However, the models’ performance hit a hard limitation when attempting to process the 200,000 token haystack test. They could not complete this evaluation because the document size exceeded their maximum context window capacity of 200,000 tokens. This is a significant constraint compared to competitors like Google’s Gemini, which handles context windows exceeding one million tokens, and OpenAI’s models with substantially larger processing capabilities.

This limitation has practical implications for users working with extensive documentation. Legal professionals analyzing lengthy contracts, researchers processing comprehensive academic papers, or analysts reviewing detailed financial reports may find Claude’s context restrictions problematic. The inability to process the full 200,000 token test suggests that real-world documents approaching this size could trigger truncation or require manual segmentation.

Verdict: Gemini is the better model for long context tasks

You can check on both the need and the haystack, here.



Conclusion

Claude 4 is great, and better than ever—but it’s not for everyone.

Power users who need its creativity and coding capabilities will be very pleased. Its understanding of human dynamics also makes it ideal for business strategists, communications professionals, and anyone needing sophisticated analysis of multi-stakeholder scenarios. The model’s transparent reasoning process also benefits educators and researchers who need to understand AI decision-making paths.

However, novice users wanting the full AI experience may find the chatbot a little lackluster. It doesn’t generate video, you cannot talk to it, and the interface is less polished than what you can find in Gemini or ChatGPT.

The 200,000 token context window limitation affects Claude users processing lengthy documents or maintaining extended conversations, and it also implements a very strict quota that may affect users expecting long sessions.

In our opinion, it is a solid “yes” for creative writers and vibe coders. Other types of users may need some consideration, comparing pros and cons against alternatives.

Edited by Andrew Hayward

Generally Intelligent Newsletter

A weekly AI journey narrated by Gen, a generative AI model.

Source link