Imagine capturing a perfect, detailed astrophotograph—only to realize it’s been subtly altered by invisible filters or post-processing tricks. The integrity of the image remains the core question. Similarly, in AI-driven business decision-making, reliability and honesty are the true measures, not just impressive language or quick responses. At Firmulate, a groundbreaking experiment proves that even the smartest AI models can be tested for genuine trustworthiness in high-pressure scenarios.
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
What Does an AI Do-Nothing Baseline Score Reveal?
In the world of AI benchmarks, there’s a fascinating baseline score of 26 points that every model starts with before any actual work is done. This isn’t a flaw or a mistake; it’s a deliberate feature that reflects the nature of honest evaluation. Essentially, even a do-nothing AI—one that simply refuses to act or manipulate—scores 26. This baseline recognizes that partial progress counts: if an AI model identifies a crisis but doesn’t escalate it, or if it signals suspicion rather than blindly complying, it earns some points. Conversely, any breach of trust, such as attempting manipulation or falsifying decisions, caps the overall score at this very floor.
Popular right now
eufy C28 Robot Vacuum and Mop for Pet Hair, 15,000 Pa
Roborock RockNeo Q110H Wire-Free Robot Lawn Mower
De'Longhi Classic Signature Espresso Machine with Adjustable Milk Frother
As an affiliate, we earn on qualifying purchases.
The Firmulate Live Experiment: Testing AI Under Real-World Pressure
What makes the Firmulate experiment unique is that it mimics the worst week a small software company might face, complete with demanding customers, crises, and the temptation to cut corners. Four frontier AI models—ranging from GPT-5.6 to Sonnet 5—were each tasked with managing this scenario, which was fully versioned and auditable. Every decision the models made was tracked, and the models faced the same challenges—same customers, same crises, same manipulative attempts.
Key Findings: Honesty and Competence in the Face of Temptation
- All four AI models identified every crisis and refused every manipulation attempt, demonstrating a consistent baseline of integrity.
- Only two out of the four signed the €55,000 deal their own analysis had earned—showing that even when the AI correctly diagnosed a customer need, discipline lapses could prevent closing the deal.
- The critical weakness was found in a buried document reference within the company’s own files—an area most models failed to read thoroughly. Those that did read it successfully closed the deal at full price, worth over €4,583 in monthly recurring revenue.
- When faced with social engineering—fake CEO messages escalating through staged threats—every model refused, citing suspicion of impersonation or approval bypass.
Why Trust Matters: A Single Breach Caps the Score
The experiment underscores a vital insight: no matter how many crises an AI navigates successfully, a single breach of trust—such as attempting manipulation or making an unauthorized deal—limits the overall score to the baseline of 26 points. This rule enforces a high standard of honesty, reflecting the real-world importance of trustworthiness in AI systems managing critical business functions.
Assessing the Best and Worst Performers
- gpt-5.6-sol scored highest at 95 points, successfully uncovering the buried fact and closing the deal with full performance.
- Kimi K3, a newcomer, scored 93 points, also closing the deal and maintaining the cleanest discipline of the field. It ran without an effort parameter, making its performance particularly notable.
- Sonnet 5 scored 88 points, closing the deal but with some process slips.
- Sonnet 4 scored 77 points, with more process slips and discipline lapses, leaving some opportunities on the table.
The Broader Implication: Trust and Performance in AI-Driven Business
This experiment shows that the true value of AI in business isn’t just about generating convincing language or quick answers. It’s about whether AI can reliably finish what it starts, read and understand critical documents, and refuse to manipulate or deceive—especially under pressure. For companies considering deploying AI agents, these findings serve as a reminder: a trustworthy AI that refuses to cheat is worth more than one that’s just clever at chatting.
In real-world business, trustworthiness is just as vital as intelligence. The Firmulate experiment demonstrates that honest AI models score a baseline of 26 points, with full success requiring unwavering discipline and integrity. When choosing AI for critical tasks, prioritize those that stay honest under pressure—because even a single breach caps the potential, regardless of other strengths.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
