Editor’s note: Before Substack started using Pangram, I used AI to help me write a piece about AI. Despite that the ideas were mine, a subsequent check on Pangram appropriately assessed that 100% of the text was generated by AI. (True story). I wasn’t that alarmed, as I disclose openly that I use AI. However, as an experiment, I went back to my original outline and wrote the piece by hand to see how vulnerable the content itself was to being detected as AI-generated, looking for a false positive. It didn’t happen. This time, the content was correctly assessed as 100% human written. The text follows below.
A drawing that resonated with me
A few weeks ago while browsing Reddit, I came across a drawing that depicted a phenomenon I’d been observing with my consulting clients that I hadn’t described so succinctly. After a bunch of reverse image searches on Google, I couldn’t track down the original creator, but I copied it here.
The diagram shows three columns for ideas, execution, and usage for both the “before AI” and “after AI” eras. In the “before AI” era, it depicts a small group of people generating ideas, a slightly larger group focused on execution and scaling of those ideas, with a large swath of downstream users. In the “after AI” era, it depicts a very large group of people generating ideas, a relatively fixed number focused on execution, but very few successful downstream users.
Of course, one interpretation is that the “after AI” version is better because there are more people empowered to generate ideas and bring them forward. However, I think the real story is in the shrinking downstream usage column.
What I’m seeing in consulting work
I’ve personally observed a version of this phenomenon in my consulting work. AI adoption is certainly growing across functions, and the apparent productivity gains appear impressive at first. Sales people who used to spend an entire day putting together a proposal can now do it in minutes. Support engineers who used to have to escalate issues to more experienced colleagues now get immediate suggestions on potential next steps. I’ve seen similar results across many functions where the next steps almost always involve consulting with AI. And, in many cases, I think the productivity benefits are real.
Where I’ve been seeing potential breakdowns are that the work products don’t end up effecting the desired results downstream. I have always believed that it’s easier to replicate success than diagnose failure. Repeating successes is just key to scaling, and the ability to repeat success depends on consistently applying lessons learned.
The problem is that the AI often doesn’t really draw on the true organizational learning when it generates proposals, recommendations, analyses, and plans. General purpose AI tools, such as ChatGPT or Claude, are trained on publicly available data, and even when augmented with corporate data, that internal data often entails a mixture of outdated presentations, unresolved email threads, and backward looking data not representative of the current business situations. As such, the output generated by AI tools in use often sounds credible and well-reasoned but fails to incorporate the key insights known by the broader organization to produce results.
More concretely, by not rigidly applying lessons learned, there can be sales presentations that don’t result in closed deals, customer support solutions that don’t result in more long-term success in the accounts, or product features that simply don’t get used.
To me, the big part missing in discussions with my clients about AI strategy is how to utilize AI to build on and enhance organizational learning. So much of the focus has been on making individuals more capable, and how to develop the individual’s AI skills on the journey from AI-assisted to AI-augmented, to AI-native. Still, the employee’s ability to utilize AI agents in their work doesn’t automatically translate into stronger organizational learning.
What changed my thinking
Instead, what is changing my thinking about AI is how individuals often use AI to develop their own individual ideas. Some of this is good, for sure. Still, every company I have worked with already has more ideas than it knows what to do with. Sales teams have ideas. Product teams have ideas. Customers have ideas. Executives have ideas. The bottleneck was almost never generating possibilities. The bottleneck was almost always prioritizing which possibilities deserved attention and then learning from the results.
That realization caused me to think about playbooks differently. I used to view a playbook primarily as a set of instructions. A sales playbook was something that told salespeople how to run a deal. A support playbook told teams how to generate customer successes. Today, I think a better description is that a playbook is a common learning surface.
For example, scaling in sales can happen when 100 salespeople all engage with a common customer profile, messaging, and sales process. This allows the group to refine its customer objection handling and understand which value propositions resonate. They can learn together to discover which customer segments perform better. While variances with some individual sales people happen and are interesting, they become even more valuable when they occur within the same basic framework, so lessons can be applied by others, too.
When people are operating against a shared playbook, the rest of the organization can also learn from their experiences. Sales Operations can identify patterns. Product marketing can adjust positioning. Leadership can rethink priorities. The important thing is not that everyone follows the playbook perfectly. The important thing is that everyone is contributing information back into a common system of learning.
The virtuous cycle comes from the organization’s ability not only to adhere to the playbook but to keep improving it. This is where I believe AI systems should play a critical role.
The part that worries me
I’m not worried about AI helping individuals in lowering the bar to create new ideas. A salesperson can generate a custom pitch. A marketer can create a new positioning approach. A support engineer can design a brand-new workflow. Much of this output is surprisingly good. In some cases it may even be better than what exists in the official playbook.
The issue is that every individualized version creates its own learning loop. There is a big difference between a hundred people improving one playbook and a hundred people improving a hundred different playbooks. In both situations people are learning, but only one of them creates organizational leverage. Utilizing the same, shared framework makes it easier to compare results, identify patterns, and refine the system. When everyone is working from their own AI-generated approach, the individual learnings become harder to aggregate and harder to interpret.
Individual ideas and deviations from the playbook have always existed. Great salespeople have always improvised and good managers have always tolerated a certain amount of drift from the script. What made that manageable was that deviation happened slowly enough and visibly enough for organizations to observe it and decide whether the change should become part of the playbook. AI changes that equation.
The problem is that widespread AI usage makes deviation fast, inexpensive, and often invisible. The deviation on its own is fine. The problems arise when organizations are unable to see the deviations clearly and learn from the outcomes. Without that feedback loop, everyone ends up choosing their own adventure. That may be empowering for individuals, but it does not necessarily help the organization learn.
The proposal that should never have existed
I observed this in my own consulting work. In that organization, the sales playbook called for the salespeople to identify the members of the actual buying committee before qualifying an opportunity. Numerous past deals demonstrated that involving the right stakeholders upfront made the deal more likely to renew, expand, and produce long-term customer value.
However, applying that lesson learned always involves more work for the salespeople, particularly when there are quarterly sales targets. In more than one example, AI was used to create highly customized, thoughtful, and persuasive proposals aimed at specific individuals in the account most likely to sign on their own, without the rest of the buying committee. The deals closed in the short term. The problem is that the organization had already learned that this kind of deal often performs poorly over time, and, consistent with those learnings, those deals resulted in customer churn.
In these cases, the proposals weren’t bad. AI didn’t create any mistakes. It simply made it easier to route around the institutional learning embedded in the playbook.
Why this reminds me of product marketing
This lesson in sales is consistent with one of my earlier writings about product marketing, where I focused on separating meaning from voice for AI content systems. In that piece, I argued that product marketing should own meaning, brand should own voice, and segment marketing should own a personalization layer for the type of customer targeted. The concept is that the AI tooling should be wired to render those shared assets into the form needed (e.g., sales email, web copy, blog post, etc.). This framework allows insights to be captured once, improved collectively, and adapted as appropriate.
Getting back to the diagram above, I believe the highest leverage use of AI isn’t helping every employee generate their own ideas, but rather helping the people responsible for the strategies and idea curation learn faster from the people executing it. When used this way, AI then becomes the enabler for accelerating organizational learning rather than fragmenting it.
The question I keep coming back to
While many are touting AI as a way to improve individual productivity because those gains are easier to observe, I want to focus on how AI can make organizations learn faster.
This shift toward using AI to strengthen organizational learning will involve a lot of instrumentation. How does feedback from the field actually reach the people responsible for improving the system? How does the next version of the playbook incorporate the learnings? How do we use AI to strengthen the connection between execution and learning?
To me, the Usage column is the most important in the drawing. While everyone notices the explosion in ideas, I don’t believe that more ideas necessarily lead to scale. I believe organizations scale when large numbers of people contribute to the same playbook to generate results. More ideas without more usage isn’t progress. It’s just more ideas.
Great thoughts, and thank you for openly sharing them, Stephen.
Your point that idea generation is becoming abundant seems to shift the real bottleneck somewhere else: from producing possibilities to deciding which possibilities deserve scarce attention, time, and capital.
This becomes even more difficult when abundance is collective, coordinated or not. Individuals can generate more ideas with AI, but so can teams, departments, and entire organizations. The number of possibilities can therefore grow much faster than our capacity to test them.
That makes me wonder about the logical status of judgment at this stage.
AI may help us compare, structure, challenge, or rank ideas. But AI itself is an abstraction of the world rather than direct contact with the world. Sometimes that abstraction may be extremely useful; sometimes it may be misleading precisely because the relevant future has not happened yet.
So when several ideas are internally coherent and none has yet encountered reality, on what basis can we legitimately prefer one over another?
Is there a logically defensible way to judge which novel idea deserves to survive before any of them has been exposed to the real world, or can AI at best help us decide which uncertainty to test first while remaining fundamentally unable to tell us which idea will actually survive contact with reality across desirability, feasibility, and viability (let's call this survivability)?
And if survivability can only emerge through contact with reality, does the scarcity created by AI-generated abundance shift from “good judgment” to something even more specific: the ability to design the smallest real-world encounter that can discriminate between competing possibilities before too many resources are committed?