Back in 2022, Charter wrote about Ideaflow, a book by Jeremy Utley and Perry Klebahn of Stanford’s d.school that argued that you find good ideas by generating an enormous amount of them and then testing them to find the winners.
They argued that creativity can be measured by “the number of novel ideas a person or group can generate around a given problem in a given amount of time,” even if they’re of unknown quality. They called this metric “ideaflow,” contending that it’s “the only business metric that matters.”
Ideaflow was released before the explosion of generative AI, but the premise has intrigued us since we read it. We went back to Utley, who is a vocal proponent of the benefits of integrating genAI into your work, to understand how he’s updated the Ideaflow thesis. Here are excerpts from our conversation, edited for space and clarity:
In Ideaflow, you argue that generating 2,000 ideas might yield one successful outcome. How has AI changed how you think about that and what’s possible now that wasn’t when you wrote the book?
It collapses the innovation timelines substantially, not only for idea generation, but also for hypothesis testing. Furthermore, the bandwidth for experimentation increases a lot. So theoretically the capacity for innovation changes dramatically. Myself and a colleague from Singularity did some research on real teams in real organizations seeking to solve problems and seeking to innovate. Our big question was: How much more creative are teams with access to AI? I thought, maybe naively, ‘are they two times more creative? Could they be a hundred times more creative?’ And the truth is, the results were pretty ‘mid.’ Harvard Business Review, when they wrote it up, said, ‘Don’t let genAI limit your team’s creativity.’ The reason that they gave it that headline was because what we found was contrary to our expectations. AI-enabled teams didn’t outperform the unassisted teams.
In most cases, they actually underperformed. As we dug into the data, we realized it’s the same old cognitive bias at play. If you remember Ideaflow, there’s this cognitive bias that Abraham and Edith Luchins called the Einstellung bias. Herbert Simon called it ‘satisficing.’ Humans are satisfied by that which suffices—it’s our instinct for ‘good enough.’ We studied these teams in Europe and the US across heavy industry, industrial goods, banking, insurance, and healthcare technology. What we found was basically AI helps people ‘satisfice’ faster than ever. They actually didn’t do better per standard creativity measure, in terms of volume and variation, novelty [or] usefulness. They did worse, but they did worse because of the same underlying cognitive bias that’s always plagued our problem solving.
What you could say is AI just amplifies our underlying human cognitive bias. AI can generate 2,000 ideas very simply. People don’t ask for 2,000 ideas. What we found in this study again and again and again was [when] folks superficially prompt the AI, they get superficial responses. They go, ‘this is great, let’s go get coffee.’ And then they just stop. It’s like Herbert Simon’s rolling over in his grave. This is what’s always plagued problem solving. It’s our human tendency to give up.
What does it look like when you use AI to be more creative?
You have to shift your orientation from treating AI like a tool to working with AI like a teammate. The folks who work with AI like a teammate get much better outcomes. But the problem is it’s actually harder. Some people want working with AI to be magical. [But] they underperform people who want working with AI to deliver better work. It does deliver better work, but they are tired. They’re like, ‘Man, this was hard.’ So that’s an interesting tension. The people who were most satisfied were the people who did least well. The people who were least satisfied were the people who did best.
I say a lot, ‘Don’t use AI. Work with it.’ And if you say, ‘I use it every day,’ I know you’re underperforming because the people who perform well aren’t using AI at all. They’re working with it. It’s a dynamic thought partner and collaborator.
What are some of the things we look for? One is conversational turns. Is there a dynamic back-and-forth? One is the role of questions. Who’s asking the questions? Is the AI allowed to ask questions or is the human the question asker? And [is] the AI the answer giver? What’s the level of critique? What’s the level of response, refinement, iteration? What’s the amount of context that the human provides to the AI? If you think about a simple task, and it’s really helpful to think just in terms of hiring a new employee, how would you treat a new employee?
The simplest example is [what] if you bring on a new employee and said to them, ‘Would you write our all hands memo?’ Is that all you would say to them? Do you tell them what your voice is? Do you give them examples of old memos that have worked well and that haven’t? Do you tell them, ‘If you have any questions, feel free to ask me. My door’s always open.’
There’s all these things that you say to a normal human, but to an AI, you go, ‘Write an all hands memo.’ And [then] you’re like, ‘It sucks. It’s no good.’ Well, it probably doesn’t suck. But AI has a tendency to perform to one’s expectations. And, for most people, the truth is they kind of hope it sucks and therefore it does. But not because it’s incapable. It’s a self-fulfilling prophecy. It’s an expectation fulfillment.
If you raise your level of expectations, you actually get way better outcomes. Right now, there’s this weird bifurcation. Some people expect the AI to do a lot and they’re willing to put in the work and they’re getting Iron Man-suit 10x, 100x outcomes. [Others] are kind of skeptical and wary, and they aren’t willing to put in the effort.
Read Charter’s recent research playbook, Leading in the age of AI: Practices for the new era.