In the last few years, we’ve seen a shift in both the types of products PMs are building and the tools they use to build those products. The AI-powered tools we build are probabilistic, context-dependent, and always evolving. Their value is defined less by feature presence and more by output quality, trust, and relevance.

At the same time, AI is expanding what PMs can do, enabling faster experimentation, functional prototyping, and earlier validation of ideas. 

In other words, AI has accelerated product development. But it’s also exposed shortfalls in the traditional PM playbook. Many PM rituals, like prototyping, usability testing, A/B testing, and roadmapping, were designed for deterministic products with more predictable behavior. AI products don’t scale with that playbook, which forces PMs to rethink the product development process. 

PMs need to adapt to this changing landscape. Here are a few ways to do that. 

The processes that PMs relied on in the past no longer fit when building AI products (and, to a lesser extent, when using AI tools). PMs who want to continue to build useful products need to be aware of the ways AI has changed their work, including: 

Alongside the challenges, PMs building AI products and using AI tools to help them build those products are also facing new opportunities. To seize those opportunities, they need to embrace new ways of thinking. 

AI products and tools are non-deterministic and, by their nature, somewhat unpredictable. As Mixpanel’s Daniel Schmidt puts it, 

“AI is like having a genius toddler in your codebase. The ceiling on what you can accomplish is extremely high, but the potential for things to go wrong is also much higher than traditional products. Engineers and product managers are learning to harness that potential while also limiting the risks.”

The first step to harnessing that potential is asking the right questions and being clear-eyed about how AI (and building AI products) changes processes. Here are a few questions that AI PMs can ask themselves at different stages of product development. 

One of the biggest changes that AI has brought to product development is vibe coding, or the ability to use AI to build apps and simple digital products without deep engineering knowledge.  

For PMs, vibe coding offers valuable opportunities to test and flesh out ideas, without calling on engineering and design resources. “Inevitably, most product ideas are not going to work. That’s just the nature of product work,” Daniel says. “Vibe coding with tools like Replit allows me to realize the flaws in my ideas a lot faster, and I don’t need to devote design and engineering resources to a project until I’ve tested it out on my own first,” he explains. 

“Any product team that’s operating solely in the world of Figma instead of vibe coding prototypes is going to be slower than the competition. And they will overinvest in product ideas that just aren’t going to work, instead of discarding them at the early stages.”

Important questions to consider during the ideation and roadmapping phase:

In many cases, AI products don’t have a static user interface, and interactions with AI products are entirely personalized and based on user inputs. That’s the nature of probabilistic products: We don’t know what AI is going to produce based on a specific input. 

This presents new problems for product evaluation and quality control. Tools like tracing and AI evals help to solve these issues, but human oversight (at least in the early stages) has become more important than ever. 

“There’s a temptation for PMs to immediately jump to a higher level of sophistication and immediately use tracing and evals, when really the first step needs to be reading transcripts of what happened and seeing what the user’s input was and what the AI response was…You can’t skip that step of having a human interpret what went right and what went wrong. Later down the line, you can graduate into more automations and AI evals. Jumping to automations prematurely is a mistake.”

Qualitative and quantitative insights from tools like session replay and heatmaps can also give a more complete picture of the user experience. 

Important questions to consider for evaluation and QA:

One of the new challenges that AI poses is the distance between proof of concept, early prototypes, and building an actual product that will work reliably for real users. “There’s a very big gap between having a good demo of an AI feature and having an AI feature that you can truly launch and bring to market,” explains Daniel. Product teams have to understand this gap when they’re prototyping and building, as it can affect budgets, viability, timelines, and decision-making. 

Important questions to ask for prototyping and discovery:

Since every user journey is different with AI products, understanding those journeys and creating the best possible experiences becomes more difficult. It’s also more challenging to diagnose issues, since it’s harder to understand what they are: Is it a negative result, a prompting problem, or an engineering problem? 

“Understanding what’s working, what’s not, and how your product is designed is trickier when every user could get a result that you can’t predict, and you don’t fully know or control what they’re going to be experiencing,” Daniel says. 

Important questions to consider for UX: 

At Mixpanel, our PMs are figuring out how to elevate their craft in the AI era. We may not have all the answers, but we want to share what we’re doing, where we feel like we’re onto something, where we’re getting it wrong, and bring you (our community) along for the journey. We’d love to hear about your AI experiments too. Join the Mixpanel Community!