## The traditional PM frameworks that don’t fully apply anymore

The processes that PMs relied on in the past no longer fit when building AI products (and, to a lesser extent, when using AI tools). PMs who want to continue to build useful products need to be aware of the ways AI has changed their work, including:

- **Low-fi prototyping in tools like Figma doesn’t replicate personalization, relevance, or model reasoning**. Instead, AI PMs are using tools like [Replit](https://replit.com/) to vibe code basic proof-of-concept versions.
- **A/B testing may struggle because results vary by user context, not treatment alone**. AI’s probabilistic nature means that even two identical inputs won’t necessarily lead to the same output.
- **Traditional roadmaps need to move from linear cycles or feature checklists to ongoing model iteration and monitoring** to account for model changes in accuracy and [AI drift](https://www.ibm.com/think/topics/model-drift).
- **User research has to shift from usability to** [**corpus testing**](https://kenja.com/2024/11/27/understanding-ai-101-corpus/) **, prompt evaluation, and scenario simulation**, using human insights but also agentic AI tools for evaluation.
- **Success metrics shift from binary feature adoption to more intangible evals** such as output quality.

## The new questions AI PMs are asking (or should ask)

Alongside the challenges, PMs building AI products and using AI tools to help them build those products are also facing new opportunities. To seize those opportunities, they need to embrace new ways of thinking.

AI products and tools are non-deterministic and, by their nature, somewhat unpredictable. As Mixpanel’s Daniel Schmidt puts it,

> _“AI is like having a genius toddler in your codebase. The ceiling on what you can accomplish is extremely high, but the potential for things to go wrong is also much higher than traditional products. Engineers and product managers are learning to harness that potential while also limiting the risks.”_

### Ideation and roadmapping

One of the biggest changes that AI has brought to product development is [vibe coding](https://mixpanel.com/blog/vibe-coding-meets-data-analytics/), or the ability to use AI to build apps and simple digital products without deep engineering knowledge.

**Important questions to consider during the ideation and roadmapping phase:**

1. How do we create tangible differentiation when core AI capabilities are increasingly commoditized and considered as table stakes?
2. What does a roadmap look like when we need to iterate on model quality, rather than new features?
3. How do we balance speed to market with long-term trust and quality?

### Evaluation and QA

In many cases, AI products don’t have a static user interface, and interactions with AI products are entirely personalized and based on user inputs. This presents new problems for product evaluation and quality control.

**Important questions to consider for evaluation and QA:**

1. How do we test something whose quality is heavily dependent on input, user, business, or any other context?
2. If model accuracy changes over time (and drifts), what does 'ship' even mean?
3. What is the MVP for an AI feature, and what’s considered good enough?
4. How do we think about launch blockers in the context of relevance and accuracy?

### Prototyping and discovery

**Important questions to ask for prototyping and discovery:**

1. How do we validate the value of an AI feature if a low-fidelity prototype can’t replicate context or reflect any meaningful accuracy or relevance?
2. Do we need to prototype with real customer data to get meaningful feedback?
3. Do we still need to prioritize testing for usability or testing the UI?

### User experience

**Important questions to consider for UX:**

1. What’s the role of UX in expectation setting for the user?
2. What’s the role of UX when it comes to accuracy or quality? (e.g., Are you supposed to expose uncertainty, or should you hide it?)

## Join us as we explore how PMs PM in the AI era

At Mixpanel, our PMs are figuring out how to elevate their craft in the AI era. We may not have all the answers, but we want to share what we’re doing, where we feel like we’re onto something, where we’re getting it wrong, and bring you (our community) along for the journey.