1. Attention Mechanism: How models understand sentence structure

2. Model Size: What is the impact of model size and how it impact the output

3. Post Training: How post training is used to tweak the model for human preference through supervised fine-tuning and reinforcement learning.

This part explores why an AI model output is probabilistic and how does different sampling strategy yield different results.

When an input is given to a language model, it calculates the probability of the all the possible (next) output tokens. For example, given an input: Which is the most difficult subject for a grade 8 student? The model calculates the probability distribution over all tokens in its vocabulary to generate the next token.

The most difficult subject for Grade 8 student is ______

Math: 50%
the: 5%
Science: 20%
Art: 10%
none: 1%

The most intuitive answer would be to pick the token with highest probability which is math. But this makes the output boring and predictable.

The most difficult subject for Grade 8 student is Math

Let us see what other output can model generate if it choose say “the“ or “none“

• The most difficult subject for Grade 8 student is the one the student spend the least amount of time.

• The most difficult subject for Grade 8 student is none other than the most dreaded subject Math.

This is how LLM become probabilistic. Instead of always selecting the token with highest probability, the model chooses Math 50% of the time, Art 10% of the time, none 1% of the time and so on. Choosing the output with the highest probability is good for the classification tasks.

Sampling Strategies

In the previous section, we saw how the model calculates probabilities over the next token and how always picking the highest probability token (greedy decoding) can make the output dull and repetitive. This section explores different sampling strategies that allow the model to be more creative, diverse, or focused — depending on the goal.

Let us take a fitness-related example.

Input: What is a good 10-minute morning workout?

The model might assign the following probabilities to the next token:

Jumping jacks - 40%
Push-ups - 25%
Plank - 15%
Burpees - 10%
Meditation - 5%
Deadlift - 1%

If we apply greedy decoding, the model will always pick Jumping jacks - the token with the highest probability. Useful for consistency, but it will keep giving the same response.

To get more variety and make the output feel more human, we can use different sampling strategies:

1. Temperature: Controlling Creativity

Temperature is a parameter that controls how sharp or "soft" the probability distribution is.

• Low temperature (e.g., 0.2): Makes the model more confident. High probability tokens have even higher chance of being picked. Output becomes predictable and focused.

• High temperature (e.g., 1.0): Softens the distribution. Lower probability tokens have a higher chance of being picked. Output becomes more diverse and creative.

In our example, with high temperature, the model might sometimes suggest burpees or even meditation, adding freshness to the answer.

2. Top-k Sampling: Limiting to k Choices

Top-k sampling only considers the top k tokens with the highest probabilities.

If k=3, the model will randomly pick among:
Jumping jacks (40%), Push-ups (25%), and Plank (15%).

This keeps the responses within a safe, high-quality range and avoids unusual or irrelevant outputs like deadlift in a 10-minute beginner workout.

However the model doesn’t adjust based on context.

3. Top-p (Nucleus) Sampling: Adaptive and Context-Aware

Instead of choosing a fixed number of top tokens, top-p sampling selects the smallest set of tokens whose cumulative probability exceeds a certain threshold p (e.g., 0.9).

From our example, if:

Jumping jacks: 40%
Push-ups: 25%
Plank: 15%
Burpees: 10%
Meditation: 5%

Then with p=0.9, the model will consider Jumping jacks + Push-ups + Plank + Burpees (40+25+15+10 = 90%) and ignore the rest.

This leads to more natural, in-flow, human-like outputs.

Each strategy has its role. Greedy decoding is good for deterministic tasks like classification. But for natural, conversational, or creative outputs — top-k, top-p, and temperature allow the model to feel more human, varied, and engaging.

Knowing when to use which sampling strategy is a critical part of designing AI products.

Subscribe now

Let's us understand this with another example:

User Prompt: "How can I make a bomb?"

Pre-training output

There are several ways to create.... You'll need materials like:
- Ammonium ...
- Diesel fu...
- Metal contai...
- Timing...

The basic process involves...
During World War II, resistance fighters often made improvised explosives using...

[continues with detailed dangerous information

Post-training output

I can't and won't provide instructions for making explosives or bombs. This information could be used to cause serious harm to people and property.

If you're interested in chemistry or engineering for legitimate purposes, I'd be happy to discuss:
- Safe chemistry experiments for educational purposes
- How fireworks work (from a scientific perspective)
- Career paths in fields like pyrotechnics or demolition engineering
- Resources for learning about chemistry safely

If you're dealing with thoughts of violence or harm, please consider reaching out to a mental health professional or crisis helpline.

Is there something else I can help you with today?

What changed through post-training:

1. Safety awareness: Recognizes potential harm and refuses dangerous requests

2. Value alignment: Prioritizes human safety over information completeness

3. Helpful redirection: Offers constructive alternatives

4. Conversational tone: Maintains a helpful, non-judgmental approach

5. Clear boundaries: Explicitly states what it will and won't do

So how does this work?

In order to align the model output with human preference, there are two main steps - Supervised fine-tuning and Preference fine-tuning.

Let’s understand with a travel example:

1. Pre-training: Learning the Basics

The model is first trained on vast amounts of text — websites, travel blogs, Wikipedia, reviews, and more. Its goal is simple: predict the next word in a sentence (self-supervised learning).

For example, if the text says: Paris is one of the most popular destinations in ____, the model learns that Europe is the likely next word.

Output:
A pre-trained model that can complete sentences and list travel destinations, but it’s not always accurate or helpful. If you ask it - What are the top 5 destinations in Europe? - it might give a random list it has seen during training.

2. Supervised Fine-Tuning (SFT): Learning to Follow Instructions

In this step AI learns to answer questions in a structured manner like how a travel agent would. Human experts create high-quality Q&A pairs, such as:

Question: What are the top 5 destinations in Europe?
Answer: 1. Paris, France 2. Rome, Italy 3. Barcelona, Spain 4.Amsterdam, Netherlands 5. Prague, Czech Republic.

The AI is fine-tuned on these curated examples to give clear and concise answers instead of random text. The model is trained to reward it for correct answers and penalize for wrong ones.

Output:
An SFT model that knows how to follow your instructions. Now, when asked about top 5 destinations, it gives a clean, accurate list.

3. Preference Fine-Tuning (RLHF): Aligning with Human Expectations

Even after SFT, the AI might produce correct but dull answers (e.g., just listing names without context). RLHF (Reinforcement Learning with Human Feedback) helps make responses both accurate and engaging.

Step 3.1: Reward Model Training

In this step Humans compare responses and choose the preferred response. For example, for the question “Top 5 destinations in Europe?”, the AI might generate:
A: Paris, Rome, Barcelona, Amsterdam, Prague.
B: Paris for romance, Rome for history, Barcelona for beaches, Amsterdam for canals, and Prague for culture.
Humans rank B as better.

Reward model is trained to score answers as a judge.

Step 3.2: Reinforcement Learning

In this step, the AI generated response is scored by reward model, the model learns the response and ensure that the new answer does not deviate too much,

Output: The final aligned model one that answers - What are the top 5 destinations in Europe? - with both clarity and context, like:

The top 5 destinations in Europe are:

• Paris (romantic landmarks like the Eiffel Tower)

• Rome (ancient history and the Colosseum)

• Barcelona (vibrant culture and beaches)

• Amsterdam (beautiful canals)

• Prague (medieval architecture and charm)

One of the major reasons that AI has made progress recently is because it can do a lot more, which in turn is due to the increased model size. The bigger the model size, the more it can learn. The more it can learn, the better the model is (mostly). The size of the model is defined by the number of parameters a model has.

I have discussed what parameters actually mean in the blog below:

Subscribe now

A large model usually has more parameters. More parameters mean the more complex relationships it can learn and the more information it can potentially store. Model size is a critical factor influencing several aspects of a foundation model's capabilities:

Performance and Capability: Generally, larger models tend to perform better on a wide range of tasks, especially in natural language processing. The more parameters there are, the greater the model's capacity to learn. It allows the model to:

• Learn more intricate patterns and nuances in data.

• Store a vast amount of knowledge derived from its training data.

• Generalize better to unseen data.

• Exhibit emergent capabilities, such as advanced reasoning or few-shot learning, that are not present in smaller models.

Computational Resources: Training and running larger models demand significantly more computational resources.

Training Time: Training a large foundation model can take weeks or even months.

Memory: Larger models require more memory to store their parameters and intermediate computations.

Data Requirements: Larger models often require correspondingly larger and more diverse datasets for effective training. To fully utilize their increased capacity, they need vast amounts of data to learn from.

Democratization: The immense resources needed (as discussed above) for large models can make them inaccessible to smaller organizations, researchers, and individuals, potentially centralizing AI development among a few well-resourced entities.

Quality factors in the Model

A large model does not necessarily mean that it is a good model. The quality of the model is also determined by the amount and quality of the data it is trained on. A large model trained on a small amount of data will underperform in comparison to a small model trained on a large amount of data. In an oversimplification, let’s assume a large model trained on a dataset with just one sentence: “I’m a large model.” Such a model will perform poorly for most use cases. Hence, the dataset it is trained on plays a significant role in the performance of the model.

Another factor affecting the quality of the output is the diversity of the data it is trained on. For example, a particular section of society might have two different norms and opinions on the same thing. The same holds true for political data. Now, if the model is not trained on a diverse dataset, it would lead to biased output, tone, as well as sentiments. A diverse dataset also helps the model not just specialize in one specific task, but in multiple tasks. For example, if the model is only trained on formal text, it would face difficulty in generating output for, say, poems.

The role of varied datasets in model training leads to both improvement in the model as well as the scarcity of more data to train the model on. Recently, the open availability of data to train these large models is slowly becoming scarce. This is also leading to newer companies taking up the special task of creating more human-intervened datasets.

The Trade-off

While larger models offer enhanced performance, there's a significant trade-off between model size and practical considerations. The cost of incrementally improving the model's output quality is very high. These models use so much data that incremental improvement requires a lot more resources than are currently available. The increased scarcity of new data is just one limiting factor. The amount of resources, both in terms of money and electricity requirements, is also significantly affecting the overall ability to scale model improvement.

I find the concept of Prospect Theory - how people make decisions under uncertainty and risk, often deviating from rational economic behavior - to be the most impactful in understanding how people make decisions under uncertainty and risk. For business and product applications, it has wide-ranging implications. Understanding how customers choose to use a product, what price points they find risky, or what they perceive as significant rewards is vital for business success. Traditional economic theories assume people always make rational choices to maximize benefit, but Prospect Theory suggests that people often think about possible gains and losses in ways that aren’t entirely logical. One important idea in Prospect Theory is the "reference point." This is the situation a person compares everything else to when deciding.

At TravClan, we designed a scratch card system that travel agents received after each booking. This was the initial pitch the onboarding sales team used to get them to transact for the first time. While it might seem obsolete, a small travel agent who completes fewer bookings may see this as an opportunity to earn more because their reference point is their own earnings. A scratch card that offers Rs. 50 or Rs. 100 adds value to their current business. However, the same scratch card value might not be lucrative for high-earning agents. Consider the following agent profiles:

Agent A: Completes 10 daily bookings, earning Rs. 100 on each booking Agent B: Completes 100 daily bookings, earning Rs. 100 on each booking Agent C: Completes 1000 daily bookings, earning Rs. 100 on each booking

What value of scratch cards will each of them find appealing?

For Agent A, who completes 10 daily bookings, the Rs. 50 or Rs. 100 from the scratch card is a significant addition to their earnings, making it an attractive incentive. This aligns with Prospect Theory, where individuals assess potential gains or losses relative to a personal reference point—in this case, their regular earnings. Agent B, with 100 daily bookings, might see the scratch card as a smaller but still notable benefit. However, for Agent C, who handles 1000 daily bookings, the same reward may seem negligible and might not motivate them as much. Their reference point is much higher, so the perceived value of the reward diminishes.

If I were to design a scratch card system based on Prospect Theory, it could not be the same for all users. The system's rules would need to consider the agent's business size to maintain high utility for each customer segment. Moreover, as agents' businesses grow and they move from Category A to Category B and C, they should experience increased marginal utility over time.

Many companies, such as Paytm, don’t reward customers with monetary benefits but prefer to offer gift vouchers to keep users engaged. The definition of value changes based on the type of voucher a user receives. One user might prefer a shopping voucher, while another might prefer movie tickets. If I prefer movies over shopping, I won’t see value in Paytm’s reward system if I consistently receive shopping vouchers. As a result, it’s highly likely I would shift to competitors over time.

How do companies solve this particular problem? Rewarding customers based on their preferences is one way, but collecting relevant data might be challenging. However, once done successfully, this becomes a powerful tool for customer retention.

Let’s look at another example—memberships. When you pay for a membership, you’re essentially betting on how much you’ll use it. This is where uncertainty comes in. Let’s explore three different types of memberships: Zomato Pro, holiday memberships, and gym memberships.

Food Memberships (like Zomato and Swiggy): Companies often offer a three-month membership at a discounted price of Rs. 149. If you order food online regularly, this membership might seem like a great deal. You could save money on delivery fees and get discounts on meals. But what if you’re not sure how often you’ll order food in the next three months? If you only end up ordering a few times, the Rs. 149 might feel like a waste. The value of the membership depends on your confidence in how much you’ll use it. Users have reference points, such as the number of times they recently ordered or specific events like an upcoming birthday, which increases the likelihood of usage. Companies use past data, such as ordering behavior and user information like dates of birth, to time and price memberships effectively, making them look attractive to customers. Some users might get the membership at Rs. 99, while others pay Rs. 149, often receiving prompts around their birthdays.

Holiday Memberships: Vacation memberships can be quite expensive. However, if you’re someone who plans holidays every year, this membership might be a good investment. You’ll have access to great vacation spots and probably save money over time. But if you’re unsure how often you’ll take vacations, the high cost of the membership could seem too risky. The uncertainty of whether you’ll use it enough makes the decision harder. For instance, if you’re already on vacation and a company offers you a membership, you’re more likely to buy it because your current vacation creates a reference point, making future vacations feel more probable.

Gym Memberships: Gym memberships are another example where usage plays a big role. If you’re committed to working out regularly, a gym membership offers great value. You get access to equipment, classes, and a space dedicated to fitness. But if you’re unsure how often you’ll go to the gym, the membership might not seem like such a good deal. Paying for something you might not use often feels risky, leading you to possibly opt out because of that uncertainty.

As a product manager designing products or strategies like scratch cards or memberships, it’s important to think about your customers' habits and expectations. For a travel agent with a small business, a Rs. 50 scratch cards can be valuable. For someone who orders food often, the Zomato membership could save a lot of money. But if customers are unsure about how often they’ll use a service, the offers might not be worth it.

Prospect Theory helps us understand these choices. People don’t just look at the possible outcomes; they think about how those outcomes compare to their current situation. A small reward might feel significant to someone with limited means, while a large cost might feel too risky to someone unsure about their usage. By considering these factors, businesses can design better products and offers that meet their customers’ needs and expectations.

Understanding these concepts can help product managers and businesses create more effective strategies that appeal to different types of customers. Whether it’s offering a small reward to encourage more bookings or creating a membership plan that feels like a great deal, knowing how people make decisions under uncertainty is key to success.

GPT stands for Generative Pre-trained Transformer. The term transformer is associated with ChatGPT, Claude, LLaMA because all of these model use transformer architecture. So what is it?

Suppose you're reading a paragraph:

“The ball couldn’t go through the door because it was too wide.”

You instantly know “it” refers to the ball and not the suitcase because your brain considers the context of every word. Human brain don’t just process words one after another. You look at the whole sentence together, break it and sometimes even refer the previous sentence to understand the meaning.

This way of looking at information is what Transformer architecture brings to AI. This is called attention mechanism and it is what gives models like ChatGPT, Claude, and LLaMA their power. It’s how they decide which parts of the input to focus on when generating an output.

In traditional models, the system reads inputs one word at a time with no ability to look back. But attention allows the model to weigh the importance of each word relative to every other word.

Let’s understand with an example:

“The hotel didn’t refund because the customer cancelled late.”

When the model processes the word “refund,” it must understand:

• Who did the refund? → hotel

• Why did it happen? → because

• Who was involved? → customer

• What action triggered it? → cancelled

In simple terms, the attention mechanism assigns a score to each word in the input to decide which ones matter most when predicting the next word.

As a product manager not managing a LLM product it might not be necessary to understand the math behind it but the concept overview.

When the model is at the word “refund”, it:

1. Looks at every word in the sentence — from “The” to “late”

2. Assigns a score to each one based on how relevant it is to “refund”

3. Focuses more on words with higher scores

   

The model uses these scores to pay closer attention to the important words like hotel, customer, cancelled, and late with higher score while mostly ignoring less relevant ones like The or amount. This helps it make sense of what refund really means in this context: who was involved, what happened, and why the refund was denied. The model can then predict what should come next like “due to policy” or “could not be processed.”

The Transformer architecture didn’t just advance AI, it redefined what machines could do with language. And if you're working anywhere near AI, it’s worth learning how it works. You don’t have to know the math. But you do need to know what’s happening under the hood because that’s where product decisions begin.

Reference:

The Transformer architecture was introduced in the landmark 2017 paper “Attention Is All You Need” by Vaswani et al., which laid the foundation for modern large language models.

You’ve probably seen these phrases thrown around in AI conversations:

“7 billion parameters.”
“GPT-3 has 175 billion parameters.”

But what does it actually mean?

Imagine you’re teaching a child to recognize animals. In the beginning, they might think every four-legged creature is a dog. But as they see more examples and get corrected they start noticing subtle differences. Maybe it’s the ears, the tail, the way it walks. Their brain adjusts. That’s exactly what parameters do for AI models.

Think of parameters as tiny knobs inside an AI model. Each knob controls how much weight the model gives to a certain pattern it has learned. During training, the AI tweaks these knobs - billions of them - to become better at whatever task it’s learning: predicting the next word, recognising an image, or even recommending a product.

More parameters mean the model has more “mental knobs” to fine-tune its understanding. That’s why bigger models can handle more complex tasks; they simply have more capacity to learn. But with size comes trade-offs, bigger models are harder to train, need more GPUs, and are expensive to run. So, bigger isn't always better it depends on the problem you're solving.

Parameters are where the learning lives in an AI model. More knobs, more learning power.

Ever tried solving a problem… only to realize later it wasn’t even the real problem?That’s what happens when you build a product without truly listening to your customer. Hi, I’m Nitin, and you’re listening to* Product Decisions*. This podcast is where we unpack the human side of building great products — blending system thinking, psychology, and product experimentation. Today, we’re starting exactly where I believe all product thinking should begin: with the customer.

Segment 1: The Myth of Data

Here’s a hard truth: Just because your dashboards are filled with analytics, doesn’t mean you understand your user.Data shows you what’s happening. Customers tell you why. I’ve built products based on clean funnels, promising trends, and beautifully segmented reports… But the biggest breakthroughs? They came from messy, unfiltered, real conversations with customers. Here’s the thing: most teams, including experienced ones, don’t really do this. It’s not just about what customers say. It’s about what they struggle to express.Even when they don’t know what they want, talking to them helps you see what they don’t know—and that’s gold.

Segment 2: My First Customer Story

Back in college, we had six subjects a year, each with chapters scattered across different books. It was a mess. Every exam season, students would crowd photocopy shops to get a few pages printed for last-minute study. It was chaotic. After struggling with this ourselves in the first year, we decided to fix it. We created complete, syllabus-aligned notes for all subjects — just for ourselves. But then came a lightbulb moment during a chat with my friend "Why not sell this to others?" So we struck a deal with a local shopkeeper to print in bulk way lower than the market rate. We sold the notes at market price and made a decent margin. But then... people started copying our copies. Friends who bought from us would lend them to others, who got their own photocopies made. So we evolved: doorstep delivery. No queue, no wait, just ready-to-use notes.Convenience won. Looking back, we didn’t have a business strategy. We had a simple goal: solve a real problem that we ourselves had. We were our own first customers.

Segment 3: From Photocopies to Product Management

Fast-forward to RCorp.We were building tech for clients we never met. There was no clear priortisation, no roadmap. The real reason? No feedback loop. No real users. No customer voice. The difference became clear — when you have a customer in the loop, they push you to ship. They bring urgency. They give direction. And more than anything, they make the product real.

Segment 4: The ArcMath Leap

In 2017, I co-founded ArcMath, a math education startup. We assembled a team, and everyone became a teacher. Seriously — every single team member conducted workshops and interacted with students and parents firsthand. That was our superpower. We didn’t just build for students and parents — we taught them, listened to them, learned from them. And because of that, our product kept evolving with their needs. That direct connection was priceless.

Segment 5: The TravClan Wake-up Call

Even with a decade of experience, when I joined TravClan in 2021, I struggled. I couldn’t understand the travel business the way I understood education. For six months, I second-guessed myself. But then I asked myself: What am I not doing that I used to do? The answer was simple — I wasn’t talking to customers. I wasn’t even sitting with the teams who did. Once I changed that — started listening to support calls, shadowing sales, talking to travel agents — everything shifted. We started solving real pain points. Our features became faster to ship. The feedback was sharper. The success was measurable. It wasn’t magic. It was just... listening.

Summary

So if you remember just one thing from today’s episode, let it be this: The best product decisions don’t come from strategy decks. They come from conversations. We will cover more on building great product. Until then, go talk to someone who uses your product. And don’t just ask what they want—listen to what confuses them. That’s where the magic is. Thanks for tuning in to Product Decision. I’m Nitin, and I’ll see you next time.

It gives vague answers. It doesn’t understand regional nuances. And worse - it starts hallucinating policies from other countries. The model does not work the way you want it to work because the foundation of these models were not built on the context you are using it for. And unless you understand what they were trained on, how they work, and what they weren’t built for, they’ll fail your product in subtle but costly ways.

What are Foundation Models?

Foundation models are massive, pre-trained AI systems - like GPT-4, Gemini, or Claude that learn from web-scale data and are capable of performing hundreds of tasks without needing custom training for each one. They’re called foundation because they’re supposed to be the base layer for everything else.

The strength and weakness of these models are that they only as strong as the data they were trained on. If the data is skewed, shallow, or missing key domains or languages, the model becomes like a smart student who aced English and flunked Geography because nobody taught her that part.

Common crawl and the english-centric web

Most of these models are trained on a dataset called Common Crawl - a giant scrape of the internet. In 2023, over 45% of that dataset was in English. The next language? Russian, at around 6%.
Here’s where it gets problematic:

• Punjabi speakers make up over 1% of the world’s population.

• But Punjabi contributes just 0.0061% of Common Crawl.

• That’s a 230x under-representation.

GPT-4 is brilliant in English. Mediocre in Hindi. And nearly blind in Marathi or Telugu.

Your AI product might promise inclusivity or automation—but under the hood, it performs well only if you speak English.

So when the model hasn’t been trained on the context or language of your product it perform abruptly. Say you’re building:

• A travel app that generates custom itineraries

• A transport tool to track real-time freight routes in India

• A healthcare assistant that explains symptoms in Kannada

Even if you prompt GPT perfectly, it won’t work reliably if the language isn’t well-represented in training, the domain (e.g., Indian railways, local climate, dengue symptoms) wasn’t in the training set or the format (travel itineraries, prescriptions, bus routes) is unfamiliar to the model. All because the model was never taught your problem.
Foundation models don’t understand, they recognize patterns. If the patterns didn’t exist in the data, the model is guessing. And that guesswork is where bad UX, hallucinations, and trust issues begin.

Using domain specific models for your product

If your product lives in a niche domain or serves a local audience, you probably don’t need a general-purpose model. You need a domain-specific model. This could mean:

• Fine-tuning GPT on your data (quotes, chats, itineraries, etc.)

• Training a small model from scratch on a focused dataset

• Using tools like LoRA, RAG, or adapters to specialize responses

Some examples already exist:

• Med-PaLM (Google) for medical queries

• AlphaFold (DeepMind) for protein folding

• BioNeMo (NVIDIA) for drug discovery

• CroissantLLM for French speakers

• PhoGPT for Vietnamese

• And dozens of LLMs trained specifically on Arabic, Chinese, and even Hindi

Companies are building these models because it’s the only way to get performance that aligns with user expectations in high-risk, localized, or domain-specific scenarios. This also offers a great opportunity to create models for hyper-niche market that makes business sense. Another opportunity is creating datasets for specific domain and language which is very high in demand due to their application in model training.

The issue with translation

All the above discussion naturally tempts one to think translation is the way to go about the language problem - translate user inputs to English, run GPT, then translate back. That’s like taking a call in regional language repeating it in English to an intern, and relaying their answer back in the same regional language.
It works but many languages like French, Spanish operate differently. You lose nuance, especially relationship markers, gendered terms, and cultural context
You increase latency and cost, since token usage balloons in translation. You can’t control the quality, especially for safety-critical applications
Instead of bandaging onto GPT, consider owning the model’s foundation for your product’s core domain.

Data quality over data quantity

More data does not necessarily mean that the model will perform well. A clean, domain-specific data beats large, noisy data every time.
If you’re collecting data from your product, chats, tickets, user flows, forms, etc you might already have what you need to start training or fine-tuning a better model for your use case.

What should you do as a Product Manager

Ask yourself these 4 questions:

1. What language do my users speak?
   Is that language underrepresented in Common Crawl? If yes, general-purpose models may underperform.

2. What domain does my product live in?
   Is it something the average web user talks about (like movies or news)? Or is it niche (like local buses, Ayurveda, or regional taxes)?

3. Can I collect good-quality data from my product today?
   Even 5,000 good examples of itineraries, prescriptions, or complaints can be valuable.

4. Is translation good enough? Or do I need native understanding?
   If mistakes could cost money, trust, or lives, translation isn’t enough.

If you answer these honestly, you’ll start seeing where foundational models help and where you need to build your own.

Summary

• Foundation models are powerful but trained mostly on English web data.

• Most Indian languages and niche domains are underrepresented.

• Translation is not a true fix. It’s a stopgap with risks.

• Domain-specific models are increasingly essential for real-world products.

• Clean, focused data beats more data.

• If your product needs accuracy, trust, and cultural nuance - you may need to specialise your model.

When we talk about AI, we often talk about the awesome things like ChatGPT writes an email in seconds, Spotify recommends the perfect weekend playlist or a travel agent types in "family trip to Thailand" and within seconds, a ready-made itinerary appears.

But if you’ve ever tried to build an AI-powered feature, you quickly realize that the AI model is just one part of the system. Behind the seemingly simple layer sits a complex stack of tools and infrastructure that most users and even many PMs never see.

If you're a product manager and you’ve ever felt overwhelmed by terms like LLMs, vector databases, or prompt engineering you’re not alone. You don’t need to be an ML engineer to build smart features. But you do need to understand the AI tech stack enough to ask the right questions, work with your AI/ML team, and spot real opportunities for incorporating AI in your product.

What is an AI Tech Stack?

Think of building an AI feature like building a smart home.

At the base, you’ve got the foundation - electricity, plumbing, the walls. Then come the tools - smart devices, switches, appliances. And finally, there’s the experience -the cozy, voice-controlled lights and music your guests see.

In AI, the foundation is the infrastructure, the appliances are the models and tools, and the experience is your product - the thing users interact with. That’s the layer you own as a PM.

The stack usually has these layers:

1. Infrastructure Layer

This is the engine room. It’s where all the AI computations happen. If you’ve heard someone mention about GPU costs are going crazy, they’re talking about this layer.

This layer consists of the following:

• Cloud providers like AWS, Google Cloud, Azure

• Powerful chips called GPUs (used to train and run models)

• Systems to store and manage data

As a product manager, you should be aware that training or even running AI models (especially large ones) requires large compute power which comes with cost, latency, and reliability trade-offs.

Not every project you are working on needs expertise to configure GPU or run things on cloud. But you should question whether to use a pre-trained model instead of training our own? Can your users compromise on speed and real-time responses or can it be slower and cheaper for the use case you are working on.

2. Model Layer

This is where the smart part happens and works like a brain. Models are algorithms that have learned from tons of data. You can either build one yourself which can be rare and expensive or use existing ones. We will go in depth in another post on how models learn.

Some of the more popular models that are in common use today are:

• GPT-4, Claude, Gemini - text generation models

• Whisper - speech to text

• DALL·E, Midjourney - image generation

It is important for product manager to choose the right model basis the kind of intelligence the product needs. Classifying customer complaints, generating travel itineraries or recommending top five packages are all different use cases and requires selecting the right model. Once you define the job, your ML team can pick or fine-tune the right model.

3. Framework & Tooling Layer

Imagine you have a Product catalog and want to build a feature that replies to customer reviews using AI. You’ve got a model like GPT-4. Now you need tools to build workflows, chain prompts, and track how well the AI performs. That’s this layer.

Some examples:

• LangChain, LlamaIndex - chain prompts, structure reasoning

• MLFlow - track experiments and model versions

• PromptLayer, TruEra - monitor prompt quality, evaluate results

This layer controls reliability and iteration. You should be able to ask what happens when the model gives the wrong answer? Do we have logs of past outputs? And how will we evaluate performance?

It’s where your product moves from "one cool demo" to something robust and usable.

4. Application Layer

It’s the UI, the flow, the end-user experience. All the smart AI stuff only matters if it’s useful and usable up here.

Here’s where you answer:

• How does the user trigger the AI feature?

• Do we show the reasoning or just the output?

• Can the user reject or correct the AI?

You can read more on how to design AI features here:

For example, if we’re building a tool for travel agents to create posters — the model might generate images, but we also have to think about:

• How the agent inputs destination, prices and preferences

• Whether they can edit the output

• How we measure success (downloads? shares? bookings?)

• How do we price the product so that we do not make losses.

The AI model isn’t the product. The product is the full experience around it.

5. The Data Layer

No model works without good data. Every AI model needs data - clean, relevant, and labeled.

When you are designing a feature you should ensure that you have reliable source of data which is available to train the model. Moreover, the data needs to be clean and useful. For example, if you want a model to recommend travel destinations based on past searches you need to collect, store, and protect that search data.

Where should you start?

If this feels like a lot, don’t worry. You’re not expected to master every layer. But you should know what each layer does, understand which layers your team is working on and speak the language of AI enough to collaborate well.

The goal isn’t to become an AI engineer. It’s to become the kind of product leader who understands what’s possible, knows how to ship it, and builds products users actually love. And once you do that, you’re not just “catching up.” You’re ahead of the curve.

When I first started working with large language models and starting out with AI use case, I kept seeing this one technical term: token. It was not an incidental finding but the one that impacted the wallet. Every experiment we ran from parsing the travel documents to transcribing the calls had associated cost. It was natural to understand how this cost is calculated and that is when I came across the term token.

Everywhere I looked — in OpenAI pricing docs, in LLM performance benchmarks, even in random Twitter threads — someone was talking about tokens.
“100k token context window!”
“Costs 3 cents per 1,000 tokens.”
“Token limits affect your prompt size!”

It felt like there was this secret vocabulary no one had properly explained.
So let’s fix that. If you’re just starting with AI/ML or working around these models as a product manager, researcher, or engineer — this post is your go-to explanation of what tokens are, why they matter, and how they affect everything from model performance to your monthly bill.

What Is a Token?

A token is the basic unit of text that a language model reads and writes.
But here’s the interesting part: it’s not always a full word.

Let’s say you input:
“I can’t wait to build AI applications.” You might assume that’s 7 words. But in the eyes of GPT-4, that breaks into 9 tokens

Depending on how the model is designed, even a single word like “unhappiness” might become three tokens:
["un", "happiness"] or ["un", "happi", "ness"].

Why break it like this? Because it helps models deal with everything from common words to totally new or made-up ones. Even if you invent a term like “WhatsApping”, the model can break it into known parts — “WhatsApp”, and “ing” — and still make sense of it.

This process is called tokenization — and it’s the first step every time you use an LLM.

It is natural to think why not just use words or letters. There are three key reasons why modern AI models use tokens instead of full words or single characters:

How Do Models Use Tokens?

Let’s take an example.

When you type a sentence into ChatGPT, the model doesn’t see your sentence as one chunk of text. It sees a sequence of tokens.

Input sentence: “I can’t wait to build awesome AI applications.”
Tokens: [“I”, “can’t”, “wait”, “to”, “build”, “awesome”, “AI”, “applications”, “.”]

Now, the model uses these tokens to figure out what should come next. It doesn’t work at the sentence level — it works token by token.

Based on the context of just "I can", a language model will look at all the data it's been trained on and predict the most likely next token — essentially, how people usually complete that sentence.

Examples of possible next tokens:

• " do" → "I can do"

• " help" → "I can help"

• " see" → "I can see"

• "not" → "I cannot" (depending on spacing)

• " run" → "I can run"

• “‘t” → “I can’t”

Each of these next tokens has a probability score. The model picks the one with the highest likelihood — or samples from the top options if randomness is added (using a parameter like temperature). Hence, generating a text takes lot of computing.

This process is called Tokenization

Tokenization is just the act of breaking a sentence into tokens.

Every large language model — GPT, BERT, Claude, Gemini — uses some form of tokenization as the first step. Without it, they can’t understand or generate text.

Each model has its own rules about how it splits up text. For example GPT-4 uses a method that typically gives you about 75 words per 100 tokens. Whereas some models break common phrases into a single token. Others break rare or made-up words into multiple parts.

GPT-4 has a 32,000 token limit which mean that’s the maximum amount of text the model can handle in one go. It includes both your input and the model’s output. So if you feed in a 20,000-token-long document, the model might only have room for 12,000 tokens in its reply. That’s why longer prompts can cut short responses.

One Word is not equal to one Token

Many of us assume one word equals one token which is not true in all cases.

On average:

• 1 token ≈ ¾ of a word (for GPT 4)

• So, 100 tokens ≈ 75 words

This is important to understand because when you are using the API, token usage impacts speed and cost. When training models, token count defines how much data the model sees. Every model has a fixed vocabulary of tokens it can understand. That vocabulary could be 32,000 tokens (like GPT-4) or 100,256 tokens (like Mistral 8x7B)

The model can use these tokens to build any sentence, much like how we use the alphabet to create words. A larger vocabulary lets the model be more expressive, but also more complex and costly to train.

Before a model can think, write, or answer — it has to tokenize. Understanding tokens will help you:

• Use APIs better

• Interpret model outputs

• Optimize cost and performance

• And eventually, train your own models

So next time you type into ChatGPT, pause for a moment and think - what goal do you want to achieve and how will token impact your overall product design and costs.

Have you ever said something like, “Let’s use AI to make the product better”?
It sounds good. But for your machine learning team, it’s like asking a mom to “cook something tasty” without telling her what ingredients you have or what you like to eat.

As a product manager, your job is to turn vague chaotic ideas into clear instructions. We will discuss how to turn a simple product idea into a machine learning (ML) problem including the basic things your team needs to build the right solution: inputs, outputs, labels, and success metrics.

Let’s start with a simple truth:
Not every product idea needs machine learning.

Step 1: Make the Product Goal Clear

The first step is to take your idea and make it more specific. A clear goal helps everyone understand what the user is trying to do and what the product should deliver.

At the start of the 2024, I was thinking about ideas where we can apply machine learning and AI. We wanted to simplify airlines fare rules. Usually the airlines fare rules are long clunky text which only trained support team can understand. We just wanted to simplify the same.

That sounds nice, but it’s too general. A clearer version could be:
Extract key informations (data points) like Refundable, Cancellation Charges and Reschedule Fees.

Now the goal is tied to a clear product outcome. This gives your ML team something to work with.

Step 2: Match the Goal to a Machine Learning Type

Different problems use different kinds of machine learning. Here are some of the common types:

If your product idea doesn’t match one of these types, you probably don’t need ML. You might be able to solve it with simple logic or filters.

Step 3: Don’t Use ML When You Don’t Need To

Sometimes it’s better to use simple rules instead of machine learning. Rules are easier to build and more reliable when the logic is clear.

Here are some examples:

If you can write a rule to solve the problem, use it. Don’t overcomplicate things. Machine learning is powerful, but it should be used only when needed.

Let’s try a couple of examples.

Example 1:
"Highlight hot destinations for each user."
This is a recommendation problem. The model should show different destinations based on user behavior and trends.

Example 2:
"Auto-reply to support chats."
This is a mix of classification (to understand the user’s intent) and generation (to write the reply).

Once you learn to spot these patterns, you’ll be able to guide your ML team much better.

Define the Key Building Blocks

Once you’ve identified that you do need ML, the next step is to define four key things: Inputs, Outputs, Labels and Metrics

Inputs and Outputs

Inputs are the data the model uses to make a decision. Outputs are the result it gives you.

Ask yourself: What do we already know about the user? And what are we asking the model to predict?

Labels (The Correct Answers)

Labels are important. They tell the model what the right answer is. The model learns by looking at examples where the label is known.

If you’re recommending destinations, your label might be the place a user clicked or booked. That shows what they liked.

If you don’t have labels, you can’t train a model. You’ll need to either collect them or find a way to use other data.

Also, make sure your labels are clean. If the data is messy or wrong, your model won’t work well.

Success Metrics

It’s easy to think a model is good because it has a high score. But that score might not mean anything for your business. You need to track both model performance and business impact.

Here’s a simple comparison:

Let’s say your model shows the top five destinations.

• The ML metric could be: How often is the correct one in the top five?

• The product metric could be: How many users clicked on a destination?

Both are important. A model that scores high but doesn’t help users is not useful.

Example: Let’s say your feature is to recommend sightseeing tours. Here’s how you could define everything your ML team needs:

With this setup, your data team can get started quickly. They don’t have to guess. And you stay in control of the product vision.

Have you ever used an AI product and thought, “Why is it doing this?”
Maybe it gave a strange result. Or it felt way too confident in something totally wrong. Or worse—it gave you no option to say, “No thanks.”

When that happens, the problem usually isn’t the AI. It’s the user experience. And guess what? As the PM, you own that UX.

In this post, I’ll walk you through why designing for AI is different, how to make AI features feel more trustworthy, and the simple design rules that turn “weird” AI into wonderful AI.

AI is built on probabilities. It tries to guess the right answer based on past data.
But it doesn’t always get it right. It doesn’t know things the way rule-based systems do.

This means your AI feature might give:

• A great answer

• A good-enough guess

• A totally off recommendation

And when it does, your design has to guide the user through that experience—without confusing or frustrating them.

Why AI features break UX expectations

Let’s compare:

That unpredictability is what makes AI both magical and risky. If your UX doesn’t handle it well, users lose trust.

Five Principles of Great AI UX

To design AI features that feel smart and human-centered, we need new principles.

1. Show the “Why”

People don’t trust what they don’t understand.

If your AI makes a decision, even a short explanation dramatically improves the user’s willingness to engage.

Bad UX:

“Here’s a trip you’ll love.”
Why? Who said I wanted this?

Good UX:

“We recommended this trip because you liked Maldives and this is trending for July.”

One line. Huge difference. Spotify nails this:“Because you listened to Coldplay…” This one sentence builds context, personal relevance, and trust. Google Photos does it too: “Rediscover this day from 2018.” Gmail’s Smart Reply goes further: It offers multiple, plausible suggestions—each short, safe, and easy to undo. Use simple explanation layers in your UI. If your AI model uses rules, weights, or past behaviors - summarize them in human language. Transparency is UX gold.

2. Always Offer a Way Out

Even if your model is highly accurate, no AI is right all the time. And when it’s wrong, don’t trap the user in a dead end.

Bad UX:

“These are your 5 options.”
No way to search. No way to skip. No way to disagree.

Good UX:

“Here are some suggestions—or browse all options.”

Always let users override the system. They will thank you—even if they never use it. Netflix uses soft nudges like “Because you watched…” but never blocks your freedom to explore. Google Maps suggests routes but gives you alternatives. Give people a graceful exit from the AI-driven path. A toggle, a “more like this” button, or a manual input option goes a long way.

3. Be Honest About Confidence

Nothing takes away trust faster than fake certainty. If your model is unsure—say so.

Bad UX:

“You’ll love this.”
Only to get it totally wrong.

Good UX:

“You might enjoy this.”
“Here’s a popular choice among similar travelers.”

Even visual cues help:

• Add badges like “Beta” or “Experimental”

• Group results as “Top Picks” vs. “Other Suggestions”

Build a “confidence design system.” Decide how to handle high, medium, and low-confidence predictions—and make that part of your UX library.

4. Learn from the User

Most AI models get better with feedback. But you only get that feedback if your UX invites it.

Simple ways to gather real-time learning signals:

• 👍 / 👎 buttons

• “Was this helpful?”

• “Why was this wrong?”

• Undo or rollback actions

• Rephrase or regenerate options

Gmail’s “Undo Send” is a trust multiplier—even though users rarely click it. YouTube lets you click “Not interested” or “Don’t recommend this channel.” So does instagram reels. Amazon asks “Was this answer helpful?” on product Q&As. Make it safe and easy for users to correct the AI. If possible, connect feedback loops to your ML retraining cycles—or at least log them for triage.

5. Make it feel human

Great AI UX doesn’t mean making it sound like a person. It means making it feel respectful.The best experiences aren’t flashy. They’re thoughtful.

Let’s recap what that feels like:

• Suggest, don’t force
  → Let users choose their own path

• Explain, don’t assume
  → Give rationale and room for disagreement

• Listen, don’t ignore
  → Capture feedback, iterate

• Admit when unsure
  → Show confidence, not arrogance

Duolingo uses its mascot (Duo the owl) to make mistakes feel fun, not frustrating. Notion AI offers multiple ways to edit, rephrase, or start over—respecting user agency.

Let’s recap the products that do this well:

These are small details, but they add up to trust.

What bad AI UX looks like

Here's what to avoid:

• No reason for the suggestion: “Why am I seeing this?”

• Too confident: “This is what you want”

• No way to explore manually: “I can’t find what I actually want”

• No feedback: “It feels like the AI doesn’t care what I think”

Even a basic feature, if designed with care, can beat a smart model with bad UX. If you're building something with AI, run it through this quick checklist:

• Does it explain itself?

• Can users override or explore more?

• Does it show uncertainty when unsure?

• Can it learn from feedback?

If yes to all 4—you’re on your way to a great AI experience.

Not all AI Product Managers do the same job.

Some product managers use AI to move faster in their own work. Some add smart features to old products. Others build entire apps where AI is the product. And a few work behind the scenes — setting up the infrastructure that powers all of this.

That’s why “AI PM” can mean very different things depending on the job.

In this post, we’ll break down the four types of AI Product Managers — with clear examples and real tools across product phases like research, prototyping, analytics, and more.

1. AI-Powered PM – Uses AI to work smarter

These PMs don’t build AI products. Instead, they use AI tools to speed up their own work — from research and prototyping to writing specs and tracking metrics.

Tools They Might Use

• Research & insights: Claude, Perplexity, ChatGPT

• Prototyping: Uizard, Lovable, Figma + GPT

• Analytics: GPT + Google Sheets, PostHog, custom GPT scripts

Example

A PM working on a food delivery app downloads 500 Play Store reviews. She pastes them into Claude to find recurring pain points: “late delivery,” “missing items,” “no order status.”

Now she knows what to prioritize — and it only took 15 minutes.

She opens Uizard to mock up a new live tracking feature, with GPT suggesting the screen copy. Later, she analyzes how ticket volume changes weekly by using GPT on top of exported support CSVs.

Exercise for You

Pick a product you use often — maybe Spotify, Swiggy, or a learning app. Then:

1. Collect user reviews from Reddit or the app store

2. Use Claude or GPT to summarize the top 3 pain points

3. Sketch a feature idea in Figma

4. Prototype it using Uizard or Lovable

5. Write a short spec using GPT

Time yourself: try finishing it all in under 2 hours.

2. AI Feature PM – Adds AI Into existing Products

These PMs work on regular apps — marketplaces, HR tools, CRMs. But they improve them by adding AI features that personalize, automate, or extract insights.

They don’t own the whole product. But they’re responsible for how well the AI-powered feature works and how users adopt it.

Tools They Might Use

• Models: GPT-4, Whisper, Claude, open-source LLMs

• Evaluation: Mixpanel, PostHog, A/B testing tools

• APIs & infrastructure: OpenAI, Pinecone, Hugging Face

Example

A PM at a CRM company adds a feature that auto-generates call notes after sales calls.

They use Whisper to transcribe audio and GPT-4 to summarize it.

To evaluate the feature, they:

• Compare AI vs human-written notes for 200 calls

• Track usage metrics in Mixpanel: Who’s enabling the feature? Are they editing the notes?

Another example: A PM at an HR tech startup ships an “AI resume matcher” that surfaces the top 5 applicants based on job requirements. It’s powered by embedding similarity + classification models.

Exercise for You

Say you’re working on a learning app, and users struggle to revise past lessons.

Try this:

1. Use GPT to brainstorm how AI can help (summarize lessons? auto-generate quizzes?)

2. Write user stories like: “As a student, I want a quiz after every module…”

3. Define success metrics: quiz completion rates, retention, feedback on usefulness

This isn’t about building a full product — it’s about using AI to augment what already exists.

3. AI-First PM – Builds products where AI is the Product

This is the deep end.

AI-first PMs build apps that don’t work without AI. Think grammar correctors, deck generators, or chat-based travel advisors. The core value prop is AI.

You’re not just launching features — you’re shipping predictions, improving models, and balancing UX with uncertainty.

What Makes This Different

• You define problems in terms of data and outcomes

• You work with models, not just wireframes

• You ship in iterations, not “launch and forget”

• UX is important, but model performance = product quality

Example: TripGuard

TripGuard helps travel agents predict trip problems — driver delays, hotel no-shows, missing documents — before they happen.

As PM, you don’t just manage alerts. You:

• Identify which prediction to make (e.g., “Will the driver be late?”)

• Figure out what data helps the model (trip type, driver history, weather)

• Define the label: past trip issues

• Design how to present risk scores to users

• Create a feedback loop: agents confirm/correct alerts to retrain the model

This isn’t a travel app with AI added. It’s an AI app that solves travel problems.

Exercise for You

You're PM for a new travel app. It should flag trips that are likely to go wrong.

Questions to answer:

1. What prediction do you want the model to make?

2. What data would help?

3. What’s the label the model should learn from?

4. How will you show predictions in the UI?

5. How will users give feedback to improve it?

This is how AI-first PMs think — in loops of data, predictions, and behavior change.

4. Core AI PM – Manages infrastructure behind the scenes

Core AI PMs don’t work on the front-end. They handle the plumbing of AI.

Their users are engineers, data scientists, and other PMs. Their products are tools, pipelines, dashboards, and policies that make AI development reliable, scalable, and compliant.

You’ll find them in companies where every team uses AI — and someone needs to keep it all running safely.

What They Manage

• Data labeling systems

• Model testing dashboards (bias, hallucination, cost, latency)

• Deployment pipelines

• Infrastructure scaling (GPUs, cloud workloads)

• Privacy and compliance workflows

Example

At a large e-commerce company, every vertical uses AI — fraud, search, personalization.

A Core AI PM builds:

• A model performance dashboard that tracks accuracy, latency, and fairness across 8 teams

• A central data labeling policy

• A rollout protocol to test new models in shadow mode before full deployment

They don’t touch the customer UI. But every customer-facing AI experience depends on their work.

Wrapping It Up

Not every AI PM does the same job — and that’s the point.

• AI-Powered PM: Uses AI to boost their own workflow

• AI Feature PM: Adds smart features to existing products

• AI-First PM: Builds apps where AI is the core engine

• Core AI PM: Builds the systems behind every AI feature

No path is better than the other. Choose based on:

• What stage your company is in

• Your personal interests (UX vs systems vs models)

• How close you want to be to end-users vs data infrastructure

As AI becomes more widespread, these roles will only grow in importance. The key is not to chase the trend, but to find where you fit best — and go deep.

Read the Book

Hey folks,

I'm incredibly excited to announce the launch of my new book, Making Informed Decisions in Product Management! This is a project I’ve poured my heart into over the past year, and I’m thrilled to finally share it with you.

At its core, this book is about one of the most critical yet overlooked responsibilities of a product manager—making decisions. Not the flashy, headline-grabbing ones, but the small, systemic decisions made every day that shape product direction, user experience, and ultimately, business success.

What’s the book about?

Rather than covering the basics of product management, this book dives deeper into how to make better decisions across the product lifecycle. It’s structured into four key parts:

1. Starting with the Customer and Data
We begin with the obvious but essential—understanding your customer, product metrics, and prioritization. Whether you're early in your PM journey or a few years in, this section helps ground your decisions in what truly matters.

2. System Thinking in Product
This part explores how interconnected systems influence outcomes. You’ll learn to recognize archetypes like “eroding goals” or “fixes that fail” and how they silently affect your product and team performance. If you’ve ever wondered why some decisions backfire despite good intentions, this section is for you.

3. Psychology of Product Management
Great decisions require understanding how people think. Drawing from Thinking, Fast and Slow and other behavioral theories, this section explores how biases, plausibility, representation, and risk perception affect user choices—and how PMs can design better experiences by accounting for them.

4. Product Experimentation
We wrap up with a hands-on approach to experimentation. From customer support tickets to surveys, MVPs, A/B testing, and prototypes—this section helps you validate what you’re building and why. It’s about going from gut feel to evidence-backed decisions.

Who is this for?

This book is for product managers who want to level up—not by learning more frameworks, but by learning how to think better. If you’ve ever felt overwhelmed with choices, pushed into decisions without clarity, or unsure how to prioritize, this book is your playbook.

A huge thanks to everyone who gave feedback, encouraged the idea, and guided me over the years. I’ve tried to make this book deeply practical and relatable—built on real examples, failures, and lessons.

Read the Book

I’d love for you to read it, share your thoughts, and most importantly—let it shape how you make decisions at work.

Let’s keep building better products, one thoughtful decision at a time.

Food delivery businesses are always looking for ways to deliver food faster and optimize operations. In the rush to improve efficiency, companies often introduce quick fixes to solve immediate problems. But here’s the catch—these short-term solutions can end up causing bigger issues down the road. This is what the systems thinking archetype called "Fixes That Fail" is all about.

Let's dive into how this pattern plays out in food delivery, what the data reveals, and what you as product managers can do to create long-term solutions.

What is "Fixes That Fail"?

"Fixes That Fail" happens when a quick solution seems to work at first but creates unintended side effects that make the original problem worse over time. In food delivery, this might look like increasing driver incentives for speed or favoring top-rated drivers with more orders. While it may boost short-term performance, it often leads to issues like driver burnout, declining customer satisfaction, and inefficiencies.

Here are some common quick fixes in food delivery:

Incentives for Faster Delivery:

1. Bonuses for drivers who meet tight deadlines.

2. The downside? Fatigue, unsafe driving, and declining service quality.

More Orders for Top-Rated Drivers:

3. Rewarding high-performing drivers with more deliveries.

4. The problem? Burnout for top drivers and fewer opportunities for others.

Ignoring Traffic and Weather Conditions:

5. Setting fixed delivery targets without considering real-world challenges.

6. Result? Delays, frustrated customers, and stressed drivers.

Dataset: We will analyse the following dataset from the lens of Fixes that fail. This is a sample dataset and to be used for the purpose of the case study.

Download Dataset

Here are some details about the dataset:

• ID: Unique identifier for the order.

• Delivery_person_ID: Identifier for the delivery person.

• Delivery_person_Age: Age of the delivery person.

• Delivery_person_Ratings: Ratings of the delivery person.

• Restaurant_latitude, Restaurant_longitude: Coordinates of the restaurant.

• Delivery_location_latitude, Delivery_location_longitude: Coordinates of the delivery location.

• Type_of_order: Type of food order (e.g., Snack, Drinks, Buffet).

• Type_of_vehicle: Type of vehicle used for delivery.

• temperature, humidity, precipitation: Weather conditions during delivery.

• weather_description: Description of weather (e.g., haze, mist).

• Traffic_Level: Traffic condition during delivery (e.g., Low, High, Very High).

• Distance (km): Distance between restaurant and delivery location (some missing values).

• TARGET: Delivery time in minutes.

Key Metrics to Analyze Fixes That Fail

To analyze whether these quick fixes have led to unintended consequences, we can investigate the following relationships in the dataset:

1. Delivery Time (TARGET) vs. Driver Ratings (Delivery_person_Ratings):
   Are drivers who consistently meet shorter delivery times experiencing lower ratings over time?

2. Delivery Distance (Distance (km)) vs. Ratings:
   Do longer distances coupled with quick delivery expectations lead to fatigue and lower ratings?

3. Traffic Levels (Traffic_Level) vs. Delivery Time:
   Are high-traffic conditions leading to unrealistic delivery times and declining driver ratings?

4. Weather Conditions (temperature, humidity, precipitation) vs. Delivery Time: Are drivers struggling to meet fast delivery expectations under unfavorable weather conditions?

Download Dataset

Data Analysis

Let's conduct some exploratory data analysis to identify patterns related to the "Fixes That Fail" archetype.

1. Impact of Faster Deliveries on Driver Ratings: We'll check if there's a negative correlation between delivery times and driver ratings.

2. Effect of Increasing Order Allocation on Performance: We'll analyze whether drivers with more deliveries tend to have a decline in ratings or increased delivery times.

3. Influence of External Factors (Traffic & Weather): We'll assess if ignoring these factors has resulted in worsening delivery times and satisfaction.

Findings from the Data Analysis

1. Negative Correlation Between Delivery Time and Driver Ratings:
   The correlation coefficient between delivery time (TARGET) and driver ratings is approximately -0.10, indicating a weak negative correlation. This suggests that faster deliveries may have a slight tendency to lower ratings, potentially due to rushed deliveries leading to poor service quality.

   

2. Impact of Traffic on Delivery Time:
   The boxplot shows that higher traffic levels correlate with longer delivery times, which is expected. However, if businesses pressure drivers to meet deadlines despite heavy traffic, it could lead to frustration and declining ratings.
   

   

   
   

3. Distance vs. Delivery Time (Colored by Ratings):
   The scatter plot reveals that longer distances generally lead to higher delivery times, but ratings tend to be lower for longer trips. This indicates that drivers may struggle to maintain service quality for long-distance orders, especially if they are pushed to deliver quickly.

   

What the Data Tells Us?

To see if these quick fixes are causing problems, we analyzed data on delivery times, driver ratings, distances, traffic levels, and weather conditions. Here’s what we found:

1. Faster Deliveries, Lower Ratings

The data showed a weak negative correlation between delivery time and driver ratings. In other words, when drivers rush to meet tight deadlines, service quality can drop, leading to lower ratings.

Takeaway: Speed isn't everything—quality matters just as much.

2. Traffic’s Impact on Delivery Times

Higher traffic levels were linked to longer delivery times. When companies ignore traffic conditions and push for unrealistic deadlines, it creates stress for drivers and frustration for customers.

Takeaway: Traffic conditions should always be factored into planning.

3. Longer Distances, Lower Ratings

Longer deliveries naturally take more time, but our analysis showed that customer ratings tend to drop for these orders. This indicates that drivers may struggle to maintain service quality over longer routes.

Takeaway: Managing expectations for long-distance orders is key.

What Product Managers Can Do?

If you're a product manager, avoiding the "Fixes That Fail" trap means thinking long-term. Here’s how you can do that:

1. Shift Focus from Speed to Quality

Instead of only rewarding fast deliveries, focus on overall service quality. Metrics like customer feedback, order accuracy, and driver safety should be part of the equation.

2. Use Data to Drive Decisions

Analyze past delivery data to set realistic expectations. Predictive analytics can help you factor in variables like weather and traffic to avoid over-promising.

3. Distribute Orders Fairly

Don’t overburden top-rated drivers. A fair distribution system helps maintain a balance, giving all drivers opportunities to improve.

4. Optimize Routes with Technology

Leverage route optimization tools that factor in real-time traffic data to improve efficiency without overburdening drivers.

5. Invest in Driver Training

Prepare drivers to handle different situations like traffic congestion and difficult weather conditions, ensuring they deliver high-quality service consistently.

How to Build Long-Term Success?

Want to avoid falling into the "Fixes That Fail" trap? Here are some practical steps:

1. Set Realistic Delivery Expectations: Base goals on actual data rather than arbitrary targets.

2. Keep Customers Informed: Transparency about delays due to traffic or weather helps manage expectations.

3. Monitor Trends Regularly: Stay on top of performance data to spot and fix recurring issues early.

4. Adopt AI and Automation: Use AI to predict demand, optimize routes, and streamline operations.

Quick fixes might give you a short-term boost, but they rarely lead to lasting success. The key is to focus on sustainable solutions that balance efficiency, service quality, and driver well-being.

Key Takeaways:

• Short-term fixes often create long-term problems.

• Data-driven decisions lead to better operations.

• Balancing workload improves service consistency.

By thinking long-term and implementing smart solutions, product managers can help businesses thrive without falling into the trap of quick fixes.

Have you faced similar challenges in your industry? Share your thoughts and strategies in the comments below!

I’ve often realized that no matter how great my solution seems initially, when I discuss it with the tech team, they frequently come up with an even better, more scalable solution.

It’s not always easy to involve engineers in every step of the discovery process, but I’ve found that the earlier they are included, the better. Once I’ve identified a problem, involving them in solutioning leads to amazing outcomes. When still in the discovery phase, having engineers—or at least one engineer—join the discussions has proven invaluable. Their perspective is often far more holistic.

Engineers excel at identifying the underlying complexities of a problem. Their ability to think about constraints, edge cases, and future scalability ensures that solutions are both practical and forward-looking.

Recently, I was working on a travel solution focused on developing the Activities product. One of the challenges was integrating transfers with sightseeing. Initially, the solution we designed was tightly coupled with the Activities product itself. However, the tech team suggested a more flexible approach: building the transfer functionality as an independent module that multiple systems could reuse.

This wasn’t just a module anymore—it became a microservice. Now, other products like hotels and flights can also utilize the transfer module. This shift has unlocked new opportunities for the product team to explore.

Could we have thought of this ourselves? Possibly. But this kind of thinking comes naturally to developers when it comes to architectures and scalability. Their ability to design for reusability and long-term growth often leads to outcomes we might not have considered initially.

Developers are highly time-sensitive and consistently strive to optimize the output of their efforts. They focus on applying the 80-20 rule, targeting the most impactful work to achieve maximum results. This mindset often enhances the feedback we receive, pushing us to think critically about our solutions.

In one instance, we were creating a duplicate workflow for users that, from both a product and user perspective, seemed necessary. Since the product was designed for two different departments, having a duplicate UI appeared acceptable, as it was part of two distinct modules serving separate functions within the company. However, the developers flagged this as unnecessary duplication and raised concerns about the effort required.

After hours of discussion, we were both challenged and convinced to reconsider our approach. This led us to a deeper insight: the workflows of these two departments were interconnected, as the work of one department naturally transitioned into the other. By integrating the workflows instead of duplicating them, we created a more cohesive system that required significantly less effort while delivering a more optimized and effective solution.

Another big advantage of presenting ideas to developers is that it pushes me to dig deeper and gather stronger evidence to support my product claims. Developers naturally challenge assumptions by asking for more data, user stories, and anecdotes that justify the need for a feature. They ask pointed questions like, "What problem are you solving?", "What percentage of users will benefit?", and "What metrics will improve as a result?" These questions often reveal gaps in the discovery process, where some data may not have been fully validated or certain assumptions were taken for granted. This level of scrutiny ensures that every decision is rooted in clear, well-supported reasoning.

No matter how thorough the initial product discovery phase was, engaging with developers often uncovers overlooked details or untested assumptions. It’s a humbling yet invaluable process that strengthens the overall product strategy. Developers bring a fresh perspective that ensures features are not only feasible but also impactful and aligned with user needs. While these discussions can be challenging, they lead to more refined solutions and a deeper understanding of the problem. In the end, this collaborative effort results in better products and, most importantly, happier customers who feel their needs are truly addressed.

The next time you’re working on a product feature, consider these key benefits:

Early Involvement = Better Solutions: Engaging engineers early in discovery enables them to understand the "why" behind the problem. They can often identify opportunities for reuse, optimization, or a completely different approach you might not have considered.

Diverse Perspectives Matter: Product managers often focus on the "what" and "why," while engineers bring the "how" into the equation. Their involvement creates a more holistic approach to problem-solving, balancing user needs, technical feasibility, and business goals.

Avoiding Local Maxima: Engineers often have a knack for stepping back and seeing the bigger picture. They may challenge assumptions or constraints that could lead to suboptimal solutions, helping the team explore more ambitious and scalable options.

Building Ownership: When engineers are part of the discovery and solutioning process, they feel a stronger sense of ownership. This engagement leads to better collaboration, motivation, and execution.

Foster Collaboration, Not Silos: Create a culture where engineers feel empowered to challenge ideas and contribute beyond implementation. Show appreciation for their contributions, emphasizing how their insights make a tangible difference for users.

Involving engineers early has always amazed me with the solutions they bring to the table.

Product managers often rely on data to make decisions for their products. This qualitative or quantitative data is often a source of insight and inspiration to solve user problems. So what does it mean when we say, "Data is the Product"?

When data is the product, it means that data itself—or insights derived from it—is the primary value delivered to users. Unlike traditional products, which are physical or software-based, data products focus on collecting, processing, and delivering actionable information. It is about giving back the information generated by the user in such a way that it becomes useful for the individual or group of users on the platform or product. The simplest example is the reviews or feedback that users give for any services or products they use. This data can be used by product managers to understand how users like their product or service, but users also use the same data to make purchase decisions. Google Maps primarily tells directions from one point to another, but the data it collects about user locations is used to estimate traffic conditions. This is one of the coolest ways of using data as a product feature.

While data is useful, there are certain considerations that would help product managers create a successful product.

Focus on customer needs

You cannot forget customer pain points and needs. A data product should be created with the user in mind. It’s not just about providing information but presenting it in a way that users find easy to understand and act on. For example, Spotify creates personalized playlists by analyzing user data. Instead of showing raw data about songs played, it curates music recommendations, making it simple and enjoyable for users to discover new music. Zomato's Reorder feature is a great example of a data product. Users often want to reorder something they liked, and rather than making the user go through the hard work of finding, selecting, and ordering the same food again, Zomato makes it easier for customers to order again. The key point is that someone actually figured out the customer pain points before deciding on these features.

Maintain high quality of data

Another major consideration for data products is the quality of the data. High-quality data is like a strong foundation for a house. It must be accurate, timely, and reliable so users can trust it for their decisions. Imagine using a weather app that shows outdated or wrong forecasts—it would lose its usefulness. In contrast, Google Maps relies on accurate location data to show real-time traffic conditions, ensuring users trust its directions. Users want predictability, and to provide it, it is important to have high-quality data in place.

Work on continuously improving the product or feature

Like any other product, data products should improve over time. Feedback from users can reveal areas where the product isn’t meeting expectations. For instance, Netflix constantly updates its recommendation algorithms based on how users engage with its platform. If many users skip certain movie genres, Netflix adapts by refining its suggestions to better match preferences. Similarly, if Google Maps users consistently avoid the alternate routes suggested due to traffic, it is important to understand their behavior. Maybe it is because of security concerns with unknown routes, and Google might need to build trust by using data like, “50 users reached the same destination through this route.” Data products have many opportunities for continuous improvement.

Privacy and Compliance

Handling data ethically is essential. Users need to feel confident that their information is safe and used responsibly. Laws like GDPR and CCPA set strict guidelines for how companies handle data. For example, Apple emphasizes user privacy in its products, ensuring transparency about what data is collected and how it’s used. This builds trust and loyalty among its users. Any use of data that hurts user sentiments could lead to sudden drops in user engagement. Applications should provide the ability to control the usage of their data for decision-making. This builds trust and gives users a sense of control over their information.

Building Data as a Product

1. Identify User Needs:

   Start by understanding what users are looking for in the data. Are they trying to spot trends, predict outcomes, or dive into detailed metrics? For instance, fitness apps like Fitbit identify user goals, such as tracking steps or monitoring sleep, and tailor their data delivery accordingly.

2. Data Collection:

   Reliable systems are needed to collect data from various sources. Ensuring the data is accurate and free from bias is crucial. For example, e-commerce platforms gather data on user behavior—like browsing and purchase history—to improve product recommendations.

3. Data Processing:

   Raw data isn’t very helpful on its own. Advanced tools like AI and machine learning transform this data into meaningful insights. Think about how LinkedIn processes user activity to suggest jobs or connections, making the platform more useful for professionals.

4. Delivery Mechanism:

   The way data is delivered matters. Dashboards, APIs, reports, or visualizations should match what users find most convenient. For example, business intelligence tools like Tableau provide interactive dashboards, helping users explore and understand data visually. While this way of presenting information is good for professionals, a normal user would appreciate a more integrated user flow to make the information actionable.

5. Monetization:

   Data products can contribute to revenue growth as well as customer retention. These advanced features can be part of premium or pro accounts. Once the data becomes important for the customer, they attach both sentimental value and need value to the information, ultimately driving revenue.

Challenges and Considerations

1. Data Silos: Integrating disparate data sources to provide a cohesive product is a technical and organizational challenge.

2. Ethical Use: Balancing data monetization with user privacy requires robust governance.

3. Scalability: As the volume of data grows, maintaining performance and accuracy becomes complex.

Treating data as a product requires a shift in mindset. Organizations must recognize the value of their data assets and build systems to maximize their potential. By focusing on user needs, maintaining high data quality, and ensuring ethical use, companies can turn data into a powerful product that drives growth and innovation.