Real-World Machine Learning: How Top Companies Ship AI to Production

Pulse EngineeringDecember 24, 20259 min

At Pulse, we're building what we like to call a financial operating system: a single place to track your portfolio, research investments, stay on top of the markets, manage your budget, monitor your net worth, and more. And like most modern software, machine learning plays a role in making it useful.

Our AI chat uses Retrieval-Augmented Generation (RAG) to pull in relevant market data and news before responding, which helps keep answers grounded in reality rather than pure generation. We cluster news articles to surface themes. We use embeddings to make search a bit smarter than exact keyword matching.

None of this is groundbreaking. We're standing on the shoulders of teams who've solved these problems at scales we can only aspire to, and who've been generous enough to share how they did it.

This post is a curated tour through some of the most valuable ML case studies from companies operating at massive scale. Whether you're an engineer looking for implementation patterns or just curious about how these systems work, there's something here for you.


Stripe Radar: Real-Time Fraud Detection

StripeCase Study

Every online payment is a bet. The merchant bets the customer is legitimate. The payment processor bets the card isn't stolen. Get it wrong in one direction, and you lose money to fraud. Get it wrong in the other, and you block legitimate customers who never come back.

Stripe processes billions of transactions annually, which gives them something invaluable: a view across the entire internet economy. Their fraud detection system, Radar, uses deep learning trained on patterns that no single merchant could ever see.

What Makes It Work
Network effects (patterns learned from millions of businesses)
Under 100 ms decision latency
Continuous learning from new fraud vectors

The insight is as much strategic as technical. Stripe's moat isn't their model architecture; it's their data flywheel. Every transaction makes the system smarter.

Read Stripe's full breakdown →


Uber DeepETA: Predicting Arrival Times

UberCase Study

"Your driver will arrive in 8 minutes." That number needs to actually be 8 minutes. Inaccurate ETAs frustrate riders, hurt driver earnings, and erode trust in the platform.

Traditional routing algorithms treat roads as edges in a graph with fixed travel times. But anyone who's driven knows reality is messier. A Giants game affects traffic differently than a concert at Chase Center. Rain slows everything down, but not uniformly.

Uber's DeepETA uses transformer architecture (the same family of models behind ChatGPT) to process routes as sequences. Think of it like treating a journey as a "sentence" of road segments, where attention mechanisms learn which segments matter most for the final prediction.

Technical Approach
Transformer-based sequential modeling
Real-time traffic integration
Segment-level attention for route understanding

The result is ETA predictions that account for complex, non-linear interactions that rule-based systems simply can't capture.

Dive into Uber's technical deep-dive →


DoorDash: Reducing Dasher Wait Times

DoorDashCase Study

Here's a scenario that costs DoorDash millions: a Dasher arrives at a restaurant and waits 20 minutes for an order that isn't ready. The Dasher earns less per hour. The customer gets lukewarm food. DoorDash pays for idle time.

The solution sounds simple: predict when orders will be ready and dispatch Dashers to arrive just in time. The execution is anything but.

DoorDash's engineering team built a time-series forecasting system that considers restaurant-specific prep patterns, current order volume, time of day, day of week, and menu item complexity. A burger takes different time than a custom sushi roll.

But the most valuable part of their case study is the lifecycle discussion:

"The hardest part wasn't building the model; it was integrating it into operations, monitoring for drift, and continuously improving it. ML products are never done."

Lifecycle Stages
Problem definition, Model development, Integration
Monitoring, Drift detection, Retraining
Continuous improvement loop

Read DoorDash's full lifecycle story →


Netflix: In-Video Search

NetflixCase Study

You remember a specific scene (maybe a dialogue, a location, a visual moment) but not which movie it's from. Traditional search is useless. You can't Google "that scene where the guy says the thing in the rain."

Netflix built a system that actually understands video content. Not just metadata, but the visual and audio content itself: detecting scenes, objects, dialogue, and mood across their entire library.

Combined with semantic search, this enables queries that would have been impossible a few years ago. Describe what you're looking for in natural language, and the system finds the moment.

Scale Challenges
15,000+ titles to process
Millions of hours of video content
Real-time search latency requirements

The engineering challenge: building inference infrastructure that processes their entire catalog cost-effectively.

Explore Netflix's approach →


Spotify: Podcast Search

SpotifyCase Study

Podcasts contain incredible content locked inside hours of audio. Traditional search relies on titles and descriptions, which means you can find "The Joe Rogan Experience" but not the specific episode where they discussed psychedelics and neuroscience.

Spotify's solution: transcribe everything, then build semantic embeddings that capture meaning rather than just keywords. Search "how to start a business" and surface relevant discussions across thousands of podcasts, even if those exact words were never spoken.

Pipeline
Speech-to-text at scale
Semantic embedding generation
Vector similarity search

They also built ML-generated podcast previews: short audio clips that give you a taste of an episode's content, generated automatically. ML can make content more accessible, not just recommend it.

Learn how Spotify built it →


Airbnb: Categories with Human-in-the-Loop

AirbnbCase Study

How do you categorize millions of unique homes into delightful categories like "Treehouses," "Castles," or "A-frames"? Pure ML makes embarrassing mistakes, confidently labeling a suburban house as a castle because it has a turret-shaped window. Pure human curation doesn't scale to millions of listings.

Airbnb's answer: human-in-the-loop ML. The model makes initial predictions. Human curators review edge cases and correct errors. Those corrections feed back into training data. The model improves. Rinse, repeat.

The Feedback Loop
ML predicts category
Human curators review uncertain cases
Corrections become training data
Model improves, cycle continues

This philosophy (using ML to scale human judgment rather than replace it) produces results that feel magical. The A-frame cabin you didn't know you wanted, surfaced because the system learned what "A-frame" really means from thousands of human decisions.

Read Airbnb's human-in-the-loop approach →


GitHub Copilot: Enterprise LLMs at Scale

GitHubCase Study

Building an LLM-powered product that millions of developers trust for code suggestions is a different beast than building a chatbot. Wrong answers don't just look bad; they could introduce bugs or security vulnerabilities into production systems.

GitHub's team shared hard-won lessons from building Copilot at scale:

Production Challenges
Latency must feel instant (users won't wait)
Cost per suggestion must be sustainable at scale
Quality must be measurable and improvable
Responsible AI: don't regurgitate copyrighted code

Their key insight: "Evaluation is everything. If you can't measure improvement, you can't ship it."

They built extensive evaluation infrastructure before scaling the product, because you can't improve what you can't measure.

GitHub's enterprise LLM lessons →


Zillow Zestimate: Neural Home Valuation

ZillowCase Study

Estimate the value of any home in America (including ones that haven't sold in decades) and do it accurately enough that people make major financial decisions based on your number.

The Zestimate evolved from simple comparable-sales calculations to ensemble ML to deep neural networks. The current system processes structured data (bedrooms, bathrooms, square footage) alongside unstructured data (listing descriptions, photos) to understand that "chef's kitchen" adds more value than "functional kitchen."

Evolution
V1: Comparable sales math
V2: Ensemble models
V3: Neural networks with NLP
Accuracy: About 2% median error

But Zillow's story also includes a cautionary tale. Their iBuying business (Zillow Offers) failed partly due to model overconfidence, a reminder that even sophisticated ML systems have uncertainty bounds that matter.

Inside the Neural Zestimate →


Patterns Worth Remembering

After reading through these case studies, some patterns stand out:

Start simple, add complexity when needed. Almost every case study describes beginning with rules or simple models, then graduating to deep learning after proving value. Don't start with transformers.

Human-in-the-loop scales judgment, not just labor. The best systems (Airbnb Categories, Uber's fraud review) combine ML efficiency with human expertise. Full automation isn't always the goal.

Your data is your moat. Stripe can detect fraud patterns because they see the entire internet economy. Your competitive advantage is data others don't have access to.

ML products are never done. DoorDash emphasizes the lifecycle: deploy, monitor, retrain, improve, repeat. Budget for ongoing investment.

Evaluation infrastructure is non-negotiable. GitHub's team is blunt: if you can't measure improvement, you can't ship it. Build evals before you build models.


Keep Exploring

This post barely scratches the surface. The full curated list contains 300+ case studies from 80+ companies, maintained by the community on GitHub:

A Curated List of ML System Design Case Studies


Have a favorite ML case study we missed? We'd love to hear about it.

Share this article