Home ML Solutions Quest ★0 XP

From idea to a working model

A model that works in a notebook is not a solution. In this quest you take one through its whole life: business goal → data → training → evaluation → deployment → monitoring, the way Amazon did for its call centre.

🗺️ 8 short levels 🎮 mini-games ⭐ earn XP and badges ⏱️ about 70 minutes
Start level 1 See the big picture first

This is your warm-up. After this quest, the full Day 3 notes will feel easy.

Your progress wheel: clear a level to light up a segment

1

Level 1 of 8, about 10 minutes

The ML lifecycle

Goal: name the 7 phases and say what happens in each.

The ML lifecycle is the end-to-end process of developing, deploying and maintaining a model.

It is a loop, not a straight line. You go back and improve again and again.

Tap a phase

Explore all 7 phases to earn XP.

Explored 0 of 7

Memory trick: B-P-D-M-D-M-R
Business goal → Problem framing → Data → Model → Deploy → Monitor → Retrain
“Bright People Design Models, Deploy, Monitor, Repeat.”
🔢 Game: put the lifecycle in order
Click the phases in the right order. Get it wrong and you start again.
Who is in the room? Business stakeholders, subject matter experts, data scientists, data engineers, ML architects, product managers. The lifecycle only works when they talk to each other. That teamwork has a name: MLOps (level 8).

🏁 Checkpoint

Where does every ML project start?

The model is live but accuracy is slipping. Which phase?

🎉 Level 1 cleared! +50 XP
2

Level 2 of 8, about 10 minutes

Real story: the Amazon call centre

Goal: follow one real project through the whole lifecycle.

😖 The problem

“Press 1 for Returns. Press 2 for Kindle. Press 3 for…”. Amazon sells almost everything, so the menu could never cover it all. Customers picked the wrong option, reached the wrong agent, and got transferred again and again. With hundreds of millions of calls a year, that wasted time and money, and annoyed customers.

🎬 Follow the project, step by step
Press “Next step” to move through the lifecycle with the Amazon team.
1. Business goalFewer transfers, happier customers
2. Problem framingPredict which skill solves the call → multiclass classification
3. Data collectionPast calls with the correct skill = labels → supervised learning
4. PreprocessingClean data, study labels, merge all Kindle skills into one
5. Visualisation40% returns, 30% Prime, 30% Kindle…
6. TrainingSplit data 80 / 10 / 10 before training
7. TuningHyperparameters + better features
8. Deploy & monitorTransfers went down. Goal met!
🧩

Features they used

What were the customer’s recent orders? Does the customer own a Kindle? Are they a Prime member?

🏷️

The label

The agent skill that actually solved the call.

🔧

Smart data cleaning

Several Kindle skills became one “Kindle” label. Fewer, cleaner labels = simpler model.

🎚️

Tuning the learning rate

Too fast: never settles on a good answer. Too slow: never gets there in time.

🕵️ Game: you are the ML team
Answer the questions the Amazon team had to answer.

🏁 Checkpoint

“Which of 20 agent skills should this call go to?” is…

Past calls already had the right skill recorded. So this is…

🎉 Level 2 cleared! +50 XP
3

Level 3 of 8, about 8 minutes

Split your data

Goal: explain why we hold data back, and what each split is for.

The goal of ML is to generalise: to work on data the model has never seen.
⚠️ Testing a model on its training data rewards memorising, not learning. It is like giving students the exam paper they practised on.
✂️ Try it: split 10,000 examples
Move the slider. Common splits are 80/10/10 or 70/15/15.

📘

Training set

The model learns from this.

📝

Validation set

Check and improve: tune hyperparameters, compare versions.

🎓

Test set

The final, honest check. Used once, at the very end.

Why two hold-out sets? If you tune against the test set, you slowly “learn” it, and the final score becomes too optimistic. The validation set takes that role, so the test set stays truly unseen.

🏁 Checkpoint

Which set is used to tune hyperparameters?

Why not evaluate on the training data?

🎉 Level 3 cleared! +50 XP
4

Level 4 of 8, about 12 minutes

Amazon SageMaker AI: one tool per phase

Goal: match each SageMaker AI feature to its lifecycle phase.

Amazon SageMaker AI is a fully managed ML service: collect and prepare data, build and train models, deploy them and monitor their predictions, all in one place.
🧹 Prepare data

Data Wrangler

  • Low-code / no-code: import, clean, transform, analyse
  • Add your own Python steps
🧹 Prepare data at scale

Processing API, EMR, AWS Glue

  • Run scripts and notebooks on managed infrastructure
  • scikit-learn, MXNet, PyTorch
🗄️ Store features

Feature Store

  • Create, share and reuse features
  • Same features for training and inference
🏋️ Train

Training jobs

  • Built-in or custom algorithms
  • Model artifacts saved to Amazon S3
🖱️ Train without code

SageMaker Canvas

  • No-code predictions and AutoML
🚀 Start from a ready model

SageMaker JumpStart

  • Pre-trained open-source models and solution templates
🔬 Compare runs

SageMaker Experiments

  • Track data, algorithms and parameters
  • See the effect of each change on accuracy
🎚️ Find the best settings

Automatic Model Tuning

  • Runs many jobs with different hyperparameters
  • Picks the best by your chosen metric
📦 Deploy

Inference options

  • Real-time, batch, asynchronous, serverless (level 8)
👀 Monitor

Model Monitor

  • Continuous or scheduled checks
  • Alerts on data quality, model quality, bias drift, feature attribution drift

Where you work: SageMaker Studio

A web interface that opens everything: prepare data, train, deploy, monitor. Inside it you can use:

📓 JupyterLab

Notebooks, code and data.

🖱️ Canvas

No-code ML and AutoML.

📊 RStudio

For the R language.

💻 Code Editor

Based on VS Code, with extensions.

🚀 JumpStart

Pre-trained models to start from.

⚖️ Model evaluations

Check LLM quality and responsible AI.

🧩 Game: pick the SageMaker feature
Which feature does the job?

🏁 Checkpoint

You want to clean and transform data with almost no code. Use…

Run many training jobs with different hyperparameters and keep the best:

🎉 Level 4 cleared! +50 XP
5

Level 5 of 8, about 8 minutes

Where do models come from?

Goal: choose between a ready model and building your own.

Four ways to get a model in SageMaker AI, from least effort to most:

1

🎁 Pre-trained models (JumpStart)

Least effort. Deploy as-is, or fine-tune first. From popular model hubs, with ready-made solution templates.

2

🧰 Built-in algorithms

More effort. AWS provides the algorithm; you bring the data. Scales well to large datasets.

3

🐍 Your code in a pre-made image

No built-in fits? Write your own script using scikit-learn, TensorFlow, PyTorch, MXNet or Chainer in an AWS-provided container.

4

🐳 Your own Docker image

Most effort, most control. Install anything you need.

Rule of thumb: start at step 1 and only move down the ladder when you must. Effort and cost go up at every step.

Built-in algorithms by problem type

🏷️

Supervised

Classification and regression: predict a label or a number.

🧩

Unsupervised

Clustering, dimension reduction, topic modelling, anomaly detection.

🖼️

Image processing

Image classification, object detection, computer vision, time series.

📝

Text analysis

Document classification and summarisation, topic modelling, transcription and translation.

The notes list only the most common ones. The SageMaker AI documentation has the full list.

🎯 Game: which option?
Pick the cheapest option that solves the problem.

🏁 Checkpoint

Which option needs the least effort?

You need an unusual library that no AWS image has. Use…

🎉 Level 5 cleared! +50 XP
6

Level 6 of 8, about 12 minutes

Model fitting: the three students

Goal: spot underfitting and overfitting, and know the fixes.

Model fitting = training a model to learn the link between input (X) and output (Y). For example: study hours → marks.

Three students, three models

🎚️ Try it: find the good fit
Slide the model complexity and watch both errors. Aim for the lowest error on new data.
Training error
New data error
ModelTraining accuracyTesting accuracyBiasVarianceGeneralisation
UnderfittingLowLowHighLowPoor
Good fitHighHighModerateModerateExcellent
OverfittingVery highLowLowHighPoor
📏

Bias

Error from assuming things are too simple. The model can’t capture the real relationship.

🌀

Variance

Sensitivity to the exact training examples. Change the data a little and the model changes a lot.

Salary example. Underfit: salary depends only on age. Good fit: experience, education, skills, location, industry. Overfit: the model “learns” employee ID and office chair number, which mean nothing for new employees.
🩺 Game: diagnose the model
Read the two scores and give your diagnosis.
🔧

Fix underfitting

More complex model (linear → polynomial or random forest), add features, train longer, reduce regularisation.

🛡️

Fix overfitting

More data, cross-validation, simpler model, regularisation (L1/L2), prune trees, dropout, early stopping, feature selection, ensembles.

AlgorithmUnderfitting riskOverfitting risk
Linear regressionHighLow
Decision treeLowVery high
Random forestLowLow
XGBoostLowMedium
KNN, small KLowHigh
KNN, large KHighLow
Neural networkLowHigh without regularisation
Key takeaway The goal is not the highest training accuracy. It is good generalisation: doing well on new, unseen data.

🏁 Checkpoint

Training loss keeps falling while validation loss rises. This is…

High bias and low variance means…

🎉 Level 6 cleared! +50 XP
7

Level 7 of 8, about 10 minutes

Measuring a model

Goal: read a confusion matrix and pick the right metric.

Evaluation happens after training, on the validation and test sets. Which numbers you look at depends on the problem type.

📧 Try it: spam filter, move the line
Each square is an email. 🟥 real spam, 🟦 real mail. The model gives each a spam score. Move the threshold and watch the metrics fight each other.
Model says SPAM
Model says OK
Really spam
caught (TP)
missed (FN)
Really fine
false alarm (FP)
correct (TN)
Accuracy
Precision
Recall
F1 score

🎯

Precision

Of everything it flagged, how much was right? Low precision = real mail lands in spam.

🕸️

Recall

Of all the real spam, how much did it catch? Low recall = spam in your inbox.

⚖️

F1 score

One number balancing precision and recall.

📈

AUC

How well the model separates the two classes across all thresholds.

⚠️ Accuracy can lie. If only 2 emails in 100 are spam, a model that says “not spam” every time is 98% accurate and catches nothing.

Predicting numbers instead? Use regression metrics

MAE

Average size of the error, in the same unit (₹, °C).

RMSE

Like MAE but punishes big mistakes much more.

R²

How much of the variation the model explains. 1.0 is perfect.

📐 Game: choose the metric
Which number matters most here?

🏁 Checkpoint

A cancer screening model must not miss sick patients. Optimise…

Predicting house prices in rupees. A good metric is…

🎉 Level 7 cleared! +50 XP
8

Level 8 of 8, about 12 minutes

Deploy it, then keep it alive (MLOps)

Goal: choose a deployment option and explain what MLOps adds.

Deployment = putting the model and its resources into production so it can make predictions.
🏠

Self-hosted API

Your own servers or containers. Full control, maybe cheaper, but you manage web servers, load balancers and databases.

☁️

Managed API

SageMaker AI hosts it: one click or one API call, automatic scaling, HTTPS endpoints that can hold several models.

Four SageMaker AI inference options

⚡

Real-time

Interactive, low latency. A live endpoint.

📦

Batch transform

Big datasets, no permanent endpoint. Also good for cleaning data before training.

⏳

Asynchronous

Queues requests. Large payloads (up to 1 GB), long processing (up to 1 hour).

💤

Serverless

Quiet periods between spikes, and you can accept a cold start. No infrastructure to manage.

🚚 Game: pick the deployment option
Which inference option fits?

MLOps = people + process + technology

MLOps means running the whole ML lifecycle in a systematic, repeatable way: develop, deploy, monitor, retrain. It is DevOps, adapted for models.
Why models need it. A normal app behaves the same next year. A model is sensitive to data: one that works today can quietly get worse in weeks, because the world changed. So it needs monitoring and retraining, not just a release.

5 benefits

⚡ Productivity

Self-service environments and curated data. Less waiting.

✅ Reliability

CI/CD means quick deployments with consistent quality.

🔁 Repeatability

Automate every step, so results can be reproduced.

🔍 Auditability

Version data, code and models: show exactly how a model was built.

📊 Data and model quality

Enforce policies against bias, track drift over time.

🤝 Collaboration

Data scientists, engineers and IT operations working as one team.

4 key principles

🗃️ Version control

Track changes to data, code and models. Roll back when needed.

🤖 Automation

Automate ingestion, preprocessing, training, validation and deployment. Automated tests catch problems early.

🔄 CI/CD

Continuous integration, delivery, training (retrain automatically) and monitoring.

🏛️ Model governance

Documentation, reviews and approvals before deployment. Check fairness, bias and ethics. Protect sensitive data.

🏭 Try it: the automated pipeline
Press “Next step” to watch a model travel from build to production.
1. Model buildNew data arrives → pipeline trains a new model
2. EvaluationQuality gate: meets the baseline metrics?
3. ApprovalRegistered in the Model Registry and approved
4. DeploymentSource → build → staging → tests → production
5. In productionMonitor infrastructure, data quality and model performance

AWS services for MLOps

StepService
Prepare dataSageMaker Data Wrangler, Processing API
Store featuresSageMaker Feature Store
TrainSageMaker training jobs
Compare runsSageMaker Experiments
Process and evaluateSageMaker Processing
Catalogue and approve modelsSageMaker Model Registry
DeploySageMaker inference options
MonitorSageMaker Model Monitor
Tie it all togetherSageMaker Model Building Pipelines

🏁 Checkpoint

Requests up to 1 GB that take several minutes need…

Which MLOps principle lets you prove how a model was built?

🎉 Level 8 cleared! +50 XP. Your wheel is complete!
🗺️

The big picture: mind map

Everything on one page. Click a branch to jump to its level.

🃏

Flashcards

Tap to flip. Mark each card. Keep going until the “review again” pile is empty.

⚡

Myth or fact?

Quick-fire. Decide fast.

👹

Boss battle

10 questions. 3 hearts. Can you beat the Drift Monster?

Quest progress

ML Solutions Builder

📖 Next step: read the Day 3 notes

You know the map now. In the notes, look for:

  • The exact wording of the 7 lifecycle phases
  • The full Amazon call centre story, phase by phase
  • Every SageMaker AI feature and what it does
  • The complete underfitting / good fit / overfitting comparison tables
  • The MLOps benefits, principles and AWS service map