Linear Regression for Beginners: Fit the Line, Predict with y = mx + b

Learning objectives#

By the end you will be able to:

1. State what linear regression does in one clear sentence.
2. Name every symbol in \(y = mx + b\) and use the equation to predict.
3. Explain least squares, residual, \(R^2\), and RMSE in plain words.
4. Compute slope \(m\) and intercept \(b\) from a small table by hand.
5. Run a complete Python 3 script (standard library only) that fits a line, prints \(R^2\) and RMSE, and predicts for a new \(x\).
6. List when simple linear regression is a sensible choice for Cloud/DevOps-style numeric forecasts — and when it is not.
7. Answer beginner interview questions and MCQs on the same vocabulary.

Prerequisites#

• Arithmetic: add, subtract, multiply, divide, square, square root (phone calculator is fine).
• Python 3 for Lab 2 only: run a script from a terminal. No NumPy or scikit-learn required.
• No prior ML or statistics course.
• Helpful later (not required here): skim Mathematics for Machine Learning after this page if you want vectors and gradient descent next.

Setup (Lab 2 only)#

python3 --version   # 3.10+ recommended; this page was checked on 3.13.5

No pip install is required for the main lab. Optional check only: numpy via np.polyfit if you already have it.

Core concept: what linear regression is#

Linear regression is a method that finds a straight-line equation describing how one number (the output) changes when another number (the input) changes. You give it labeled pairs \((x, y)\). It returns slope \(m\) and intercept \(b\). For a new \(x\), you compute:

When to use it (beginner checklist)

1. You want to predict a continuous number (price, marks, cost, latency in ms).
2. The relationship looks roughly like a straight line, not a strong curve.
3. You have past pairs of (input, known output).

Do not use simple linear regression when the pattern is clearly curved, when the target is a category (spam / not spam), or when you have almost no data.

How it works: the line \(y = mx + b\)#

The equation of a straight line in two dimensions is:

Reading the slope

• If \(m = 92\), then when \(x\) goes up by 1, predicted \(y\) goes up by 92.
• If \(m = -3\), then when \(x\) goes up by 1, predicted \(y\) goes down by 3.

Reading the intercept

• If \(b = 60\), then when \(x = 0\), the model says \(y = 60\).
• Sometimes \(x = 0\) is not a realistic case (a house of size 0). The intercept is still required by the math.

One prediction step

Given \(m\) and \(b\), for any new \(x\):

How we find the best line: least squares#

Many lines could pass near your points. Least squares picks the one line that makes the total squared prediction error as small as possible.

Steps in words

1. For each data point, the line gives a predicted \(y\).
2. The residual for that point is \(\text{actual } y - \text{predicted } y\).
3. Square each residual (positive and negative errors both count as cost; large errors cost more).
4. Add all those squared residuals.
5. Choose \(m\) and \(b\) that make this sum as small as possible.

Residual#

• Residual \(> 0\) → the line predicted too low.
• Residual \(< 0\) → the line predicted too high.
• Residual \(= 0\) → the line hit the point exactly.

For ordinary simple linear regression with an intercept, the sum of residuals on the training points is 0.

Fit formulas (simple linear regression, one input)#

Let there be \(n\) points \((x_1, y_1), \ldots, (x_n, y_n)\).

You do not need calculus for beginner use. Plug the numbers into these formulas.

Simple vs multiple (brief)#

This tutorial focuses on simple linear regression. Multiple regression uses the same idea (reduce squared error) but usually needs a library or matrix method.

Assumptions (checklist level)#

1. Linearity — relationship is close to a straight line.
2. Independence — points are not a hidden chain you are ignoring.
3. Homoscedasticity — residual spread is roughly similar across \(x\) (not tiny on the left and huge on the right).
4. No single-point control — one outlier should not alone dictate the line.
5. Residuals look roughly random — no clear leftover curve or clusters.

If these fail badly, predictions can mislead even when \(R^2\) looks high.

Judging fit: \(R^2\) and RMSE#

After you fit the line, measure how good it is on the data you used.

RMSE (Root Mean Squared Error)#

• Same units as \(y\).
• Lower is better.
• Plain reading: typical prediction error size is about the RMSE.

Convention used on this page: \(\mathrm{RMSE} = \sqrt{\mathrm{SS_{res}} / n}\). Some tools use \(n-2\) for an unbiased residual-variance estimator in regression. State which formula you use when comparing numbers.

\(R^2\) (R-squared, coefficient of determination)#

• Near 1 → the line explains most of the variation in \(y\) on this data.
• Near 0 → the line is barely better than always guessing \(\bar{y}\).
• \(R^2\) does not prove the model is correct for new data. It describes fit on the points you fitted.

Practical example 1 — House price from size#

Story: Five houses. Input \(x\) = size in thousands of square feet. Output \(y\) = selling price in thousands of dollars.

\(n = 5\)

Means#

Slope and intercept#

Fitted line:

Meaning: Each extra 1000 sq ft adds about $92,000 to the predicted price. A size of 0 would give a baseline prediction of $60,000 (math intercept; not a real house).

Predictions and residuals#

\(R^2\) and RMSE#

(RMSE is in $1000s, so about $5,657.)

New prediction#

For size 2.2 (thousand sq ft):

Predicted price ≈ $262,400.

Practical example 2 — Study hours to marks#

Story: Five students. Input \(x\) = hours studied. Output \(y\) = marks out of 100.

\(n = 5\)

Means#

Slope and intercept#

Fitted line:

Meaning: Each extra study hour adds 7.5 marks to the prediction. With 0 hours, the baseline prediction is 26.5 marks.

Predictions and residuals#

\(R^2\) and RMSE#

New prediction#

For 6 hours:

Predicted marks ≈ 71.5.

All arithmetic in both examples was verified with Python 3.13.5 before this draft.

Code example — fit Example 2 with plain Python#

Complete, runnable script. Standard library only (csv, math). Save as fit_study_marks.py next to a CSV, or paste the whole block.

import csv
import math

# --- data (same as Example 2) ---
rows = [
    {"hours": "2", "marks": "40"},
    {"hours": "3", "marks": "50"},
    {"hours": "5", "marks": "65"},
    {"hours": "7", "marks": "80"},
    {"hours": "8", "marks": "85"},
]

xs = [float(r["hours"]) for r in rows]
ys = [float(r["marks"]) for r in rows]

n = len(xs)
mean_x = sum(xs) / n
mean_y = sum(ys) / n
num = sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys))
den = sum((x - mean_x) ** 2 for x in xs)
m = num / den
b = mean_y - m * mean_x

print(f"n = {n}")
print(f"mean_x = {mean_x}")
print(f"mean_y = {mean_y}")
print(f"slope m = {m}")
print(f"intercept b = {b}")
print(f"equation: y = {m}*x + {b}")
print()
print("x\tactual\tpredicted\tresidual")

ss_res = 0.0
ss_tot = 0.0
for x, y in zip(xs, ys):
    pred = m * x + b
    res = y - pred
    ss_res += res * res
    ss_tot += (y - mean_y) ** 2
    print(f"{x}\t{y}\t{pred:.4f}\t\t{res:.4f}")

r2 = 1 - ss_res / ss_tot
rmse = math.sqrt(ss_res / n)
print()
print(f"R2 = {r2:.6f}")
print(f"RMSE = {rmse:.6f}")
print(f"prediction for 6 hours: {m * 6 + b}")

Expected output (verified):

n = 5
mean_x = 5.0
mean_y = 64.0
slope m = 7.5
intercept b = 26.5
equation: y = 7.5*x + 26.5

x   actual  predicted   residual
2.0 40.0    41.5000     -1.5000
3.0 50.0    49.0000     1.0000
5.0 65.0    64.0000     1.0000
7.0 80.0    79.0000     1.0000
8.0 85.0    86.5000     -1.5000

R2 = 0.994898
RMSE = 1.224745
prediction for 6 hours: 71.5

Hands-on lab#

Lab 1 — Pen and paper (4 points)#

Goal: Fit \(m\) and \(b\) by hand. Confirm every intermediate value.

\(n = 4\)

Steps

1. Compute \(\bar{x}\) and \(\bar{y}\).
2. Fill columns: \(x\), \(y\), \(x-\bar{x}\), \(y-\bar{y}\), product, \((x-\bar{x})^2\).
3. Sum the last two columns; compute \(m\) and \(b\).
4. Predict for each training \(x\); compute residuals.
5. Optional: RMSE.

Expected intermediates

Line: \(y = 2.3x + 0.5\)

Checks: sum of residuals \(= 0\); prediction for \(x = 5\) is \(2.3\times 5 + 0.5 = 12.0\).

Lab 2 — Python with a tiny CSV#

Goal: Same math as Example 2, from a CSV, standard library only. Zero cloud cost.

mkdir -p lr-lab2 && cd lr-lab2

cat > study_marks.csv << 'CSV'
hours,marks
2,40
3,50
5,65
7,80
8,85
CSV

python3 << 'PY'
import csv, math

xs, ys = [], []
with open("study_marks.csv", newline="") as f:
    reader = csv.DictReader(f)
    for row in reader:
        xs.append(float(row["hours"]))
        ys.append(float(row["marks"]))

n = len(xs)
mean_x = sum(xs) / n
mean_y = sum(ys) / n
num = sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys))
den = sum((x - mean_x) ** 2 for x in xs)
m = num / den
b = mean_y - m * mean_x

print(f"n = {n}")
print(f"mean_x = {mean_x}")
print(f"mean_y = {mean_y}")
print(f"slope m = {m}")
print(f"intercept b = {b}")
print(f"equation: y = {m}*x + {b}")
print()
print("x\tactual\tpredicted\tresidual")
ss_res = 0.0
ss_tot = 0.0
for x, y in zip(xs, ys):
    pred = m * x + b
    res = y - pred
    ss_res += res * res
    ss_tot += (y - mean_y) ** 2
    print(f"{x}\t{y}\t{pred:.4f}\t\t{res:.4f}")

r2 = 1 - ss_res / ss_tot
rmse = math.sqrt(ss_res / n)
print()
print(f"R2 = {r2:.6f}")
print(f"RMSE = {rmse:.6f}")
print(f"prediction for 6 hours: {m * 6 + b}")
PY

Verification checklist

Cleanup

cd ..
rm -rf lr-lab2
test ! -d lr-lab2 && echo "cleanup ok"

Common mistakes#

1. Swapping \(x\) and \(y\) — Put the thing you want to predict on the \(y\) side. \(x\) = input you know; \(y\) = output you predict.
2. Forgetting the intercept — Predicting with only \(m\cdot x\). For hours→marks, \(7.5\times 6 = 45\) is wrong; correct is \(7.5\times 6 + 26.5 = 71.5\).
3. Treating high \(R^2\) as proof the model is "true" — \(R^2\) measures fit on the data you used. It does not prove cause, and it does not guarantee good predictions on new data.
4. Ignoring residuals — A leftover curve or growing spread means a straight line may be the wrong shape, even if \(R^2\) is high.
5. Using regression for classification labels — Yes/no categories need a classification method, not plain simple linear regression treated as continuous scores.
6. Mixing RMSE denominators — This page uses \(\sqrt{\mathrm{SS_{res}}/n}\). Some tools use \(n-2\). Compare only when the formula matches.
7. Letting one outlier dominate — A single extreme point can pull \(m\) and \(b\). Scan min/max (or plot) before trusting the line.

Real-world applications (Cloud / DevOps angle)#

These are sensible shapes for simple linear regression when the relationship looks roughly linear and you have numeric history. They are teaching scenarios, not claimed client case studies.

How you would use it in practice (process, not a product claim):

1. Export a CSV of past pairs (usage, cost) or (load, latency).
2. Fit \(y = mx + b\) with the formulas or the Python lab pattern.
3. Read RMSE in the same units as \(y\) (dollars, ms).
4. Check residuals: if they curve upward at high load, a straight line is the wrong shape for that range.
5. Predict for a planned \(x\) (for example, "what bill if we add 20 VMs?") and treat it as a baseline estimate, not a guarantee.

When the bill has large fixed fees, step pricing, or strong non-linear discounts, simple linear regression on raw totals can mislead — start with a scatter plot of your own data first.

Interview questions (beginner)#

IQ1. What is linear regression in one minute?#

Linear regression finds a straight-line equation that relates numeric inputs to a numeric output. For one input it is \(y = mx + b\). We choose \(m\) and \(b\) with least squares so the total squared prediction error on the training points is as small as possible. Then we use the equation to predict \(y\) for new \(x\) values.

IQ2. What is the difference between slope and intercept?#

The slope \(m\) says how much the prediction changes when the input increases by one unit. The intercept \(b\) is the prediction when the input is zero. Both are required to write the full line, even if "input = 0" is not a realistic case in the business story.

IQ3. What does least squares mean?#

For each point we compute residual = actual − predicted, square it, and add those squares. Least squares picks the line whose sum of squared residuals is the smallest. Squaring makes all errors positive and penalizes large misses more than small ones.

IQ4. How do you know if the line is a good fit?#

Look at RMSE (typical error size in \(y\) units — lower is better) and \(R^2\) (fraction of \(y\) variation explained — closer to 1 is stronger on that dataset). Also check residuals for patterns. Good fit on training data is not the same as good predictions on new data.

IQ5. Simple vs multiple linear regression?#

Simple uses one input: \(y = mx + b\). Multiple uses several inputs: \(y = m_1 x_1 + m_2 x_2 + \cdots + b\). The goal is the same — reduce squared error — but multiple regression needs more data care and usually a library or matrix method.

MCQs (10)#

Original teaching questions for this course — not claimed from any real exam paper. Answers at the end of this section.

Q1. What does linear regression try to find?A) A straight line that best fits numeric data for prediction
B) A cluster label for each data point
C) A password for a cloud account
D) A sorting order for filenames

Q2. In \(y = mx + b\), what is \(m\)?A) The residual of the last point
B) The slope — how much predicted \(y\) changes when \(x\) increases by 1
C) The \(R^2\) score
D) The number of data points

Q3. What is a residual?A) The mean of all \(x\) values
B) The intercept when \(x = 0\)
C) Actual \(y\) minus predicted \(y\)
D) The slope divided by the intercept

Q4. Least squares chooses the line by minimizing which quantity?A) The sum of the absolute values of \(x\)
B) The sum of squared residuals
C) The product of all \(y\) values
D) The maximum \(x\) value

Q5. \(R^2\) is closest to which plain meaning?A) The cloud region where the model runs
B) How many rows are missing in the CSV
C) What fraction of the variation in \(y\) the line explains (near 1 is a strong fit on that data)
D) The slope rounded to 2 decimals

Q6. You have \(m = 7.5\) and \(b = 26.5\) for marks vs study hours. What is the prediction for 6 hours?A) 71.5
B) 45.0
C) 26.5
D) 7.5

Q7. Which pair is a sensible use of simple linear regression?A) Predict house price from house size
B) Predict whether an email is spam or not spam using only a yes/no label as the modeling goal without a numeric target
C) Sort log files by name
D) Restart a Kubernetes pod

Q8. RMSE is best described as:A) A count of independent variables
B) A typical size of prediction error, in the same units as \(y\)
C) Always a value between 0 and 1
D) The intercept of the line

Q9. In simple linear regression, the dependent variable is:A) Always called \(m\)
B) The input you already know (\(x\))
C) The output you want to predict (\(y\))
D) The residual table header

Q10. A model that memorizes training quirks and then predicts poorly on new data is showing (beginner wording):A) Under-storage
B) Overfitting
C) Perfect causality
D) Zero residual by definition on all future data

Answer key#

Practice exercises#

E1. Using Example 1's line \(y = 92x + 60\), predict price for \(x = 1.8\) and \(x = 2.7\).
(Expected: \(92\times 1.8 + 60 = 225.6\); \(92\times 2.7 + 60 = 308.4\).)

E2. Repeat Lab 1 without looking at the answer key until you have \(m\) and \(b\). Then check: \(m\) must be \(2.3\), \(b\) must be \(0.5\).

E3. For Example 2, compute the residual at \(x = 5\) by hand.
(Predicted \(7.5\times 5 + 26.5 = 64\); residual \(65 - 64 = +1\).)

E4. Change Lab 2's CSV to add one more row 9,90, re-fit, and record new \(m\), \(b\), and prediction for 6 hours. Compare to the original \(71.5\).
(You should re-run the script; do not invent the new coefficients without computing them.)

E5. In one sentence: why square residuals instead of only summing them?
(Squaring makes every error contribute a non-negative cost and penalizes large misses more; with an intercept, the raw sum of residuals for the best line is already 0.)

Summary#

1. Linear regression fits \(y = mx + b\) to numeric pairs so you can predict a continuous output.
2. Least squares chooses \(m\) and \(b\) by minimizing the sum of squared residuals.
3. Use \(R^2\) (higher toward 1 is stronger on that data) and RMSE (lower is better, same units as \(y\)) together — and still look at residuals.
4. Verified reference numbers from this page: houses \(y = 92x + 60\), \(R^2 \approx 0.9925\), pred@2.2 → 262.4; marks \(y = 7.5x + 26.5\), \(R^2 \approx 0.9949\), pred@6 → 71.5; Lab 1 \(m = 2.3\), \(b = 0.5\).
5. Next depth: vectors, cost surfaces, and gradient descent in Mathematics for Machine Learning. Systems path: RAG Tutorial in Python.

Next level: more than one input. Linear Regression with Multiple Regression (Expert) derives least squares in matrix form, works a six-row cloud-bill example by hand, and covers diagnostics (VIF, Cook's distance, Breusch–Pagan), gradient descent, Ridge and Lasso, with three runnable Python labs.

Further learning#

• Pushpjeet — Mathematics for Machine Learning (vectors, gradients, tiny linear regression from scratch with NumPy).
• Pushpjeet — RAG Tutorial in Python (build a retrieval pipeline after you have the ML vocabulary).
• Pushpjeet — AI & Machine Learning Foundations.
• Student cheat sheet: Linear Regression Beginner cheat sheet.
• Interactive quiz: Try the quiz.

FAQ#

Do I need scikit-learn for this tutorial?
No. Lab 2 uses only Python's csv and math modules. Libraries are a later convenience once the formulas are clear.

Is a high \(R^2\) enough to ship a forecast?
No. Check residuals, whether a straight line makes sense, and how the model behaves on new data. \(R^2\) alone is not a go-live gate.

Why keep the intercept if \(x = 0\) is unrealistic?
Because the math of the line needs \(b\). Interpreting \(b\) as a real-world case when \(x = 0\) never occurs is optional; dropping \(b\) from predictions is a common error.

How does this connect to Maths for ML?
Here you fit \(m\) and \(b\) with closed-form least squares. In the Maths for ML tutorial you see the same linear prediction as a weighted sum, then improve weights with gradient descent.

Will there be a multiple-regression follow-up?
Yes — see Linear Regression with Multiple Regression (Expert). This page stays on one input.

Continue