Linear Regression Theory Guide

Rules & Common Mistakes

• Exam Trap: The Line Goes on Forever, Reality Does Not

  Your regression equation is mathematically infinite, allowing you to plug in *any* $X$ value. But if your model was trained on houses ranging from 1,000 to 3,000 sq ft, asking it to predict a 20,000 sq ft mansion is pure fantasy. This is called extrapolation. If an exam asks whether it's safe to predict far outside the training window (like $X=50,000$), the answer is almost always no — the model has no reality check for a world it has never seen.
• Exam Trap: Extreme X-Values Act Like Magnets

  Not all outliers are created equal. An outlier sitting far off on the $Y$-axis just tugs your regression line slightly, but a point sitting far out on the $X$-axis has high leverage. It acts like a magnet nailed to the end of a seesaw, physically tilting the entire slope $m$ toward itself. On an exam, if a scatter plot has one lonely point far to the right, flag it immediately before doing any calculations.
• Lab Trap: The Dummy Variable Trap (Multicollinearity)

  Feeding the model two features that carry the exact same information (like Temperature in °C and °F) causes the coefficients to become wildly unstable. The same crash happens with One-Hot Encoding: if you create 3 dummy columns for 3 categories without dropping one, they always add up to 1, giving the model a shortcut that breaks the math. Always drop one dummy column (the reference category) to avoid this Dummy Variable Trap.
• Lab Trap: Fitting a Straight Line to Curved Data

  Linear Regression has no conscience — it will blindly fit a straight line to a U-shaped curve or a scattered wave without throwing a single error. You will get a slope, an intercept, and an $R^2$ score, but your predictions will be completely garbage for most of the data. Always plot your scatter plot first. If the relationship physically curves, a straight line is the wrong tool entirely, and you will need polynomial features instead.

Strengths, Weaknesses & When To Use It

When to use it:Linear Regression is your Day 1 Baseline for any regression task — always run it first before reaching for anything fancier. It shines when you need to explain *why* a prediction was made to non-technical stakeholders, like showing a marketing team exactly how much each euro of ad spend drives sales. Hard rule: if your scatter plot shows a curve, a wave, or a U-shape, do not use it as-is. You'd need to manually engineer polynomial features first, or pick a different model entirely.

Simple Linear Regression vs. Multiple Linear Regression

Simple Linear Regression is the foundation; Multiple Linear Regression is the real-world upgrade. The core goal is identical — minimize the error between your predicted line and the actual data points — but Multiple scales that logic up to handle several input features simultaneously, giving you a far more powerful and realistic model.

• Lines vs. Hyperplanes: Simple Linear Regression takes one independent variable and draws a clean 2D line described by $Y=mx+b$. The moment you add a second input feature, you leave 2D space entirely — Multiple Linear Regression fits a multi-dimensional hyperplane through the data, one dimension per feature.
• The Math Jump: Simple Linear Regression can be solved by hand using a standard 5-column deviation table — a calculator is enough. Multiple Linear Regression requires linear algebra: you build an $X$ matrix of all your input features, compute its transpose $X^T$, and solve the Normal Equation by finding the inverse of $X^TX$. Same concept, completely different computational weight.
• Isolating Variables: Simple gives you one slope to interpret. Multiple gives you a separate coefficient for every feature, and this is where it gets powerful — each coefficient tells you the impact of that one variable while holding every other variable mathematically constant. That ability to isolate causes is what makes regression genuinely useful in the real world.

Implementation Pseudocode

// LINEAR REGRESSION — Eager Learner
// Unlike KNN (lazy learner) which memorizes data and calculates at prediction time,
// Linear Regression does ALL the hard math during training.
// The result: prediction is a single multiplication — instant, O(1), forever.

// ============================================================
// FUNCTION 1: TRAINING — Build the model from the dataset
// ============================================================
FUNCTION trainLinearRegression(dataset):

    n = LENGTH(dataset)
    sumX = 0
    sumY = 0

    // --- STEP 1: Calculate Means ---
    FOR EACH point IN dataset:
        sumX = sumX + point.x
        sumY = sumY + point.y
    END FOR

    meanX = sumX / n
    meanY = sumY / n
    // EXAM WARNING: Do NOT round meanX or meanY here.
    // Premature rounding poisons every deviation you calculate next.

    // --- STEP 2 & 3: Calculate Deviations, Products, and Squared Deviations ---
    sumProduct    = 0
    sumSquaredX   = 0

    FOR EACH point IN dataset:
        devX       = point.x - meanX
        devY       = point.y - meanY
        // SANITY CHECK: If you summed ALL devX values across every row,
        // the total must equal exactly 0. Same for all devY values.
        // If it doesn't, your meanX or meanY is wrong — stop and recalculate.

        productDev   = devX * devY
        squaredDevX  = devX * devX

        sumProduct   = sumProduct  + productDev
        sumSquaredX  = sumSquaredX + squaredDevX
    END FOR

    // --- STEP 4: Calculate Slope (m) and Intercept (b) ---
    m = sumProduct / sumSquaredX
    b = meanY - (m * meanX)
    // EXAM TRAP — The Double Negative: If m is negative AND meanX is positive,
    // you get b = meanY - (negative * positive) = meanY + something.
    // Write out the sign explicitly on paper — do not do this step in your head.

    RETURN model = { m: m, b: b }

END FUNCTION

// ============================================================
// FUNCTION 2: PREDICTION — Use the trained model instantly
// ============================================================
FUNCTION predictLinearRegression(model, unknownX):

    // --- STEP 5: Assemble the Line Equation and Predict ---
    RETURN (model.m * unknownX) + model.b
    // This is O(1) — one multiplication, one addition, done.
    // The model can predict for ANY value of unknownX, including values
    // far outside the original training data (extrapolation).
    // Just remember: extrapolating too far gives fantasy numbers.

END FUNCTION

Time & Space Complexity

Linear Regression Exam Questions Students Always Get Wrong

• My intercept ($b$) is negative, but I am predicting house prices which can't be negative. Is my math wrong?

  No — your math is fine. The intercept simply anchors the line at $X=0$. If a house with zero square feet doesn't exist in reality, that anchor point is purely mathematical. Never interpret $b$ as a real-world prediction unless $X=0$ is actually meaningful in your problem.

• If I swap my $X$ and $Y$ columns, will I get the exact same line just flipped?

  No — and this is a classic exam trap. The algorithm minimizes vertical error distances toward $Y$. Swap the columns and it minimizes horizontal distances instead. You get a completely different line with a different slope. The two lines are not mirror images of each other.

• I added up all my prediction errors (residuals) and got exactly zero. Did I make a mistake?

  That means you did it perfectly. OLS mathematically places the line through the center of gravity of the data. The positive residuals above the line will always cancel out the negative ones below it exactly. A residual sum of zero is the proof your line is correctly placed.

• My $R^2$ score is only 0.40. Does this mean my model is useless?

  Not necessarily. When predicting noisy real-world behavior — human decisions, market trends, medical outcomes — low $R^2$ scores are completely normal. It just means your single $X$ variable doesn't explain all the variance in $Y$. The trend can still be statistically valid and genuinely useful.

• My model has a 0.95 $R^2$ score. Does this prove that spending more on ads directly causes higher sales?

  No — never write 'causes' on an exam. Regression measures mathematical correlation, not real-world causation. A hidden third variable, like a holiday season, could be driving both metrics simultaneously. High $R^2$ only tells you the variables move together, not which one is pulling the strings.