Multiple Linear Regression Theory Guide

Rules & Common Mistakes

• Lab Trap: The Singular Matrix Error

  If your calculator or Python crashes trying to compute $(X^TX)^{-1}$, the matrix is singular — uninvertible. This almost always means two columns are perfectly correlated, like weight in kg and weight in lbs. The math breaks because it literally cannot separate their individual impacts on $\hat{Y}$.
• Exam Trap: The Illusion of a Rising $R^2$

  Adding any new variable — even today's cloud count — will mathematically force standard $R^2$ to stay flat or increase. It never penalizes garbage inputs. Always report Adjusted $R^2$ instead; it actively punishes you for adding variables that don't genuinely improve the model.
• Exam Trap: Big Coefficients Don't Mean Big Impact

  If $\beta_1=1000$ and $\beta_2=0.5$, feature 1 is not necessarily more important. Coefficients scale to match their units — millimeters vs miles produce wildly different numbers for the same real-world effect. To compare true feature importance, standardize your $X$ variables before fitting the model.
• Lab Trap: The Dummy Variable Trap

  When One-Hot Encoding categories — like 3 cities — always drop one column. All 3 dummy columns sum perfectly to 1, which clashes directly with the intercept's built-in column of 1s. That perfect correlation makes $X^TX$ singular and crashes the entire Normal Equation calculation.

Strengths, Weaknesses & When To Use It

When to use it:Multiple Linear Regression is the gold standard when the goal is predicting a continuous number and explaining exactly why that prediction was made — think proving to a bank precisely why a property is valued at a specific price. Every coefficient is auditable, defensible, and human-readable. But if interpretability is not the priority and the data is complex, curved, or highly non-linear, skip MLR entirely. A Random Forest or Neural Network will outperform it without breaking a sweat.

Multiple Linear Regression vs. Logistic Regression

This is the fundamental crossroads of machine learning. Multiple Linear Regression answers 'how much?' — predicting a continuous number with no ceiling or floor. Logistic Regression answers 'which one?' — predicting a probability to classify an outcome as yes or no.

• The Output — Infinite vs. Bounded: MLR outputs any real number from negative to positive infinity — a house price, a stock value, a temperature. Logistic Regression takes that same output and squashes it through a Sigmoid curve, compressing everything into a 0-to-1 probability. A 0.80 output means an 80% chance the event happens.
• The Exam Name Trap — It's Not Actually Regression: Despite the word 'Regression' in its name, Logistic Regression is a classification algorithm. If an exam question asks to predict a category — Spam vs. Not Spam, Malignant vs. Benign, Pass vs. Fail — and you reach for Multiple Linear Regression, that is an instant fail. The name is a deliberate trap.
• Hyperplane vs. Decision Boundary: MLR fits a hyperplane through the data, estimating the exact value of a continuous output at every point. Logistic Regression fits a boundary between groups of data points, separating one category from another. Same linear math underneath — completely different mission.

Implementation Pseudocode

// MULTIPLE LINEAR REGRESSION — Trading Loops for Linear Algebra
// Simple Regression scanned rows one by one in a FOR EACH loop.
// Multiple Regression replaces all of that with matrix operations.
// The Normal Equation calculates every single coefficient simultaneously
// in one mathematical sweep — intercept AND all feature slopes at once.

// ============================================================
// FUNCTION 1: TRAINING — Build the beta vector from matrices
// ============================================================
FUNCTION trainMultipleRegression(X_matrix, Y_vector):

    // --- STEP 1: Build the Design Matrix (Add the Column of 1s) ---
    FOR EACH row IN X_matrix:
        prepend 1 to the front of row
    END FOR
    // EXAM TRAP: This is the single most common exam mistake.
    // That leading 1 in every row is what allows the Normal Equation
    // to calculate the intercept b_0. Skip it and your entire beta
    // vector is wrong — every coefficient shifts to compensate.

    // --- STEP 2: Compute the Transpose ---
    X_transpose = transpose(X_matrix)
    // Flips rows into columns. A matrix of shape (n x d)
    // becomes (d x n). Required for the matrix multiplication to work.

    // --- STEP 3: Apply the Normal Equation ---

    // First: compute X_transpose * X_matrix
    XTX = matrixMultiply(X_transpose, X_matrix)

    // Second: invert the result
    XTX_inverse = matrixInverse(XTX)
    // LAB TRAP: If matrixInverse() throws a Singular Matrix error,
    // stop immediately. This means X^TX cannot be inverted.
    // Root cause is almost always Multicollinearity (two columns carry
    // identical information) or the Dummy Variable Trap (you kept all
    // dummy columns instead of dropping one). Fix the data, not the math.

    // Third: multiply inverse by X_transpose
    XTX_inv_XT = matrixMultiply(XTX_inverse, X_transpose)

    // Fourth: multiply by Y to get the final beta vector
    beta_vector = matrixMultiply(XTX_inv_XT, Y_vector)
    // beta_vector contains [b_0, b_1, b_2, ..., b_n] top to bottom.
    // b_0 is the intercept. Every value after it is a partial slope.

    RETURN beta_vector

END FUNCTION

// ============================================================
// FUNCTION 2: PREDICTION — Instant dot product
// ============================================================
FUNCTION predictMultipleRegression(beta_vector, unknown_X_vector):

    // --- STEP 1: Prepend 1 to match the training matrix shape ---
    prepend 1 to the front of unknown_X_vector
    // Must mirror exactly what was done during training.
    // That leading 1 pairs with b_0 to add the intercept to the result.

    // --- STEP 2: Dot product = instant prediction ---
    RETURN dotProduct(unknown_X_vector, beta_vector)
    // This is an O(d) operation where d = number of features.
    // Each feature value is multiplied by its coefficient and summed.
    // Trained on 10 rows or 10 million — prediction cost never changes.

END FUNCTION

Time & Space Complexity

MLR Exam & Lab Questions Students Always Get Wrong

• My intercept $b_0$ is negative, but my output (like house prices) cannot be negative. Is my math wrong?

  No — the math is fine. The intercept is purely the baseline value when every single $X$ feature equals exactly zero. If a zero-square-foot house doesn't exist in reality, that anchor point is just a mathematical necessity, not a real-world prediction. The hyperplane still fits correctly.

• I have a 'Color' column (Red, Green, Blue). Can I just change them to 1, 2, and 3 before training?

  Never. Encoding Red=1, Green=2, Blue=3 tells the algorithm that Blue is mathematically three times Red — a completely invented ranking that poisons every coefficient. Always use One-Hot Encoding to create binary columns, then drop one category to avoid the Dummy Variable Trap.

• Do I have to standardize or normalize my $X$ values before running the Normal Equation?

  For raw predictions, no — the coefficients automatically scale to match their units. But if the question asks which feature has the most impact or asks to compare $\beta$ values directly, standardize first. Without it, the feature measured in the largest units wins by default.

• What happens if my dataset has 100 rows but I feed it 120 different features?

  The Normal Equation crashes instantly. With more features than rows, $X^TX$ becomes singular and uninvertible — there are more unknowns than equations, so no unique solution exists. Fix it by dropping features, collecting more data, or switching to a penalized model like Ridge Regression.

• My $R^2$ score goes up every time I add a new feature, but my lab instructor still failed the model. Why?

  Standard $R^2$ is mathematically forced to rise with every new column added — even pure noise. The model isn't actually improving; the metric is just broken for this use case. Adjusted $R^2$ penalizes unnecessary features and is the only valid measure when comparing models with different feature counts.