Linear Regression

Introduction

Linear regression is one of the simplest and most fundamental algorithms in machine learning. It models the relationship between variables by fitting a linear equation to observed data.

The Mathematics

For simple linear regression with one input variable, we model:

y = mx + b

Where:

  • y: predicted output
  • x: input feature
  • m: slope (weight)
  • b: y-intercept (bias)

We find the best parameters by minimizing the Mean Squared Error (MSE):

MSE = (1/n) × Σ(yᵢ - ŷᵢ)²

Interactive Linear Regression

Click to add data points, then train the model using gradient descent or fit analytically:

Click to add data points

Controls

Model Parameters

Slope (m) = 0.000
Intercept (b) = 0.000
MSE Loss = 0.0000

Tips:

  • Red lines show residuals (errors)
  • Gradient descent iteratively improves the fit
  • Analytical solution gives the optimal fit instantly

Gradient Descent vs Analytical Solution

Gradient Descent

  • Iterative optimization algorithm
  • Works for any differentiable loss function
  • Scales to large datasets and complex models
  • Foundation for training neural networks
  • Requires tuning learning rate

Analytical Solution

  • Closed-form solution using calculus
  • Gives optimal parameters directly
  • Only works for linear regression
  • Computationally expensive for large datasets
  • No hyperparameters to tune

Key Takeaways

  • Linear regression finds the best-fitting line through data points
  • The goal is to minimize the squared errors between predictions and actual values
  • Gradient descent is a general optimization technique that works for many ML algorithms
  • Understanding linear regression helps grasp concepts like loss functions, optimization, and overfitting
  • It's the foundation for more complex algorithms like neural networks