Logistic Regression

Introduction

Logistic regression is a classification algorithm that predicts the probability of a binary outcome. Despite its name, it's used for classification, not regression. It uses the sigmoid function to map any input to a value between 0 and 1.

The Sigmoid Function

The sigmoid (logistic) function is the key to logistic regression:

σ(z) = 1 / (1 + e^(-z))

Properties:

  • Output range: (0, 1)
  • σ(0) = 0.5
  • Smooth, differentiable
  • Has a nice derivative: σ'(z) = σ(z)(1 - σ(z))

The sigmoid function maps any real value to (0, 1)

Interactive Logistic Regression

Click on the left (red region) to add class 0 points, or on the right (blue region) for class 1 points:

Blue curve: sigmoid function | Purple line: decision boundary (p = 0.5)

Controls

Model Parameters

Weight (w) = 0.000
Bias (b) = 0.000
Cross-Entropy Loss = 0.0000

Binary Cross-Entropy Loss

Logistic regression uses binary cross-entropy as its loss function:

L = -[y log(p) + (1-y) log(1-p)]

Where:

  • y: true label (0 or 1)
  • p: predicted probability

This loss function penalizes confident wrong predictions more heavily than uncertain predictions.

From Linear to Logistic

Linear Regression

  • Predicts continuous values
  • Output: y = wx + b
  • Range: (-∞, +∞)
  • Loss: Mean Squared Error

Logistic Regression

  • Predicts probabilities
  • Output: p = σ(wx + b)
  • Range: (0, 1)
  • Loss: Cross-Entropy

Key Takeaways

  • Logistic regression is a linear classifier that outputs probabilities
  • The sigmoid function maps linear combinations to probabilities
  • Decision boundary occurs where the probability equals 0.5
  • Cross-entropy loss is appropriate for probability predictions
  • Can be extended to multi-class classification (softmax regression)
  • Forms the basis for neural network output layers in classification tasks