ReLU
Also: rectified linear unit, Leaky ReLU
The rectified linear unit, max(0, x), is the default hidden-layer activation in deep networks: it is cheap to compute and its gradient is exactly 1 for positive inputs, which eases the vanishing-gradient problem. Leaky ReLU keeps a small slope for negatives so units never go fully silent.
ReLU(x) = max(0, x)
Example
ReLU([−2, 0.5, 3]) = [0, 0.5, 3]. The gradient is 0 for the −2 entry and 1 for the other two.