# 常用激活函数及其梯度可视化

5924

## Sigmoid

$$\text{Sigmoid}(x) = \sigma(x) = \frac{1}{1 + e^{-x}}$$

## LogSigmoid

$$\text{LogSigmoid}(x) = \log\left(\frac{ 1 }{ 1 + e^{-x}}\right)$$

## SoftSign

$$\text{SoftSign}(x) = \frac{x}{ 1 + |x|}$$

## HardSigmoid

$$\text{Hardsigmoid}(x) = \begin{cases} 0 & \text{if~} x \le -3, \\ 1 & \text{if~} x \ge +3, \\ x / 6 + 1 / 2 & \text{otherwise} \end{cases}$$

## Tanh

$$\text{Tanh}(x) = \tanh(x) = \frac{e^{x} - e^{-x}} {e^{x} + e^{-x}}$$

## ReLU

$$\text{ReLU}(x) = \max(0, x)$$

## ReLU6

$$\text{ReLU6}(x) = \min(\max(0,x), 6)$$

## PReLU

$$f(x)= \begin{cases} x, &x \gt 0\\ ax, & x \le 0 \end{cases}$$

$PReLU$的参数$a$在训练中学习得到。

## Leaky ReLU

$$\text{LeakyRELU}(x) = \begin{cases} x, & \text{ if } x \geq 0 \\ \alpha x, & \text{ otherwise } \end{cases}$$

## RReLU

$$\text{RReLU}(x) = \begin{cases} x & \text{if } x \geq 0 \\ ax & \text{ otherwise }, a \sim \mathcal{U}(\text{lower}, \text{upper}) \end{cases}$$

## ELU

$$f(x)= \begin{cases} x, &x \gt 0\\ \alpha(e^x - 1), & x \le 0 \end{cases}$$

## CELU

$$f(x)= \begin{cases} x, &x \gt 0\\ \alpha(e^{\frac{x}{\alpha}} - 1), & x \le 0 \end{cases}$$

## SELU

$$\text{SELU}(x) = \text{scale} * (\max(0,x) + \min(0, \alpha * (\exp(x) - 1)))$$

## GeLU

$$\text{GELU}(x) = x * \Phi(x), \Phi(x) \text{为高斯分布的累积分布函数}$$

## SoftPlus

$$\text{Softplus}(x) = \frac{1}{\beta} * \log(1 + e^{\beta * x})$$

## Mish

$$\text{Mish}(x) = x * \text{Tanh}(\text{Softplus}(x))$$

## Silu

$$\text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ 为 sigmoid.}$$

## HardSwish

$$\text{Hardswish}(x) = \begin{cases} 0 & \text{if~} x \le -3, \\ x & \text{if~} x \ge +3, \\ x \cdot (x + 3) /6 & \text{otherwise} \end{cases}$$

## TanhShrink

$$\text{Tanhshrink}(x) = x - \tanh(x)$$

## SoftShrink

$$\text{SoftShrinkage}(x) = \begin{cases} x - \lambda, & \text{ if } x \gt \lambda \\ x + \lambda, & \text{ if } x \lt -\lambda \\ 0, & \text{ otherwise } \end{cases}$$

## HardShrink

$$\text{HardShrink}(x) = \begin{cases} x, & \text{ if } x \gt \lambda \\ x, & \text{ if } x \lt -\lambda \\ 0, & \text{ otherwise } \end{cases}$$

Article Tags
[本]通信工程@河海大学 & [硕]CS@清华大学

0
5924
0

More Recommendations

July 18, 2022
June 15, 2022
June 8, 2022
June 1, 2022