【pytorch】使用numpy实现pytorch的softmax函数与cross

公式
S o f t m a x ( x i ) = exp ⁡ ( x i ) ∑ j exp ⁡ ( x j ) {Softmax}(x_{i}) = frac{exp(x_i)}{sum_j exp(x_j)} Softmax(xi​)=∑j​exp(xj​)exp(xi​)​

import numpy as np
import torch
import torch.nn.functional as F


def my_softmax(x):
    exp_x = np.exp(x)
    return exp_x/np.sum(exp_x)

x = np.array([1.5, 2.2, 3.1, 0.9, 1.2, 1.7])
xt = torch.from_numpy(x)


print('F.softmax:', F.softmax(xt))
print('my_softmax', my_softmax(x))

结果一致

从源码上来看，torch.nn.functional.F.softmax实际上调用的是Tensor自身的softmax函数

公式
L o g _ s o f t m a x ( x i ) = l n exp ⁡ ( x i ) ∑ j exp ⁡ ( x j )            ( 1 )                                         = x i − l n ∑ j exp ⁡ ( x j )            ( 2 ) {Log_softmax}(x_{i}) = lnfrac{exp(x_i)}{sum_j exp(x_j)} space space space space space space space space space space (1) newline space space space space space space space space space space space space space space space space space space space space space space space space space space space space space space space space space space space space space space space = x_i - ln{sum_j exp(x_j)} space space space space space space space space space space (2) Log_softmax(xi​)=ln∑j​exp(xj​)exp(xi​)​          (1)                                       =xi​−lnj∑​exp(xj​)          (2)

import numpy as np
import torch
import torch.nn.functional as F



def my_softmax(x):
    exp_x = np.exp(x)
    return exp_x/np.sum(exp_x)

def my_log_softmax1(x):
    return np.log(my_softmax(x))

def my_log_softmax2(x):
    return x-np.log(np.sum(np.exp(x)))

x = np.array([1.5, 2.2, 3.1, 0.9, 1.2, 1.7])
xt = torch.from_numpy(x)


print('F.softmax:', F.log_softmax(xt))
print('my_softmax', my_log_softmax1(x))
print('my_softmax', my_log_softmax2(x))

结果一致

This criterion combines log_softmax and nll_loss in a single function

官网的公式如下：
ℓ ( x , y ) = L = { l 1 , … , l N } ⊤ , l n = − w y n log ⁡ exp ⁡ ( x n , y n ) ∑ c = 1 C exp ⁡ ( x n , c ) ⋅ 1 { y n ≠ ignore_index } ell(x, y) = L = {l_1,dots,l_N}^top, newline quad l_n = - w_{y_n} log frac{exp(x_{n,y_n})}{sum_{c=1}^C exp(x_{n,c})} cdot mathbb{1}{y_n not= text{ignore_index}} ℓ(x,y)=L={l1​,…,lN​}⊤,ln​=−wyn​​log∑c=1C​exp(xn,c​)exp(xn,yn​​)​⋅1{yn​​=ignore_index}

这里有权重参数weights，我们简单来看先省去这个weights。即
l n = − log ⁡ exp ⁡ ( x n , y n ) ∑ c = 1 C exp ⁡ ( x n , c ) ⋅ 1 { y n ≠ ignore_index } quad l_n = - log frac{exp(x_{n,y_n})}{sum_{c=1}^C exp(x_{n,c})} cdot mathbb{1}{y_n not= text{ignore_index}} ln​=−log∑c=1C​exp(xn,c​)exp(xn,yn​​)​⋅1{yn​​=ignore_index}

这个公式是这么来的呢？

首先cross_entropy并不是深度学习里独有的，它是个数学公式如下：
假设p和q两组概率集合
c r o s s _ e n t r o p y ( p , q ) = − ∑ k p ( k ) l o g ( q ( k ) ) cross_entropy(p, q) = -sum_{k}{p(k) log(q(k)}) cross_entropy(p,q)=−k∑​p(k)log(q(k))
换到我们的上下文中，Y是标签， P是观测概率集合，则有下面
c r o s s _ e n t r o p y ( Y , P ) = − ∑ k Y ( k ) l o g ( P ( k ) ) cross_entropy(Y, P) = -sum_{k}{Y(k) log(P(k)}) cross_entropy(Y,P)=−k∑​Y(k)log(P(k))
Y实际上是会转为one hot形式的向量，所以 对于y=k的情况下，Y(k) = 1;其他都是 Y(k) =0,则有如下，
c r o s s _ e n t r o p y ( Y , P ) = − ∑ k l o g ( P ( k ) )    cross_entropy(Y, P) = - sum_{k}{log(P(k)}) spacespace cross_entropy(Y,P)=−k∑​log(P(k))  

但是一般而言，这里的P(y)都会使用softmax来计算，所以有
c r o s s _ e n t r o p y ( Y , P ) = − ∑ k l o g _ s o f t m a x ( P ( k ) ) ) cross_entropy(Y, P) = - sum_{k}{log_softmax(P(k))}) cross_entropy(Y,P)=−k∑​log_softmax(P(k)))

参考文档

1. https://zhuanlan.zhihu.com/p/145341251
2. https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss
3. https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/