最近为了对比神经网络在不同语言中的运行效率，于是手搓了一个简单模型在不同语言中的代码，在这儿记录一下。
基于MINIST手写数字数据集，采用3层全连接网络，由Pytorch、Python、C++、CUDA四个版本实现。
- 实验结果
- Pytorch版
- Python版
2.1.Numpy版
2.2.纯Python版 - C++版
- CUDA版
- 纯Python实现的全连接神经网络，矩阵没有使用Numpy，将高维数组全部以1维列表的形式进行处理。
- 为了简化代码，目前batchsize只能设置为1
- 完整代码，Github：
from functions import *
from layers import *
from dataloader import *
import time
class Model:
def __init__(self,model_size=(784,1024,1024,10)):
self.model_size=model_size
self.params={}
self.params['w1']=FC_init(self.model_size[0]*self.model_size[1])
self.params['b1']=[0.0]*self.model_size[1]
self.params['w2']=FC_init(self.model_size[1]*self.model_size[2])
self.params['b2']=[0.0]*self.model_size[2]
self.params['w3']=FC_init(self.model_size[2]*self.model_size[3])
self.params['b3']=[0.0]*self.model_size[3]
self.grads={}
self.grads['w1']=[0.0]*(self.model_size[0]*self.model_size[1])
self.grads['b1']=[0.0]*(self.model_size[1])
self.grads['w2']=[0.0]*(self.model_size[1]*self.model_size[2])
self.grads['b2']=[0.0]*(self.model_size[2])
self.grads['w3']=[0.0]*(self.model_size[2]*self.model_size[3])
self.grads['b3']=[0.0]*(self.model_size[3])
self.layer1=FC(self.params['w1'],self.model_size[0],self.model_size[1],self.params['b1'])
self.layer1_act=Relu()
self.layer2=FC(self.params['w2'],self.model_size[1],self.model_size[2],self.params['b2'])
self.layer2_act=Relu()
self.layer3=FC(self.params['w3'],self.model_size[2],self.model_size[3],self.params['b3'])
self.layer4=Softmax_with_loss()
def forward(self,x):
x = self.layer1.forward(x)
x = self.layer1_act.forward(x)
x = self.layer2.forward(x)
x = self.layer2_act.forward(x)
x = self.layer3.forward(x)
return x
def loss(self,x,t):
return self.layer4.forward(x,t)
def gradient(self,x,t):
y=self.forward(x)
loss_value=self.loss(y,t)
#反向传播部分
dout=self.layer4.backward()
dout=self.layer3.backward(dout)
dout=self.layer2_act.backward(dout)
dout=self.layer2.backward(dout)
dout=self.layer1_act.backward(dout)
dout=self.layer1.backward(dout)
#存储grads
self.grads['w1'],self.grads['b1']=self.layer1.dw,self.layer1.db
self.grads['w2'], self.grads['b2'] = self.layer2.dw, self.layer2.db
self.grads['w3'], self.grads['b3'] = self.layer3.dw, self.layer3.db
return loss_value
def SGD(self):
for key in self.params.keys():
SGD_update(self.grads[key],self.params[key])
DNN=Model()
(x_train, t_train), (x_test, t_test) = load_mnist(normalize=True, flatten=True, one_hot_label=True)
train_size = x_train.shape[0]
test_size = x_test.shape[0]
loss_list=[]
t1=time.time()
for i in range(train_size):
x=x_train[i].tolist()
t=t_train[i].tolist()
loss=DNN.gradient(x,t)
DNN.SGD()
# print(i,loss)
loss_list.append(loss)
if i%500==0:
print(i,"loss_avg:",sum(loss_list)/len(loss_list))
print("Time:",time.time()-t1)
loss_list=[]
t1=time.time()
dataloader.py
# coding: utf-8
try:
import urllib.request
except ImportError:
raise ImportError('You should use Python 3.x')
import os.path
import gzip
import pickle
import os
import numpy as np
url_base = 'http://yann.lecun.com/exdb/mnist/'
key_file = {
'train_img': 'dataset/train-images-idx3-ubyte.gz',
'train_label': 'dataset/train-labels-idx1-ubyte.gz',
'test_img': 'dataset/t10k-images-idx3-ubyte.gz',
'test_label': 'dataset/t10k-labels-idx1-ubyte.gz'
}
dataset_dir = os.path.dirname(os.path.abspath(__file__))
save_file = dataset_dir + "/mnist.pkl"
train_num = 60000
test_num = 10000
img_dim = (1, 28, 28)
img_size = 784
def _download(file_name):
file_path = dataset_dir + "/" + file_name
if os.path.exists(file_path):
return
print("Downloading " + file_name + " ... ")
urllib.request.urlretrieve(url_base + file_name, file_path)
print("Done")
def download_mnist():
for v in key_file.values():
_download(v)
def _load_label(file_name):
file_path = dataset_dir + "/" + file_name
print("Converting " + file_name + " to NumPy Array ...")
with gzip.open(file_path, 'rb') as f:
labels = np.frombuffer(f.read(), np.uint8, offset=8)
print("Done")
return labels
def _load_img(file_name):
file_path = dataset_dir + "/" + file_name
print("Converting " + file_name + " to NumPy Array ...")
with gzip.open(file_path, 'rb') as f:
data = np.frombuffer(f.read(), np.uint8, offset=16)
data = data.reshape(-1, img_size)
print("Done")
return data
def _convert_numpy():
dataset = {}
dataset['train_img'] = _load_img(key_file['train_img'])
dataset['train_label'] = _load_label(key_file['train_label'])
dataset['test_img'] = _load_img(key_file['test_img'])
dataset['test_label'] = _load_label(key_file['test_label'])
return dataset
def init_mnist():
'''
Note：已将数据集下载至本地，第一次加载会将数据集保存成pickle
'''
# download_mnist()
dataset = _convert_numpy()
print("Creating pickle file ...")
with open(save_file, 'wb') as f:
pickle.dump(dataset, f, -1)
print("Done!")
def _change_one_hot_label(X):
T = np.zeros((X.size, 10))
for idx, row in enumerate(T):
row[X[idx]] = 1
return T
def load_mnist(normalize=True, flatten=True, one_hot_label=False):
"""读入MNIST数据集
Parameters
----------
normalize : 将图像的像素值正规化为0.0~1.0
one_hot_label :
one_hot_label为True的情况下，标签作为one-hot数组返回
one-hot数组是指[0,0,1,0,0,0,0,0,0,0]这样的数组
flatten : 是否将图像展开为一维数组
Returns
-------
(训练图像, 训练标签), (测试图像, 测试标签)
"""
if not os.path.exists(save_file):
init_mnist()
with open(save_file, 'rb') as f:
dataset = pickle.load(f)
if normalize:
for key in ('train_img', 'test_img'):
dataset[key] = dataset[key].astype(np.float32)
dataset[key] /= 255.0
if one_hot_label:
dataset['train_label'] = _change_one_hot_label(dataset['train_label'])
dataset['test_label'] = _change_one_hot_label(dataset['test_label'])
if not flatten:
for key in ('train_img', 'test_img'):
dataset[key] = dataset[key].reshape(-1, 1, 28, 28)
print('训练图片数量：', dataset['train_img'].shape[0], '测试图片数量：', dataset['test_img'].shape[0])
return (dataset['train_img'], dataset['train_label']), (dataset['test_img'], dataset['test_label'])
if __name__ == '__main__':
data = load_mnist(normalize=True, flatten=True, one_hot_label=True)
print(type(data[0][0][0]))
layers.py
from functions import *
class FC:
def __init__(self,w,input_size,output_size,b):
self.w=w
self.b=b
self.input_size=input_size
self.output_size=output_size
self.x=None
self.dw=None
self.db=None
def forward(self,x):
self.x=x
output=[0.0]*self.output_size
for i in range(self.output_size):
for j in range(self.input_size):
output[i]+=x[j]*self.w[i+j*self.output_size]
output[i]+=self.b[i]
return output
def backward(self,dout):
dx=[0.0]*self.input_size
self.dw=[0.0]*(self.input_size*self.output_size)
w_T=matrix_transpose(self.w,self.input_size,self.output_size)
for i in range(self.input_size):
for j in range(self.output_size):
dx[i]+=dout[j]*w_T[i+j*self.input_size]
for i in range(self.input_size):
for j in range(self.output_size):
self.dw[i*self.output_size+j]=self.x[i]*dout[j]
self.db=dout
return dx
class Relu:
def __init__(self):
self.mask=None
def forward(self,x):
out=[0.0]*len(x)
self.mask=[0]*len(x)
for i in range(len(x)):
if x[i]>0:
out[i]=x[i]
self.mask[i]=1
else:
out[i]=0
self.mask[i]=0
return out
def backward(self,dout):
dx=[0.0]*len(dout)
for i in range(len(dout)):
if self.mask[i]>0:
dx[i]=dout[i]
else:
dx[i]=0
return dx
class Softmax_with_loss:
def __init__(self):
self.y=None
self.loss=None
self.t=None
def forward(self,x,t):
self.t=t
self.y=softmax(x)
loss_value=cross_entropy_error(self.y,self.t)
return loss_value
def backward(self):
dx=[0.0]*len(self.y)
for i in range(len(self.y)):
dx[i]=self.y[i]-self.t[i]
return dx
functions.py
import numpy as np
import math
import random
def matrix_transpose(a,shape0,shape1):
b=[0]*(shape0*shape1)
for i in range(shape1):
for j in range(shape0):
b[i*shape0+j]=a[i+shape1*j]
return b
def softmax(input):
max_value=max(input)
exp_a=[0.0]*len(input)
y=[0.0]*len(input)
for i in range(len(input)):
exp_a[i]=math.exp(input[i]-max_value)
sum_exp_a=sum(exp_a)
for i in range(len(input)):
y[i]=exp_a[i]/sum_exp_a
return y
def cross_entropy_error(y,t):
delta=1e-7
out=0
for i in range(len(y)):
if t[i]!=0:
out=-(t[i]*math.log(y[i]+delta))
return out
def FC_init(size):
array=[0.0]*size
for i in range(size):
array[i]=random.random()
return array
def SGD_update(grads,params):
learning_rate=0.01
if len(grads)!=len(params):
print("SGD错误，输入长度不一致")
else:
for i in range(len(grads)):
params[i]-=learning_rate*grads[i]
