Modeling, Inference, Learning
什么是深度学习？
什么是神经网络？
激活函数
前向神经网络
卷积神经网络
Recurrent Neural Networks（RNN循环神经网络)
Seq2Seq with Attention
02 用PyTorch构建深度学习模型PyTorch可以做什么？
- 深度学习模型框架
TensorFlow ay/net
PyTorch Caffe
mxnet Keras - PyTorch与其他框架的对比
（1）PyTorch：动态计算图 Dynamic Computation Graph
Tensorflow：静态计算图 Static Computation Graph
（2）PyTorch代码通俗易懂，非常接近Python原生代码，不会让人感觉是 完全在学习一门新的语言。
（3）拥有Facebook支持，社区活跃。
- PyTorch
PyTorch是一个基于Python的科学计算库，特点：
①类似于NumPy，但是它可以使用GPU；
②可以用它定义深度学习模型，可以灵活地进行深度学习模型的训练和使用。
常用的操作
（1）构造一个未初始化的5x3矩阵：
x = torch.empty(5，3) print(x)
结果：
tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00],
[0.0000e+00, 4.7339e+30, 1.4347e-19],
[2.7909e+23, 1.8037e+28, 1.7237e+25],
[9.1041e-12, 6.2609e+22, 4.7428e+30],
[3.8001e-39, 0.0000e+00, 0.0000e+00]])
（2）构建一个随机初始化的矩阵：
x = torch.rand(5, 3) print(x)
结果：
tensor([[0.4821, 0.3854, 0.8517],
[0.7962, 0.0632, 0.5409],
[0.8891, 0.6112, 0.7829],
[0.0715, 0.8069, 0.2608],
[0.3292, 0.0119, 0.2759]])
（3）构建一个全部为0，类型为long的矩阵：
x = torch.zeros(5, 3, dtype=torch.long) print(x)
结果：
tensor([[0, 0, 0],
[0, 0, 0],
[0, 0, 0],
[0, 0, 0],
[0, 0, 0]])
（4）从数据直接直接构建tensor：
x = torch.tensor([5.5, 3]) print(x)
结果：
tensor([5.5000, 3.0000])
（5）从一个已有的tensor构建一个tensor。这些方法会重用原来tensor的特征，例如，数据类型，除非提供新的数据。
x = x.new_ones(5, 3, dtype=torch.double) # new_* methods take in sizes print(x) x = torch.randn_like(x, dtype=torch.float) # override dtype! print(x) # result has the same size
结果：
tensor([[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.]], dtype=torch.float64)
tensor([[ 1.4793, -2.4772, 0.9738],
[ 2.0328, 1.3981, 1.7509],
[-0.7931, -0.0291, -0.6803],
[-1.2944, -0.7352, -0.9346],
[ 0.5917, -0.5149, -1.8149]])
（6）加法运算
y = torch.rand(5, 3) print(x + y)
结果：
tensor([[ 1.7113, -1.5490, 1.4009],
[ 2.4590, 1.6504, 2.6889],
[-0.3609, 0.4950, -0.3357],
[-0.5029, -0.3086, -0.1498],
[ 1.2850, -0.3189, -0.8868]])
（7）另一种加法的操作
`print(torch.add(x, y)`)
结果：
tensor([[ 1.7113, -1.5490, 1.4009],
[ 2.4590, 1.6504, 2.6889],
[-0.3609, 0.4950, -0.3357],
[-0.5029, -0.3086, -0.1498],
[ 1.2850, -0.3189, -0.8868]])
（8）in-place加法
adds x to yy.add_(x) print(y)
结果：
tensor([[ 1.7113, -1.5490, 1.4009],
[ 2.4590, 1.6504, 2.6889],
[-0.3609, 0.4950, -0.3357],
[-0.5029, -0.3086, -0.1498],
[ 1.2850, -0.3189, -0.8868]])
（9）如果只有一个元素的tensor，使用.item()方法可以把里面的value变成Python数值。
x = torch.randn(1) print(x) print(x.item())
结果：
tensor([0.4726]) 0.4726296067237854
（10）Numpy和Tensor之间的转化
在Torch Tensor和NumPy array之间相互转化非常容易。Torch Tensor和NumPy array会共享内存，所以改变其中一项也会改变另一项。把Torch Tensor转变成NumPy Array
a = torch.ones(5) print(a) b = a.numpy() print(b) a.add_(1) print(a) print(b) import numpy as np a = np.ones(5) b = torch.from_numpy(a) np.add(a, 1, out=a) print(a) print(b)
结果：
tensor([1., 1., 1., 1., 1.]) [1. 1. 1. 1. 1.] tensor([2., 2., 2., 2., 2.]) [2. 2. 2. 2. 2.] [2. 2. 2. 2. 2.] tensor([2., 2., 2., 2., 2.], dtype=torch.float64)
第1课课后复习导图
用numpy实现两层神经网络
一个全连接ReLU神经网络，一个隐藏层，没有bias。用来从x预测y，使用L2 Loss。
这一实现完全使用numpy来计算前向神经网络，loss，和反向传播。
numpy ndarray是一个普通的n维array。它不知道任何关于深度学习或者梯度(gradient)的知识，也不知道计算图(computation graph)，只是一种用来计算数学运算的数据结构。
import numpy as np
# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10
# Create random input and output data
x = np.random.randn(N, D_in)
y = np.random.randn(N, D_out)
# Randomly initialize weights
w1 = np.random.randn(D_in, H)
w2 = np.random.randn(H, D_out)
learning_rate = 1e-6
for t in range(500):
# Forward pass: compute predicted y
h = x.dot(w1)
h_relu = np.maximum(h, 0)
y_pred = h_relu.dot(w2)
# Compute and print loss
loss = np.square(y_pred - y).sum()
print(t, loss)
# Backprop to compute gradients of w1 and w2 with respect to loss
# loss = (y_pred - y) ** 2
grad_y_pred = 2.0 * (y_pred - y)
#
grad_w2 = h_relu.T.dot(grad_y_pred)
grad_h_relu = grad_y_pred.dot(w2.T)
grad_h = grad_h_relu.copy()
grad_h[h < 0] = 0
grad_w1 = x.T.dot(grad_h)
# Update weights
w1 -= learning_rate * grad_w1
w2 -= learning_rate * grad_w2
结果：
0 34399246.46047344 1 29023199.257758312 2 25155679.85447208 3 20344203.603057466 4 14771404.625789404 5 9796072.99431371 6 6194144.749997159 7 3948427.3657580013 8 2637928.1726997104 9 1879876.2597949505 10 1424349.925182723 11 1131684.579785501 12 930879.9521737935 13 783503.167740541 14 669981.8287784329 15 579151.6288421676 16 504610.5781504087 17 442295.18952143926 18 389647.44224490353 19 344718.3535892912 20 306120.2245707266 21 272728.24885829526 22 243778.8617292929 23 218485.92082002352 24 196304.70602822883 25 176774.2980280186 26 159509.34934842546 27 144200.52956072442 28 130597.06878493169 29 118484.47548850597 30 107661.24303895692 31 97973.75762285746 32 89291.0096051952 33 81500.46898789635 34 74477.4654945682 35 68139.90452489533 36 62418.87519034026 37 57241.53801123622 38 52545.34658231941 39 48280.5552386464 40 44399.73653914068 41 40864.495617471934 42 37640.08489317873 43 34695.77852549495 44 32004.894008637555 45 29545.09481447049 46 27292.93700341219 47 25232.87780747312 48 23342.570881009553 49 21606.76105421809 50 20015.62357395961 51 18551.83281521863 52 17204.31407669751 53 15962.736948706759 54 14818.242254751764 55 13762.251705340486 56 12787.060032590252 57 11885.95797873141 58 11053.123737613136 59 10282.617503272711 60 9569.805676161515 61 8909.467534754986 62 8297.782408129178 63 7731.121277369748 64 7205.863671952578 65 6718.146999962471 66 6265.473531640673 67 5845.100232214373 68 5454.557838660972 69 5091.658572234415 70 4754.393958028546 71 4440.682575260731 72 4148.70793229529 73 3877.022931816484 74 3624.088506535617 75 3388.5746286682042 76 3169.088547995476 77 2964.637382505168 78 2774.073275503305 79 2596.433385302534 80 2430.76267026859 81 2276.0929913609607 82 2131.752323451521 83 1997.0334011418258 84 1871.251515936368 85 1753.7448614349362 86 1643.919519574932 87 1541.289735192464 88 1445.3733798948592 89 1355.6688030350501 90 1271.7809967407718 91 1193.2972539295215 92 1119.8689894828083 93 1051.1890596219616 94 986.9044505648076 95 926.7286776893059 96 870.3673474483486 97 817.5707566117906 98 768.1200077715573 99 721.7693127164074 100 678.327084576388 101 637.5984844921132 102 599.3471700265131 103 563.480144773489 104 529.8443636950776 105 498.2900261297218 106 468.69076555164696 107 440.9141759159077 108 414.8406348102356 109 390.36201589159975 110 367.377986904459 111 345.794153113363 112 325.53293001498525 113 306.50254567681907 114 288.6084052220689 115 271.79635277136003 116 255.99599171996437 117 241.14900382305748 118 227.1967308563582 119 214.07331707113855 120 201.72960304299005 121 190.1215354419851 122 179.20332461067048 123 168.93031453109492 124 159.26460190296683 125 150.18028740393805 126 141.6310351604903 127 133.5845565128207 128 126.00708973959819 129 118.87233941614235 130 112.15340039819878 131 105.82589029792051 132 99.86499912782936 133 94.24914703127945 134 88.95813583258435 135 83.97204514587689 136 79.27316202972057 137 74.84491080200985 138 70.66991561621043 139 66.7338564785546 140 63.02257660379944 141 59.52313997962988 142 56.22322968024125 143 53.111763701729714 144 50.175933060002905 145 47.40606340622237 146 44.793256923660664 147 42.328047476976025 148 40.00144856286997 149 37.80536851048289 150 35.73343107253782 151 33.77738183530186 152 31.930774425664392 153 30.187309589532056 154 28.541632056826323 155 26.987624733348596 156 25.52056466134328 157 24.135140772349633 158 22.82592867935182 159 21.589141054848028 160 20.42054265142189 161 19.31668484832083 162 18.27373881475532 163 17.288158658562804 164 16.35722592792538 165 15.47708242556972 166 14.64564401606641 167 13.859533873388633 168 13.116375098259695 169 12.413894134407165 170 11.749589132060152 171 11.121654032769754 172 10.527755403701851 173 9.966244729776452 174 9.43532189744742 175 8.933071972010405 176 8.458011087399335 177 8.008768655278885 178 7.583910598664408 179 7.181953229124071 180 6.801495862472622 181 6.441562835264972 182 6.100994571216053 183 5.778713643608878 184 5.473781077364653 185 5.185201108079436 186 4.910778594973227 187 4.651157323562962 188 4.405448410225631 189 4.172946150445695 190 3.953151092232221 191 3.744908981306408 192 3.5478244092334004 193 3.3613194717897596 194 3.1847221013676847 195 3.017529877609752 196 2.859281816477389 197 2.7094552588397214 198 2.567567563134434 199 2.4332311668647577 200 2.3060182574075156 201 2.1855656999370474 202 2.0714842934675706 203 1.963429527723876 204 1.8610826001126994 205 1.7641435814092026 206 1.6723644556551025 207 1.5854221408589446 208 1.5030539196422792 209 1.425000717598524 210 1.351040918987503 211 1.2809841760961524 212 1.2146023546998657 213 1.151699249786657 214 1.09209327048869 215 1.035622604251914 216 0.9821312023078086 217 0.931398400382507 218 0.8833282657308957 219 0.8377648680868359 220 0.7946102894611751 221 0.7536759265506683 222 0.7148752979674705 223 0.6781061363107517 224 0.6432513423030822 225 0.6102056251947237 226 0.5788694783475372 227 0.5491564979966296 228 0.5209877387920085 229 0.49428305592302185 230 0.4689618516832248 231 0.4449467772589872 232 0.4221771538530943 233 0.4005837129745524 234 0.3801063836563353 235 0.3606948589685684 236 0.34227967093321354 237 0.3248108971814142 238 0.3082453595318127 239 0.29253330742340067 240 0.27763364296473714 241 0.26349628059737484 242 0.25008661105388497 243 0.23736707183615013 244 0.2253000957467423 245 0.2138523361624442 246 0.20298921216044458 247 0.19268673137415065 248 0.1829142719891026 249 0.17364132891347556 250 0.1648403028301571 251 0.1564884304843211 252 0.14856550672688007 253 0.14104626799439057 254 0.13391149996667578 255 0.12714129899063353 256 0.12071506432087209 257 0.11461697949344261 258 0.10882972424002835 259 0.10333904654553265 260 0.09812616827155932 261 0.0931780565261463 262 0.08848217530870164 263 0.08402546175986916 264 0.07979485919700613 265 0.07577840283923465 266 0.07196528340443757 267 0.06834633090302783 268 0.06491078821045225 269 0.06164958715982291 270 0.05855323394724137 271 0.055613749333235304 272 0.05282348135771675 273 0.050174186967151216 274 0.04765819132181068 275 0.045271438294485024 276 0.04300319389006948 277 0.04085019680263658 278 0.03880575227345778 279 0.03686467008847949 280 0.035020834986065126 281 0.033270091765219535 282 0.03160778576828356 283 0.03002859285883898 284 0.028528860354370092 285 0.02710476057141562 286 0.025752260418639834 287 0.024468202569374195 288 0.023248340960852463 289 0.022089463426076428 290 0.020988894619024936 291 0.0199437951726752 292 0.018950933816160725 293 0.01800781808698391 294 0.01711204608500793 295 0.016261162547766717 296 0.015452712469537512 297 0.014684969147452612 298 0.013955631973752351 299 0.013262568423046399 300 0.012604847094777376 301 0.011979553027742364 302 0.011385439315700433 303 0.010820996391120769 304 0.010284594931599429 305 0.009774981235579705 306 0.009290757906358697 307 0.008830808784232147 308 0.008393777417063167 309 0.007978536761423337 310 0.007583920779683997 311 0.007208915598860074 312 0.006852552626458435 313 0.006513949532633323 314 0.006192232365425878 315 0.005886426734936253 316 0.0055958430902459 317 0.005319724577409574 318 0.00505742623277029 319 0.004808048657916753 320 0.004571049645157161 321 0.00434578735628591 322 0.00413168646938554 323 0.003928177611013847 324 0.0037348015880361344 325 0.0035511516515125325 326 0.00337649547244047 327 0.003210468217165082 328 0.0030526001815046697 329 0.0029025877721886788 330 0.002759953496700082 331 0.0026243799521717052 332 0.0024955282428992384 333 0.002373029911103254 334 0.0022565698554979346 335 0.0021458674153799783 336 0.0020406457772717784 337 0.0019405998694530123 338 0.0018454766429137211 339 0.001755053210723282 340 0.0016691117022354827 341 0.0015873764219450975 342 0.001509661141624596 343 0.0014357855252587638 344 0.0013655404837908268 345 0.0012987690170491828 346 0.0012352672659498882 347 0.0011748904401224442 348 0.0011174828717238274 349 0.001062929250585763 350 0.0010110289742089754 351 0.0009616701215304243 352 0.0009147416941442063 353 0.0008701229479127731 354 0.0008276969799764454 355 0.0007873380259436494 356 0.0007489547972389264 357 0.000712451804667229 358 0.000677739371845171 359 0.000644728435464334 360 0.0006133353967986715 361 0.000583485133709109 362 0.0005550950412157807 363 0.0005280932731431832 364 0.0005024110280030845 365 0.00047798591228805434 366 0.0004547507774551706 367 0.0004326546423136559 368 0.0004116382458083261 369 0.0003916440959886334 370 0.0003726296356534275 371 0.0003545443586216977 372 0.000337347352488608 373 0.00032099061370803334 374 0.0003054229784132819 375 0.00029061647064382485 376 0.0002765299098361774 377 0.0002631327221101076 378 0.0002503865963973947 379 0.0002382599294869431 380 0.00022672670184804494 381 0.00021575299560298047 382 0.00020531375263207438 383 0.000195381616896771 384 0.00018593500698085453 385 0.00017694494225329907 386 0.00016839225855899982 387 0.00016025517275686525 388 0.00015251350815142156 389 0.0001451491411549753 390 0.0001381428245892601 391 0.00013147417414693054 392 0.00012512977608770297 393 0.00011909308605343111 394 0.00011334857979979945 395 0.00010788480695473414 396 0.00010268704883570024 397 9.773868892276339e-05 398 9.303020197524704e-05 399 8.85491663624475e-05 400 8.428485316645869e-05 401 8.022778747190388e-05 402 7.636668153099922e-05 403 7.269236014951034e-05 404 6.919607836124983e-05 405 6.586822433827189e-05 406 6.270255584885866e-05 407 5.968876673919747e-05 408 5.682053107105221e-05 409 5.409095915616159e-05 410 5.1493171389161614e-05 411 4.902052422909128e-05 412 4.666751255362057e-05 413 4.442778311925004e-05 414 4.229665906499858e-05 415 4.026848185397114e-05 416 3.833731276858471e-05 417 3.6499489902161296e-05 418 3.47508278392673e-05 419 3.308721418833935e-05 420 3.150227517115802e-05 421 2.9993581650856873e-05 422 2.855746554164107e-05 423 2.719082047553995e-05 424 2.5889677603252346e-05 425 2.4650964056310747e-05 426 2.3472062284026367e-05 427 2.234982200401504e-05 428 2.1281362386681205e-05 429 2.026427108363315e-05 430 1.9295935390705645e-05 431 1.837421991578651e-05 432 1.7497081369101593e-05 433 1.6661834251218388e-05 434 1.586646319751679e-05 435 1.5109303894458321e-05 436 1.4388405284291706e-05 437 1.370208486427819e-05 438 1.3048788200557073e-05 439 1.2426643170448882e-05 440 1.1834592393040145e-05 441 1.1270674124878297e-05 442 1.0733978520453249e-05 443 1.0222848634947958e-05 444 9.736143369136732e-06 445 9.27267469036151e-06 446 8.831344021766777e-06 447 8.411269141029111e-06 448 8.01120400023348e-06 449 7.630301830674667e-06 450 7.267514155906956e-06 451 6.922046453620718e-06 452 6.593111073936982e-06 453 6.279880429162564e-06 454 5.981637837795765e-06 455 5.697647106767495e-06 456 5.427145298606709e-06 457 5.169599426636958e-06 458 4.924305367745114e-06 459 4.690669682311872e-06 460 4.468174841019597e-06 461 4.256343870589204e-06 462 4.054581997203786e-06 463 3.862428795746305e-06 464 3.67941417005127e-06 465 3.50524041752366e-06 466 3.339264640996386e-06 467 3.1811641594010322e-06 468 3.0305596775569415e-06 469 2.8871296080016895e-06 470 2.7505179893217363e-06 471 2.6204108493977605e-06 472 2.496484420691004e-06 473 2.3784400359079537e-06 474 2.266014000554243e-06 475 2.1589189914752615e-06 476 2.0569210781071096e-06 477 1.959750825498873e-06 478 1.867203798935969e-06 479 1.7790548691072608e-06 480 1.6950719025924979e-06 481 1.6150665241638997e-06 482 1.5388752061694276e-06 483 1.4662834016989446e-06 484 1.3971344832895556e-06 485 1.3312570753140638e-06 486 1.2684946752376657e-06 487 1.2087236535163552e-06 488 1.1517952524068044e-06 489 1.0975341827852709e-06 490 1.0458478627989778e-06 491 9.966057254836819e-07 492 9.49690406945525e-07 493 9.04995815168244e-07 494 8.624191220382796e-07 495 8.218529042471487e-07 496 7.831982460890191e-07 497 7.463699242750524e-07 498 7.112838972272693e-07 499 6.778580009634641e-07暑期编程PK赛 得CSDN机械键盘等精美礼品！
