深度学习入门：基于Python的理论与实现.pdf

PDF: https://github.com/jash-git/Jash-good-idea-20200304-001

程序可以从以下网址下载 http://www.ituring.com.cn/book/1921 [https://github.com/jash-git/Jash-good-idea-20200304-001]

第1 章　Python入门· ··········································· 1
1.1 Python是什么· ········································· 1
1.2 Python的安装· ········································· 2
1.2.1　Python版本· ····································· 2
1.2.2　使用的外部库· ···································· 2
1.2.3　Anaconda发行版· ································· 3
1.3 Python解释器· ········································· 4
1.3.1　算术计算········································· 4
1.3.2　数据类型········································· 5
1.3.3　变量············································ 5
1.3.4　列表············································ 6
1.3.5　字典············································ 7
1.3.6　布尔型·········································· 7
1.3.7　if 语句· ·········································· 8
1.3.8　for 语句········································· 8
1.3.9　函数············································ 9
1.4 Python脚本文件· ······································· 9
vi 目录
1.4.1　保存为文件······································· 9
1.4.2　类· ············································ 10
1.5 NumPy· ·············································· 11
1.5.1　导入NumPy· ···································· 11
1.5.2　生成NumPy数组· ································ 12
1.5.3　NumPy 的算术运算······························· 12
1.5.4　NumPy的N维数组· ······························ 13
1.5.5　广播··········································· 14
1.5.6　访问元素········································ 15
1.6 Matplotlib············································ 16
1.6.1　绘制简单图形· ··································· 16
1.6.2　pyplot 的功能· ··································· 17
1.6.3　显示图像········································ 18
1.7 小结················································· 19
第2 章　感知机················································ 21
2.1 感知机是什么· ········································· 21
2.2 简单逻辑电路· ········································· 23
2.2.1　与门··········································· 23
2.2.2　与非门和或门· ··································· 23
2.3 感知机的实现· ········································· 25
2.3.1　简单的实现······································ 25
2.3.2　导入权重和偏置· ································· 26
2.3.3　使用权重和偏置的实现· ··························· 26
2.4 感知机的局限性· ······································· 28
2.4.1　异或门········································· 28
2.4.2　线性和非线性· ··································· 30
2.5 多层感知机············································ 31
2.5.1　已有门电路的组合· ······························· 31
目录 vii
2.5.2　异或门的实现· ··································· 33
2.6 从与非门到计算机· ····································· 35
2.7 小结················································· 36
第3 章　神经网络·············································· 37
3.1 从感知机到神经网络· ··································· 37
3.1.1　神经网络的例子· ································· 37
3.1.2　复习感知机······································ 38
3.1.3　激活函数登场· ··································· 40
3.2 激活函数·············································· 42
3.2.1　sigmoid 函数· ···································· 42
3.2.2　阶跃函数的实现· ································· 43
3.2.3　阶跃函数的图形· ································· 44
3.2.4　sigmoid 函数的实现· ······························ 45
3.2.5　sigmoid 函数和阶跃函数的比较······················ 46
3.2.6　非线性函数······································ 48
3.2.7　ReLU函数· ····································· 49
3.3 多维数组的运算· ······································· 50
3.3.1　多维数组········································ 50
3.3.2　矩阵乘法········································ 51
3.3.3　神经网络的内积· ································· 55
3.4 3 层神经网络的实现· ···································· 56
3.4.1　符号确认········································ 57
3.4.2　各层间信号传递的实现· ··························· 58
3.4.3　代码实现小结· ··································· 62
3.5 输出层的设计· ········································· 63
3.5.1　恒等函数和softmax 函数· ·························· 64
3.5.2　实现softmax 函数时的注意事项· ···················· 66
3.5.3　softmax 函数的特征· ······························ 67
viii 目录
3.5.4　输出层的神经元数量· ····························· 68
3.6 手写数字识别· ········································· 69
3.6.1　MNIST数据集· ·································· 70
3.6.2　神经网络的推理处理· ····························· 73
3.6.3　批处理········································· 75
3.7 小结················································· 79
第4 章　神经网络的学习· ······································· 81
4.1 从数据中学习· ········································· 81
4.1.1　数据驱动········································ 82
4.1.2　训练数据和测试数据· ····························· 84
4.2 损失函数·············································· 85
4.2.1　均方误差········································ 85
4.2.2　交叉熵误差······································ 87
4.2.3　mini-batch 学习· ································· 88
4.2.4　mini-batch 版交叉熵误差的实现· ···················· 91
4.2.5　为何要设定损失函数· ····························· 92
4.3 数值微分·············································· 94
4.3.1　导数··········································· 94
4.3.2　数值微分的例子· ································· 96
4.3.3　偏导数········································· 98
4.4 梯度·················································100
4.4.1　梯度法·········································102
4.4.2　神经网络的梯度· ·································106
4.5 学习算法的实现· ·······································109
4.5.1　2 层神经网络的类·································110
4.5.2　mini-batch 的实现· ·······························114
4.5.3　基于测试数据的评价· ·····························116
4.6 小结·················································118
目录 ix
第5 章　误差反向传播法· ·······································121
5.1 计算图················································121
5.1.1　用计算图求解· ···································122
5.1.2　局部计算········································124
5.1.3　为何用计算图解题· ·······························125
5.2 链式法则··············································126
5.2.1　计算图的反向传播· ·······························127
5.2.2　什么是链式法则· ·································127
5.2.3　链式法则和计算图· ·······························129
5.3 反向传播··············································130
5.3.1　加法节点的反向传播· ·····························130
5.3.2　乘法节点的反向传播· ·····························132
5.3.3　苹果的例子······································133
5.4 简单层的实现· ·········································135
5.4.1　乘法层的实现· ···································135
5.4.2　加法层的实现· ···································137
5.5 激活函数层的实现· ·····································139
5.5.1　ReLU层· ·······································139
5.5.2　Sigmoid 层······································141
5.6 Affine/Softmax层的实现·································144
5.6.1　Affine层· ·······································144
5.6.2　批版本的Affine层· ·······························148
5.6.3　Softmax-with-Loss 层· ····························150
5.7 误差反向传播法的实现· ·································154
5.7.1　神经网络学习的全貌图· ···························154
5.7.2　对应误差反向传播法的神经网络的实现· ··············155
5.7.3　误差反向传播法的梯度确认························158
5.7.4　使用误差反向传播法的学习························159
5.8 小结·················································161
x 目录
第6 章　与学习相关的技巧· ·····································163
6.1 参数的更新············································163
6.1.1　探险家的故事· ···································164
6.1.2　SGD· ··········································164
6.1.3　SGD的缺点· ····································166
6.1.4　Momentum······································168
6.1.5　AdaGrad········································170
6.1.6　Adam· ·········································172
6.1.7　使用哪种更新方法呢· ·····························174
6.1.8　基于MNIST数据集的更新方法的比较· ···············175
6.2 权重的初始值· ·········································176
6.2.1　可以将权重初始值设为0 吗· ························176
6.2.2　隐藏层的激活值的分布· ···························177
6.2.3　ReLU的权重初始值·······························181
6.2.4　基于MNIST数据集的权重初始值的比较· ·············183
6.3 Batch Normalization· ···································184
6.3.1　Batch Normalization 的算法· ·······················184
6.3.2　Batch Normalization 的评估· ·······················186
6.4 正则化················································188
6.4.1　过拟合·········································189
6.4.2　权值衰减········································191
6.4.3　Dropout· ·······································192
6.5 超参数的验证· ·········································195
6.5.1　验证数据········································195
6.5.2　超参数的最优化· ·································196
6.5.3　超参数最优化的实现· ·····························198
6.6 小结·················································200
目录 xi
第7 章　卷积神经网络· ·········································201
7.1 整体结构··············································201
7.2 卷积层················································202
7.2.1　全连接层存在的问题· ·····························203
7.2.2　卷积运算········································203
7.2.3　填充···········································206
7.2.4　步幅···········································207
7.2.5　3 维数据的卷积运算· ······························209
7.2.6　结合方块思考· ···································211
7.2.7　批处理·········································213
7.3 池化层················································214
7.4 卷积层和池化层的实现· ·································216
7.4.1　4 维数组· ·······································216
7.4.2　基于im2col 的展开· ·······························217
7.4.3　卷积层的实现· ···································219
7.4.4　池化层的实现· ···································222
7.5 CNN的实现· ··········································224
7.6 CNN的可视化· ········································228
7.6.1　第1 层权重的可视化·······························228
7.6.2　基于分层结构的信息提取· ·························230
7.7 具有代表性的CNN·····································231
7.7.1　LeNet· ·········································231
7.7.2　AlexNet········································232
7.8 小结·················································233
第8 章　深度学习··············································235
8.1 加深网络··············································235
8.1.1　向更深的网络出发· ·······························235
8.1.2　进一步提高识别精度· ·····························238
xii 目录
8.1.3　加深层的动机· ···································240
8.2 深度学习的小历史· ·····································242
8.2.1　ImageNet· ······································243
8.2.2　VGG· ··········································244
8.2.3　GoogLeNet· ·····································245
8.2.4　ResNet· ········································246
8.3 深度学习的高速化· ·····································248
8.3.1　需要努力解决的问题· ·····························248
8.3.2　基于GPU的高速化· ······························249
8.3.3　分布式学习······································250
8.3.4　运算精度的位数缩减· ·····························252
8.4 深度学习的应用案例· ···································253
8.4.1　物体检测········································253
8.4.2　图像分割········································255
8.4.3　图像标题的生成· ·································256
8.5 深度学习的未来· ·······································258
8.5.1　图像风格变换· ···································258
8.5.2　图像的生成······································259
8.5.3　自动驾驶········································261
8.5.4　Deep Q-Network（强化学习）· ·······················262
8.6 小结·················································264
附录A　Softmax-with-Loss 层的计算图· ···························267
A.1 正向传播· ············································268
A.2 反向传播· ············································270
A.3 小结· ················································277
参考文献· ····················································279

—-

第1 章　Python入門· ··········································· 1
1.1 Python是什麼· ········································· 1
1.2 Python的安裝· ········································· 2
1.2.1　Python版本· ····································· 2
1.2.2　使用的外部庫· ···································· 2
1.2.3　Anaconda發行版本· ································· 3
1.3 Python解譯器· ········································· 4
1.3.1　算術計算········································· 4
1.3.2　資料類型········································· 5
1.3.3　變數············································ 5
1.3.4　列表············································ 6
1.3.5　字典············································ 7
1.3.6　布林型·········································· 7
1.3.7　if 語句· ·········································· 8
1.3.8　for 語句········································· 8
1.3.9　函數············································ 9
1.4 Python指令檔· ······································· 9
vi 目錄
1.4.1　保存為檔······································· 9
1.4.2　類· ············································ 10
1.5 NumPy· ·············································· 11
1.5.1　導入NumPy· ···································· 11
1.5.2　生成NumPy陣列· ································ 12
1.5.3　NumPy 的算數運算······························· 12
1.5.4　NumPy的N維陣列· ······························ 13
1.5.5　廣播··········································· 14
1.5.6　訪問元素········································ 15
1.6 Matplotlib············································ 16
1.6.1　繪製簡單圖形· ··································· 16
1.6.2　pyplot 的功能· ··································· 17
1.6.3　顯示圖像········································ 18
1.7 小結················································· 19
第2 章　感知機················································ 21
2.1 感知機是什麼· ········································· 21
2.2 簡單邏輯電路· ········································· 23
2.2.1　及閘··········································· 23
2.2.2　反及閘和或閘· ··································· 23
2.3 感知機的實現· ········································· 25
2.3.1　簡單的實現······································ 25
2.3.2　導入權重和偏置· ································· 26
2.3.3　使用權重和偏置的實現· ··························· 26
2.4 感知機的局限性· ······································· 28
2.4.1　異或閘········································· 28
2.4.2　線性和非線性· ··································· 30
2.5 多層感知機············································ 31
2.5.1　已有門電路的組合· ······························· 31
目錄 vii
2.5.2　異或閘的實現· ··································· 33
2.6 從反及閘到電腦· ····································· 35
2.7 小結················································· 36
第3 章　神經網路·············································· 37
3.1 從感知機到神經網路· ··································· 37
3.1.1　神經網路的例子· ································· 37
3.1.2　複習感知機······································ 38
3.1.3　啟動函數登場· ··································· 40
3.2 啟動函數·············································· 42
3.2.1　sigmoid 函數· ···································· 42
3.2.2　階躍函數的實現· ································· 43
3.2.3　階躍函數的圖形· ································· 44
3.2.4　sigmoid 函數的實現· ······························ 45
3.2.5　sigmoid 函數和階躍函數的比較······················ 46
3.2.6　非線性函數······································ 48
3.2.7　ReLU函數· ····································· 49
3.3 多維陣列的運算· ······································· 50
3.3.1　多維陣列········································ 50
3.3.2　矩陣乘法········································ 51
3.3.3　神經網路的內積· ································· 55
3.4 3 層神經網路的實現· ···································· 56
3.4.1　符號確認········································ 57
3.4.2　各層間信號傳遞的實現· ··························· 58
3.4.3　代碼實現小結· ··································· 62
3.5 輸出層的設計· ········································· 63
3.5.1　恒等函數和softmax 函數· ·························· 64
3.5.2　實現softmax 函數時的注意事項· ···················· 66
3.5.3　softmax 函數的特徵· ······························ 67
viii 目錄
3.5.4　輸出層的神經元數量· ····························· 68
3.6 手寫數位識別· ········································· 69
3.6.1　MNIST資料集· ·································· 70
3.6.2　神經網路的推理處理· ····························· 73
3.6.3　批次處理········································· 75
3.7 小結················································· 79
第4 章　神經網路的學習· ······································· 81
4.1 從數據中學習· ········································· 81
4.1.1　資料驅動········································ 82
4.1.2　訓練資料和測試資料· ····························· 84
4.2 損失函數·············································· 85
4.2.1　均方誤差········································ 85
4.2.2　交叉熵誤差······································ 87
4.2.3　mini-batch 學習· ································· 88
4.2.4　mini-batch 版交叉熵誤差的實現· ···················· 91
4.2.5　為何要設定損失函數· ····························· 92
4.3 數值微分·············································· 94
4.3.1　導數··········································· 94
4.3.2　數值微分的例子· ································· 96
4.3.3　偏導數········································· 98
4.4 梯度·················································100
4.4.1　梯度法·········································102
4.4.2　神經網路的梯度· ·································106
4.5 學習演算法的實現· ·······································109
4.5.1　2 層神經網路的類·································110
4.5.2　mini-batch 的實現· ·······························114
4.5.3　基於測試資料的評價· ·····························116
4.6 小結·················································118
目錄 ix
第5 章　誤差反向傳播法· ·······································121
5.1 計算圖················································121
5.1.1　用計算圖求解· ···································122
5.1.2　局部計算········································124
5.1.3　為何用計算圖解題· ·······························125
5.2 鏈式法則··············································126
5.2.1　計算圖的反向傳播· ·······························127
5.2.2　什麼是鏈式法則· ·································127
5.2.3　鏈式法則和計算圖· ·······························129
5.3 反向傳播··············································130
5.3.1　加法節點的反向傳播· ·····························130
5.3.2　乘法節點的反向傳播· ·····························132
5.3.3　蘋果的例子······································133
5.4 簡單層的實現· ·········································135
5.4.1　乘法層的實現· ···································135
5.4.2　加法層的實現· ···································137
5.5 啟動函數層的實現· ·····································139
5.5.1　ReLU層· ·······································139
5.5.2　Sigmoid 層······································141
5.6 Affine/Softmax層的實現·································144
5.6.1　Affine層· ·······································144
5.6.2　批版本的Affine層· ·······························148
5.6.3　Softmax-with-Loss 層· ····························150
5.7 誤差反向傳播法的實現· ·································154
5.7.1　神經網路學習的全貌圖· ···························154
5.7.2　對應誤差反向傳播法的神經網路的實現· ··············155
5.7.3　誤差反向傳播法的梯度確認························158
5.7.4　使用誤差反向傳播法的學習························159
5.8 小結·················································161
x 目錄
第6 章　與學習相關的技巧· ·····································163
6.1 參數的更新············································163
6.1.1　探險家的故事· ···································164
6.1.2　SGD· ··········································164
6.1.3　SGD的缺點· ····································166
6.1.4　Momentum······································168
6.1.5　AdaGrad········································170
6.1.6　Adam· ·········································172
6.1.7　使用哪種更新方法呢· ·····························174
6.1.8　基於MNIST資料集的更新方法的比較· ···············175
6.2 權重的初始值· ·········································176
6.2.1　可以將權重初始值設為0 嗎· ························176
6.2.2　隱藏層的啟動值的分佈· ···························177
6.2.3　ReLU的權重初始值·······························181
6.2.4　基於MNIST資料集的權重初始值的比較· ·············183
6.3 Batch Normalization· ···································184
6.3.1　Batch Normalization 的演算法· ·······················184
6.3.2　Batch Normalization 的評估· ·······················186
6.4 正則化················································188
6.4.1　過擬合·········································189
6.4.2　權值衰減········································191
6.4.3　Dropout· ·······································192
6.5 超參數的驗證· ·········································195
6.5.1　驗證資料········································195
6.5.2　超參數的最優化· ·································196
6.5.3　超參數最優化的實現· ·····························198
6.6 小結·················································200
目錄 xi
第7 章　卷積神經網路· ·········································201
7.1 整體結構··············································201
7.2 卷積層················································202
7.2.1　全連接層存在的問題· ·····························203
7.2.2　卷積運算········································203
7.2.3　填充···········································206
7.2.4　步幅···········································207
7.2.5　3 維資料的卷積運算· ······························209
7.2.6　結合方塊思考· ···································211
7.2.7　批次處理·········································213
7.3 池化層················································214
7.4 卷積層和池化層的實現· ·································216
7.4.1　4 維陣列· ·······································216
7.4.2　基於im2col 的展開· ·······························217
7.4.3　卷積層的實現· ···································219
7.4.4　池化層的實現· ···································222
7.5 CNN的實現· ··········································224
7.6 CNN的視覺化· ········································228
7.6.1　第1 層權重的視覺化·······························228
7.6.2　基於分層結構的資訊提取· ·························230
7.7 具有代表性的CNN·····································231
7.7.1　LeNet· ·········································231
7.7.2　AlexNet········································232
7.8 小結·················································233
第8 章　深度學習··············································235
8.1 加深網路··············································235
8.1.1　向更深的網路出發· ·······························235
8.1.2　進一步提高識別精度· ·····························238
xii 目錄
8.1.3　加深層的動機· ···································240
8.2 深度學習的小歷史· ·····································242
8.2.1　ImageNet· ······································243
8.2.2　VGG· ··········································244
8.2.3　GoogLeNet· ·····································245
8.2.4　ResNet· ········································246
8.3 深度學習的高速化· ·····································248
8.3.1　需要努力解決的問題· ·····························248
8.3.2　基於GPU的高速化· ······························249
8.3.3　分散式學習······································250
8.3.4　運算精度的位數縮減· ·····························252
8.4 深度學習的應用案例· ···································253
8.4.1　物體檢測········································253
8.4.2　圖像分割········································255
8.4.3　圖像標題的生成· ·································256
8.5 深度學習的未來· ·······································258
8.5.1　圖像風格變換· ···································258
8.5.2　圖像的生成······································259
8.5.3　自動駕駛········································261
8.5.4　Deep Q-Network（強化學習）· ·······················262
8.6 小結·················································264
附錄A　Softmax-with-Loss 層的計算圖· ···························267
A.1 正向傳播· ············································268
A.2 反向傳播· ············································270
A.3 小結· ················································277
參考文獻· ····················································279