反向传播算法推导—全连接神经网络
反向传播算法是人工神经网络训练时采用的一种通用方法，在现代深度学习中得到了大 规模的应用。全连接神经网络(多层感知器模型，MLP)，卷积神经网络(CNN)，循环神 经网络(RNN)中都有它的实现版本。算法从多元复合函数求导的链式法则导出，递推的 计算神经网络每一层参数的梯度值。算法名称中的“误差”是指损失函数对神经网络每一层 临时输出值的梯度。反向传播算法从神经网络的输出层开始，利用递推公式根据后一层的误 差计算本层的误差，通过误差计算本层参数的梯度值，然后将差项传播到前一层(w, x,)+b这个神经元接受的输入信号为向量(),向量()为输入向量的组合权重,为徧置项,是标量。神经儿对输入冋量进行加权求和,并加上偏置项最后经过激活函数变换产生输出为表述简洁,我们把公式写成向量和矩阵形式。对每个神经元,它接受的来自前一层神经元的输入为向量,本节点的权重向量为,偏置项为,该神经元的输出值为先计算输入向量与权重向量的内积,加上偏置项,再送入一个函数进行变换,得到输出这个函数称为激活函数,典型的是函数。为什么需要激活函数以及什么样的函数可以充当激活函数,在之前的公众号文章“理解神经网终的激活函数”中已经进行了介绍。神绎网络一般有多个层。第一层为输入层,对应输入向量,神绎元的数量等于特征向量的维数,这个层不对数据进行处理,只是将输入向量送入下一层中进行计算。中间为隐含层,可能有多个。最后是输出层,神经元的数量等于要分类的类别数,输出层的输岀值被用来做分类预测。下面我们来看一个简单神经网络的例了,如下图所示这个网络有层。第一层是输入层,对应的输入向量为,有个神经元,写成分量形式为(),它不对数据做任何处理,直接原样送入下一层。中间层有个神经元,接受的输入数据为向量,输出向量为,写成分量形式为。第三个层为输出层,接受的输入数据为向量,输出向量为,写成分量形式为()。第一层到第层的权重矩阵为(,第二层到第三层的权重矩阵为()。权重矩阵的每一行为一个权重向量,是层所有神经元到本层某一个神经儿的连接权重,这里的上标表小层数如果激活函数选用函数,则第二层神经元的输出值为+(-(+0)+(1+(0)(-(()第三层神经元的输出值为如果把代入上面二式中,可以将输出向量表示成输出向量的函数。通过调整权重矩阵和偏置项可以实现不同的函数映射,因此神经网终就是一个复合函数需要解决的·个核心问题是·旦神经网络的结构(即神经元层数,每层神经元数量)桷定之后,怎样得到权重矩阵和偏置项。这些参数是通过训练得到的,这是本文推导的核心任务个简单的例子首先以前面的层神经网络为例,推导损失函数对神经网络所有参数梯度的计算方法假设训练样本集中有个样本()。其中为输入向量,为标签向量。现在要确定神经网络的映射函数:什么样的函数能很好的解释这批训练栟本?答案是神经网络的预测输出要尽可能的接近样本的标签值,即在训练集上最小化预测误差,如果使用均方误差,则优化的目标为:∑‖()-其中()和都是向量,求和项内部是向量的范数平方,即各个分量的平方和。上面的误差也称为欧氏距离损失函数,除此之外还可以使用其他损失函数,如交叉熵、对比损失等。优化目标函数的自变量是各层的权重矩阵和梯度向量,一般情况下无法保证目标函数是凸函数,因此这不是一个凸优化问题,有陷入局部极小值和鞍点的风险(对于这些概念和问题之前的公众号文章“理解梯度下降法”,“理解凸优化”中己经做了详细介绍)这是神经网络之前一直被诟病的一个问题。可以使用梯度下降法进行求解,使用梯度下降法需要计算出损失函数对所有权重矩阵、偏置向量的梯度值,接下来的关键是这些梯度值的计算。在这里我们先将问题简化,只考虑对单个样本的损失函数()-‖后面如果不加说明,都使用这种单样木的损失函数。如果计算出了对单个样木损失函数的棁度值,对这些梯度值计算均值即可得到整个目标函数的梯度值。和(要被代入到网络的后一层中,是复合函数的内层变量,我们先考虑外层的和。权重矩阵是一个x的矩阵,它的两个行分别为向量(和是个维的列向量,它的两个元素为()和()。网络的输入是向量,第一层映射之后的输出是向量首先计算损失函数对权重矩阵每个元素的偏导数,将欧氏距离损尖函数展开,有((+))(())6(如果,即对权重矩阵第行的元素求导,上式分了中的后半部分对来说是常数。根据链式法则有S()+()O如果,即对矩阵第二行的元素求导,类似的有:可以统一写成可以发现,第一个下标决定了权重矩阵的第行和偏置向量的第个分量,第二个下标决定了向量的第个分量。这可以看成是一个列向量与一个行向量相乘的结果,写成矩阵形式为上式中乘法⊙为向量对应元素相乘,第二个乘法是矩阵乘法。是个维列向量,+也是一个维列向量,两个向量执行⊙运算的结果还是个维列向量。是一个元素的列向量,其转置为维行向量,前面这个:维列向量与的乘积为的矩阵,这正好与矩阵的尺寸相等。在上面的公式中,权重的偏导数在求和项中由部分组成,分别是网络输出值与真实标签值的误差激活区数的导数+(),本层的输入值。神经网络的输出值、激活函数的导数值本层的输入值都可以在正向传播吋得到,因此可以晑效的计算出来。对所有训练样本的偏导数计算均值,可以得到总的偏导数对偏置项的偏导数为:如果上式分子中的后半部分对来说是常数,有:()⊥()如果类似的有这可以统写成:写成矩阵形式为偏置项的导数由两部分组成,分别是神经网络预测值与真实值之间的误差,激活函数的导数值,与权重矩阵的偏导数相比唯一的区别是少了。接下来计算对和的偏导数,由于是复合函数的内层,情况更为复杂。()是个的短阵,它的个行向量为(),(,(,(。偏置项()是维向量,个分量分别是(),(,(),(。首先计算损失函数对的元素的偏导数:而上式分子中的两部分都有,因此都与有关。为了表述简活,我们令:根据链式法则有:其巾((和和都是标量和()是两个()向量的内积,的每一个分量都是()的函数。接下来计算和这里的一是个向量,衣示的每个分量分别对求导。当时有:后面个分量相对于求导变量(都是常数。类似的当时有:()0)(()和时的结果以此类推。综合起来有:同理有:()十如果令合并得到()()[()-)。()。()写成矩阵形式为()最后计算偏置项的偏导数()类似的我们得到:合并后得到()写成矩阵形式为:(0)至此,我得到了这个简单网络对所有参数的偏导数,接下来我们将这种做法推广到更般的情况。从上面的结果可以看岀一个规律,输出层的权重矩阵和偏置向量梯度计算公式中共用了()-)()对」隐含层也有类似的结果完整的算法现在考虑一般的情况。假设有个训练样本(),其中为输入向量,为标签向量。训练的目标是最小化样木标签值与神经网络预测值之闩的误差,如果使用均方误差,则优化的目标为:其中为神经网络所有参数的集合,包括各层的权重和偏置。这个最优化问题是·个不带约束条件的问题,可以用梯度下降法求解。上面的误差函数定义在整个训练样本集上,梯度下降法每一次迭代利用了所有训练样本,称为批量棁度卜降法。如果样木数量很大,每次迭代都用所有样木进计算成木太高。为了解决这个问题,可以采用单样本梯度下降法,我们将上面的损失函数写成对单个样本的损失函数之和:定义对单个样本()的损失函数为)=-()如果采用单个样本进行迭代,梯度下降法第次迭代时参数的更新公式为:nV如果要用所有样本进行迭代,根据单个样本的损失函数梯度计算总损失梯度即可,即所有样本梯度的均值用梯度下降法求解需要初始化优化变量的值。一般初始化为一个随机数,如用正态分布(a)产生这些随机数,其中G是一个很小的正数到日前为止还有一个关键问题没有解决:日标函数是一个多层的复合函数,因为神经网络中每一层都有权重矩阵和偏置向量,且每一层的输出将会作为下一层的输入。因此,直接计算损失函数对所有权重和偏置的梚度很复杂,需要使用复合函数的求导公式进行递推计算几个重要的结论在进行推导之前,我们首先来看下面几种复合函数的求导。又如下线性映射函数:其中是维向量,是×的矩阵,是维向量。问题:假设有函数,如果把看成常数,看成的函数,如何根据函数对的梯度值Ⅴ计算函数对的梯度值Ⅴ?根据链式法则,由于只和有关,和其他的≠无关,因此有:c∑(对于的所有元素有:写成矩阵形式为:问题:如果将看成常数,将看成的函数,如何根据V计算Ⅴ?由于任意的和所有的都有关系,根据链式法则有写成矩阵形式为这是一个对称的结果,在计算函数映射时用矩阵乘以向量得到,在求梯度时用矩阵的转置乘以的梯度得到的梯度。问题:如果有向量到向量的映射:
- 2020-12-09下载
- 积分：1
凸优化在信号处理与通信中的应用Convex Optimization in Signal Processing and Communications
凸优化理论在信号处理以及通信系统中的应用 比较经典的通信系统凸优化入门教程ContentsList of contributorspage IxPrefaceAutomatic code generation for real- time convex optimizationJacob Mattingley and stephen Boyd1.1 Introduction1.2 Solvers and specification languages61. 3 Examples121. 4 Algorithm considerations1.5 Code generation261.6 CVXMOD: a preliminary implementation281.7 Numerical examples291. 8 Summary, conclusions, and implicationsAcknowledgments35ReferencesGradient-based algorithms with applications to signal-recoveryproblemsAmir beck and marc teboulle2.1 Introduction422.2 The general optimization model432.3 Building gradient-based schemes462. 4 Convergence results for the proximal-gradient method2.5 A fast proximal-gradient method2.6 Algorithms for l1-based regularization problems672.7 TV-based restoration problems2. 8 The source-localization problem772.9 Bibliographic notes83References85ContentsGraphical models of autoregressive processes89Jitkomut Songsiri, Joachim Dahl, and Lieven Vandenberghe3.1 Introduction893.2 Autoregressive processes923.3 Autoregressive graphical models983. 4 Numerical examples1043.5 Conclusion113Acknowledgments114References114SDP relaxation of homogeneous quadratic optimization: approximationbounds and applicationsZhi-Quan Luo and Tsung-Hui Chang4.1 Introduction1174.2 Nonconvex QCQPs and sDP relaxation1184.3 SDP relaxation for separable homogeneous QCQPs1234.4 SDP relaxation for maximization homogeneous QCQPs1374.5 SDP relaxation for fractional QCQPs1434.6 More applications of SDP relaxation1564.7 Summary and discussion161Acknowledgments162References162Probabilistic analysis of semidefinite relaxation detectors for multiple-input,multiple-output systems166Anthony Man-Cho So and Yinyu Ye5.1 Introduction1665.2 Problem formulation1695.3 Analysis of the SDr detector for the MPsK constellations1725.4 Extension to the Qam constellations1795.5 Concluding remarks182Acknowledgments182References189Semidefinite programming matrix decomposition, and radar code design192Yongwei Huang, Antonio De Maio, and Shuzhong Zhang6.1 Introduction and notation1926.2 Matrix rank-1 decomposition1946.3 Semidefinite programming2006.4 Quadratically constrained quadratic programming andts sdp relaxation201Contents6.5 Polynomially solvable QCQP problems2036.6 The radar code-design problem2086.7 Performance measures for code design2116.8 Optimal code design2146.9 Performance analysis2186.10 Conclusions223References226Convex analysis for non-negative blind source separation withapplication in imaging22Wing-Kin Ma, Tsung-Han Chan, Chong-Yung Chi, and Yue Wang7.1 Introduction2297.2 Problem statement2317.3 Review of some concepts in convex analysis2367.4 Non-negative, blind source-Separation criterion via CAMNS2387.5 Systematic linear-programming method for CAMNS2457.6 Alternating volume-maximization heuristics for CAMNS2487.7 Numerical results2527.8 Summary and discussion257Acknowledgments263References263Optimization techniques in modern sampling theory266Tomer Michaeli and yonina c. eldar8.1 Introduction2668.2 Notation and mathematical preliminaries2688.3 Sampling and reconstruction setup2708.4 Optimization methods2788.5 Subspace priors2808.6 Smoothness priors2908.7 Comparison of the various scenarios3008.8 Sampling with noise3028. 9 Conclusions310Acknowledgments311References311Robust broadband adaptive beamforming using convex optimizationMichael Rubsamen, Amr El-Keyi, Alex B Gershman, and Thia Kirubarajan9.1 Introduction3159.2 Background3179.3 Robust broadband beamformers3219.4 Simulations330Contents9.5 Conclusions337Acknowledgments337References337Cooperative distributed multi-agent optimization340Angelia Nedic and asuman ozdaglar10.1 Introduction and motivation34010.2 Distributed-optimization methods using dual decomposition34310.3 Distributed-optimization methods using consensus algorithms35810.4 Extensions37210.5 Future work37810.6 Conclusions38010.7 Problems381References384Competitive optimization of cognitive radio MIMO systems via game theory387Gesualso Scutari, Daniel P Palomar, and Sergio Barbarossa11.1 Introduction and motivation38711.2 Strategic non-cooperative games: basic solution concepts and algorithms 39311.3 Opportunistic communications over unlicensed bands411.4 Opportunistic communications under individual-interferenceconstraints4151.5 Opportunistic communications under global-interference constraints43111.6 Conclusions438Ackgment439References43912Nash equilibria: the variational approach443Francisco Facchinei and Jong-Shi Pang12.1 Introduction44312.2 The Nash-equilibrium problem4412. 3 EXI45512.4 Uniqueness theory46612.5 Sensitivity analysis47212.6 Iterative algorithms47812.7 A communication game483Acknowledgments490References491Afterword494Index49ContributorsSergio BarbarossaYonina c, eldarUniversity of rome-La SapienzaTechnion-Israel Institute of TechnologyHaifaIsraelAmir beckTechnion-Israel instituteAmr El-Keyiof TechnologyAlexandra universityHaifEgyptIsraelFrancisco facchiniStephen boydUniversity of rome La sapienzaStanford UniversityRomeCaliforniaItalyUSAAlex b, gershmanTsung-Han ChanDarmstadt University of TechnologyNational Tsing Hua UniversityDarmstadtHsinchuGermanyTaiwanYongwei HuangTsung-Hui ChangHong Kong university of scienceNational Tsing Hua Universityand TechnologyHsinchuHong KongTaiwanThia KirubarajanChong-Yung chiMcMaster UniversityNational Tsing Hua UniversityHamilton ontarioHsinchuCanadaTaiwanZhi-Quan LuoJoachim dahlUniversity of minnesotaanybody Technology A/sMinneapolisDenmarkUSAList of contributorsWing-Kin MaMichael rebsamenChinese University of Hong KongDarmstadt UniversityHong KonTechnologyDarmstadtAntonio de maioGermanyUniversita degli studi di napoliFederico iiGesualdo scutariNaplesHong Kong University of Sciencealyand TechnologyHong KongJacob MattingleyAnthony Man-Cho SoStanford UniversityChinese University of Hong KongCaliforniaHong KongUSAJitkomut songsinTomer michaeliUniversity of californiaTechnion-Israel instituteLoS Angeles. CaliforniaogyUSAHaifaMarc teboulleTel-Aviv UniversityAngelia NedicTel-AvUniversity of Illinois atIsraelUrbana-ChampaignInoSLieven VandenbergheUSAUniversity of CaliforniaLos Angeles, CaliforniaUSAAsuman OzdaglarMassachusetts Institute of TechnologyYue WangBoston massachusettsVirginia Polytechnic InstituteUSAand State UniversityArlingtonDaniel p palomarUSAHong Kong University ofScience and TechnologyYinyu YeHong KongStanford UniversityCaliforniaong-Shi PangUSAUniversity of illinoisat Urbana-ChampaignShuzhong zhangIllinoisChinese university of Hong KongUSAHong KongPrefaceThe past two decades have witnessed the onset of a surge of research in optimization.This includes theoretical aspects, as well as algorithmic developments such as generalizations of interior-point methods to a rich class of convex-optimization problemsThe development of general-purpose software tools together with insight generated bythe underlying theory have substantially enlarged the set of engineering-design problemsthat can be reliably solved in an efficient manner. The engineering community has greatlybenefited from these recent advances to the point where convex optimization has nowemerged as a major signal-processing technique on the other hand, innovative applica-tions of convex optimization in signal processing combined with the need for robust andefficient methods that can operate in real time have motivated the optimization commu-nity to develop additional needed results and methods. The combined efforts in both theoptimization and signal-processing communities have led to technical breakthroughs ina wide variety of topics due to the use of convex optimization This includes solutions tonumerous problems previously considered intractable; recognizing and solving convex-optimization problems that arise in applications of interest; utilizing the theory of convexoptimization to characterize and gain insight into the optimal-solution structure and toderive performance bounds; formulating convex relaxations of difficult problems; anddeveloping general purpose or application-driven specific algorithms, including thosethat enable large-scale optimization by exploiting the problem structureThis book aims at providing the reader with a series of tutorials on a wide varietyof convex-optimization applications in signal processing and communications, writtenby worldwide leading experts, and contributing to the diffusion of these new developments within the signal-processing community. The goal is to introduce convexoptimization to a broad signal-processing community, provide insights into how convexoptimization can be used in a variety of different contexts, and showcase some notablesuccesses. The topics included are automatic code generation for real-time solvers, graphical models for autoregressive processes, gradient-based algorithms for signal-recoveryapplications, semidefinite programming(SDP)relaxation with worst-case approximationperformance, radar waveform design via SDP, blind non-negative source separation forimage processing, modern sampling theory, robust broadband beamforming techniquesdistributed multiagent optimization for networked systems, cognitive radio systems viagame theory, and the variational-inequality approach for Nash-equilibrium solutionsPrefaceThere are excellent textbooks that introduce nonlinear and convex optimization, providing the reader with all the basics on convex analysis, reformulation of optimizationproblems, algorithms, and a number of insightful engineering applications. This book istargeted at advanced graduate students, or advanced researchers that are already familiarwith the basics of convex optimization. It can be used as a textbook for an advanced graduate course emphasizing applications, or as a complement to an introductory textbookthat provides up-to-date applications in engineering. It can also be used for self-study tobecome acquainted with the state of-the-art in a wide variety of engineering topicsThis book contains 12 diverse chapters written by recognized leading experts worldwide, covering a large variety of topics. Due to the diverse nature of the book chaptersit is not possible to organize the book into thematic areas and each chapter should betreated independently of the others. a brief account of each chapter is given nextIn Chapter 1, Mattingley and Boyd elaborate on the concept of convex optimizationin real-time embedded systems and automatic code generation. As opposed to genericsolvers that work for general classes of problems, in real-time embedded optimization thesame optimization problem is solved many times, with different data, often with a hardreal-time deadline. Within this setup the authors propose an automatic code-generationsystem that can then be compiled to yield an extremely efficient custom solver for theproblem familyIn Chapter 2, Beck and Teboulle provide a unified view of gradient-based algorithmsfor possibly nonconvex and non-differentiable problems, with applications to signalrecovery. They start by rederiving the gradient method from several different perspectives and suggest a modification that overcomes the slow convergence of the algorithmThey then apply the developed framework to different image-processing problems suchas e1-based regularization, TV-based denoising, and Tv-based deblurring, as well ascommunication applications like source localizationIn Chapter 3, Songsiri, Dahl, and Vandenberghe consider graphical models for autore-gressive processes. They take a parametric approach for maximum-likelihood andmaximum-entropy estimation of autoregressive models with conditional independenceconstraints, which translates into a sparsity pattern on the inverse of the spectral-densitymatrix. These constraints turn out to be nonconvex. To treat them the authors proposea relaxation which in some cases is an exact reformulation of the original problem. Theproposed methodology allows the selection of graphical models by fitting autoregressiveprocesses to different topologies and is illustrated in different applicationsThe following three chapters deal with optimization problems closely related to SDPand relaxation techniquesIn Chapter 4, Luo and Chang consider the SDP relaxation for several classes ofquadratic-optimization problems such as separable quadratically constrained quadraticprograms(QCQPs)and fractional QCQPs, with applications in communications and signal processing. They identify cases for which the relaxation is tight as well as classes ofquadratic-optimization problems whose relaxation provides a guaranteed, finite worstcase approximation performance. Numerical simulations are carried out to assess theefficacy of the SDP-relaxation approach
- 2020-12-10下载
- 积分：1
