Verilog-IEEE Std 1364 -2005 IEEE Standard
Verilog的IEEE标准，比较新的一个吧应该是IEEE Std 1364 TM-2005(Revision of IEEE Std 1364-2001)lEE Standard for VerilogHardware Description LanguageSponsorDesign Automation Standards Committeeof theIEEE Computer SocietyAbstract: The Verilog hardware description language(HDL) is defined in this standard. VerilogHDL is a formal notation intended for use in all phases of the creation of electronic systems. Because it is both machine-readable and human-readable, it supports the development, verificationsynthesis, and testing of hardware designs; the communication of hardware design data; and themaintenance, modification, and procurement of hardware. The primary audiences for this standardare the implementors of tools supporting the language and advanced users of the languageKeywords: computer, computer languages, digital systems, electronic systems, hardware, hardware description languages, hardware design, HDL, PLI, programming language interface, Verilog,Verilog HDL, verilog PllThe Institute of Electrical and Electronics Engineers, Inc3 Park Avenue. New york. NY 10016-5997 USACopyright@ 2006 by the Institute of Electrical and Electronics Engineers, IncAll rights reserved Published 7 April 2006. Printed in the United states of AmericaIEEE is a registered trademark in the U.S. Patent Trademark Office, owned by the Institute of Electrical and ElectronicsEngineers, IncorporatedVerilog is a registered trademark of Cadence Design Systems, IncPrint:|sBN0-738148504SH95395PDFSBN0-7381-48512SS95395No part of this publication may be reproduced in any form, in an electronic retrieval system or otherwise, without the priorwritten permission of the publisher.IEEE Standards documents are developed within the IEee Societies and the Standards CoordinatingCommittees of the iEEE Standards Association (IEEE-SA)Standards board The ieee develops its standardsthrough a consensus development process, approved by the american National Standards Institute, which bringstogether volunteers representing varied viewpoints and interests to achieve the final product. Volunteers are notnecessarily members of the Institute and serve without compensation. While the ieee administers the processand establishes rules to promote fairness in the consensus development process, the ieee does not independentlyevaluate, test, or verify the accuracy of any of the information contained in its standards.Use of an IEEE Standard is wholly voluntary. The ieee disclaims liability for any personal injury, property orother damage of any nature whatsoever, whether special, indirect, consequential, or compensatory, directly orindirectly resulting from the publication, use of, or reliance upon this, or any other iEEE Standard documentThe ieee does not warrant or represent the accuracy or content of the material contained herein, and expresslydisclaims any express or implied warranTy, including any implied warranty of merchantability or fitness for a specific purpose, or that the use of the material contained herein is free from patent infringement. IEEE Standardsdocuments are supplied "AS IsThe existence of an IEEE Standard does not imply that there are no other ways to produce, test, measurepurchase, market, or provide other goods and services related to the scope of the IEEE Standard. Furthermore, theviewpoint expressed at the time a standard is approved and issued is subject to change brought about throughdevelopments in the state of the art and comments received from users of the standard Every IeeE Standard issubjected to review at least every five years for revision or reaffirmation. When a document is more than fiveyears old and has not been reaffirmed, it is reasonable to conclude Chat ils contents, although still of some valuedo not wholly reflect the present state of the art. Users are cautioned to check to determine that they have thelatest edition of any IEEE StandardIn publishing and making this document available, the IEeE is not suggesting or rendering professional or otherervices for, or on behalf of, any person or entity. Nor is the Ieee undertaking to perform any dutyother person or entity to another. Any person utilizing this, and any other ieee Standards document, should relupon the advice of a competent professional in determining the exercise of reasonable care in any givencircumstancesInterpretations: Occasionally questions may arise regarding the meaning of portions of standards as they relate tospecific applications. When the need for interpretations is brought to the attention of IEEe, the Institute will initiateaction to prepare appropriate responses. Since ifff Standards represent a consensus of concerned interests, it isimportant to ensure that any interpretation has also received the concurrence of a balance of interests. For thisreason, IEEE and the members of its societies and standards coordinating committees are not able to provide aninstant response to interpretation requests except in those cases where the matter has previously received formalconsideration. At lectures, symposia, seminars, or educational courses, an individual presenting information onIEee standards shall make it clear that his or her views should be considered the personal views of that individuarather than the formal position, explanation, or interpretation of the IeeeComments for revision of IEEE Standards are welcome from any interested party, regardless of membership aftil-iation with IEEE. Suggestions for changes in documents should be in the form of a proposed change of texttogether with appropriate supporting comments Comments on standards and requests for interpretations shouldaddressed toIEEE-SA Standards Board445 Hoes lanePiscataway, NJ 08854USANoTE-Attention is called to the possibility that implementation of this standard may require use of subjectmatter covered by patent rights. By publication of this standard, no position is taken with respect to theexistence or validity of any patent rights in connection therewith. The IEf shall not be responsible foridentifying patents for which a license may be required by an ieee standard or for conducting inquiries into thelegal validity or scope of those patents that are brought to its attentionAuthorization to photocopy portions of any individual standard for internal or personal use is granted by the Institute of Electrical and Electronics Engineers, Inc, provided that the appropriate fee is paid to Copyright ClearanceCenter. To arrange for payment of licensing fee, please contact Copyright Clearance Center, Customer Service222 Rosewood Drive, Danvers, MA01923 USA; +19787508400. Permission to photocopy portions of any indi-idual standard for educational classroom use can also be obtained through the Copyright Clearance CenterIntroductionThis introduction is not a part of IEEE Std 1364-2005, IEEE Standard for Verilog Hardware Description LanguageThe Verilog hardware description language(HDL) became an IEEE standard in 1995 as IEEE Std 13641995. It was designed to be simple, intuitive, and effective at multiple levels of abstraction in a standardtextual format for a variety of design tools, including verification simulation, timing analysis, test analysisand synthesis. It is because of these rich features that verilog has been accepted to be the language of choiceby an overwhelming number of integrated circuit(IC)designersVerilog contains a rich set of built-in primitives, including logic gates, user-definable primitives, switches,and wired logic. It also has device pin-to-pin delays and timing checks. The mixing of abstract levels isessentially provided by the semantics of two data types: nets and variables. Continuous assignments, inwhich expressions of both variables and nets can continuously drive values onto nets, provide the basicstructural construct. Procedural assignments, in which the results of calculations involving variable and netvalues can be stored into variables, provide the basic behavioral construct. a design consists of a set of modules, each of which has an input/output(1/O)interface, and a description of its function, which can be structural, behavioral, or a mix. These modules are formed into a hierarchy and are interconnected with netsThe Verilog language is extensible via the programming language interface(PLI)and the verilog procedural interface(VPi) routines. The Pli/vPi is a collection of routines that allows foreign functions to accessinformation contained in a Verilog hdl description of the design and facilitates dynamic interaction withsimulation. Applications of PLI/VPI include connecting to a Verilog HDL simulator with other simulationand computer-assisted design(CAD)systems, customized debugging TaskS, delay calculators, andannotatorsThe language that influenced Verilog HDL the most was HILO-2, which was developed at Brunel University in England under a contract to produce a test generation system for the British Ministry of DefensehiLO-2 successfully combined the gate and register transfer levels of abstraction and supported verificationsimulation, timing analysis, fault simulation, and test generationIn 1990, Cadence Design Systems placed the Verilog HDL into the public domain and the independentOpen Verilog International(oVi)was formed to manage and promote Verilog HDL. In 1992, the Board ofDirectors of ovi began an effort to establish Verilog hdl as an ieeE standard. In 1993, the first IEeEworking group was formed; and after 18 months of focused efforts, Verilog became an IEEE standard asIEEE Std 1364-1995After the standardization process was complete, the IEEe P1364 Working Group started looking for feedback from IEEE 1364 users worldwide so the standard could be enhanced and modified accordingly. Thisled to a five-year effort to get a much better Verilog standard in IEEE Std 1364-2001With the completion of IEEE Std 1364-2001, work continued in the larger Verilog community to identifyoutstanding issues with the language as well as ideas for possible enhancements. As Accellera began work-ing on standardizing System Verilog in 2001, additional issues were identified that could possibly have led toincompatibilities between Verilog 1364 and System Verilog. The IEEE P1364 Working group was established as a subcomittee of the System Verilog P1800 Working Group to help ensure consistent resolution ofsuch issues. The result of this collaborative work is this standard ieee Std 1364-2005Copyright C 2006 IEEE. All rights reservedNotice to usersErrataErrata,ifanyforthisandallotherstandardscanbeaccessedatthefollowingurL:http:/stan-dards.ieee.org/reading/ieee/updates/errata/index.html. Users are encouraged to check this URL for errataperiodicaInterpretationsCurrentinterpretationscanbeaccessedatthefollowingUrl:http:/standards.ieeeorg/reading/ieee/interp/Index. htmlPatentsAttention is called to the possibility that implementation of this standard may require use of subject mattercovered by patent rights. By publication of this standard, no position is taken with respect to the existence orvalidity of any patent rights in connection therewith. The IEee shall not be responsible for identifyingpatents or patent applications for which a license may be required to implement an IEEE standard or forconducting inquiries into the legal validity or scope of those patents that are brought to its attentionParticipantsAt the time this standard was completed, the ieee P1364 Working group had the following membershipJohny Srouji, IBM, IEEE SyStem verilog Working Group chairTom Fitzpatrick, Mentor Graphics Corporation, ChairNeil Korpusik, Sun Microsystems, Inc, Co-chairStuart sutherland sutherland hdl inc. editorShalom Bresticker, Intel Corporation, Editor through February 2005The Errata Task Force had the following membershipKaren pynopsys,rKurt baty. WFSDB ConsultinDennis marsa. XilinxStefen Boyd, Boyd TechnologyFrancoise Martinolle, Cadence Design Systems, IncShalom Bresticker, Intel CorporationMike McNamara, Verisity, LtdDennis Brophy, Mentor Graphics CorporationDon Mills, LCDM EngineeringCliff Cummings, Sunburst Design, IncAnders nordstrom, Cadence Design Systems, IncCharles dawson, Cadence Design Systems, IncKaren Pieper, Synopsys, IncTom Fitzpatrick, Mentor Graphics CorpoBrad Pierce, Synopsys, IncRonald goodstein, first shot Logic simulation andSteven Sharp Cadence Design Systems, IncAlec Stanculescu. Fintronic USA IncDesignStuart Sutherland. Sutherland HDL IncMark Hartog, Synopsys incGordon Vreugdenhil, Mentor Graphics CorporationJames Markevitch, Evergreen Technology GroupJason Woolf, Cadence design Systems, IncCopyright C 2006 IEEE. All rights reservedThe behavioral Task Force had the following membership:Steven Sharp, Cadence Design Systems, InC, ChairKurt Baty, WFSDB ConsultingJay lawrence. Cadence design Systems. IncStefen Boyd, Boyd TechnologyFrancoise Martinolle, Cadence Design Systems, IncShalom Bresticker, Intel CorporationKathryn McKinley, Cadence Design Systems, IncDennis brophy, Mentor graphics corporationMichael mcnamara. Verisity LtdCliff Cummings, Sunburst Design, IncDon Mills, LCDM EngineeringSteven Dovich, Cadence Design Systems, IncMehdi Mohtashemi, Synopsys, IncTom Fitzpatrick, Mentor Graphics CorporationKaren Pieper, Synopsys, IncRonald Goodstein, First Shot Logic Simulation andBrad Pierce, Synopsys, IncDesignDave Rich, Mentor Graphics CorporationKeith Gover, Mentor Graphics CorporationSteven Sharp, Cadence Design Systems, IncMark Hartoog, Synopsys, IncAlec Stanculescu. Fintronic USAEnnis Hawk, Jeda TechnologiesStuart Sutherland. Sutherland hdl. IncAtsushi kasuya, Jcda TechnologicsGordon Vrcugdcnhil, Mentor Graphics CorporationThe PLI Task Force had the following membershipCharles Dawson, Cadence Design Systems, Inc, ChairGhassan Khoory, Synopsys, Inc Co-chairTapati Basu, Synopsys, IncMichael rohleder. Freescale Semiconductor. IncSteven Dovich, Cadence Design Systems, IncRob Slater, Freescale Semiconductor IncRalph duncan, Mentor Graphics CorporationJohn Stickley, Mentor Graphics CorporationJim garnett, Mentor Graphics CorporationStuart Sutherland. Sutherland HDL. incJoao geada CLK Design AutomationBassam Tabbara. Novas software. IncAndrzej litwiniuk, Synopsys, IncJim Vellenga, Cadence Design Systems, IncFrancoise Martinolle, Cadence Design Systems, IncDoug Warmke, Mentor Graphics CorporationSachchidananda Patel, Synopsys, IncIn addition, the working group wishes to recognize the substantial efforts of past contributorsMichael McNamara, Cadence Design Systems, Inc1364 Working Group past chair(through September 2004)Alec Stanculescu, Fintronic USA, 1304 Working Group past vice-chair(through June 2004)Stefen Boyd, Boyd Technology, ETF past co-chair(through November 2004)The following members of the entity balloting commitlee voted on this standard. Balloters may have votedfor approval, disapproval, or abstentionAccelleraIntel CorporationBluespec, Inc.Mentor Graphics CorporationCadence Dcsign Systcms, IncSun microsystems, IncIntronic u.s.aSunburst design, IncIBMSutherland hdl lncInfineon TechnologiesSynopsys, IncCopyright C 2006 IEEE. All rights reservedWhen the IEEE-SA Standards Board approved this standard on 8 November 2005, it had the followingmembershipSteve M. mills. chairRichard h. hulett vice chairDon wright Past chairJudith gorman. secretaryMark d. bowmanWilliam B HopfT W. olsenDennis B. BrophyLowell G. JohnsonGlenn parsonsJoseph brudeHerman KochRonald c. petersenRichard coxJoseph L. Koepfinger*Gary s. RobinsonBob davisDavid J lawFrank stoneJulian forster kDaleep c mohlaMalcolm v thadenJoanna n. gueninPaul nikolichRichard l. townsendS. HalpJoe d. watseRaymond hapemanHoward L, wolfmanAlso included are the following nonvoting Ieee-Sa Standards board liaisonsSatish K. aval, NRC RepresentativeRichard Deblasio, DOE RepresentativeAlan H. Cookson, NIST RepresentativeMichelle d, trIEEE Standards Project EditoCopyright C 2006 IEEE. All rights reservedContentsOverview1. 2 Conventions used in this standard1.3SIption1 4 Use of color in this standard1.5 Contents of this standard1.6 Deprecated clauses……….….….….…1.7 Header file listings....18 Examples…………………1.9PNormative references63. Lexical conventions83.1 Lexical tokens3.2 White space3. 3 Comments中··…·········:···············中·····“:·:·:·4·····“··········3. 4 Operators3. 5 Numberssteger constants3.5.2 Real constants123.5.3 Conversion123.6 Strings123.6. 1 String variable declaration133.6.2 String manipulation133.6.3 Special cha3.7 Identifiers. keds, and syste143.7.1 Escaped identifiers143.7.2 Keywords153.7.3 System tasks and functions3.7.4 Compiler directives153.8 Attrib163.8.1 Examples3.8.2 SyIata typ··4.1 Val214.2 Nets and variables4.2.1 Net declarations4.2.2 Variable declarations4.3V4.3.1g v244.3.2 Veclor net accessibility44.4 Strengths4.4.1 Charge strength4.4.2 Drive strength.....卓········中····“·········:·····················:········254.5 Implicit declarations4.6 Net types………264.6.1 Wire and tri nets264.6.2 Wired nets4.6.3TCopyright C 2006 IEEE. All rights reserved4.6.4 Trio and tri l nets4.6.5 Unresolved nets4.6.6 Supply nets324.7 Regs324.8 Integers, reals, times, and realtime4.8.1 Operators and real numbers4.8.2 Conversion4.9 Arrays4.9.1Net arrays…·中·····:··344.9.2 reg and variable arrays344.9.3 Memories4.10 Parameters…………………354.10.1 Module parameters…364.10.2 Local parameters(localparam““374.10.3 Specify parameters……384. Name spaces…39Expressions……5.1 Operators41Operators with real operands425.1.2 Operator precedence………5.1.3 Using integer numbers in expressions…….445.1.4 Expression evaluation order……2451. 5 Arithmetic operators5.1.6 Arithmetic expressions with regs and integers5. 1.7 Relational operators485.1.8Equ495.1.9 Logical operators…495.1.10 bitw isators505.1.11 Reduction operators5.112 Shift operators…….535. 1. 13 Conditional operator535.1. 14 Concatenations545.2 Operands………5.2. 1 Vector bit-Select and part-select addressing..565.2.2 Array and memory addressing.......,.……5.2.3ings585.3 Minimum, typical, and maximum delay expressions5.4 Expression bit lengths5.4.1 Rules for expression bit lengths65.4.2 Examole of expression bit-length problerpl635.4.3 Example of self-determined expressions.645.5 Signed expressions5.5.1 Rules for expression types.....5.5.2 Steps for evaluating5.5.3 Steps for evaluating an assignment5.54 Handling X and Z in signed expressions………5.6 Assignments and truncation6. Assignments……………686.1 Continuous assignments686. 1. 1 The net declaration assignment6.1.2 The continuous assignment statement·····中···:·:·4·中·····6.1.371Copyright C 2006 IEEE. All rights reserved
- 2021-05-06下载
- 积分：1
反向传播算法推导—全连接神经网络
反向传播算法是人工神经网络训练时采用的一种通用方法，在现代深度学习中得到了大 规模的应用。全连接神经网络(多层感知器模型，MLP)，卷积神经网络(CNN)，循环神 经网络(RNN)中都有它的实现版本。算法从多元复合函数求导的链式法则导出，递推的 计算神经网络每一层参数的梯度值。算法名称中的“误差”是指损失函数对神经网络每一层 临时输出值的梯度。反向传播算法从神经网络的输出层开始，利用递推公式根据后一层的误 差计算本层的误差，通过误差计算本层参数的梯度值，然后将差项传播到前一层(w, x,)+b这个神经元接受的输入信号为向量(),向量()为输入向量的组合权重,为徧置项,是标量。神经儿对输入冋量进行加权求和,并加上偏置项最后经过激活函数变换产生输出为表述简洁,我们把公式写成向量和矩阵形式。对每个神经元,它接受的来自前一层神经元的输入为向量,本节点的权重向量为,偏置项为,该神经元的输出值为先计算输入向量与权重向量的内积,加上偏置项,再送入一个函数进行变换,得到输出这个函数称为激活函数,典型的是函数。为什么需要激活函数以及什么样的函数可以充当激活函数,在之前的公众号文章“理解神经网终的激活函数”中已经进行了介绍。神绎网络一般有多个层。第一层为输入层,对应输入向量,神绎元的数量等于特征向量的维数,这个层不对数据进行处理,只是将输入向量送入下一层中进行计算。中间为隐含层,可能有多个。最后是输出层,神经元的数量等于要分类的类别数,输出层的输岀值被用来做分类预测。下面我们来看一个简单神经网络的例了,如下图所示这个网络有层。第一层是输入层,对应的输入向量为,有个神经元,写成分量形式为(),它不对数据做任何处理,直接原样送入下一层。中间层有个神经元,接受的输入数据为向量,输出向量为,写成分量形式为。第三个层为输出层,接受的输入数据为向量,输出向量为,写成分量形式为()。第一层到第层的权重矩阵为(,第二层到第三层的权重矩阵为()。权重矩阵的每一行为一个权重向量,是层所有神经元到本层某一个神经儿的连接权重,这里的上标表小层数如果激活函数选用函数,则第二层神经元的输出值为+(-(+0)+(1+(0)(-(()第三层神经元的输出值为如果把代入上面二式中,可以将输出向量表示成输出向量的函数。通过调整权重矩阵和偏置项可以实现不同的函数映射,因此神经网终就是一个复合函数需要解决的·个核心问题是·旦神经网络的结构(即神经元层数,每层神经元数量)桷定之后,怎样得到权重矩阵和偏置项。这些参数是通过训练得到的,这是本文推导的核心任务个简单的例子首先以前面的层神经网络为例,推导损失函数对神经网络所有参数梯度的计算方法假设训练样本集中有个样本()。其中为输入向量,为标签向量。现在要确定神经网络的映射函数:什么样的函数能很好的解释这批训练栟本?答案是神经网络的预测输出要尽可能的接近样本的标签值,即在训练集上最小化预测误差,如果使用均方误差,则优化的目标为:∑‖()-其中()和都是向量,求和项内部是向量的范数平方,即各个分量的平方和。上面的误差也称为欧氏距离损失函数,除此之外还可以使用其他损失函数,如交叉熵、对比损失等。优化目标函数的自变量是各层的权重矩阵和梯度向量,一般情况下无法保证目标函数是凸函数,因此这不是一个凸优化问题,有陷入局部极小值和鞍点的风险(对于这些概念和问题之前的公众号文章“理解梯度下降法”,“理解凸优化”中己经做了详细介绍)这是神经网络之前一直被诟病的一个问题。可以使用梯度下降法进行求解,使用梯度下降法需要计算出损失函数对所有权重矩阵、偏置向量的梯度值,接下来的关键是这些梯度值的计算。在这里我们先将问题简化,只考虑对单个样本的损失函数()-‖后面如果不加说明,都使用这种单样木的损失函数。如果计算出了对单个样木损失函数的棁度值,对这些梯度值计算均值即可得到整个目标函数的梯度值。和(要被代入到网络的后一层中,是复合函数的内层变量,我们先考虑外层的和。权重矩阵是一个x的矩阵,它的两个行分别为向量(和是个维的列向量,它的两个元素为()和()。网络的输入是向量,第一层映射之后的输出是向量首先计算损失函数对权重矩阵每个元素的偏导数,将欧氏距离损尖函数展开,有((+))(())6(如果,即对权重矩阵第行的元素求导,上式分了中的后半部分对来说是常数。根据链式法则有S()+()O如果,即对矩阵第二行的元素求导,类似的有:可以统一写成可以发现,第一个下标决定了权重矩阵的第行和偏置向量的第个分量,第二个下标决定了向量的第个分量。这可以看成是一个列向量与一个行向量相乘的结果,写成矩阵形式为上式中乘法⊙为向量对应元素相乘,第二个乘法是矩阵乘法。是个维列向量,+也是一个维列向量,两个向量执行⊙运算的结果还是个维列向量。是一个元素的列向量,其转置为维行向量,前面这个:维列向量与的乘积为的矩阵,这正好与矩阵的尺寸相等。在上面的公式中,权重的偏导数在求和项中由部分组成,分别是网络输出值与真实标签值的误差激活区数的导数+(),本层的输入值。神经网络的输出值、激活函数的导数值本层的输入值都可以在正向传播吋得到,因此可以晑效的计算出来。对所有训练样本的偏导数计算均值,可以得到总的偏导数对偏置项的偏导数为:如果上式分子中的后半部分对来说是常数,有:()⊥()如果类似的有这可以统写成:写成矩阵形式为偏置项的导数由两部分组成,分别是神经网络预测值与真实值之间的误差,激活函数的导数值,与权重矩阵的偏导数相比唯一的区别是少了。接下来计算对和的偏导数,由于是复合函数的内层,情况更为复杂。()是个的短阵,它的个行向量为(),(,(,(。偏置项()是维向量,个分量分别是(),(,(),(。首先计算损失函数对的元素的偏导数:而上式分子中的两部分都有,因此都与有关。为了表述简活,我们令:根据链式法则有:其巾((和和都是标量和()是两个()向量的内积,的每一个分量都是()的函数。接下来计算和这里的一是个向量,衣示的每个分量分别对求导。当时有:后面个分量相对于求导变量(都是常数。类似的当时有:()0)(()和时的结果以此类推。综合起来有:同理有:()十如果令合并得到()()[()-)。()。()写成矩阵形式为()最后计算偏置项的偏导数()类似的我们得到:合并后得到()写成矩阵形式为:(0)至此,我得到了这个简单网络对所有参数的偏导数,接下来我们将这种做法推广到更般的情况。从上面的结果可以看岀一个规律,输出层的权重矩阵和偏置向量梯度计算公式中共用了()-)()对」隐含层也有类似的结果完整的算法现在考虑一般的情况。假设有个训练样本(),其中为输入向量,为标签向量。训练的目标是最小化样木标签值与神经网络预测值之闩的误差,如果使用均方误差,则优化的目标为:其中为神经网络所有参数的集合,包括各层的权重和偏置。这个最优化问题是·个不带约束条件的问题,可以用梯度下降法求解。上面的误差函数定义在整个训练样本集上,梯度下降法每一次迭代利用了所有训练样本,称为批量棁度卜降法。如果样木数量很大,每次迭代都用所有样木进计算成木太高。为了解决这个问题,可以采用单样本梯度下降法,我们将上面的损失函数写成对单个样本的损失函数之和:定义对单个样本()的损失函数为)=-()如果采用单个样本进行迭代,梯度下降法第次迭代时参数的更新公式为:nV如果要用所有样本进行迭代,根据单个样本的损失函数梯度计算总损失梯度即可,即所有样本梯度的均值用梯度下降法求解需要初始化优化变量的值。一般初始化为一个随机数,如用正态分布(a)产生这些随机数,其中G是一个很小的正数到日前为止还有一个关键问题没有解决:日标函数是一个多层的复合函数,因为神经网络中每一层都有权重矩阵和偏置向量,且每一层的输出将会作为下一层的输入。因此,直接计算损失函数对所有权重和偏置的梚度很复杂,需要使用复合函数的求导公式进行递推计算几个重要的结论在进行推导之前,我们首先来看下面几种复合函数的求导。又如下线性映射函数:其中是维向量,是×的矩阵,是维向量。问题:假设有函数,如果把看成常数,看成的函数,如何根据函数对的梯度值Ⅴ计算函数对的梯度值Ⅴ?根据链式法则,由于只和有关,和其他的≠无关,因此有:c∑(对于的所有元素有:写成矩阵形式为:问题:如果将看成常数,将看成的函数,如何根据V计算Ⅴ?由于任意的和所有的都有关系,根据链式法则有写成矩阵形式为这是一个对称的结果,在计算函数映射时用矩阵乘以向量得到,在求梯度时用矩阵的转置乘以的梯度得到的梯度。问题:如果有向量到向量的映射:
- 2020-12-09下载
- 积分：1
