'}hD2ddlZddlmZddlmZmZmZmZmZmZm Z ddl m Z m Z ddgZ GddeZd d ed ed ed e d zd ze_ d"de ede ede e edede ede ede ede ededededededefdZde ede ede e ede ede edededededededefdZde ede ede e ede ede edededededededefdZde ede ede e ede ede edededededededed dfd!Zy)#N)Tensor) Optimizer_use_grad_for_differentiable_default_to_fused_or_foreach_differentiable_doc _foreach_doc _maximize_doc _fused_doc)ListOptionalSGDsgdc peZdZ d ddddddedeededeeffdZfd Zd Zed d Z xZ S)rFN)maximizeforeachdifferentiablefusedrrrrc 6|dkrtd||dkrtd||dkrtd|t|||||||| |  } |r|dks|dk7r tdt | || | r"d|_| r t d |r t d yy) NgzInvalid learning rate: zInvalid momentum value: zInvalid weight_decay value: ) lrmomentum dampening weight_decaynesterovrrrrrz8Nesterov momentum requires a momentum and zero dampeningTz)`fused` does not support `differentiable`z0`fused` and `foreach` cannot be `True` together.) ValueErrordictsuper__init___step_supports_amp_scaling RuntimeError) selfparamsrrrrrrrrrdefaults __class__s N/var/www/html/test/engine/venv/lib/python3.12/site-packages/torch/optim/sgd.pyrz SGD.__init__ s 86rd;< < c>7zBC C # ;L>JK K2I%1H!)7'5UD Q)q.WX X * .2D +"#NOO"#UVV ct|||jD]\}|jdd|jdd|jdd|jdd|jdd^y)NrFrrrr)r __setstate__ param_groups setdefault)r!stategroupr$s r%r(zSGD.__setstate__$sv U#&& -E   Z /   Z /   Y -   -u 5   We ,  -r&cd}|dD]}|j|j||j|j|jjrd}|j|}|j|j d|S)NFr"Tmomentum_buffer)gradappend is_sparser+get)r!r,params_with_gradd_p_listmomentum_buffer_listhas_sparse_gradpr+s r% _init_groupzSGD._init_group-sx JAvv! ''*'66##&*O 1 $++EII6G,HI Jr&cd}|$tj5|}ddd|jD]}g}g}g}|j||||}t ||||d|d|d|d|d|d||d|d t |d dt |d d t ||D]\}} |j|} | | d <|S#1swYxYw)zPerforms a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrrrrr grad_scale found_inf) rrrrrrr6rrr:r;r.)torch enable_gradr)r8rgetattrzipr+) r!closurelossr,r3r4r5r6r7r.r+s r%stepzSGD.step<s  ""$ !y !&& ;E! H#% "..u6FRfgO  $">2z*; ,z*z* /i(Gn"4t<!$ T: < '**:?*/WEJfj(-tW>BWU]^bUcW!%W6>tnW2- "'"'r&a Implements stochastic gradient descent (optionally with momentum). .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)}, \: \lambda \text{ (weight decay)}, \\ &\hspace{13mm} \:\mu \text{ (momentum)}, \:\tau \text{ (dampening)}, \:\textit{ nesterov,}\:\textit{ maximize} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{if} \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}\textbf{if} \: \mu \neq 0 \\ &\hspace{10mm}\textbf{if} \: t > 1 \\ &\hspace{15mm} \textbf{b}_t \leftarrow \mu \textbf{b}_{t-1} + (1-\tau) g_t \\ &\hspace{10mm}\textbf{else} \\ &\hspace{15mm} \textbf{b}_t \leftarrow g_t \\ &\hspace{10mm}\textbf{if} \: \textit{nesterov} \\ &\hspace{15mm} g_t \leftarrow g_{t} + \mu \textbf{b}_t \\ &\hspace{10mm}\textbf{else} \\[-1.ex] &\hspace{15mm} g_t \leftarrow \textbf{b}_t \\ &\hspace{5mm}\textbf{if} \: \textit{maximize} \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} + \gamma g_t \\[-1.ex] &\hspace{5mm}\textbf{else} \\[-1.ex] &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma g_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} Nesterov momentum is based on the formula from `On the importance of initialization and momentum in deep learning`__. a Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-3) momentum (float, optional): momentum factor (default: 0) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) dampening (float, optional): dampening for momentum (default: 0) nesterov (bool, optional): enables Nesterov momentum (default: False) z z a Example: >>> # xdoctest: +SKIP >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9) >>> optimizer.zero_grad() >>> loss_fn(model(input), target).backward() >>> optimizer.step() __ http://www.cs.toronto.edu/%7Ehinton/absps/momentum.pdf .. note:: The implementation of SGD with Momentum/Nesterov subtly differs from Sutskever et. al. and implementations in some other frameworks. Considering the specific case of Momentum, the update can be written as .. math:: \begin{aligned} v_{t+1} & = \mu * v_{t} + g_{t+1}, \\ p_{t+1} & = p_{t} - \text{lr} * v_{t+1}, \end{aligned} where :math:`p`, :math:`g`, :math:`v` and :math:`\mu` denote the parameters, gradient, velocity, and momentum respectively. This is in contrast to Sutskever et. al. and other frameworks which employ an update of the form .. math:: \begin{aligned} v_{t+1} & = \mu * v_{t} + \text{lr} * g_{t+1}, \\ p_{t+1} & = p_{t} - v_{t+1}. \end{aligned} The Nesterov version is analogously modified. Moreover, the initial value of the momentum buffer is set to the gradient value at the first step. This is in contrast to some other frameworks that initialize it to all zeros. r"r4r5r6rrr:r;rrrrrrc|6|4tjjst|dd\}}nd}d}|d}|d}|r)tjjr t d|r)tjjr t d|r%tjjst }n-|r%tjjst }nt}|||||| | | | || || y)zlFunctional API that performs SGD algorithm computation. See :class:`~torch.optim.SGD` for details. NF)r use_fusedz6torch.jit.script not supported with foreach optimizersz4torch.jit.script not supported with fused optimizers) rrrrrr6rr:r;)r<jit is_scriptingrr _multi_tensor_sgd _fused_sgd_single_tensor_sgd)r"r4r5r6rrr:r;rrrrrrfuncs r%rrs45=yy%%'9&QVbghNE7GE }599))+STT '')QRRuyy--/  uyy--/!  " ( r&c||Jt|D]\} } | s|| n||  }|dk7r|j| |}|dk7rm|| }|)tj|j }||| <n%|j |j |d|z | r|j||}n|}| j || y)Nralphar) enumerateaddr<clonedetachmul_add_)r"r4r5r:r;rrrrrrr6iparamd_pbufs r%rOrOs  )"33 3f%#5!)hqk | 1 ''%|'4C q=&q)C{kk#&--/*-$Q'"''1y='Aggcg2 3rc ")#r&gradscN||Jt|dk(rytj|||gd} | jD]\\} }}}| xrt d|D}| rt j |}|dk7r3| rt j|| |nt j|| |}|dk7rg}d}tt|D]}||d}n|j||!|r2t j||t j||d|z ng}tt|D]y}||4t j||jx}x||<|||<n-||}|j|j||d|z |j|{| rt j|||n|}|st j| || tt| D]}| |j||| y)NrT with_indicesc34K|]}|jywrC)r1).0r/s r% z$_multi_tensor_sgd..<s8aD8asrRFr)lenr"_group_tensors_by_device_and_dtypevaluesanyr< _foreach_neg _foreach_add_ _foreach_addranger0 _foreach_mul_rVrWrXrY)r"r^r5r:r;rrrrrrr6grouped_tensors device_params device_gradsdevice_momentum_buffer_listindicesdevice_has_sparse_gradbufsall_states_with_momentum_bufferrZr]s r%rMrM(sT  )"33 3 6{aBBFESgChw{|OQ`QgQgQi1BM C-'BW!0!aS8aT`8a5a  --l;L 1 ##L-|T$11, Uab q=D.2 +3:;< @.q196;3KK ;A >?  @/##D(3##D,a)mLs#>?@%A215=!KK Q8??ABB9!.s@aT @sFr`)NN) rrrrrr is_first_stepr:r;) r deviceallrTr< empty_likerrfitemsto _fused_sgd_)r"r^r5r:r;rrrrrrr6grad_scale_dictfound_inf_dictno_momentum_bufferr{rZgrnr|dtyperorprq_device_grad_scaledevice_found_infs r%rNrNos KLL9C9Oz((*5UYO6?6Ki&& 2QUN!Q@+?@@[I[E[Me$ :DAq&+&6&6q&9  # :BB ,-ECO\k\q\q\s XXT=,8SVW.8++  !_,*4--*?' / 7   ^+)2f)=v&-f5    $B*E%'(&  r&)NNNNN)r<r optimizerrrrrr r r typingr r __all__r__doc__rGfloatrrOrMrNryr&r%rs* VVV! %.Z)Zz"D        E/^) _X @!%"& $'+&*>V >v,>"8F#34>  > $ >~>V$>F#>>> >>> !>@##tF|##!%f##-1(62B-C##$,F#3###+6"2 ## &+ ##"'##!###(##"&##"&##)-##LDBd6lDB!&\DB,0&1A,BDB#+6"2DB"*&!1 DB %* DB!&DB DB"'DB!%DB!%DB(,DBN4 L4 <4 x/04  4  4 4 4  4 4 4 4 4  4 r&