'}h+8ddlZddlmZddlmZmZmZmZmZmZm Z ddl m Z m Z ddgZ GddeZd d ed ed ed ze_ dd e ede ede ede ede ede edededededededededefdZd e ede ede ede ede edededededededededefdZd e ede ede ede ede edededededededededefdZy) N)Tensor) Optimizer_default_to_fused_or_foreach_use_grad_for_differentiable_differentiable_doc _foreach_doc _maximize_doc _view_as_real)ListOptionalRMSproprmspropcheZdZ ddeededeffd ZfdZdZed dZ xZ S) rforeachmaximizedifferentiablec  d|kstd|d|kstd|d|kstd|d|kstd|d|kstd|t|||||||| |  } t | || y)NgzInvalid learning rate: zInvalid epsilon value: zInvalid momentum value: zInvalid weight_decay value: zInvalid alpha value: ) lrmomentumalphaepscentered weight_decayrrr) ValueErrordictsuper__init__) selfparamsrrrrrrrrrdefaults __class__s R/var/www/html/test/engine/venv/lib/python3.12/site-packages/torch/optim/rmsprop.pyrzRMSprop.__init__ sby6rd;< <cz6se<= =h7zBC Cl";L>JK Ke|4UG<= =%)   *ct|||jD]\}|jdd|jdd|jdd|jdd|jdd^y)NrrrFrrr)r __setstate__ param_groups setdefault)rstategroupr"s r#r&zRMSprop.__setstate__0sv U#&& 6E   Z +   Z /   Y -   Z /   -u 5  6r$cFd}|dD]}|j|tj|z}|j||jjr t d|j|j|j |} t| dk(rd| d<tj|tj| d<|ddkDr(tj|tj| d <|d r(tj|tj| d <|j| d|ddkDr|j| d |d r|j| d |d rt| dtr t d | dxxdz cc<|S)NFr z)RMSprop does not support sparse gradientsrstep) memory_format square_avgrmomentum_bufferrgrad_avgrz`step` can't be a tensorr) gradtorch is_complexappend is_sparse RuntimeErrorr)len zeros_likepreserve_format isinstancer) rr*params_with_gradgrads square_avgsmomentum_buffer_list grad_avgs has_complexpr)s r# _init_groupzRMSprop._init_group9s x$ Avv~ 5++A. .K  # #A &vv"#NOO LL JJqME5zQ !f &+&6&6U%:%:'l#$q(/4/?/?)>)>0E+,$(-(8(8)>)>)E*%   u\2 3Z 1$$++E2C,DEZ   z!23%&:eFmV+L"#=>> &MQ MI$ Jr$c:d}|$tj5|}ddd|jD]W}g}g}g}g}g}|j||||||} t ||||||d|d|d|d|d|d|d|d |d | Y|S#1swYqxYw) zPerforms a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrrrrrr) rrrrrrrrrr@)r2 enable_gradr'rBr) rclosurelossr*r;r<r=r?r>r@s r#r,z RMSprop.stepbs  ""$ !y !&& E! EKI#% **52BE;XlnwxK  $;Gn%L">2z*z*i(z*$%56'  6 = ! !s BB) g{Gz?gGz?g:0yE>rrFNFF)N) __name__ __module__ __qualname__r boolrr&rBrr, __classcell__)r"s@r#rr sg  "&$#+$#+#+#+J6'R"'"'r$a Implements RMSprop algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \alpha \text{ (alpha)},\: \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)},\: \mu \text{ (momentum)},\: centered\\ &\textbf{initialize} : v_0 \leftarrow 0 \text{ (square average)}, \: \textbf{b}_0 \leftarrow 0 \text{ (buffer)}, \: g^{ave}_0 \leftarrow 0 \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}if \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}v_t \leftarrow \alpha v_{t-1} + (1 - \alpha) g^2_t \hspace{8mm} \\ &\hspace{5mm} \tilde{v_t} \leftarrow v_t \\ &\hspace{5mm}if \: centered \\ &\hspace{10mm} g^{ave}_t \leftarrow g^{ave}_{t-1} \alpha + (1-\alpha) g_t \\ &\hspace{10mm} \tilde{v_t} \leftarrow \tilde{v_t} - \big(g^{ave}_{t} \big)^2 \\ &\hspace{5mm}if \: \mu > 0 \\ &\hspace{10mm} \textbf{b}_t\leftarrow \mu \textbf{b}_{t-1} + g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \\ &\hspace{10mm} \theta_t \leftarrow \theta_{t-1} - \gamma \textbf{b}_t \\ &\hspace{5mm} else \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \hspace{3mm} \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `lecture notes `_ by G. Hinton. and centered version `Generating Sequences With Recurrent Neural Networks `_. The implementation here takes the square root of the gradient average before adding epsilon (note that TensorFlow interchanges these two operations). The effective learning rate is thus :math:`\gamma/(\sqrt{v} + \epsilon)` where :math:`\gamma` is the scheduled learning rate and :math:`v` is the weighted moving average of the squared gradient. a Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-2) momentum (float, optional): momentum factor (default: 0) alpha (float, optional): smoothing constant (default: 0.99) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) centered (bool, optional) : if ``True``, compute the centered RMSProp, the gradient is normalized by an estimation of its variance weight_decay (float, optional): weight decay (L2 penalty) (default: 0) z z r r<r=r?r>rrrr@rrrrrrc |t||d\}}|r)tjjr t d|r%tjjst }nt }||||||| | | | | ||||y)zsFunctional API that performs rmsprop algorithm computation. See :class:`~torch.optim.RMSProp` for details. NF) use_fusedz6torch.jit.script not supported with foreach optimizers) rrrrrrrrr@)rr2jit is_scriptingr6_multi_tensor_rmsprop_single_tensor_rmsprop)r r<r=r?r>rrrr@rrrrrr_funcs r#rrs01&.TYZ 7599))+STTuyy--/$%   !%r$c ht|D]\}}||}| s|n| }||}|dk7r|j||}tj|}|r?tj|}tj|}tj|}|j |j ||d|z | rT||}|rtj|}|j|d|z |j||dj}n|j}| r|j|}n|j|}| dkDrS||}|rtj|}|j | j|||j|| |j||| y)Nrrrvalue) enumerateaddr2r3 view_as_realmul_addcmul_lerp_addcmulsqrt_sqrtadd_addcdiv_)r r<r=r?r>rrrrrrrrr@iparamr1r.is_complex_paramr0avgbufs r#rQrQs$f%%15Qx#t$ ^ 1 88E86D ++E2 &&u-E%%d+D++J7J''d!e)'D  |H --h7 NN4U +$$Xxr$BHHJC//#C ''#,C((3-C a<&q)C((- HHX  ' 'c 2 JJs2#J & NN4RCN 0K%1r$c t|dk(ry| rJdtj|||||g}|jD]\\}}}}}}| r9||g}| dkDr|j || r|j |t |g|| rt j|}|dk7r3| rt j|||nt j|||}t j||t j|||d|z | r_t j||d|z t j|||d}t j|t j||n+t j|}t j||| dkDrHt j|| t j |||t j||| t j |||| y)Nrz#_foreach ops don't support autogradrUrrVrX)r7r"_group_tensors_by_device_and_dtypevaluesr4r r2 _foreach_neg _foreach_add_ _foreach_add _foreach_mul__foreach_addcmul__foreach_lerp__foreach_addcmul_foreach_sqrt_ _foreach_sqrt_foreach_addcdiv_)r r<r=r?r>rrrrrrrrr@grouped_tensorsgrouped_params grouped_gradsgrouped_square_avgsgrouped_grad_avgsgrouped_momentum_buffer_listrRstate_and_gradsrgs r#rPrP9s$ 6{aDDD BBFES^`ikDABO/>/E/E/G%S , '>=*=?P % ,.ABO!|&&'CD&&'89 . ;? ; !..}=M 1 ##M>V % 2 2=.Xd e  /7  3]MYZ]bYbc   !2M1u9 M(()<>OQbjlmC   %   S )%%&9:C   S ) a<    rs YYY! i @i@F*T      U:J# 2 L2 <2f2F| 2 v, 2d^2222 2 2 2 !2"#2$%2j71 L71 <71f71F| 71 v, 71 71 71 71717171717171t=S L=S <=Sf=SF| =S v, =S =S =S =S=S=S=S=S=S=Sr$