'}h0ddlZddlmZddlmZmZmZmZmZmZm Z ddl m Z m Z ddgZ GddeZd d ed ed ed ze_ dd e ede ede ede ede edededededededefdZd e ede ede ede edededededededefdZd e ede ede ede edededededededefdZy)N)Tensor) Optimizer_use_grad_for_differentiable_default_to_fused_or_foreach_differentiable_doc _foreach_doc _maximize_doc _view_as_real)ListOptionalRproprpropcfeZdZ d dddddeededeffdZfdZd Zed d Z xZ S) rNF)foreachmaximizedifferentiablerrrcd|kstd|d|dcxkrdcxkr|dksntd|dd|dt||||||}t | ||y) NgzInvalid learning rate: rg?rzInvalid eta values: z, )lretas step_sizesrrr) ValueErrordictsuper__init__) selfparamsrrrrrrdefaults __class__s P/var/www/html/test/engine/venv/lib/python3.12/site-packages/torch/optim/rprop.pyrzRprop.__init__ sby6rd;< <T!W,s,T!W,3DG9BtAwiHI I!)   *ct|||jD]8}|jdd|jdd|jdd:y)NrrFr)r __setstate__ param_groups setdefault)rstategrouprs r r#zRprop.__setstate__%sV U#&& 6E   Y -   Z /   -u 5 6r!c d}|dD]H}|j|tj|z}|j||j}|jr t d|j||j |} t| dk(rd| d<tj|tj| d<|jjr*tj|t|d|d| d <ntj||d| d <|j| d|j| d | dxxd z cc<K|S) NFrz'Rprop does not support sparse gradientsrstep memory_formatprevr step_sizer) gradtorch is_complexappend is_sparse RuntimeErrorr&len zeros_likepreserve_formatdtype full_likecomplex) rr'rgradsprevsr has_complexpr.r&s r _init_groupzRprop._init_group,s= x Avv~ 5++A. .K MM! 66D~~"#LMM LL JJqME5zQ !f % 0 0U%:%:!f 77%%geDk5;.OP+&*/uT{)KE+& LLv '   eK0 1 &MQ M= >r!c2d}|$tj5|}ddd|jD]S}g}g}g}g}|d\}} |d\} } |d} |d} |j|||||}t ||||| | || | | |d| U|S#1swYmxYw)zPerforms a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrr) step_size_min step_size_maxetaminusetaplusrrrr<)r/ enable_gradr$r>r)rclosurelossr'rr:r;rrBrCr@rArrr<s r r)z Rprop.stepOs  ""$ !y !&& EFEEJ %f Hg+0+> (M=I&GZ(H**5&% SK ++!!$%56'  6 = ! !s B  B)g{Gz?)g?g333333?)gư>2)N) __name__ __module__ __qualname__r boolrr#r>rr) __classcell__)rs@r rr sd   +#'$+$+++46!F"'"'r!a Implements the resilient backpropagation algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \theta_0 \in \mathbf{R}^d \text{ (params)},f(\theta) \text{ (objective)}, \\ &\hspace{13mm} \eta_{+/-} \text{ (etaplus, etaminus)}, \Gamma_{max/min} \text{ (step sizes)} \\ &\textbf{initialize} : g^0_{prev} \leftarrow 0, \: \eta_0 \leftarrow \text{lr (learning rate)} \\ &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm} \textbf{for} \text{ } i = 0, 1, \ldots, d-1 \: \mathbf{do} \\ &\hspace{10mm} \textbf{if} \: g^i_{prev} g^i_t > 0 \\ &\hspace{15mm} \eta^i_t \leftarrow \mathrm{min}(\eta^i_{t-1} \eta_{+}, \Gamma_{max}) \\ &\hspace{10mm} \textbf{else if} \: g^i_{prev} g^i_t < 0 \\ &\hspace{15mm} \eta^i_t \leftarrow \mathrm{max}(\eta^i_{t-1} \eta_{-}, \Gamma_{min}) \\ &\hspace{15mm} g^i_t \leftarrow 0 \\ &\hspace{10mm} \textbf{else} \: \\ &\hspace{15mm} \eta^i_t \leftarrow \eta^i_{t-1} \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1}- \eta_t \mathrm{sign}(g_t) \\ &\hspace{5mm}g_{prev} \leftarrow g_t \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to the paper `A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm `_. a Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-2) etas (Tuple[float, float], optional): pair of (etaminus, etaplus), that are multiplicative increase and decrease factors (default: (0.5, 1.2)) step_sizes (Tuple[float, float], optional): a pair of minimal and maximal allowed step sizes (default: (1e-6, 50)) z z rr:r;rrrrr<r@rArBrCc |t||d\} }|r)tjjr t d|r%tjjst } nt } | |||||| | | ||| y)zpFunctional API that performs rprop algorithm computation. See :class:`~torch.optim.Rprop` for details. NF) use_fusedz6torch.jit.script not supported with foreach optimizers)r@rArBrCrrr<)rr/jit is_scriptingr3_multi_tensor_rprop_single_tensor_rprop)rr:r;rrrrr<r@rArBrC_funcs r rrs,1&.TYZ 7599))+STTuyy--/"#  ##% r!cRt|D]\} } || } |s| n| } || }|| }tj| rTtj| } tj|}tj| } tj|}| r.| j |j j }n| j |j }|||jd<|||jd<d||jd<|j|j||| j tj} d| |j|<| j| j |d|j| y)Nrrr*value) enumerater/r0 view_as_realmulclonesigngtlteqmul_clamp_r6addcmul_copy_)rr:r;rr@rArBrCrrr<iparamr.r,r-r]s r rRrRshf%5Qx#t$QxqM   E "%%d+D%%d+D&&u-E**95I 88DJJL)..0D88D>&&(D"TWWQZ#TWWQZTWWQZ t##M=Azz(=(=z>"#TWWX  tyy{IR8 4;r!c~t|dk(ry| rJdtj||||g} | jD]s\\} } }}}| rt | | ||t j | |}|rt j|t j|| |rt j||} t j||D]>}|||jd<|||jd<d||jd<@t j|||D]}|j||t| } t!t| D]}d| |||j|<~| Dcgc]}|j#}}t j$| ||dvycc}w)Nrz#_foreach ops don't support autogradrrVrW)r4r"_group_tensors_by_device_and_dtypevaluesr r/ _foreach_mul _foreach_neg__foreach_copy__foreach_sign_r^r_r` _foreach_mul_rblistranger]_foreach_addcmul_)rr:r;rr@rArBrCrrr<grouped_tensorsgrouped_params grouped_grads grouped_prevsgrouped_step_sizesrSsignsr]r-rer. grad_signss r rQrQ s 6{aDDD BBFESXZdCefOSbSiSiSk'ZO K.-8JQ  .-HZ [""=-@     & ]M:     .%  U# !D&D 'D  D  ! .6+ ;I   ]M : ; ]+ s=)* 8A67M! U1X[[2 3 8 /<rs YYY! G mIm`"D      E0 r# - L- <- <-V -d^--------`, L, <, <,V ,  ,,,,,,,^<Z L<Z <<Z <<ZV <Z  <Z<Z<Z<Z<Z<Z<Zr!