|h9 XdZddlZddlZddlZddlmcmZddlm Z ddl m Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZdd lmZGd d ej4j6j8Zd ZeddgGddeZGddeZ Gdde!Z"GddZ# ej4jHjJjMddej4jHjJjOddddgejjSd#ee_y#e($riej4jHjJjMd d!ej4jHjJjOd"dddgYwxYw)$zBase class of optimizer.N)logging)backend) initializers)utils)learning_rate_schedule)tf_utils) keras_export) doc_controlscBeZdZdZ d,dZdZdZdZdZdZ e jd Z e jd d Zd Zd-d ZdZedZej*dZedZej*dZeej0dZej*dZdZe jdZdZd.dZ d/dZd0fd Zd-dZ dZ!d/dZ"dZ#d-dZ$d Z%d!Z&d"Z'd#Z(d$Z)d%Z*d&Z+e,d-d'Z-ed(Z.d)Z/d*Z0d+Z1xZ2S)1_BaseOptimizerzBOptimizer base class, which only supports non-distribute use case.Tc  V||_||_||_||_||_||_| r;t jr'tjjdrd|_ nd|_ tjdk(r[tjdk(rDtj d|j"j$d|j"j$d|r=|d kDs|d krt'd ||r#t)|t*r|d krt'd |||_||_|j2|j&t'd |jd|jdg|_|j3i|_|j7|j9| y)NGPUTFDarwinarmz8At this time, the v2.11+ optimizer `tf.keras.optimizers.zv` runs slowly on M1/M2 Macs, please use the legacy TF-Keras optimizer instead, located at `tf.keras.optimizers.legacy.`.rzC`ema_momentum` must be in the range [0, 1]. Received: ema_momentum=z\`ema_overwrite_frequency` must be an integer > 1 or None. Received: ema_overwrite_frequency=zOAt most one of `clipnorm` and `global_clipnorm` can be set. Received: clipnorm=z, global_clipnorm=.)name weight_decayclipnormglobal_clipnorm clipvalueuse_emarcan_jit_compiletfconfiglist_physical_devices jit_compileplatformsystem processorrwarning __class____name__ ValueError isinstanceint ema_momentumema_overwrite_frequency _variables_no_dependency_sharded_variable_builders_create_iteration_variable_process_kwargs) selfrrrrrrr(r)rkwargss `/var/www/html/test/engine/venv/lib/python3.12/site-packages/tf_keras/src/optimizers/optimizer.py__init__z_BaseOptimizer.__init__'s (  ."  ((* //6#D $D  ??  (X-?-?-AU-J OO((,(?(?'@A/04~~/F/F.Gr K  a$ == $)=)=)I..2mm_=##'#7#7"8;  +/*=*=b*A' '') V$ctj5tjddtjd|_ddd|j j |jy#1swY/xYw)z'Create the iterations counter variable.r iterationFrdtype trainableN)r init_scopeVariableint64 _iterationsr*appendr/s r1r-z)_BaseOptimizer._create_iteration_variablepsZ ]]_  "{{ 288u D   t//0   s -A00A9c,|jdd|jdd}|r-tjd|jjdhd}|D]8}||vr%t |d|jjdt |dy)Nis_legacy_optimizerlrz`lr` is deprecated in TF-Keras optimizer, please use `learning_rate` or use the legacy optimizer, e.g.,tf.keras.optimizers.legacy.r>decaygradient_aggregatorgradient_transformersz is deprecated in the new TF-Keras optimizer, please check the docstring for valid arguments, or use the legacy optimizer, e.g., tf.keras.optimizers.legacy.zY is not a valid argument, kwargs should be empty for `optimizer_experimental.Optimizer`.)poprr"r#r$r% TypeError)r/r0rA legacy_kwargsks r1r.z_BaseOptimizer._process_kwargszs ($/ ZZd # OO..2nn.E.E-FaI    AM! c237..2I2I1J!M c?? r3c Htd|jjd)Na You are trying to restore a checkpoint from a legacy TF-Keras optimizer into a v2.11+ Optimizer, which can cause errors. Please update the optimizer referenced in your code to be an instance of `tf.keras.optimizers.legacy.Optimizer`, e.g.: `tf.keras.optimizers.legacy.r)r%r#r$)r/r0s r1 _create_or_restore_slot_variablez/_BaseOptimizer._create_or_restore_slot_variables. + ,0>>+B+B*C2  G  r3c|jS).Get a unique identifier of the given variable. _unique_idr/variables r1_var_keyz_BaseOptimizer._var_keys """r3cg}|D]}t|tjr|j}|j}tj |\}}tj j||tj|d}|jtj|||j|j||S)aBDeduplicate sparse gradient. For sparse gradients, i.e., gradient is of type `tf.IndexedSlices`, it is possible that `gradient.indices` has duplicated indices. This function adds up values for the duplicated indices, and returns a `tf.IndexedSlices` with indices of unique values. r) r&r IndexedSlicesvaluesindicesuniquemathunsorted_segment_sumshaper= dense_shape) r/gradsprocessed_gradsgradrTrUunique_indicesnew_index_positions summed_valuess r1_deduplicate_sparse_gradz'_BaseOptimizer._deduplicate_sparse_grads -D$ 0 01,,68ii6H3 3 " < </.1I!1L!  &&$$%~t7G7G  &&t, - r3ct)aiFunction to update variable value based on given gradients. This method must be implemented in customized optimizers. Args: gradient: backpropagated gradient of the given variable. variable: variable whose value needs to be updated. Returns: An `Operation` that applies the specified gradients. NotImplementedErrorr/gradientrPs r1 update_stepz_BaseOptimizer.update_steps "!r3)rc&|j||S)aA wrapper of `update_step` to enable XLA acceleration. Due to `tf.function` tracing mechanism, for (gradient, variable) pairs of the same shape and dtype, the execution graph always invoke the first pair it has seen. Thus, we need a `key` argument to make each (gradient, variable) pair unique. In additions, XLA cannot understand string input, so the key is an integer. Args: gradient: backpropagated gradient of the given variable. variable: variable whose value needs to be updated. key (int): a unique key that identifies the variable. Returns: An `Operation` that applies the specified gradients. ) _update_step)r/rfrPkeys r1_update_step_xlaz_BaseOptimizer._update_step_xlas$  844r3ct|ddy|j||jvr0td|jd|j j d|j||y)NrNz(The optimizer cannot recognize variable a . This usually means you are trying to call the optimizer to update different parts of the model separately. Please call `optimizer.build(variables)` with the full list of trainable variables before the training loop or use legacy optimizer `tf.keras.optimizers.legacy.r)getattrrQ _index_dictKeyErrorrr#r$rgres r1riz_BaseOptimizer._update_stepsw 8\4 0 8  == "$*:*: ::8==/J/ 04~~/F/F.Gq J  8,r3cht|s|td|d|d|tj}t|r@|5t|s|j ||}t|r|}ddd|j ||}t t||S#1swY0xYw)aBCompute gradients of loss on trainable variables. Args: loss: `Tensor` or callable. If a callable, `loss` should take no arguments and return the value to minimize. var_list: list or tuple of `Variable` objects to update to minimize `loss`, or a callable returning the list or tuple of `Variable` objects. Use callable when the variable list would otherwise be incomplete before `minimize` since the variables are created at the first time `loss` is called. tape: (Optional) `tf.GradientTape`. If `loss` is provided as a `Tensor`, the tape that computed the `loss` must be provided. Returns: A list of (gradient, variable) pairs. Variable is always present, but gradient can be `None`. NzB`tape` is required when a `Tensor` loss is passed. Received: loss=z, tape=r)callabler%r GradientTapewatchrflistzip)r/lossvar_listtaper[s r1compute_gradientsz _BaseOptimizer.compute_gradientss$~$,""&wtfA7  <??$D D> *)JJx(vH%'zH  * dH-Cx()) * *s 6B((B1c<g}|jr[|jdkDrL|D]E}||j||jtj||jG|S|jr2|jdkDr#tj ||jdS|j rh|j dkDrY|D]R}||j||jtj||j |j T|S|S)Nr)clip_value_minclip_value_max)rr=r clip_by_normrclip_by_global_normr clip_by_value)r/r[ clipped_gradsgs r1_clip_gradientsz_BaseOptimizer._clip_gradientss ==T]]Q. L9!((+!((DMM)JK  L !   D$8$81$<))%1E1EFqI I >>dnnq0 9!((+!((((,0NN?+/>> !  r3c|jS)zThe number of training steps this `optimizer` has run. By default, iterations would be incremented by one every time `apply_gradients()` is called. )r<r>s r1 iterationsz_BaseOptimizer.iterations8sr3cJt|ddrtd|d||_y)N_builtFzCannot set `iterations` to a new Variable after the Optimizer weights have been created. Here it is attempting to set `iterations` to z.Usually this means you are trying to set `iterations` after calling `apply_gradients()`. Please set `iterations` before calling `apply_gradients()`.)rm RuntimeErrorr<rOs r1rz_BaseOptimizer.iterationsAs; 45 )55=J?CC $r3ct|dr |j td|j}t|tj r |j S|S)N_learning_ratezPMissing learning rate, please set self.learning_rate at optimizer creation time.)hasattrrr%r&rLearningRateSchedule_current_learning_rate)r/rAs r1 learning_ratez_BaseOptimizer.learning_rateNs[t-.$2E2E2M,   b0EE F.. . r3ct|tjr||_yt|jtjr t d|jj |y)NaThis optimizer was created with a `LearningRateSchedule` object as its `learning_rate` constructor argument, hence its learning rate is not settable. If you need the learning rate to be settable, you should instantiate the optimizer with a float `learning_rate` argument.)r&rrrrFassignr/rs r1rz_BaseOptimizer.learning_rate]se  1FF #0D ##%;%P%P K    & &} 5r3c|jS)zAlias of `learning_rate()`. `lr()` is heavily called in workflows using `optimizer_v2.OptimizerV2`, so we keep it for backward compabitliy. rr>s r1rAz_BaseOptimizer.lrps!!!r3c||_yNrrs r1rAz_BaseOptimizer.lrzs *r3ctj5t|tjrXtj ||j }tj|d|jd|_ |cdddStj|dtjdcdddS#1swYyxYw)Ncurrent_learning_rateFr6r) rr9r&rrconvert_to_tensorrr:r7rrfloatx)r/rrs r1_build_learning_ratez#_BaseOptimizer._build_learning_rate~s ]]_ 5JJ)+(<(<!$//2)%/1kk)0/55# /+ %   ;;$nn& !   sA)B<*B<<Cct|ddry|j||jr< 0"--E"((E  }!'',,4%'MM.$NM%'HH^-A-A$OM "e < ;;'!?!N$?$?#@A   >#7 8-@@B 88CC!,,m<',<,F,F(GHJ   *((*&&z'7'7'9: OO " "8 ,r3c ( t|ds|ji|_t ||fi|}|j D]\}}|dvr|}t |trg}|D]V}t|dr7|j t fd|Dr4|j F|j|Xtjjj|}||j|<||j|<|jS)a*Override in order to coalesce and track `ShardedVariable`s. If an optimizer variable's corresponding model variable is a shard of a larger `ShardedVariable`, then we track the optimizer variable in `self._variables` as a `ShardedVariable` via the logic in `add_variable_from_reference`. However, most optimizer implementations additionally keep their variables as attributes, which will be tracked via `AutoTrackable` functionality and not accumulated into `ShardedVariable`s. So, to enable restoration of these attributes in possibly different sharding configurations, we should save them as `ShardedVariable`s. Here, any optimizer attributes that are variable shards of a larger `ShardedVariable` are here replaced by the `ShardedVariable` itself, which was created in `add_variable_from_reference`. All non-sharded variables are kept as-is. If none of the model variables are sharded, this reduces to `AutoTrackable._trackable_children()`. _coalesced_children)r*rnrr<rc3PK|]}j|jk(ywrrM).0other_svsvs r1 z5_BaseOptimizer._trackable_children..Ls(+"(0%'MMX5H5H$H+"s#&)rr+rsuper_trackable_childrenitemsr&rtranyr=r __internal__trackingwrap) r/ save_typer0childrenrjvalnew_valsv_valsrrr#s @r1rz"_BaseOptimizer._trackable_childrens.t23(,':':2'>D $w29GGH$NN, 8S "G!#t,"$#& 4C&s,@A%(%;%;%=(++"4;+"("%,NN2$6 's 3 4#%//":":"?"?"H4;D,,S147D,,S17 88'''r3cL|j|||}|j|y)aMinimize `loss` by updating `var_list`. This method simply computes gradient using `tf.GradientTape` and calls `apply_gradients()`. If you want to process the gradient before applying then call `tf.GradientTape` and `apply_gradients()` explicitly instead of using this function. Args: loss: `Tensor` or callable. If a callable, `loss` should take no arguments and return the value to minimize. var_list: list or tuple of `Variable` objects to update to minimize `loss`, or a callable returning the list or tuple of `Variable` objects. Use callable when the variable list would otherwise be incomplete before `minimize` since the variables are created at the first time `loss` is called. tape: (Optional) `tf.GradientTape`. Returns: None N)ryapply_gradients)r/rvrwrxgrads_and_varss r1minimizez_BaseOptimizer.minimizeYs&*//hE ^,r3c|t|jtjrt |dr5|j j |j|jytj|j|j}tj|d|jd|_yy)NrrFr6) r&rrrrrrrrrr:r7)r/rs r1_compute_current_learning_ratez-_BaseOptimizer._compute_current_learning_rateqs    !7!L!L t56++22''8)+(<(<''8)%/1kk)0/55# /+ r3ct|dr|jr td|r%|Dcgc]}|j|c}|_ng|_|xsg|_ycc}w)aExclude variables from weight decay. This method must be called before the optimizer's `build` method is called. You can set specific variables to exclude out, or set a list of strings as the anchor words, if any of which appear in a variable's name, then the variable is excluded. Args: var_list: A list of `tf.Variable`s to exclude from weight decay. var_names: A list of strings. If any string in `var_names` appear in the model variable's name, then this model variable is excluded from weight decay. For example, `var_names=['bias']` excludes all bias variables from weight decay. rzR`exclude_from_weight_decay()` can only be configued before the optimizer is built.N)rrr%rQ_exclude_from_weight_decay _exclude_from_weight_decay_names)r/rw var_namesrPs r1exclude_from_weight_decayz(_BaseOptimizer.exclude_from_weight_decaysg 4 "t{{*  8@/,4 h'/D +/1D +09R- /sAct|dg}t|dg}|j|}|D] }||k(s y|D]$}tj||j$yy)NrrFT)rmrQresearchr)r/rPrexclude_from_weight_decay_names variable_id exclude_idrs r1_use_weight_decayz _BaseOptimizer._use_weight_decays$+ .% !+2 4b+ 'mmH- 3 Jj( 4 Dyyx}}-9 r3c*|jt|}t|dk(r |jSt |\}}|xs|j xsd}t j|5t j5|j|dddtj|}tt|dk(r|jcdddSt |\}}|j|}|j|}|j|tt ||}|j|}|D]/}|j |j#|j!|1|cdddS#1swYxYw#1swYyxYw)aApply gradients to variables. Args: grads_and_vars: List of `(gradient, variable)` pairs. name: string, defaults to None. The name of the namescope to use when creating variables. If None, `self.name` will be used. Returns: A `tf.Variable`, representing the current iteration. Raises: TypeError: If `grads_and_vars` is malformed. r optimizerN)rrtrr<rurr name_scoper9roptimizer_utilsfilter_empty_gradientsrra_apply_weight_decay_internal_apply_gradients constraintr)r/rrr[trainable_variables scope_namer5rPs r1rz_BaseOptimizer.apply_gradientss ++-n- ~ ! ### #%(.%9""5TYY5+ ]]: &  0 ./ 0-CCN4'(A-''  *-n)= &E&((/E11%8E  $ $%8 9!#e-@"ABN66~FI0 C&&2OOH$7$7$AB C1   0 0  s1*F ?E=?F A5F #F =F F  Fc&|jy|D]}|j|stj|j|j }tj|j|j }|j ||z|zyr)rrrcastrr7 assign_sub)r/rrPrAwds r1rz"_BaseOptimizer._apply_weight_decaysy    $ ! 8H%%h/WWT//@WWT..?##HrMB$67  8r3c |jr6|D]0\}}|j||t|j|2n|D]\}}|j |||j j dS)zHelper function of apply gradients. This is required for separating out distributed training logic. Args: grads_and_vars: List of (gradient, variable) pairs. r)rrkidrQrir assign_add)r/rr]rs r1rz(_BaseOptimizer._internal_apply_gradientss|   + I c%%dCDMM#4F1GH I, - c!!$, -))!,,r3c|jrd|D]^}|j|j|j|}|j |j |zd|j z |zz`yy)8Update the stored moving average using the latest value.rN)rrrnrQrr(r/rwrrs r1&_update_model_variables_moving_averagez5_BaseOptimizer._update_model_variables_moving_averagesu << >>$$T]]3%78%%/1t7H7H3HC2OO   r3c|D]>}|j|j|j|}|j|@y)z2Overwrite model variables with its moving average.N)rrnrQrrs r1-_overwrite_model_variables_with_average_valuez<_BaseOptimizer._overwrite_model_variables_with_average_valuesF C::  s!34G JJw   r3c@|jr|j|yy)aSet the final value of model's trainable variables. Sometimes there are some extra steps before ending the variable updates, such as overriding the model variables with its average value. Args: var_list: list of model variables. N)rrr/rws r1finalize_variable_valuesz'_BaseOptimizer.finalize_variable_valuess  <<  > >x H r3ct|tjrtj|St|tj r|j St|r|S|S)z=Serialize a hyperparameter that can be a numeric or callable.)r&rr serializerr:numpyrq)r/hyperparameters r1_serialize_hyperparameterz(_BaseOptimizer._serialize_hyperparameters]  2GG *33NC C nbkk 2!'') ) N #!# #r3c |j|j|j|j|j|j |j |j|jdd }|S)aReturns the config of the optimizer. An optimizer config is a Python dictionary (serializable) containing the configuration of an optimizer. The same optimizer can be reinstantiated later (without any saved state) from this configuration. Subclass optimizer should override this method to include other hyperparameters. Returns: Python dictionary. F) rrrrrrr(r)rr@) rrrrrrr(r)r)r/rs r1 get_configz_BaseOptimizer.get_config*s^II -- #33|| --'+'C'C++#(   r3czd|vr0t|dtrtj|d||d<|di|S)aCreates an optimizer from its config. This method is the reverse of `get_config`, capable of instantiating the same optimizer from the config dictionary. Args: config: A Python dictionary, typically the output of get_config. custom_objects: A Python dictionary mapping names to additional user-defined Python objects needed to recreate this optimizer. Returns: An optimizer instance. r)custom_objects)r&dictr deserialize)clsrrs r1 from_configz_BaseOptimizer.from_configFsH f $&148*@*L*L?+N+'}V}r3c,t|jS)z$Returns variables of this optimizer.) CallableListr*r>s r1rz_BaseOptimizer.variables\sDOO,,r3c Xt|dds tdt|j|D]y\}}|j|jk7rJtd|j |dt |jdt |jd|j|{y) zSet the weights of the optimizer. Args: weights: a list of `tf.Variable`s or numpy arrays, the target values of optimizer variables. It should have the same order as `self._variables`. rFzYou are calling `set_weights()` on an optimizer that has not yet been built. Please call `optimizer.build(trainable_variables)` to create the optimizer weights before calling `set_weights()`.zOptimizer variable z has shape z+ not compatible with provided weight shape rN)rmr%rur*rYrQrr)r/weightsrPweights r1 set_weightsz_BaseOptimizer.set_weightsastXu-D !$DOOW = $ Hf~~- )$--*A)B+8>>*+,$$' $5#6a9 OOF # $r3cvt|jD]!\}}|j|t|<#y)z'Get the state of this optimizer object.N)rrrr)r/storerrPs r1save_own_variablesz!_BaseOptimizer.save_own_variableszs2$T^^4 -KAx$NN,E#a&M -r3c t|jt|jk7rsd|jdt|jdt|jd}t|jdk(r|dz }t j |yt |jD]"\}}|j|t|$y)z'Set the state of this optimizer object.z)Skipping variable loading for optimizer 'z', because it has z+ variables whereas the saved optimizer has z variables. rzCThis is likely because the optimizer has not been called/built yet.N) rkeysrrrr"rrr)r/rmsgrrPs r1load_own_variablesz!_BaseOptimizer.load_own_variabless uzz| DNN 3 3;DII;G""%dnn"5!67++.uzz|+<*=\K  4>>"a'( OOC $T^^4 +KAx OOE#a&M * +r3)NNNNFGz?NTr)NrNNN) checkpoint)3r$ __module__ __qualname____doc__r2r-r.rJrQraabcabstractmethodrgrfunctionrkriryrpropertyrsetterrr do_not_generate_docsrArrrrrrrrrrrrrrrrrr classmethodr rrrr __classcell__r#s@r1r r $sL  $G%R1: #6  " "R[[T"5#5&- "*H:   $ $  66$&&"'"YY++0 2*"8HLAF8(t-0*@: .`8-   I 8*--$2- +r3r ac name: String. The name to use for momentum accumulator weights created by the optimizer. weight_decay: Float, defaults to None. If set, weight decay is applied. clipnorm: Float. If set, the gradient of each weight is individually clipped so that its norm is no higher than this value. clipvalue: Float. If set, the gradient of each weight is clipped to be no higher than this value. global_clipnorm: Float. If set, the gradient of all weights is clipped so that their global norm is no higher than this value. use_ema: Boolean, defaults to False. If True, exponential moving average (EMA) is applied. EMA consists of computing an exponential moving average of the weights of the model (as the weight values change after each training batch), and periodically overwriting the weights with their moving average. ema_momentum: Float, defaults to 0.99. Only used if `use_ema=True`. This is the momentum to use when computing the EMA of the model's weights: `new_average = ema_momentum * old_average + (1 - ema_momentum) * current_variable_value`. ema_overwrite_frequency: Int or None, defaults to None. Only used if `use_ema=True`. Every `ema_overwrite_frequency` steps of iterations, we overwrite the model variable by its moving average. If None, the optimizer does not overwrite model variables in the middle of training, and you need to explicitly overwrite the variables at the end of training by calling `optimizer.finalize_variable_values()` (which updates the model variables in-place). When using the built-in `fit()` training loop, this happens automatically after the last epoch, and you don't need to do anything. jit_compile: Boolean, defaults to True. If True, the optimizer will use XLA compilation. If no GPU device is found, this flag will be ignored. mesh: optional `tf.experimental.dtensor.Mesh` instance. When provided, the optimizer will be run in DTensor mode, e.g. state tracking variable will be a DVariable, and aggregation/reduction will happen in the global DTensor context. **kwargs: keyword arguments only used for backward compatibility.zkeras.optimizers.Optimizerz'keras.optimizers.experimental.Optimizer)v1ceZdZdZ dfd Z dfd ZfdZfdZdZ dfd Z dZ fd Z fd Z fd Z d Zd ZxZS) OptimizeraAAbstract optimizer base class. This class supports distributed training. If you want to implement your own optimizer, please subclass this class instead of _BaseOptimizer. Args: {{base_optimizer_keyword_args}} ### Usage ```python # Create an optimizer with the desired parameters. opt = keras.optimizers.SGD(learning_rate=0.1) var1, var2 = tf.Variable(1.0), tf.Variable(2.0) # `loss` is a callable that takes no argument and returns the value # to minimize. loss = lambda: 3 * var1 * var1 + 2 * var2 * var2 # Call minimize to update the list of variables. opt.minimize(loss, var_list=[var1, var2]) ``` ### Processing gradients before applying them Calling `minimize()` takes care of both computing the gradients and applying them to the variables. If you want to process the gradients before applying them you can instead use the optimizer in three steps: 1. Compute the gradients with `tf.GradientTape`. 2. Process the gradients as you wish. 3. Apply the processed gradients with `apply_gradients()`. Example: ```python # Create an optimizer. opt = tf.keras.optimizers.experimental.SGD(learning_rate=0.1) var1, var2 = tf.Variable(1.0), tf.Variable(2.0) # Compute the gradients for a list of variables. with tf.GradientTape() as tape: loss = 3 * var1 * var1 + 2 * var2 * var2 grads = tape.gradient(loss, [var1, var2]) # Process the gradients. grads[0] = grads[0] + 1 # Ask the optimizer to apply the gradients on variables. opt.apply_gradients(zip(grads, [var1, var2])) ``` ### Dynamic learning rate Dynamic learning rate can be achieved by setting learning rate as a built-in or customized `tf.keras.optimizers.schedules.LearningRateSchedule`. Example: >>> var = tf.Variable(np.random.random(size=(1,))) >>> learning_rate = tf.keras.optimizers.schedules.ExponentialDecay( ... initial_learning_rate=.01, decay_steps=20, decay_rate=.1) >>> opt = tf.keras.optimizers.experimental.SGD(learning_rate=learning_rate) >>> loss = lambda: 3 * var >>> opt.minimize(loss, var_list=[var]) ### Gradients clipping Users can clip the gradients before applying to variables by setting `clipnorm`, `clipvalue` and `global_clipnorm`. Notice that `clipnorm` and `global_clipnorm` can only have one being set. Example: >>> opt = tf.keras.optimizers.experimental.SGD(learning_rate=1, clipvalue=1) >>> var1, var2 = tf.Variable(2.0), tf.Variable(2.0) >>> with tf.GradientTape() as tape: ... loss = 2 * var1 + 2 * var2 >>> grads = tape.gradient(loss, [var1, var2]) >>> print([grads[0].numpy(), grads[1].numpy()]) [2.0, 2.0] >>> opt.apply_gradients(zip(grads, [var1, var2])) >>> # Without clipping, we should get [0, 0], but as gradients are clipped >>> # to have max value 1, we get [1.0, 1.0]. >>> print([var1.numpy(), var2.numpy()]) [1.0, 1.0] ### Using weight decay. Weight decay in certain scenarios can boost the model's performance. Keras has built-in support for weight decay in all optimizers. Users can apply weight decay by setting `weight_decay` argument. >>> opt = tf.keras.optimizers.experimental.SGD(1, weight_decay=0.004) >>> grads, var1, var2 = tf.zeros(()), tf.Variable(2.0), tf.Variable(2.0) >>> # You can exclude variables from weight decay, in this case we >>> # exclude `var2`. >>> opt.exclude_from_weight_decay(var_list=[var2]) >>> opt.apply_gradients(zip([grads, grads], [var1, var2])) >>> print([var1.numpy(), var2.numpy()]) [1.992, 2.0] ### Using exponential moving average. Empirically it has been found that using the exponential moving average (EMA) of the trained parameters of a deep network achieves a better performance than using its trained parameters directly. TF-Keras optimizers allows users to compute this moving average and overwrite the model variables at desired time. Example: ```python # Create an SGD optimizer with EMA on. `ema_momentum` controls the decay # rate of the moving average. `ema_momentum=1` means no decay and the stored # moving average is always model variable's initial value before training. # Reversely, `ema_momentum=0` is equivalent to not using EMA. # `ema_overwrite_frequency=3` means every 3 iterations, we overwrite the # trainable variables with their moving average values. opt = tf.keras.optimizers.experimental.SGD( learning_rate=1, use_ema=True, ema_momentum=0.5, ema_overwrite_frequency=3) var1, var2 = tf.Variable(2.0), tf.Variable(2.0) with tf.GradientTape() as tape: loss = var1 + var2 grads = tape.gradient(loss, [var1, var2]) # First iteration: [var1, var2] = [1.0, 1.0] opt.apply_gradients(zip(grads, [var1, var2])) print([var1, var2]) # Second iteration: [var1, var2] = [0.0, 0.0] opt.apply_gradients(zip(grads, [var1, var2])) print([var1, var2]) # Third iteration, without EMA, we should see [var1, var2] = [-1.0, -1.0], # but overwriting results in [var1, var2] = [-0.125, -0.125]. The full # calculation for the moving average of var1 is: # var1=2*0.5**3+1*(1-0.5)*0.5**2+0*(1-0.5)*0.5**1+(-1)*(1-0.5)=-0.125. opt.apply_gradients(zip(grads, [var1, var2])) print([var1, var2]) ``` When optimizer is constructed with `use_ema=True`, in custom training loop, users can explicitly call `finalize_variable_values()` to overwrite trainable variables with their EMA values. `finalize_variable_values()` is by default called at the end of `model.fit()`. ### Use with `tf.distribute.Strategy` This optimizer class is `tf.distribute.Strategy` aware, which means it automatically sums gradients across all replicas. To aggregate gradients yourself, call `apply_gradients` with `skip_gradients_aggregation` set to True. This is useful if you need to process aggregated gradients. ```python # This example is not runnable, it consists of dummy code for simple # tutorial. strategy = tf.distribute.experimental.TPUStrategy() with strategy.scope(): opt = tf.keras.optimizers.experimental.SGD() model = magic_function_that_returns_model() gradients = magic_function_that_returns_gradients() # Custom logic to aggregate gradients. gradients = strategy.reduce("SUM", gradients, axis=None) opt.apply_gradients(zip(gradients, model.trainable_variables), skip_gradients_aggregation=True) ``` ### Creating a custom optimizer If you intend to create your own optimization algorithm, please inherit from this class and override the following methods: - `build`: Create your optimizer-related variables, such as `momentums` in SGD optimizer. - `update_step`: Implement your optimizer's updating logic. - `get_config`: serialization of the optimizer, include all hyper parameters. Your optimizer would automatically be compatible with tensorflow distributed training if you subclass `optimizer_experimental.Optimizer`. c | jdd} | |_t | ||||||||| f i| tj j |_tj|_ y)zCreate a new Optimizer.meshN) rE_meshrr2r distribute get_strategy_distribution_strategy dtensor_utilsrunning_with_dtensor_strategy_run_with_dtensor) r/rrrrrrr(r)rr0r,r#s r1r2zOptimizer.__init__{svzz&$'          #   ')mm&@&@&B#!.!L!L!Nr3c6|jr+|tj|}nt|tjr{tj j j|tj j jj|j|jj}t|dr |j}n |j}tj j j||d|j |d}|j"j%||Stj&j)}|j*j-|5t.|a||||cdddS#1swYyxYw)NrrrFr)r-rrr&Tensor experimentaldtensor copy_to_meshLayout replicatedrYrrrr7 DVariablerr*r=r.r/extendedcolocate_vars_withrr) r/rrrYrr7rPstrategyr#s r1rz%Optimizer.add_variable_from_referencesN ::$!# n = M2995 " 7 7 D D!OO++22== )<)<)A)A>! ~|4&11&,,..88+%a(C(C'DE 9H OO " "8 ,O}}113H""55nE w:"M5-   s 3FFcP|jrtjdtj}tjj j |tjj jj|jd}tj5tjj j|d|_ ddd|jj|jyt|=y#1swY>xYw)Nrrr5r5r)r-rconstantr;r7r8r9r:r;r9r<r<r*r=rr-)r/init_valr#s r1r-z$Optimizer._create_iteration_variables ::{{1BHH5H..;;''..99$**19MH $&??#:#:#D#D;$E$   OO " "4#3#3 4 G . 0  s .1DD%ct|dr|j}nQtj|r ::// OOA  " !";;NK Kr3c |js |jrt| ||S|j dd}|s|r|j |}t| ||S)aApply gradients to variables. Args: grads_and_vars: List of `(gradient, variable)` pairs. name: string, defaults to None. The name of the namescope to use when creating variables. If None, `self.name` will be used. skip_gradients_aggregation: If true, gradients aggregation will not be performed inside optimizer. Usually this arg is set to True when you write custom code aggregating gradients outside the optimizer. **kwargs: keyword arguments only used for backward compatibility. Returns: A `tf.Variable`, representing the current iteration. Raises: TypeError: If `grads_and_vars` is malformed. RuntimeError: If called in a cross-replica context. rA experimental_aggregate_gradientsT)r-r3rrrErJ)r/rrskip_gradients_aggregationr0rLr#s r1rzOptimizer.apply_gradientssl2 :://7*>*E E ,2:: ., (*.N!55nENw&~D&AAr3cjyfd}tjjjj |j |y)NcZfd}|D] }|jj||d"y)Ncj|rltjj|j}tjj |j}|j ||z|zyyr)rrrrr7rr)rPrArr/s r1weight_decay_fnz^Optimizer._apply_weight_decay..distributed_apply_weight_decay..weight_decay_fn%sa))(3!3!3X^^DB!2!2HNNCB''2 (:;4r3F)group)r=update) distributionrr0rQrPr/s r1distributed_apply_weight_decayzEOptimizer._apply_weight_decay..distributed_apply_weight_decay$s9 < & %%,,oU- r3)rrrr.interimmaybe_merge_callr0)r/rrUs` r1rzOptimizer._apply_weight_decaysF    $   ""**;; *  ' '  r3c|js |jrt| |Stj j jj|j|j|Sr) r-r3rrrrr.rVrW_distributed_apply_gradients_fnr0)r/rr#s r1rz#Optimizer._internal_apply_gradients6sZ :://74^D D))11BB  0 0  ' '   r3c|js |jrt| ||j}|D]M}|j |j |j|}|jj|d|fOy)zOverwrite model variables with their moving average values. This function overwrites variables on each device. Args: var_list: list of model variables. c$|j|Sr)r)abs r1zIOptimizer._overwrite_model_variables_with_average_value..Ss!((1+r3)argsN) r-r3rrr0rrnrQr=rS)r/rwr?rrr#s r1rz7Optimizer._overwrite_model_variables_with_average_valueAs ::// G A( K.. C::  s!34G    $ $-WJ %   r3cjst |Stjj j }fd}t|tjrMtj|j}||}||dtj_ |S|tj|tj}||dtj dS)Nctjjj|tjjjj j dS)Nrr5)rr7r8r9r:r;r-)xr/s r1r^z0Optimizer._build_learning_rate..\sE"//*A*A*N*N r&&--88!8L+ r3r)rr7rFr6)r-rrrr7r8r<r&rrrrfloat32rrBrr)r/rvariable_creationinit_value_convert_fnrrCr#s` r1rzOptimizer._build_learning_rateVszz7/ > >OO33==!   1FF %'$8$8doo.% !%::O$P !+<%$jj+D ' ! ( KK RZZ 8 !  .."   r3cjrbfd}|D]W}jjj|}jj j |||fdYyy)rch|jj|zdjz |zzy)Nr)rr()rrr/s r1update_averagezHOptimizer._update_model_variables_moving_average..update_average~s2%%/1t7H7H3HC2OOr3Fr_rRN)rrrnrQr0r=rS)r/rwrhrrs` r1rz0Optimizer._update_model_variables_moving_averagezsx <<    >>$$T]]3%78++44;;^3&<   r3c  fd}|D]%\}}|jj|||fd'jrt|\} j j r^j dzj zdk(}tjtj|tj fddj jdS) z1`apply_gradients` using a `DistributionStrategy`.c jr+j||tj|Sj ||Sr)rrkrrQri)rr]r/s r1apply_grad_to_update_varzKOptimizer._distributed_apply_gradients_fn..apply_grad_to_update_varsB,,T34==;M8NOO((s33r3Frirrc&jSr)rrsr1r^z;Optimizer._distributed_apply_gradients_fn..sD$V$V %r3cyrrrr3r1r^z;Optimizer._distributed_apply_gradients_fn..sr3)true_fnfalse_fn) r=rSrrurr)rrcondrboolr) r/rTrr0rlr]r_should_overwrite_model_varsrws ` @r1rYz)Optimizer._distributed_apply_gradients_fns  4 ( ID#  ! ! ( (-TG5 )   <<~.KAx  7 7 A++OOa'00/145/6+GG7A* ))!,,r3)rNNNFrNTr)NF)r$rrrr2rr-rQrJrrrrrrrYr&r's@r1r*r*sv xz $O@HL#J1"*"L4#( %BN .  *" H" -r3r*c$eZdZfdZdZxZS)RestoredOptimizerc$t|dy)Nrv)rr2)r/r#s r1r2zRestoredOptimizer.__init__s ,-r3ctd)NzRestoring functional Optimizers from SavedModels is not currently supported. Please file a feature request if this limitation bothers you.rcr>s r1rzRestoredOptimizer.get_configs!   r3)r$rrr2rr&r's@r1rvrvs . r3rvceZdZdZdZy)r zETemporary shim to support both `opt.variables()` and `opt.variables`.c|Srrr>s r1__call__zCallableList.__call__s r3N)r$rrrr{rr3r1r r s Or3r c(eZdZdZdZdZdZdZy)rz4Accumulate variable shards into a `ShardedVariable`.cdg|z|_yr)shards)r/ num_shardss r1r2z _ShardedVariableBuilder.__init__sfz) r3ct|jjddjdd}|j|||j|<yt d|j)Npart_:rzs r1rz_ShardedVariableBuilder.builds!))99$++FFr3N)r$rrrr2rrrrr3r1rrs>* AGr3rexperimentalOptimizerc"t|tSrr&r*objs r1r^r^ JsI.r3ctSrrvprotos r1r^r^ ->-@r3r)object_factoryversionmin_producer_versionmin_consumer_version)versionstf_keras_experimentalOptimizerc"t|tSrrrs r1r^r^rr3ctSrrrs r1r^r^rr3z{{base_optimizer_keyword_args}})*rrrrtensorflow.compat.v2compatv2rabslr tf_keras.srcrrtf_keras.src.dtensorrr1tf_keras.src.optimizersr!tf_keras.src.optimizers.schedulesrtf_keras.src.utilsr tensorflow.python.util.tf_exportr tensorflow.tools.docsr rr AutoTrackabler base_optimizer_keyword_argsr*rvrtr r saved_modelloadregister_revived_typeVersionedTypeRegistrationAssertionErrorreplacerr3r1rs  !! %7<D':.k +R__--;;k +\&KR -  k-k-  k-\   4GG6OO$$::. OO ' ' , , F F@%&%& G  ; 4%%--%'B  OO$$::(. OO ' ' , , F F@%&%& G  ;  s3A&D;;A+F)(F)