[^(hm$ddlmZmZmZmZddlZddlmZddlmZm Z m Z m Z m Z m Z mZddgZGdde Zd d e d e zd ze_d eedeedeeedeeedeeedeedeedeededeeefdededeedededef dZdedeeeej0eej2efeeeeffd Zd eedeedeeedeeedeeedeedeedeededeeefdededeedededef d!Zee" d%d eedeedeeedeeedeeedeed#eedeedeedededeeefdededededef"d$Zy)&)castOptional TYPE_CHECKINGUnionN)Tensor)_disable_dynamo_if_unsupported_get_scalar_dtype _maximize_doc _params_doc OptimizerParamsTTensorListList Adafactor adafactorceZdZ dddddedeeefdedeeeefded ed ee d e ffd Z fd Z dZ e jddZxZS)rNF)foreachmaximizeparamslr beta2_decayepsd weight_decayrrc t|tr|jdk7r tdd|kstd|d|k\std||dd|dkstd|dd|dkstd|dd |kstd |d|kstd |t ||||||| } t ||| y) NrzTensor lr must be 1-elementz%Learning rate should be >= 0 but is: z#beta2_decay should be <= 0 but is: rz epsilon1 should be >= 0 but is: z epsilon2 should be >= 0 but is: ?z,Clipping threshold d should be >= 1 but is: z$weight_decay should be >= 0 but is: )rrrrrrr) isinstancernumel ValueErrordictsuper__init__) selfrrrrrrrrdefaults __class__s T/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/torch/optim/_adafactor.pyr#zAdafactor.__init__s b& !bhhjAo:; ;byDRDIJ Jk!B;-PQ Q q6 cSVm?AxHI Ic!f}?AxHI IaxKA3OP Pl"CL>RS S#%  *cft|||jD]}|jdd|dD]v}|jj |g}t |dk7s.tj|drGt|d}tj|t|d<xy)Nrrrstepdtype) r" __setstate__ param_groups setdefaultstategetlentorch is_tensorfloattensorr )r$r0grouppp_statestep_valr&s r'r-zAdafactor.__setstate__<s U#&& XE   Y -8_ X**..B/w<1$U__WV_-M$WV_5H&+ll8CTCV&WGFO  X Xr(c$|dD]}|jtj|r td|jjr td|j ||j |j|j |} t| dk(rtjdt| d<|jjdkDrt|jj} d| d <|jj| | d <t|jj} d| d <|jj| | d <n2tj|jtj | d<|j | j!d d|j | j!d d|j | j!dd|j | d y)Nrz-Adafactor does not support complex parametersz+Adafactor does not support sparse gradientsrrr+r*rrow_varcol_var) memory_formatvarianceF)gradr3 is_complex RuntimeError is_sparseappendr0r2r6r dimlistshape new_zeros zeros_likepreserve_formatr1) r$r7params_with_gradgradsrow_varscol_vars variances state_stepsr8r0 row_shape col_shapes r' _init_groupzAdafactor._init_groupFsx% .Avv~""#RSSvv"#PQQ  # #A & LL JJqME5zQ!& S8I8K Lf 66::2i(z*"4t<!$ T:'# '% N U ! !s B;;C)g{Gz?g)NgMbP?rrN)__name__ __module__ __qualname__rrr5rtuplerboolr#r-rUr3no_gradr* __classcell__)r&s@r'rrs$(!-9!#+#'#+#+ %- #+ #+ 8E?E) * #+  #+#+$#+#+JX0dU]]_55r(aImplements Adafactor algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{(lr)}, \: \tau \text{(}\beta_2\text{ decay)}, \: \theta_0 \text{(params)}, \: f(\theta) \text{(objective)}, \\ &\hspace{15mm} \: \epsilon_1, \epsilon_2 \text{ (epsilons)}, \: d \text{(clipping threshold)}, \\ &\hspace{15mm} \: \lambda \text{(weight decay)}, \: \textit{maximize} \\ &\textbf{initialize} : \: R_0 \leftarrow 0 \text{ (second moment row factor)}, \\ &\hspace{23mm} \: C_0 \leftarrow 0 \text{ (second moment col factor)}, \\ &\hspace{23mm} \: \widehat{V}_0 \leftarrow 0 \text{ (second moment for vectors)} \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}\textbf{if} \: \textit{maximize}: \\ &\hspace{10mm}G_t \leftarrow -\nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}G_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\widehat{\beta}_{2_t} \leftarrow 1 - t^{\tau} \\ &\hspace{5mm}\rho_t \leftarrow min(lr, \frac{1}{\sqrt{t}}) \\ &\hspace{5mm}\alpha_t \leftarrow max(\epsilon_2, \text{RMS}(\theta_{t-1}))\rho_t \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \gamma \lambda \theta_{t-1} \\ &\hspace{5mm}\textbf{if} \: \text{dim}(G_t) > 1: \\ &\hspace{10mm}R_t \leftarrow \widehat{\beta}_{2_t}R_{t-1}+ (1-\widehat{\beta}_{2_t})(G_t \odot G_t) \cdot 1_m \\ &\hspace{10mm}C_t \leftarrow \widehat{\beta}_{2_t}C_{t-1}+ (1-\widehat{\beta}_{2_t}) 1^\top_n \cdot (G_t \odot G_t) \\ &\hspace{10mm}\widehat{V}_t \leftarrow \frac{R_t \cdot C_t}{max(1^\top_n \cdot R_t, \epsilon_1)} \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}\widehat{V}_t \leftarrow \widehat{\beta}_{2_t}\widehat{V}_{t-1}+ (1-\widehat{\beta}_{2_t}) \cdot (G_t \odot G_t) \\ &\hspace{5mm}U_t \leftarrow \frac{G_t}{max(\sqrt{\widehat{V}_t}, \epsilon_1)} \\ &\hspace{5mm}\widehat{U}_t \leftarrow \frac{U_t}{max(1, \frac{\text{RMS}(U_t)}{d})} \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1} - \alpha_t \widehat{U}_t \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Adafactor: Adaptive Learning Rates with Sublinear Memory Cost`_. z Args: a lr (float, Tensor, optional): unlike other optimizers, Adafactor does not require a learning rate, and Shazeer, Noam, and Mitchell Stern do not use lr at all. Deviating from the paper, this implementation uses lr for applying weight decay and as the maximum value for relative step size rho_t. Note that in the paper, a constant of 0.01 is used as the maximum value for relative step size, and so we set 0.01 as the default value. (default: 1e-2) beta2_decay (float, optional): the decay rate of beta2. beta2 standardly refers to the coefficient used for computing the running average of the gradient squared. (default: -0.8) eps (Tuple[float, float], optional): epsilon1 is the term added to the denominator of the update calculation to improve numerical stability. This use of epsilon1 deviates from the algorithm written in the paper! See note below for more details. epsilon2 is the term used to avoid having too small a weight update when applying parameter scaling. (default: (None, 1e-3)) d (float, optional): the clipping threshold, used to avoid larger-than-desired updates. weight_decay (float, optional): weight decay coefficient (default: 1e-2) foreach (bool, optional): whether foreach implementation of optimizer is used. Note that the foreach implementation uses ~ sizeof(params) more peak memory than the for-loop version due to the intermediates being a tensorlist vs just one tensor. As Adafactor is commonly used when memory is prohibitive, Adafactor will default to the slower single tensor for-loop implementation unless this flag is explicitly True. This behavior is contrary to other optimizers, which will attempt defaulting to foreach on CUDA for faster runtime. (default: None) a4 .. Note:: The implementation of Adafactor subtly differs from Shazeer, Noam, and Mitchell Stern and implementations in some other frameworks with its use of learning rate and :math:`\epsilon_1`. Regarding the learning rate hyperparameter: Shazeer, Noam, and Mitchell Stern do not use lr at all, as the stated algorithm uses :math:`\rho_t` and update clipping to affect the step size. This implementation allows `lr` to influence the maximum value for :math:`\rho_t`: .. math:: \begin{aligned} &\hspace{5mm}\rho_t \leftarrow min(lr, \frac{1}{\sqrt{t}}) \end{aligned} This differs from Shazeer, Noam, and Mitchell Stern, who use a constant of 0.01 as the maximum value of :math:`\rho_t` .. math:: \begin{aligned} &\hspace{5mm}\rho_t \leftarrow min(0.01, \frac{1}{\sqrt{t}}) \end{aligned} Shazeer, Noam, and Mitchell Stern do not enforce an opinion on how weight decay should be computed, and so we use the learning rate as a coefficient for decoupled weight decay, similar to what is suggested in `Decoupled Weight Decay Regularization`_. Regarding the use of :math:`\epsilon_1`: The implementation attempts to replicate the presumed intention of Shazeer, Noam, and Mitchell Stern to use :math:`\epsilon_1` as a stabilizing term when the squared gradient becomes small. This stabilization can be written as .. math:: \begin{aligned} &\hspace{5mm}R_t \leftarrow \widehat{\beta}_{2_t}R_{t-1}+ (1-\widehat{\beta}_{2_t})(G_t \odot G_t + 1_n \cdot 1^\top_m) \cdot 1_m \\ &\hspace{5mm}C_t \leftarrow \widehat{\beta}_{2_t}C_{t-1}+ (1-\widehat{\beta}_{2_t}) 1^\top_n \cdot (G_t \odot G_t + 1_n \cdot 1^\top_m) \\ &\hspace{5mm}\widehat{V}_t \leftarrow \frac{R_t \cdot C_t}{max(1^\top_n \cdot R_t, \epsilon_1)} \\ &\hspace{5mm}U_t \leftarrow \frac{G_t}{max(\sqrt{\widehat{V}_t}, \epsilon_1)} \\ \end{aligned} where the row and column factors of gradient squared :math:`R_t` and :math:`C_t` are left alone, and we apply :math:`\epsilon_1` at the final calculation of the variance estimate :math:`\widehat{V}_t` and for the update :math:`U_t`. This is in contrast to Shazeer, Noam, and Mitchell Stern and other frameworks which apply :math:`\epsilon_1` to both row and column factors of the squared gradient, but not in the calculations after: .. math:: \begin{aligned} &\hspace{5mm}R_t \leftarrow \widehat{\beta}_{2_t}R_{t-1}+ (1-\widehat{\beta}_{2_t})(G_t \odot G_t + \epsilon_1 1_n \cdot 1^\top_m) \cdot 1_m \\ &\hspace{5mm}C_t \leftarrow \widehat{\beta}_{2_t}C_{t-1}+ (1-\widehat{\beta}_{2_t}) 1^\top_n \cdot (G_t \odot G_t + \epsilon_1 1_n \cdot 1^\top_m) \\ &\hspace{5mm}\widehat{V}_t \leftarrow \frac{R_t \cdot C_t}{1^\top_n \cdot R_t} \\ &\hspace{5mm}U_t \leftarrow \frac{G_t}{\sqrt{\widehat{V}_t}} \\ \end{aligned} .. _Adafactor\: Adaptive Learning Rates with Sublinear Memory Cost: https://arxiv.org/pdf/1804.04235 .. _Decoupled Weight Decay Regularization: https://arxiv.org/abs/1711.05101 rrNrOrPrQrRrWrXrrrrrYrZrr[c||Jdtjjrt| tsJt |D]\}}|s||n|| }||}||}||}||}| )tj |jj} |dz }|j}|| z}t| d|dzz }t| |jdj|jdzz |z}| dk7r|jd| | zz |jdkDr||Jdtj|dd j!j#|j%d}|j'||tj|d d j!j#|j%d }|j'||||z}|j#|j)d d j+| n0|Jd ||z}|j'|||j-}|j+| | z j/}|j|td |jdj|jdz|zz } |j1|| | z y)N5Grad scaling should occur outside of optimizer.step()r?rCrow_var and col_var should be defined when grad is multidimensionalr<TrGkeepdimr>)min0variance should be defined when grad is a vectorralpha)r3jit is_scriptingrr5 enumeratefinfor,ritemrpmaxnormrmul_rGsquare_div_sizelerp_meanclamp_clonersqrt_add_)!rrNrOrPrQrRrWrXrrrrrYrZrr[iparamrBstep_tr=r?rA step_floatone_minus_beta2_trho_trsrow_meancol_mean var_estimate grad_squaredupdatedenoms! r'_single_tensor_adafactorrFs0 y0?>?0 yy"e$$$f%215'uQxeAhYQ1+1+Q< <;;u{{+//D ! [[] & 3BZ_-.D%**Q-,,.%++-32FGH5P 1  JJq2 ,, - 88:>#(; UT U; 4R6>>@EEdiiPRmT  MM($5 6 4R6>>@EEdiiPRmT  MM($5 6"W,L   gllr4l@GGDGQ R$ BA B$$;L NN<): ;#>>+L$$$5<<> DCQ,,.6<<>S3HA2MNO 6%% 0e21r( tensorlistsreturnc2tj|}i}|jD]\\}}\}}||df}||df}t|dD]\}} | Jd| j dkDrN||vr|Dcgc]}gc}||<t t |D]} ||| j|| |!p||vr|Dcgc]}gc}||<t t |D]} ||| j|| |!|Scc}wcc}w)zGroups tensors by device, dtype, AND multidimensionality -- whether the tensor has multiple dims or just one dim (is a vector). This allows the foreach impl of Adafactor to assume that every group of params will either be factored or not.TFrzgrad should not be None)r "_group_tensors_by_device_and_dtypeitemsrvrGranger2rF) rgrouped_tensorsultra_grouped_tensorsdevicer,_ matrix_key vector_keyjr6rs r'._group_tensors_by_device_dtype_and_is_multidimrsl BB;OO .=-B-B-DS))+qeT* eU+ #;q>2 SIAv% @'@ @%zz|a%::EP8Q8Q)*5s;/0SA)*5a8?? Aq@QRS%::EP8Q8Q)*5s;/0SA)*5a8?? Aq@QRS S S" ! 9R 9Rs 6 D Dc  t|dk(ry||Jdt||||||g}|jD]\\}}}\}}}}}}ttt |}ttt |}ttt |}| (|Jdt j|j} tr|dJ|rt j|}t jjs=|djr.t j|t jdddnt j|dg}g}g}|D]x} |j!| j#| z|j!d| j#| zz |j!t%| d| j#d zz zt'||D!"cgc]E\}!}"t)| |!j+d j#|!j-d zz |"zG}#}!}"| dk7rt j.|d| | zz |rttt |}$ttt |}%|$d|%dJd |D&cgc]}&t j*|&d d }'}&t j.|'|'t j0|'|D&cgc]}&|&j3d c}&t j4|$|'|~'|D&cgc]}&t j*|&dd }(}&t j.|(|(t j0|(|D&cgc]}&|&j3dc}&t j4|%|(|~(t'|$|%D)*cgc] \})}*|)|*z }+})}*|$D)cgc]})|)j7dd },})t j8|,| t j0|+|,~,nnttt |}-|-dJdt j:||}.t j4|-|.|~.|-D/cgc]}/|/j=}+}/t j8|+| | zt j>|+t j.|+||+}0t'|#|0D12cgc]I\}1}2|1 t)d|2j+d j#|2j-d z|zz z K}#}1}2t j.|0|#t j||0ycc}"}!wcc}&wcc}&wcc}&wcc}&wcc}*})wcc})wcc}/wcc}2}1w)Nrrjz2dtype is needed to compute eps1 when eps1 is unsetrcpu)rrrrrkrlrmr<Trnr>rq) r2rrrrHrr3rwrr _foreach_negcompiler is_compilingis_cpu _foreach_add_r6rFrxrpzipryrzr _foreach_mul_ _foreach_div_r~_foreach_lerp_r_foreach_clamp_min_ _foreach_mulr_foreach_rsqrt_)3rrNrOrPrQrRrWrXrrrrrYrZrr[rrr, is_multidimdevice_params_ device_grads_device_row_vars_device_col_vars_device_variances_device_state_steps_ device_params device_gradsdevice_state_stepsone_minus_beta2_tsbeta2_tsrho_tssr8ralphasdevice_row_varsdevice_col_varsrB row_means col_meansr=r? var_estimates row_var_meansdevice_variances grads_squaredvupdatesars3 r'_multi_tensor_adafactorrsr. 6{a y0?>?0E (I{CO    o4 E;      T&\>: DL-8 !$v,0CD <! DC D!;;u%))D %a(4 44  --l;L ~~**,1CA1F1M1M   "ELLU$C3     2C 8# :A  % %affh+&= > OOAK 77 8 MM#b!qvvx3"78 9 :M62 1 affQinn&!'')s*:;  T 2M  % %mT :    } =#DL2CD  #/ BA B/"..|\JM  !1=BT U1AA1QWWYAMA !!-= m, M<8!1 6B#c6;;q>..0V\\^s5Ja4OPQ R   GV, M73_o4X ,T ,T"B s8$A UU#U U#)U( /U-U3$U8AU=)single_tensor_fnrc tjjstd|Ds t d|rt }nt }|||||||| | | | |||||| y)zxFunctional API that performs Adafactor algorithm computation. See :class:`~torch.optim.Adafactor` for details. c3PK|]}t|tj ywra)rr3r).0ts r' zadafactor..ls 5() 1ell#5s$&z?`state_steps` argument must contain a list of singleton tensors) rrrrrYrZrrWrXr[N)r3rrallrDrr)rrNrOrPrQrRrrWrXr[rrrrrYrZrfuncs r'rrQs6 >> & & (5-852 M  &'  !  !r()NNNF)typingrrrrr3r optimizerr r r r r rr__all__r__doc__rHr5rfrr!rerr,rrrr(r'rs87   $X Xx.^   2 7_JVEWP jS1 LS1 <S18F#$S18F#$S1HV$%S1fS1 S1S1 S1 femS1 !S1"#S1$ 5/%S1& 'S1()S1*+S1l!!  (5<< (5;;"7 =>hv  "!DP4 LP4 <P48F#$P48F#$P4HV$%P4fP4 P4P4 P4 femP4 !P4"#P4$ 5/%P4& 'P4()P4*+P4f 1IJ##'"&7 L7 <78F#$78F#$ 7 HV$% 7 f 7d^7 777 7 eVm7 !7"#7$ %7& '7()7K7r(