S^(h ddlZddlmZddlmZddlmZddlmZm Z e je Z erddl Z dZ d"deed ed d eed ed effdZ d"deed ed d eed ed effdZ d"deed ed d eed ed effdZ d#ded d d eed ed effdZ d#ded d d eed ed effdZ d#ded d d eed ed effdZeeeeeedZ d$dedededeedeef dZd#dedeefdZd#dedeefdZd#dedeefdZd#dedeefdZd#dedeefdZ d#dedeefd Z!eeeee e!dZ"d#dedeefd!Z#y)%Nwraps)Optional)PretrainedConfig)is_torch_availableloggingcBddtfd}|S)ad Decorator function to update the RoPE parameters in the forward pass, if the model is using a dynamic RoPE (i.e. a RoPE implementation that may recompute its frequencies in the forward pass). Args: rope_forward (Callable): The forward pass of the RoPE implementation. Returns: The decorated forward pass. ctj|dz}t|jdr|jj}n|jj }||kDrTt|ds)|j |j||dz\|_}|jd|jdy|jj||_ |jd|jdy) zbLongrope uses long factor if sequence is larger than original pretraining length, short otherwise.r original_max_position_embeddings long_inv_freqseq_leninv_freqF persistentN) torchmaxhasattrconfigr max_position_embeddings rope_init_fnr register_bufferoriginal_inv_freqto)self position_idsdevicerr _s ^/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/modeling_rope_utils.pylongrope_frequency_updatez6dynamic_rope_update..longrope_frequency_update+s))L)A- 4;; B C/3{{/[/[ ,/3{{/R/R , 5 541(,(9(9KK1QTU1U):)%"A  T-?-?E R&*%;%;%>%>v%FD "  T-C-CPU Vctj|dz}||jkDrA|j|j||\}|_|j d|d||_||jkrj|j|jkDrP|jj||_|j d|jd|j|_yyy)a dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow scaling) 2 - the current sequence length is in the original scale (avoid losing precision with small sequences) rrrFrN) rrmax_seq_len_cachedrrattention_scalingroriginal_max_seq_lenrr)rrrrrs r dynamic_frequency_updatez5dynamic_rope_update..dynamic_frequency_update>s ))L)A- T,, ,/3/@/@f^e/@/f ,Hd,  X% H&-D # T.. .43J3JTMfMf3f&*%;%;%>%>v%FD "  T-C-CPU V&*&?&?D # 4g .r"cd|jvr|||jn$|jdk(r|||j|||S)Ndynamic)rlongrope) rope_typer)rxrr'r! rope_forwards r wrapperz$dynamic_rope_update..wrapperQsJ  & $T< I ^^z ) %dL JD!\22r"r)r-r.r'r!s` @@r dynamic_rope_updater/s/W&@& <33 Nr"rrz torch.devicerreturnz torch.Tensorc |t|dkDrtd|d|t|dkDr |d}|d}nZ|X|j}t|dr |jnd}t |d|j |jz}t||z}d}dtjdd tj j|tj |z zz } | |fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rzUnexpected arguments: `**rope_kwargs` and `config` are mutually exclusive in `_compute_default_rope_parameters`, got `rope_kwargs`= and `config`=basedimpartial_rotary_factor?head_dimdtyperr:)len ValueError rope_thetarr5getattr hidden_sizenum_attention_headsintrarangeint64rfloat) rrr rope_kwargsr3r4r5r7attention_factorrs r _compute_default_rope_parametersrH\s *c+.2 EEPMQ_`f_g i   ;!6"%     @GPg@h < rr5r?r@rArBrrKrrCrDrrE) rrrrFr3r4rrJr5r7rGrs r _compute_dynamic_ntk_parametersrNs,c+.2 DDO=P^_e^f h   ;!6"% "-.G"HX&    @GPg@h < >6A:NTW[^ab[bTcd dDdu||AsAU[[ILLTZbgbmbmLnqttuvH % %%r"c @t|dkDrtd||j}t|dr |jnd}t |d|j |jz}t||z}|jd}|jjd} |jjd} |jjd } d |jvr|jd } |j| z }n |j} dd } | )| r| rt| || | || z } n| |} |jjd xsd}|jjdxsd }dfd}d}|tjd|dj|tj|z z}d|z }d||zz }|||||| \}}d ||||dzj|tjz }|d |z z||zz}|| fS)a Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://arxiv.org/abs/2309.00071) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rzYUnexpected arguments: `**rope_kwargs` should be unset in `_compute_yarn_parameters`, got r5r6r7rJrGmscalemscale_all_dimr rcJ|dkryd|ztj|zdzS)Nrr6g?)mathlog)scalerPs r get_mscalez,_compute_yarn_parameters..get_mscale s( A:V|dhhuo-33r" beta_fast beta_slowc|tj||dztjzz zdtj|zz S)zPInverse dimension formula to find the dimension based on the number of rotationsr8)rSrTpi) num_rotationsr4r3rs r find_correction_dimz5_compute_yarn_parameters..find_correction_dimsBdhh6-!:Kdgg:UVWW\]`d`h`him`n\noor"ctj||||}tj||||}t|dt ||dz fS)z.Find dimension range bounds based on rotationsrr)rSfloorceilrmin)low_rothigh_rotr4r3rlowhighr]s r find_correction_rangez7_compute_yarn_parameters..find_correction_range#sUjj,Wc4AXYZyy,XsDBYZ[3{CcAg...r"c||k(r|dz }tj|tj|z ||z z }tj|dd}|S)NgMbP?r9rr)rrCfloat32clamp)rarr4 linear_func ramp_funcs r linear_ramp_factorz4_compute_yarn_parameters..linear_ramp_factor)sL #: 5LC||Cu}}=Cc R KK Q2 r"r8r;)r)r<r=r>rr5r?r@rArBrKgetrrErrCr)rrrrFr3r5r7r4rJrGrPrQr rVrWrYrfrl pos_freqsinv_freq_extrapolationinv_freq_interpolationrdreinv_freq_extrapolation_factorrr]s @r _compute_yarn_parametersrrs_( ;!ghsgt u     DN *V-@-@@+1+>+>?a+b(//2RR+1+I+I(4  n$Z%?*VUcBd%de )&1 ##'' 4:I##'' 49Ip/ aa03363UX[[\I 9_ FY$67%iCGghIC%&(:3cQh(O(R(RZ`hmhshs(R(t$t!!&C"CD #@ @ A  % %%r"c t|dkDrtd||j}t|dr |jnd}t |d|j |jz}t||z}|jd}|jd} |jjd} |jjd } t|d r&|j} |j|jz } n |j} | I| dkrd} nAtjd tj| tj| z z} |r,|| kDr't!j"|t j$| } n&t!j"| t j$| } t!j&d|d t j(| j+|z }d| ||zzz }|| fS)a  Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rz]Unexpected arguments: `**rope_kwargs` should be unset in `_compute_longrope_parameters`, got r5r6r7 long_factor short_factorrJrGr r)r:rr8)r<r=r>rr5r?r@rArBrKrmr rrSsqrtrTrtensorrhrCrDrE)rrrrFr3r5r7r4rtrurJrGr ext_factorsinv_freq_shapers r _compute_longrope_parametersrzBs* ;! km     D))*LMO&8'*::$''kH$G[[+;!;X=NPXYN$w.@EUXgEghM]*n= 1, got rKrmrrr isinstancerErr)rrrKr+rrrJs r (_validate_linear_scaling_rope_parametersrs&&L  l.>.>vt.LMI (+M ))+,MM=kZ ( #F ~Z6&3,QRXQYZ[;Gr"c*|j}|jd|jdd}ddh}dh}t|j}t ||||||d}|t |t r|dkrtjd|yy)Nr+rrJr rr6rr)rrrKr+rrrrJs r )_validate_dynamic_scaling_rope_parametersrs&&L  l.>.>vt.LMI (+M78M ))+,MM=-]hi ( #F ~Z6&3,QRXQYZ[;Gr"c|j}|jd|jdd}ddh}hd}t|j}t ||||||d}|t |t r|dkrtjd||jd}|-t |t r|d krtjd ||jd } | (t | t stjd | |jd } | (t | t stjd| | xsd| xsdkrtjd| d| dyy)Nr+rrJ>rPrWrYrQrGr rr6rrGrL`rope_scaling`'s attention_factor field must be a float greater than 0, got rWz6`rope_scaling`'s beta_fast field must be a float, got rYz6`rope_scaling`'s beta_slow field must be a float, got rXrzO`rope_scaling`'s beta_fast field must be greater than beta_slow, got beta_fast=z( (defaults to 32 if None) and beta_slow=z (defaults to 1 if None)r) rrrKr+rrrrJrGrWrYs r _validate_yarn_parametersrs&&L  l.>.>vt.LMI (+MM ))+,MM=-]hi ( #F ~Z6&3,QRXQYZ[#''(:;#Z8H%-PTdghThZ[kZl m   -IZ 5%AOPY{[\  -IZ 5%AOPY{[\RIN+]^g]hi66?[@X Z ,r"c|j}|jd|jdd}hd}hd}t|j}t |||||t |dr |j nd}t|d|j|jz}t||z} |jd } t| ts*td | Drtjd | t!| | d zk(s'tjd | d zdt!| |jd} t| ts*td| Drtjd| t!| | d zk(s'tjd| d zdt!| t |drtj#dy|jd} | tjdn-t| t$r| dkrtjd| |jd} | /t| t$r| dkrtjd| yyy)Nr+r>r+rtru>rJrGr rr5r6r7ruc3HK|]}t|ttfywNrrBrE.0r,s r z0_validate_longrope_parameters..,s1dRS*Qe 2M1d "zC`rope_scaling`'s short_factor field must be a list of numbers, got r8z5`rope_scaling`'s short_factor field must have length z, got rtc3HK|]}t|ttfywrrrs r rz0_validate_longrope_parameters..2s0bQRAU|1L0brzB`rope_scaling`'s long_factor field must be a list of numbers, got z4`rope_scaling`'s long_factor field must have length r aYThis model has set a `original_max_position_embeddings` field, to be used together with `max_position_embeddings` to determine a scaling factor. Please set the `factor` field of `rope_scaling`with this ratio instead -- we recommend the use of this field over `original_max_position_embeddings`, as it is compatible with most model architectures.rJz1Missing required keys in `rope_scaling`: 'factor'rrGgr)rKrmrrrrr5r?r@rArBrlistallrrr< warning_oncerE)rrrKr+rrrr5r7r4rurtrJrGs r _validate_longrope_parametersrsI&&L  l.>.>vt.LMI@MVM ))+,MM=-]hi NNN OFE*fsl NNUV\U]^ _'++,>?  '.6:JS:Pbcsbtu;Q (r"c|j}|jd|jdd}hd}t|j}t |||||d}|t |t r|dkrtjd||d}|d }|t |t stjd ||t |t stjd |||krtjd |d ||d} | t | tstjd| | |jk\r&tjd| d|jyy)Nr+r>rJr+r|r}r rrJr6rr|r}z<`rope_scaling`'s low_freq_factor field must be a float, got z=`rope_scaling`'s high_freq_factor field must be a float, got zc`rope_scaling`'s high_freq_factor field must be greater than low_freq_factor, got high_freq_factor=z and low_freq_factor=r zP`rope_scaling`'s original_max_position_embeddings field must be an integer, got zg`rope_scaling`'s original_max_position_embeddings field must be less than max_position_embeddings, got z and max_position_embeddings=) rKrmrrrrrErrrBr) rrrKr+rrrJr|r}r s r _validate_llama3_parametersrPs&&L  l.>.>vt.LMIvM ))+,MM=kZ ( #F ~Z6&3,QRXQYZ["#45O#$67j%&HUVeUfghz2BE'JVWgVhij?* q 5o5F H (44V'W$'/zBbdg7h ^/0 2 (6+I+II u/00MfNlNlMm o Jr"ct|dd}|y|jd|jdd}tj|}| |||ytj d|dy) zO Validate the RoPE config arguments, given a `PretrainedConfig` object rKNr+rrrzTMissing validation function mapping in `ROPE_VALIDATION_FUNCTIONS` for 'rope_type'='')r?rmROPE_VALIDATION_FUNCTIONSrr)rrrKr+ validation_fns r rope_config_validationrsw6>48L  l.>.>vy.QRI-11)rs 1.   H %;~*.'+!'& % &'& ^ $'&c]'&  >5 ! '&V*.'+!&& % &&& ^ $&&c]&&  >5 ! &&T*.'+!0& % &0& ^ $0&c]0&  >5 ! 0&hPT]& ]&&4]&?G}]& >5 !]&BPT<& <&&4<&?G}<& >5 !<&@PT(, (,&4(,?G}(, >5 !(,^05. $,( $(!% llllC= l # l:[.>[XVY][ \5E \T\]`Ta \ \6F \U]^aUb \# &6# Xc]# L/*:/RU/d! (8! xPS}! L168 %-)  #3 (3- r"