S^(h>BddlmZddlmZddlmZmZmZmZm Z ddl m Z ddl m Z mZmZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd lmZmZdd l m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/m0Z0ddl1m2Z2m3Z3e-r ddl4Z4ddl4m5Z5e'r ddl6m7Z7ddl8m9Z9e(jte;ZeZQd?ZRe$d@eRGdAdBeZSGdCdDe5j|ZTdEZUe$dFeRGdGdHeQZVGdIdJee#ZWGdKdLeQeZXeGdMdNeZYdOZZdPZ[e$dQe[GdRdSeSeZ\gdTZ]y)W) dataclass)partial)CallableListOptionalTupleUnion)ACT2FN)Cache DynamicCache StaticCache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)BaseModelOutputWithPastCausalLMOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) LossKwargsadd_start_docstrings%add_start_docstrings_to_model_forwardcan_return_tupleis_torch_flex_attn_availableloggingreplace_return_docstrings)deprecate_kwarg)is_torch_available) AutoModelAutoModelForCausalLM) AriaConfigAriaTextConfigN)nn) BlockMask)make_flex_block_causal_maskr(c,eZdZdfd ZdZdZxZS)AriaTextRMSNormct|tjt j ||_||_y)z> AriaTextRMSNorm is equivalent to T5LayerNorm N)super__init__r) Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s d/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/aria/modeling_aria.pyr0zAriaTextRMSNorm.__init__As1 ll5::k#:; #c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)Nr#T)keepdim) dtypetor2float32powmeanrsqrtr5r4)r6 hidden_states input_dtypevariances r:forwardzAriaTextRMSNorm.forwardIsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r;c^t|jjd|jS)Nz, eps=)tupler4shaper5r6s r: extra_reprzAriaTextRMSNorm.extra_reprPs* ))*+6$2G2G1HIIr;)gư>)__name__ __module__ __qualname__r0rHrM __classcell__r9s@r:r-r-@s$;Jr;r-c(eZdZdZfdZdZxZS)AriaProjectorMLPa! Feed-Forward Network module for the Aria Projector. Args: in_features (`int`): Input embedding dimension. hidden_features (`int`): Hidden dimension of the feed-forward network. output_dim (`int`): Output dimension. ct|tj||d|_tj||d|_t d|_y)NFbiasgelu_new)r/r0r)Linear linear_in linear_outr act)r6 in_featureshidden_features output_dimr9s r:r0zAriaProjectorMLP.__init__asB ;eL))OZeL*%r;ch|j|j|}|j|}|SN)r\rZr[)r6rEs r:rHzAriaProjectorMLP.forwardgs- !>?  6 r;rNrOrP__doc__r0rHrQrRs@r:rTrTTs & r;rTc6eZdZdZddedeffd ZddZxZS)AriaCrossAttentionzv Aria Cross-Attention module. Args: config (`AriaConfig`): The configuration to use. config dropout_ratecBt||jj}|jj}||_t j||d|_t j||d|_ t j||d|_ t j||d|_ t j|||_ t j||_t j ||_t j ||_y)NFrVT) batch_first)r/r0 vision_configr7num_attention_heads num_headsr)rYq_projk_projv_projMultiheadAttentionmultihead_attnlinearDropoutdropout LayerNorm layer_norm layer_norm_kv)r6rfrgr7rlr9s r:r0zAriaCrossAttention.__init__vs **66 ((<< "ii [uE ii [uE ii [uE !33KX\]ii [9 zz,/ ,,{3\\+6r;c|j|j|}|j|}|j|}|j |}|j ||||\}}|j |j|}|S)a Forward pass of the AriaCrossAttention module. Args: key_value_states (`torch.Tensor`): Input tensor for key and value. hidden_states (`torch.Tensor`): Input tensor for query. attn_mask (`torch.Tensor`, *optional*, defaults to None): Attention mask. Returns: torch.Tensor: Output tensor after cross-attention.  attn_mask)rmrvrwrnrorqrtrr) r6key_value_statesrErzquerykeyvalue attn_output_s r:rHzAriaCrossAttention.forwards  DOOM:;--.>?kk*+ ,-,,UC),T Qll4;;{#;< r;)rra) rNrOrPrcr'floatr0rHrQrRs@r:rerems 7z77"r;recheZdZdZdeffd ZddejdeejfdZ xZ S) AriaProjectora  Aria Projector module. This module projects vision features into the language model's embedding space, enabling interaction between vision and language components. Args: config (`AriaConfig`): Configuration object for the model. rfct||j|_|jj |_|jj|_|jj |_ |jj |_ |jj |_ tjtj |j"|j |_t'||_tj*|j |_t/|j |j|j|_yra)r/r0projector_patch_to_query_dictpatch_to_query_dictrjr7r]rkrlkv_dim text_configr^r_r)r1r2zeros'max_value_projector_patch_to_query_dictr|re cross_attnrurvrT feed_forwardr6rfr9s r:r0zAriaProjector.__init__s #)#G#G !//;;--AA**66 %11== ,,88\\%++f.\.\^b^n^n"op ,V4,,t'7'78,T-=-=t?S?SUYUdUder;r{rzcP|jd|jd}}||jjvr*td|d|jjd|j|}|jd|j dj |dd}|M|j|jd}|j djd|jdd}|j|||}|j|j|}|S) a Forward pass of the Projector module. Args: key_value_states (`torch.Tensor`): Input tensor of shape (batch_size, num_patches, kv_dim). attn_mask (`torch.Tensor`, *optional*, default is None): Attention mask. Returns: `torch.Tensor`: Output tensor of shape (batch_size, query_number, output_dim). rr&zNumber of patches z: not found in patch_to_query_dict amongst possible values .Nr=ry)rKrkeysKeyErrorr| unsqueezerepeatrepeat_interleaverlexpandsizerrrv) r6r{rz batch_size num_patches query_numqueries attention_outouts r:rHzAriaProjector.forwards?#3"8"8";=M=S=STU=VK d66;;= =$[M1klpmEmEmJmJmLlMMNO ,,[9 **Zi(2215<? r;ra) rNrOrPrcr'r0r2TensorrrHrQrRs@r:rrs7ff( %,,AWr;rc.eZdZdZdeffd ZdZxZS)AriaSharedExpertsMLPa/ Shared Expert MLP for shared experts. Unlike routed experts, shared experts process all tokens without routing. This class reconfigures the intermediate size in comparison to the LlamaMLP. Args: config (`AriaTextConfig`): Configuration object for the Aria language model. rfc t|||_|j|_|j|j z|_t j|j|j|j|_ t j|j|j|j|_ t j|j|j|j|_ t|j|_y)NrV)r/r0rfr7intermediate_sizemoe_num_shared_expertsr)rYmlp_bias gate_projup_proj down_projr hidden_actact_fnrs r:r0zAriaSharedExpertsMLP.__init__s  !--!'!9!9F++auzz:K:R:RSK ;0A"BCN0034 !*- Q/eC(ll6>*#=>uS   Mr;c(eZdZdZfdZdZxZS)AriaGroupedExpertsGemmaP Grouped GEMM (General Matrix Multiplication) module for efficient expert computation. This module utilizes the grouped_gemm library (https://github.com/fanshiqing/grouped_gemm) for optimized performance. If the grouped_gemm library is not installed, it gracefully falls back to a sequential GEMM implementation, which may be slower but ensures functionality. Args: in_features (`int`): Number of input features. out_features (`int`): Number of output features. groups (`int`): Number of expert groups. ct|||_||_||_t j tj||||_ yra) r/r0r]rgroupsr)r1r2emptyr4)r6r]rrr9s r:r0zAriaGroupedExpertsGemm.__init__/sB &( ll5;;v{L#QR r;cLt||j|jS)au Perform grouped matrix multiplication. Args: input (`torch.Tensor`): Input tensor of shape (num_tokens, in_features). tokens_per_expert (`torch.Tensor`): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor of shape (num_tokens, out_features). )rr4cpu)r6inputrs r:rHzAriaGroupedExpertsGemm.forward6s''  KK  ! ! #  r;rbrRs@r:rrs S r;rc2eZdZdZdeddffd ZdZxZS)AriaGroupedExpertsMLPz Grouped MLP module for Mixture of Experts. Args: config (`AriaTextConfig`): Configuration object for the model. rfreturnNct|||_t|j|j dz|j |_t|j |j|j |_y)Nr#) r/r0rfrr7rmoe_num_expertsfc1fc2rs r:r0zAriaGroupedExpertsMLP.__init__Ssa  )&*<*V>VYZ>Z\b\r\rs)&*B*BFDVDVX^XnXnor;c|j||}tj|dd\}}tjj ||z}|j ||}|S)a5 Forward pass of the Grouped MLP. Args: permuted_tokens (torch.Tensor): Permuted input tokens. tokens_per_expert (torch.Tensor): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor after passing through the MLP. r#r=r)rr2chunkr) functionalsilur)r6permuted_tokensr fc1_output projectiongate fc2_outputs r:rHzAriaGroupedExpertsMLP.forwardYs\XXo/@A  ;;z1"= D]]'' 3d: XXj*;< r;rrRs@r:rrJs#p~p$p r;rc`eZdZdZdeffd ZdejdejfdZxZ S)AriaTextMoELayerz Aria Text Mixture of Experts (MoE) Layer. This layer applies a gating mechanism to route input tokens to different experts. Args: config (`AriaTextConfig`): Configuration object for the text component of the model. rfct|tj|j|j d|_t||_t||_ ||_ yNFrV) r/r0r)rYr7rrouterrexpertsrshared_expertsrfrs r:r0zAriaTextMoELayer.__init__wsO ii 2 2F4J4JQVW ,V4 26: r;rErc|j}|jd|jd}|j|}t j ||j jd\}}tjj|d}|j}t j|jjtj|j j d|j j dz j|}|} | jd} t j"| } |j%d| |j jz} |j'| |} t j(|jd|j jz| jdf| j| j*}|j-d| | |jd|j j| jd}||j/dzj1dj|}|j3|j|}||zS)a. Forward pass of the MoE Layer. Args: hidden_states (`torch.Tensor`): Input tensor of shape (batch_size, sequence_length, hidden_size). Returns: torch.Tensor: Output tensor after passing through the MoE layer. Process: 1. Route tokens to experts using the router. 2. Permute tokens based on routing decisions. 3. Process tokens through experts. 4. Unpermute and combine expert outputs. 5. Add shared expert output to the final result. r=r&)krrr)binsminmaxr)rKviewrrr2topkrfmoe_topkr)rsoftmaxr?histcflattenr@rArargsort index_selectrrr index_copy_rsumr)r6rEoriginal_shapelogits top_logits top_indicesscoresoriginal_dtyperindicesflatten_indicessorted_indicesr expert_outputunpermuted_tokensrshared_expert_outputs r:rHzAriaTextMoELayer.forwards$',,%**2}/A/A"/EF ]+"'**Vt{{7K7KQR"S K&&zr&:$**!KK    ! $ $U]] 3,, ++a/  "^   ",,r*7'44Q$++J^J^8^_ _6GH "KK \\!_t{{33 3]5G5G5J K%% ''  %%aG-222t{{7K7K]M_M_`aMbc#f&6&6r&::??A?FKKN[ $22=3E3En3UV,,,r;) rNrOrPrcr(r0r2rrHrQrRs@r:rrls/~9-U\\9-ell9-r;rc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Nr=r#r)rKr2r)rx1x2s r: rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r;c|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )rr)qrcossin position_ids unsqueeze_dimq_embedk_embeds r:apply_rotary_pos_embr sY( -- &C -- &C3w;q>C/0G3w;q>C/0G G r;rEn_reprc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r&N)rKrreshape)rEr batchnum_key_value_headsslenhead_dims r: repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr;moduler|r}r~attention_maskscalingrtc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)Nr#r r=)rr?)ptrainingr&)rnum_key_value_groupsr2r transposerKr)rrrAr@r?rtr contiguous) rr|r}r~rrrtkwargs key_states value_states attn_weights causal_maskrs r:eager_attention_forwardr!s3 ; ;$>&B\B\$\!}}d* !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  r;rEposition_embeddingsrpast_key_valuecache_positionrrc |jdd}g|d|j}|j|j|j dd} |j |j|j dd} |j |j|j dd} |\} } t| | | | \} } |'| | |d}|j| | |j|\} } t}|jjdk7r^|jjdk(r(|jddrtjd nt |jj}||| | | |f|j"sd n |j$|j&d |\}}|j(g|dj+}|j-|}||fS) Nr=r&r#)rrr.eagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rtr)rKrrmrrrnror updater$r!rf_attn_implementationgetlogger warning_oncerrr'rr rr*)r6rEr,rr-r.r input_shape hidden_shape query_statesrrrr cache_kwargsattention_interfacerrs r:rHzAriaTextAttention.forwards$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &S#7 jRUWZ#[ j  %#&snUL'5'<'d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r;)NN)rNrOrPrcr(intr0r2rrrr LongTensorrrrHrQrRs@r:r#r#sG ~ # 8+/59 /)||/)#5<<#=>/)!. /) ! /) !!1!12 /)-./) u||Xell3XeELL>Q5RR S/)r;r#cteZdZdZdedeffd Z ddejde ejde ejde e d e e d e e d e ejd e e ejejfd eede ej e e ej ej fffdZxZS)AriaTextDecoderLayerag Aria Text Decoder Layer. This class defines a single decoder layer in the language model, incorporating self-attention and Mixture of Experts (MoE) feed-forward network. Args: config (`AriaTextConfig`): Configuration object for the text component of the model. layer_idx (`int`): Index of the layer. rfr$c t||j|_t|||_t ||_t|j|j|_ t|j|j|_ y)N)rfr$r8) r/r0r7r# self_attnrmlpr- rms_norm_epsinput_layernormpost_attention_layernormr+s r:r0zAriaTextDecoderLayer.__init__\sm !--*&IN#F+.v/A/AvGZGZ[(78J8JPVPcPc(d%r;rErrr-r2 use_cacher.r,rrc |} |j|}|jd||||||||d| \}} | |z}|} |j|}|j|}| |z}|f} |r| | fz } | S)N)rErrr-r2rIr.r,)rGrDrHrE) r6rErrr-r2rIr.r,rresidualself_attn_weightsoutputss r:rHzAriaTextDecoderLayer.forwardes!,,]; ,:4>> , ')%)/) 3 ,  , ( (!=0 !55mD /  =0  "  )+ +Gr;)NNNFFNN)rNrOrPrcr(r>r0r2rrr?r boolrrr FloatTensorrHrQrRs@r:rArAOs e~e#e2637*.,1$)59KO(||(!.(u//0 ( ! ( $D> (D>(!!1!12(&eELL%,,,F&GH(-.( u  (51B1BEDUDU1U+V"WW X(r;rAc:eZdZdZeZdZddgZdZdZ dZ dZ dZ dZ y ) AriaTextPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. modelrArTpast_key_valuesFc|jj}t|tjrY|j j jd||j%|jj jyyt|tjrf|j j jd||j2|j j |jjyyt|tr(|j j jd|yt|tjrf|j j jd|t|dr2|j%|jj jyyyy)Nr3rCstdrW)rfinitializer_range isinstancer)rYr4datanormal_rWzero_ Embedding padding_idxrConv2dhasattrr6rrWs r: _init_weightsz%AriaTextPreTrainedModel._init_weightssHkk++ fbii ( MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>.  6 7 MM   & &CS & 9  * MM   & &CS & 9vv&6;;+B   &&(,C&+r;N)rNrOrPrcr' config_classbase_model_prefix_no_split_modulessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_cache_classrbrKr;r:rRrRsBL/1IJ&*#"3"N )r;rRaM This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`AriaTextConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zRThe bare Aria Model outputting raw hidden-states without any specific head on top.cFeZdZeZdZdZdgZdgZdZ dZ dZ dZ dZ dZdZdZy)AriaPreTrainedModelrSTAriaDecoderLayerrTFc|jj}t|tjrY|j j jd||j%|jj jyyt|tjrf|j j jd||j2|j j |jjyyt|tr,tjj|j|yy)Nr3rV)rW)rfrXrYr)rYr4rZr[rWr\r]r^rinit trunc_normal_r|ras r:rbz!AriaPreTrainedModel._init_weightsskk++ fbii ( MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>.  . GG ! !&,,C ! 8/r;N)rNrOrPr(rcrdrfrergrhri_supports_flex_attnrj_supports_quantized_cache_supports_static_cache_supports_attention_backendrbrKr;r:rlrlsU "L&*#+,#4"5!N  $""' 9r;rlc^eZdZddeffd Zej edZxZ S)AriaTextRotaryEmbeddingrfct|t|drG|j;|jj d|jj d|_nd|_|j |_|j |_||_ t|j |_ |j|j|\}|_ |jd|d|j|_y)N rope_scaling rope_typetypedefaultinv_freqF) persistent)r/r0r`rxr6rymax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrfr rope_init_fnattention_scalingregister_bufferr|original_inv_freq)r6rfrr|r9s r:r0z AriaTextRotaryEmbedding.__init__s  6> *v/B/B/N#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.ra) rNrOrPr(r0r2no_gradrrHrQrRs@r:rvrvs3/~/"U]]_ <