S^(h/DddlmZddlmZmZmZmZddlZddlZddlm Z ddl m Z m Z ddl mZddlmZmZdd lmZdd lmZmZd d lmZmZmZmZd dlmZej<eZ GddeZ!GddeZ"GddeZ#GddeeZ$GddeZ%y))partial)ListOptionalTupleUnionN)nn)Cache DynamicCache)FlashAttentionKwargs)BaseModelOutputWithPastCausalLMOutputWithPast)Unpack) LossKwargslogging)LlamaAttentionLlamaDecoderLayerLlamaForCausalLM LlamaModel) GraniteConfigc4eZdZdZddedeeffd ZxZS)GraniteAttentionz=Multi-headed attention from 'Attention Is All You Need' paperconfig layer_idxcHt||||j|_yN)super__init__attention_multiplierscalingselfrr __class__s i/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/granite/modular_granite.pyr zGraniteAttention.__init__&s +22 r) __name__ __module__ __qualname____doc__rrintr __classcell__r%s@r&rr#s"G3}3#33r'rcfeZdZdedeffd Z ddejdeejdeejdee dee d ee d eejd ee ejejfd e ejee ejejfffd ZxZS)GraniteDecoderLayerrrclt||||j|_t|||_y)N)rr)rr residual_multiplierr self_attnr#s r&r zGraniteDecoderLayer.__init__,s. +#)#=#= )9Mr' hidden_statesattention_mask position_idspast_key_valueoutput_attentions use_cachecache_positionposition_embeddingsreturnc |} |j|}|jd||||||||d| \}} | ||jzz}|} |j|}|j |}| ||jzz}|f} |r| | fz } | S)a. Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`Tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )r4r5r6r7r8r9r:r;)input_layernormr3r2post_attention_layernormmlp) r$r4r5r6r7r8r9r:r;kwargsresidualself_attn_weightsoutputss r&forwardzGraniteDecoderLayer.forward1sD!,,]; ,:4>> , ')%)/) 3 ,  , ( (!=43K3K#KK !55mD /  =43K3K#KK  "  )+ +Gr')NNNFFNN)r(r)r*rr,r torchTensorr LongTensorr boolr FloatTensorrFr-r.s@r&r0r0+sN}NN2637*.,1$)59KO?||?!.?u//0 ? ! ? $D> ?D>?!!1!12?&eELL%,,,F&GH? u  (51B1BEDUDU1U+V"WW X?r'r0c eZdZdeffd Z ddeejdeejdeejdee deejdee d ee d ee d eejd e e d efdZxZS) GraniteModelrc t|||j|_tjt |j Dcgc]}t||c}|_ycc}wr) rr embedding_multiplierr ModuleListrangenum_hidden_layersr0layersr#s r&r zGraniteModel.__init__tsR  $*$?$?!mmEJ6KcKcEd e  3 e  esA( input_idsr5r6past_key_values inputs_embedsr9r8output_hidden_statesr:flash_attn_kwargsr<c 0||n|jj}||n|jj}||n|jj}|du|duz r t d|j r%|j r|rtjdd}||j|}||jz}|r | t}| F||jnd} tj| | |jdz|j } || j#d}|j%||| ||} |} |j'| |}|rdnd}|rdnd}|j(d|jj*D]r}|r|| fz }|j r:|j r.|j-t/|j0fi| | | ||||| | }n|| f| ||||| |d| }|d} |sj||dfz }t|j3| } |r|| fz }t5| |r|nd|| S) Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Frr)devicer>)r5r6r7r8r9r:r;)last_hidden_staterUr4 attentions)rr8rWr9 ValueErrorgradient_checkpointingtraininglogger warning_once embed_tokensrOr get_seq_lengthrGarangeshaperZ unsqueeze_update_causal_mask rotary_embrSrR_gradient_checkpointing_funcr__call__normr )r$rTr5r6rUrVr9r8rWr:rXpast_seen_tokens causal_maskr4r;all_hidden_statesall_self_attns decoder_layer layer_outputss r&rFzGraniteModel.forward{s2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK?L] & #oom\J#7BD0d![[)H4;;+H+HI 6M#!m%55!**t}} $ A AM22H6GH! #%"' ! !.! !#.!-#2&7'#1(; !( ! *!,M =#3"55A 6D -0  -!1 1 &+/8Od+%   r') NNNNNNNNN)r(r)r*rr rrGrIrHr rKrJrr r rFr-r.s@r&rMrMss } 151537+/59$(,0/359g E,,-g !.g u//0 g "% g   1 12 g D>g $D>g 'tng !!1!12g $$89g  !g r'rMc eZdZy)KwargsForCausalLMN)r(r)r*r>r'r&rsrssr'rsc\eZdZ ddeej deej deej deeee ejfdeejdeej dee d ee d ee d eej d ee ej fd e edefdZy)GraniteForCausalLMNrTr5r6rUrVlabelsr9r8rWr:logits_to_keeprBr<c ||n|jj}| | n|jj} |jd|||||||| | d | } | j}t | t r t| dn| }|j|dd|ddf}||jjz }d}|*|jd|||jjd| }t||| j| j| jS)N) rTr5r6rUrVr9r8rWr:)logitsrv vocab_size)lossryrUr4r\r>)rr8rWmodelr[ isinstancer,slicelm_headlogits_scaling loss_functionrzrrUr4r\)r$rTr5r6rUrVrvr9r8rWr:rwrBrEr4 slice_indicesryr{s r&rFzGraniteForCausalLM.forwards.2C1N-TXT_T_TqTq$8$D $++JjJj  ,64:: , )%+'/!5) ,  ,  11 8B>SV8W~ot4]k mA}a,?@A$++444  %4%%pVFt{{OeOepiopD%#33!//))   r') NNNNNNNNNNr)r(r)r*rrGrIrHrr rrKrJr,rrsrrFr>r'r&rurus$151537KO59-1$(,0/359342 E,,-2 !.2 u//0 2 "%tE4E4E/F(F"GH 2   1 12 2 ))*2 D>2 $D>2 'tn2 !!1!122 c5<</02 *+2  2 r'ru)& functoolsrtypingrrrrrGtorch.utils.checkpointr cache_utilsr r modeling_flash_attention_utilsr modeling_outputsr rprocessing_utilsrutilsrrllama.modeling_llamarrrrconfiguration_graniter get_loggerr(r`rr0rMrsrur>r'r&rs // .BO&(bb0   H %3~3E+EPo :o d?,j>3 )3 r'