S^(hDdZddlmZmZmZmZmZddlZddlZddlm Z ddl m Z ddl m Z ddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZmZddlmZmZmZm Z m!Z!m"Z"ddl#m$Z$ejJe&Z'dZ(Gdde jRZ*ejVe*Gdde!Z,dZ-d&dZ.GddeZ/GddeZ0Gdde jRZ1Gdd e Z2Gd!d"e eZ3Gd#d$eZ4gd%Z5y)'zPyTorch Cohere model.)CallableListOptionalTupleUnionN)nn)Cache)FlashAttentionKwargs)BaseModelOutputWithPastCausalLMOutputWithPast)dynamic_rope_update)ALL_ATTENTION_FUNCTIONS)Unpack)ALL_LAYERNORM_LAYERS) LossKwargslogging)LlamaAttentionLlamaForCausalLMLlamaMLP LlamaModelLlamaRotaryEmbeddingeager_attention_forward) CohereConfigrc&eZdZdfd ZdZxZS)CohereLayerNormct|tjt j ||_||_y)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeepsbias __class__s g/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/cohere/modular_cohere.pyr!zCohereLayerNorm.__init__8s/ ll5::k#:; #c|j}|jtj}|j dd}||z j dj dd}||z tj ||jzz}|jjtj|z}|j|S)NT)keepdimr) dtypetor#float32meanpowrsqrtr&r%)r' hidden_states input_dtyper4variances r,forwardzCohereLayerNorm.forward>s#)) %((7 !!"d!3!D(--a055b$5G&-XH]H]=]1^^  u}}5 E  ,,r-)Ngh㈵>F)__name__ __module__ __qualname__r!r: __classcell__r+s@r,rr7s $ -r-rcDeZdZejedZy)CohereRotaryEmbeddingc.|jddddfjj|jddd}|dddddfj}t |j j tr/|j j dk7r|j j nd}tj|d5|j|jzjdd}tj|dd }|j|jz}|j|jz} dddj|j  j|j fS#1swYAxYw) Nrr/rmpscpuF) device_typeenabledrdimr1)inv_freqfloatexpandshape isinstancedevicetypestrr#autocast transposerepeat_interleavecosattention_scalingsinr2r1) r'x position_idsinv_freq_expandedposition_ids_expandedrEfreqsembrUrWs r,r:zCohereRotaryEmbedding.forwardLsD!MM$4-8>>@GG HZHZ[\H]_acde ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))%;C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF  FN)r;r<r=r#no_gradrr:r-r,rArAKs$U]]_ < C/0G3w;q>C/0G ::E: "GJJUJ$; ;;r-ceZdZfdZxZS) CohereMLPcJt||tj|j|j d|_tj|j|j d|_tj|j |jd|_y)NF)r*) r r!rLinearr(intermediate_size gate_projup_proj down_projr'configr+s r,r!zCohereMLP.__init__ss  4#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXr-)r;r<r=r!r>r?s@r,rqrqsYYr-rqc>eZdZdZd dedeeffd Z ddejde ejejfdeejdee d eejd e ed e ejeejee ejffd ZxZS)CohereAttentionz=Multi-headed attention from 'Attention Is All You Need' paperry layer_idxc*t||||j|_|jret|j|j f|j |_t|j|j f|j |_ yy)Nr(r)) r r! use_qk_normrnum_attention_headshead_dimlayer_norm_epsq_normnum_key_value_headsk_normr'ryr|r+s r,r!zCohereAttention.__init__s| +!--   )#77GVMbMbDK*#77GVMbMbDK r-r7position_embeddingsattention_maskpast_key_valuecache_positionkwargsreturnc |jdd}g|d|j}|j|j|} |j |j|} |j |j|} |j r"|j| } |j| } | jdd} | jdd} | jdd} |\} } t| | | | \} } |'| | |d}|j| | |j|\} } t}|jjdk7r^|jjdk(r(|j!ddrt"j%d nt&|jj}||| | | |f|j(sd n |j*|j,d |\}}|j.g|dj1}|j3|}||fS) Nr/rr)rWrUreagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.g)dropoutscaling)rMrq_projviewk_projv_projrrrrSroupdater|rry_attn_implementationgetlogger warning_oncertrainingattention_dropoutrreshape contiguouso_proj)r'r7rrrrr input_shape hidden_shape query_states key_states value_statesrUrW cache_kwargsattention_interface attn_output attn_weightss r,r:zCohereAttention.forwards$))#2. 88b8$--8 {{=166|D [[/44\B {{=166|D   ;;|4LZ0J#--a3 ))!Q/ #--a3 &S#7 jRUWZ#[ j  %#&snUL'5'<'d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r-N)NN)r;r<r=__doc__rrintr!r#Tensorrr LongTensorrr r:r>r?s@r,r{r{sG |  "+/59 7)||7)#5<<#=>7)!. 7) ! 7) !!1!12 7)-.7) u||Xell3XeELL>Q5RR S7)r-r{cpeZdZdedeffd Z ddejdeejdeejdee dee d ee d eejd ee ejejfd e ed e ejee ejejfffdZxZS)CohereDecoderLayerryr|ct||j|_t|||_t ||_t|j|j|_ y)N)ryr|r~) r r!r(r{ self_attnrqmlprrinput_layernormrs r,r!zCohereDecoderLayer.__init__sR !--()LV$.F!,,]; 6DT^^ 6 ')%)/) 3 6  6 2!2!HH]3!#::=NN  "  )+ +Gr-)NNNFFNN)r;r<r=rrr!r#rrrr boolrrr FloatTensorr:r>r?s@r,rrsl|ll2637*.,1$)59KO:||:!.:u//0 : ! : $D> :D>:!!1!12:&eELL%,,,F&GH:-.: u  (51B1BEDUDU1U+V"WW X:r-rc$eZdZdeffd ZxZS) CohereModelryc &t||tjt |j Dcgc]}t ||c}|_t||_ t|j|j|_ ycc}w)N)ryr~)r r!r ModuleListrangenum_hidden_layersrlayersrA rotary_embrr(rnormrs r,r!zCohereModel.__init__so  mmDI&JbJbDc dy  2 d  0v>#1C1C&J_J_`  esB)r;r<r=rr!r>r?s@r,rrsa|aar-rc eZdZy)KwargsForCausalLMN)r;r<r=r_r-r,rr"sr-rcneZdZfdZ ddeej deejdeej deee e ejfdeejdeej dee d ee d ee d eej d ee ejfd eedefdZxZS)CohereForCausalLMct||t||_|j|_|j |_yr)r r!rmodel logit_scaletie_word_embeddingsrxs r,r!zCohereForCausalLM.__init__&s8   ( !--#)#=#= r- input_idsrrYpast_key_values inputs_embedslabelsrroutput_hidden_statesrlogits_to_keeprrc ||n|jj}| | n|jj} |jd|||||||| | d | } | j}t | t r t| dn| }|j|dd|ddf}||jz}d}|*|jd|||jjd| }t||| j| j| jS)a0 labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. logits_to_keep (`int` or `torch.Tensor`, *optional*): If an `int`, compute logits for the last `logits_to_keep` tokens. If `0`, calculate logits for all `input_ids` (special case). Only last token logits are needed for generation, and calculating them only for that token can save memory, which becomes pretty significant for long sequences or large vocabulary size. If a `torch.Tensor`, must be 1D corresponding to the indices to keep in the sequence length dimension. This is useful when using packed tensor format (single dimension for batch and sequence length). Returns: Example: ```python >> from transformers import AutoTokenizer, CohereForCausalLM >> model = CohereForCausalLM.from_pretrained("CohereForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("CohereForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrYrrrrrr)logitsr vocab_size)lossrrr7 attentionsr_)ryrrrlast_hidden_staterNrslicelm_headr loss_functionrr rr7r)r'rrrYrrrrrrrrrrr7 slice_indicesrrs r,r:zCohereForCausalLM.forward,s+\2C1N-TXT_T_TqTq$8$D $++JjJj  ,64:: , )%+'/!5) ,  ,  11 8B>SV8W~ot4]k mA}a,?@A$***  %4%%pVFt{{OeOepiopD%#33!//))   r-) NNNNNNNNNNr)r;r<r=r!rr#rrrr rrrrrrr r:r>r?s@r,rr%s8>151537KO59-1$(,0/35934Q E,,-Q !.Q u//0 Q "%tE4E4E/F(F"GH Q   1 12 Q ))*Q D>Q $D>Q 'tnQ !!1!12Q c5<</0Q *+Q  Q r-r)rrCoherePreTrainedModel)Nr)6rtypingrrrrrr#torch.utils.checkpointr cache_utilsr modeling_flash_attention_utilsr modeling_outputsr r modeling_rope_utilsrmodeling_utilsrprocessing_utilsr pytorch_utilsrutilsrrllama.modeling_llamarrrrrrconfiguration_coherer get_loggerr;r_CONFIG_FOR_DOCModulerappendrArgrorqr{rrrr__all__r_r-r,rs.99  BO65&1(/   H % -bii-"O,<0<"<<YYF)nF)RBBJa*a?,j>X (X v r-