S^(hddZddlmZmZmZddlZddlZddlmZddlm Z ddl m Z m Z m Z ddlmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZd dlmZer ddlm Z ddl!m"Z"ejFe$Z%dZ&dZ'de(de(dejRfdZ*dejRdejRfdZ+dejRdejRdejRdejRfdZ,GddejZZ.Gdd ejZZ/Gd!d"ejZZ0Gd#d$eZ1d%Z2d&Z3ed'e2Gd(d)e1Z4ed*e2Gd+d,e1eZ5gd-Z6y).zPyTorch CodeGen model.)OptionalTupleUnionN)nn)ACT2FN)Cache DynamicCache StaticCache)GenerationMixin)AttentionMaskConverter)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardis_torch_flex_attn_availablelogging) CodeGenConfig) BlockMask)make_flex_block_causal_maskzSalesforce/codegen-2B-monornum_posdimreturncddtjd|dtj|z zz }tjdtj|tjj |j }tj tj |tj|fdS) N?i'rdtypez i , j -> i jrr)torcharangeint64einsumfloatcatsincos)rrinv_freq sinusoid_inps j/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/codegen/modeling_codegen.pycreate_sinusoidal_positionsr.4se QQekk JS PQRH<< WEKK0X0^0^0`bjkqqsL 99eii -uyy/FGQ OOxc|dddddddddf}|dddddddddf}tj| |fd}|jdS)Nrrr")r#stackflatten)r0x1x2s r-rotate_every_twor8;sS 1aCaC<B 1aADqD= B bS"I2&A 99R=r/tensorr)r*ctj|dddddddfdd}tj|dddddddfdd}||zt||zzS)Nrr)r#repeat_interleaver8)r9r)r*s r-apply_rotary_pos_embr<Cs^  ! !#aD!m"4a ;C  ! !#aD!m"4a ;C SL-f5; <>"#;N$NOr/ct|jdk(r$|jdddddj}n\t|jdk(r#|jddddj}n!t dt|j|j dd ||zfz}|j |S) zM Merges attn_head_size dim and num_attn_heads dim into n_ctx rrrrz3Input tensor rank should be one of [4, 5], but is: Nr3)lenrcpermute contiguousrRsizeview)r]r9rPattn_head_size new_shapes r- _merge_headszCodeGenAttention._merge_headsms v||  !^^Aq!Q2==?F  ! #^^Aq!Q/:: MMD$<$<<F !KK 9"Euc!!%)A)A4==Y_!`T%=%=t}}U[\!!%)A)A4==Y_!` aAq)..  ! !\%8%8 8-001D1DEO#2D  .;;vv||B'71'<"ES ?? &1a!24??!223EAq$//"334F!Q#4T__#445E1aDOO$556F(S9E(S9E))UFO4CIIufo26E&sC5C(S9Ekk!Q1% aAq)  !)-"0 L $**366-2E2E+Ft~~_klJC%)JJuc5.R[$\! \'' T5M5Mt}}] mmK0 ((5  +    &Gr/N)NNNNNNFFN)rM __module__ __qualname__rBrhrsrrr# FloatTensorr LongTensorboolrrTensorr __classcell__rLs@r-r>r>Is6X< &$ )D'+6:3715$),159L 1 12LUOL!!2!23 L u//0 L E--. LD>L$D>L!!1!12L  ellE%,,//0u||U5<<%8% c@Q:RRST V Lr/r>c\eZdZfdZdeej dej fdZxZS) CodeGenMLPct||j}tj|||_tj|||_t|j|_ tj|j|_ yr) rArBn_embdrrXfc_infc_outractivation_functionactrDrGdropout)r]intermediate_sizer^rOrLs r-rBzCodeGenMLP.__init__se MM YYy*;< ii 19= &445zz&"4"45 r/rrc|j|}|j|}|j|}|j|}|Sr)rrrr)r]rs r-rzCodeGenMLP.forwards@ =1 /  M2  ]3 r/) rMrrrBrr#rrrrs@r-rrs,6Xe.?.?%@UEVEVr/rcVeZdZd fd Z ddeej deedeej deejdeej dee dee d eejd e e ejee eje ej d ffffd Z xZS) CodeGenBlockct||j |jnd|jz}t j |j|j |_t|||_ t|||_ y)Nrkeps) rArBn_innerrr LayerNormlayer_norm_epsilonln_1r>attnrmlp)r]r^rI inner_dimrLs r-rBzCodeGenBlock.__init__sc &,nn&@FNNa&--FW LLF4M4MN $VY7 i0r/rrr{rr|rrrr.c |} |j|}|j||||||||} | d} | dd} |j|} | | z| z}|r|f| z} | S|f| ddz} | S)Nrrr{rr|rrrrr)rrr)r]rrr{rr|rrrresidual attn_outputsrrfeed_forward_hidden_statess r-rzCodeGenBlock.forwards! -0 yy'!)%/)!  #1o qr"%)XXm%<"#&@@8K $&0G%&4Gr/rr)rMrrrBrr#rr rrrrrrrrs@r-rrs1'+6:3715$),159" 1 12"UO"!!2!23 " u//0 " E--. "D>"$D>"!!1!12" uU\\"HU5<<uGXGXZ]G]A^3^-_$`` a"r/rcJeZdZdZeZdZdZdgZdZ dZ dZ dZ fdZ dZxZS)CodeGenPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. transformerTrpast_key_valuesc$t||i|yr)rArB)r]inputskwargsrLs r-rBzCodeGenPreTrainedModel.__init__:s &+F+r/ct|tjfrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyy)zInitialize the weights.)meanstdNr) isinstancerrXweightdatanormal_r^initializer_ranger@zero_ Embedding padding_idxrfill_)r]modules r- _init_weightsz$CodeGenPreTrainedModel._init_weights=s fryyl + MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r/)rMrr__doc__r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_cache_class_supports_quantized_cache_supports_static_cacherBrrrs@r-rr+sF !L%&*#'("3  $!,*r/raJ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`CodeGenConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoProcenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_attention_heads,)` or `(n_layer, num_attention_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_dim)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. past_key_values (`Cache` or `tuple(tuple(torch.FloatTensor))`, *optional*): Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Two formats are allowed: - a [`~cache_utils.Cache`] instance, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache); - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy cache format. The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the legacy cache format will be returned. If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. zaThe bare CodeGen Model transformer outputting raw hidden-states without any specific head on top.ceZdZfdZdZdZeejde e e e  dde ejde eeeeej&fde ej(d e ejd e ejd e ej(d e ej(d e ede ede ede ede ejdeee ffdZ ddej&dej&dej&dedef dZedej&dededej4dej6dej&defdZxZS) CodeGenModelc t|||j|_|j|_t j |j|j|_t j|j|_ t jt|jDcgc]}t||c}|_t j |j|j"|_t'|j(|j*|j,z|_d|_|j1ycc}w)N)rIrF)rArBrrO vocab_sizerrwterD embd_pdropdrop ModuleListrangen_layerrhrrln_fminr[n_ctxrPgradient_checkpointing post_init)r]r^irLs r-rBzCodeGenModel.__init__s   ++<< 1 14>>BJJv001 5QWQ_Q_K`aa Vq AabLLV5N5NO f//A[A[1[\&+#  bs,Ec|jSrrr]s r-get_input_embeddingsz!CodeGenModel.get_input_embeddingss xxr/c||_yrrr]new_embeddingss r-set_input_embeddingsz!CodeGenModel.set_input_embeddingss !r/batch_size, sequence_length checkpoint output_typer input_idsrr{token_type_idsrr| inputs_embedsrroutput_hidden_states return_dictrrc | | n|jj} | | n|jj} ||n|jj}| | n|jj} |du|duz r t d|j r%|jr|rtjdd}||j|}d}|rIt|ts9d}| t}n*tj|}tjd|jd}| 9||j!nd}t#j$|||z|j&} || j)d}|j+||| || }|j-||jj.}|}|(|j1d |}|j|}||z}|j3|}d ||j5d f}d}| rd nd}| rd nd}t7|j8D]}\}}| r||fz}|j r3|jr'|j;|j<|d|||||| | }n||||||||| | }|d}|dur|d}| sq|||rd ndfz}|j?|}|j1|}| r||fz}|r|nd}|r|jA}| stCd ||||fDStE||||S)Nz:You must specify exactly one of input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FTzWe detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)rrrr2rrc3&K|] }|| ywrr).0vs r- z'CodeGenModel.forward..Asbcbos)last_hidden_staterr attentions)#r^rrruse_return_dictrRrtrainingrJrKrrr r from_legacy_cachercget_seq_lengthr#r$r unsqueeze_update_causal_mask get_head_maskrrprro enumerater_gradient_checkpointing_func__call__rto_legacy_cachetupler)r]rrr{rrr|rrrrrrrreturn_legacy_cache seq_lengthpast_seen_tokensr~rtoken_type_embeds output_shapenext_decoder_cacheall_self_attentionsall_hidden_statesrblockr next_caches r-rzCodeGenModel.forwards,2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK?L] &&y$++2E2EF %  %+00Z@N $ 8 ),==M -0 J (:(:2(>? !$5b4"6BD!$&&)! ^HAu#$58H$H!**t}};;NN! aL%"  "/.#.!-'l'&7#1 $AJMD %,QZ" &9W)QYZ=[<]&]#C! ^F -0 %**<8  1]4D D +4'$ #335J):7HJ]^ '+&+*   r/ input_tensorc |jjdk(r||dk(jr|Sy|jjdk(r7t|tj r t |}t|tr|S||jnd}t|t}|jjdk(r(|s&|s$tj||||jry|j|j} }|jd} |r|j!} n1t|tj r|jdn|| zdz} |j#|| | || ||jd } |jjdk(rQ|O|jj$d vr7|s5t j&|j(} tj*| | } | S) Nflash_attention_2rflex_attentionrsdpa)rpast_key_values_length is_trainingrr2)sequence_length target_lengthr!rr batch_size)cudaxpu)r^_attn_implementationanyrr#rrrr r r _ignore_causal_mask_sdparr!rrcget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positiontypefinfor_unmask_unattended)r]r{rrrrrusing_static_cacher!rr$r%r~ min_dtypes r-r z CodeGenModel._update_causal_maskMs ;; + +/B B)~/D.I.I.K%% ;; + +/? ?.%,,7!<^!L.)4%% @O?Z?99;`a'E ;; + +v 5>PYj%>>*'7 MM  $**L,?,?v&,,Q/ +??AMnell;$$R(%7!; PP +')#))!,Q  KK , , 6*%%**o=%  E*..I0CCKQZ[Kr/r$r%r!rr&c ||jdk(r|}|Stj|j} tj||f| ||}|dk7rtj |d}|tj |||jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|jz} | dk(} |ddddddd| fj| | |ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. device (`torch.device`): The device to place the 4D attention mask on. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nrk) fill_valuer!rr)diagonalrr2r)rr#r/rfulltriur$rbexpandclonercrUr masked_fill) r{r$r%r!rrr&rr~r2 mask_length padding_masks r-r-zBCodeGenModel._prepare_4d_causal_attention_mask_with_cache_positionsyD  %.*<*<*>!*C(K*' E*..I** -0Ye\bK!##jjqA 5<< fEH^H^_acdHee eK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r/) NNNNNNNNNNNN)F)rMrrrBrrrCODEGEN_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOCrr#rrr rrrrrr  staticmethodintr!rr-rrs@r-rrs=  "++C+J+JKh+ij&+$15NR6:59371559$(,0/3&*59H E,,-H "%uU5<<5H/I(I"JKH !!2!23 H !!1!12 H u//0 H E--.H   1 12H D>H $D>H 'tnH d^H !!1!12H  u-- .H k H b#( D DllD D  D  DL7 777{{ 7  7  777r/rzM The CodeGen Model transformer with a language modeling head on top. c!peZdZdgZfdZdZdZeejde e e e  ddeej deeeeeej(fd eej*d eej d eej d eej*d eej*deej deedeedeedeedeej deee ffdZedeeej(dej(deeej(fdZxZS)CodeGenForCausalLMzlm_head.weightct||t||_t j |j |j|_|jyr) rArBrrrrXrrlm_headr)r]r^rLs r-rBzCodeGenForCausalLM.__init__sE  '/yy0A0AB  r/c|jSrrFrs r-get_output_embeddingsz(CodeGenForCausalLM.get_output_embeddingss ||r/c||_yrrHrs r-set_output_embeddingsz(CodeGenForCausalLM.set_output_embeddingss % r/rrrrr{rrr|rlabelsrrrrrrc $| | n|jj} |j|||||||| | | | |  }|d}|j|j t j }d}|`|j |j}|j||fd|jji|}|j |j}| s|f|ddz}||f|zS|St|||j|j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N) rr{rrr|rrrrrrrrr)losslogitsrrr)r^rrrFrUr#rTr loss_functionrr!rrrr)r]rrr{rrr|rrLrrrrrrtransformer_outputsr lm_logitsrNoutputs r-rzCodeGenForCausalLM.forwardsF:&1%CodeGenForCausalLM._reorder_cache...>s.jQ[*))!X[[9J9J-KLjs58Nr)rrrTs r-rz4CodeGenForCausalLM._reorder_cache..=s%  j_ij j s!rZ)rrTs `r-_reorder_cachez!CodeGenForCausalLM._reorder_cache4s -   r/) NNNNNNNNNNNNN)rMrr_tied_weights_keysrBrIrKrr=r>rr?rr@rr#rrr rrrrrrAr[rrs@r-rDrDs++&++C+J+JKh+ij&*$15NR6:59371559-1$(,0/3&*59F E,,-F "%uU5<<5H/I(I"JKF !!2!23 F !!1!12 F u//0 F E--.F   1 12F ))*F D>F $D>F 'tnF d^F !!1!12F u,, -!F k F P  uU\\23  ?D||  uU\\" #    r/rD)rDrr)7rtypingrrrr#torch.utils.checkpointr activationsr cache_utilsr r r generationr modeling_attn_mask_utilsr modeling_outputsrrmodeling_utilsrutilsrrrrrconfiguration_codegenr!torch.nn.attention.flex_attentionrintegrations.flex_attentionr get_loggerrMrJr?r@rBrr.r8r<Moduler>rrrCODEGEN_START_DOCSTRINGr=rrD__all__rr/r-rms)) !;;)>O-1 !;J   H %2!PP3P5<<P ==ELL=u||=X]XdXd= \ryy\@(+299+\ *_ *F EPgg)g gT   k /k  k \ Kr/