T^(hdZddlmZmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZmZmZdd lmZdd lmZdd lmZmZdd lmZmZmZmZdd lm Z ddl!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)e%r ddl*m+Z+ddl,m-Z-erddlm.Z.e&j^e0Z1dZ2dZ3gdZ4dZ5dZ6dZ7GddejpZ9GddejtZ;Gdde;Z<Gd d!e;Z=e;eGd#d$ejtZ?d%Z@e#d&e@Gd'd(e ZAd)ZBGd*d+eAZCe#d&e@Gd,d-eAZDGd.d/eAeZEe#d0e@Gd1d2eAZFe#d3e@Gd4d5eAZGgd6ZHy)7zPyTorch OPT model.)ListOptionalTupleUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCache StaticCache)GenerationMixin)AttentionMaskConverter)!flash_attn_supports_top_left_maskis_flash_attn_available)BaseModelOutputWithPastCausalLMOutputWithPastQuestionAnsweringModelOutput SequenceClassifierOutputWithPast)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardis_torch_flex_attn_availableloggingreplace_return_docstrings) OPTConfig) BlockMask)make_flex_block_causal_mask)_flash_attention_forwardzfacebook/opt-350mr )rizArthurZ/opt-350m-dummy-scg\(\?z 'LABEL_0'cxeZdZdZdedeffd Z d dejdedeejffd Z xZ S) OPTLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcNd|_t| ||jz|yN)offsetsuper__init__)selfr'r( __class__s b/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/opt/modeling_opt.pyr.z&OPTLearnedPositionalEmbedding.__init__Ns$  $++5}Eattention_maskpast_key_values_length position_idsc|8tj|d}||zdz j}|dd|df}t|||j zS)z3`input_ids_shape` is expected to be [bsz x seqlen].Nrdim)torchcumsumlongr-forwardr,)r/r3r4r5r0s r1r<z%OPTLearnedPositionalEmbedding.forwardTs^   <<A>L(>9A=CCEL'+A+B(BCLw|dkk9::r2rN) __name__ __module__ __qualname____doc__intr.r9 LongTensorrr< __classcell__r0s@r1r&r&Is]FsF3F'(37 ;((;!$;u//0 ;;r2r&cHeZdZdZ ddedeeffd Z ddejdee ejdeejdeejd e d eejd eejd e ejeejee ffd Z xZS) OPTAttentionz=Multi-headed attention from 'Attention Is All You Need' paperconfig layer_idxc t|||_|j|_|j |_|j|_|j|_ ||_ |-tjd|jjd|j|j z|_d|_|j|j z|jk7r&t#d|jd|j d|jdz|_t'j(|j|j|j|_t'j(|j|j|j|_t'j(|j|j|j|_t'j(|j|j|j|_y) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩bias)r-r.rH hidden_size embed_dimnum_attention_heads num_headsattention_dropoutdropout enable_biasrIlogger warning_oncer0r>head_dim is_causal ValueErrorscalingrLineark_projv_projq_projout_proj)r/rHrIkwargsr0s r1r.zOPTAttention.__init__hs  ++33// !--"      !8!8 9:,,  $..8  MMDNN *t~~ =MdnnM]$T^^$4B8 }}d* iiTEUEUV iiTEUEUV iiTEUEUV  $..$..tGWGWX r2 hidden_statespast_key_valuer3layer_head_maskoutput_attentionsr5cache_positionreturncl|j\}} } |j||jz} | j|d|j|j j dd} |j|} |j|} | j|d|j|j j dd} | j|d|j|j j dd} |#|j| | |jd|i\} } tj| | j dd}|#|ddddddd| jdf}||z}tjj!|dtj"j%| j&}|_|j|jfk7r*t)d |jfd |j|jdddd|z}tjj+||j*|j, }tj|| }|j ddj/}|j1|| |j2}|j5|}|||fS) #Input shape: Batch x Time x Channelrr+Nrdr )r8dtypez/Head mask for a single layer should be of size z , but is ptraining)sizer]rYviewrPrV transposer[r\updaterIr9matmulshaper functionalsoftmaxfloat32torjrXrRrm contiguousreshaperNr^)r/r`rar3rbrcr5rdbsztgt_len_ query_states key_states value_states attn_weights causal_mask attn_probs attn_outputs r1r<zOPTAttention.forwards}(,,.Wa{{=1DLL@ #((b$..$--PZZ[\^_` [[/ {{=1 __S"dnndmmLVVWXZ[\ #((b$..$--PZZ[\^_`  %'5'<'<L$..;K^:\( $J ||L*2F2Fq!2LM  %(Aq2HJ4D4DR4H2H)HIK'+5L}},,\r,WZZ[g[m[mn  &##%$..):: Et~~FWEXY',,./1+//2q!<|KL]]**<4< "))#wG mmK0 J66r2NNNNFNN)r>r?r@rAr rrBr.r9Tensorrboolr r<rDrEs@r1rGrGesG $(!Y!YC=!YL9=1526"'/31577||77!u||!4577!. 77 "%,,/ 77  77u||,77!.77 u||Xell3Xe_D E77r2rGcNeZdZdZfdZ d dej deeej deej deej de deej d eej d eej eej eeej ffd Z xZ S) OptFlashAttention2aB OPT flash attention module. This module inherits from `OPTAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cBt||i|t|_yr)r-r.r_flash_attn_uses_top_left_mask)r/argsr_r0s r1r.zOptFlashAttention2.__init__s# $)&) /P.Q+r2r`rar3rbrcr5rdrec  |j\}} } |j|} | j|d|j|j} |j |} |j |} | j|d|j|jjdd} | j|d|j|jjdd} |#|j| | |jd|i\} } |jr |jnd}| jdd} | jdd} | j}|tjk(rtjrtj }nMt#|j$dr|j$j&}n |jj(j}t*j-d|d | j/|} | j/|} | j/|} t1| | | || |||j2|j4 }|j7|| |j|jz}|j9|}|sd}|||fS) rgrhrr+Nrd_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)r5rRrWuse_top_left_mask)rnr]rorPrVr[r\rprqrIrmrRrjr9rvis_autocast_enabledget_autocast_gpu_dtypehasattrrHrweightrTrUrwr#rWrryr^)r/r`rar3rbrcr5rdrz query_lengthr|r}r~r attn_dropout input_dtype target_dtyperattn_weights_reshapeds r1r<zOptFlashAttention2.forwardsM -113\1{{=1 #((b$..$--P [[/ {{=1 __S"dnndmmLVVWXZ[\ #((b$..$--PZZ[\^_`  %'5'<'<L$..;K^:\( $J (,}}t||#  ))!Q/ #--a3 #(( %-- '((*$;;= &?@#{{BB #{{1177    >$  (??<8L#|4J'??<8L.     % nn"AA  !, 3 3Ct~~X\XeXeGe fmm$9:  $( !1>AAr2r) r>r?r@rAr.r9rrrrr<rDrEs@r1rrs R9=1526"'/315LB||LB!u||!45LB!. LB "%,,/ LB  LBu||,LB!.LB u||Xell3XeELL>Q5RR SLBr2rcHeZdZdZ d dej deeej deej deej dedeej deej d eej eej eeej fffd Z xZ S) OPTSdpaAttentiona@ OPT sdpa attention module. This module inherits from `OPTAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of sdpa attention and deal with padding tokens in case the input contains any of them. r`rar3rbrcr5rdrec|s|*tjdt| ||||||S|j \}} } |j |} | j |d|j|jjdd} |j|} |j|} | j |d|j|jjdd} | j |d|j|jjdd} |#|j| | |jd|i\} } |}||ddddddd| jdf}|| dkDrdnd }tj j"j%| | | ||j&r |j(nd | }|jddj+}|j || d}|j-|}|d|fS) NzOPTModel is using SDPA attention, which currently does not support output_attentions=True.failing back to eager attention. remove warning using attn_implementation="eager".)r`r3rbrarcrdrhrr+rdriTFr) attn_mask dropout_prW)rTrUr-r<rnr]rorPrVrpr[r\rqrIrsr9rrtscaled_dot_product_attentionrmrRrxr^)r/r`rar3rbrcr5rdrzq_lenr|r}r~rrrWrr0s r1r<zOPTSdpaAttention.forward*s   ;   e  7?+- /-"3- # &**, UA{{=1 #((b$..$--PZZ[\^_` [[/ {{=1 __S"dnndmmLVVWXZ[\ #((b$..$--PZZ[\^_`  %'5'<'<L$..;K^:\( $J %  %%aA/E1A1A"1E/E&EFK(/EAID5 hh))FF   !&*mmdll G "++Aq1<<> !&&sE26 mmK0 D.00r2r) r>r?r@rAr9rrrrr<rDrEs@r1rr#s9=1526"'/315>1||>1!u||!45>1!. >1 "%,,/ >1  >1u||,>1!.>1 u||Xell3XeELL>Q5RR S>1>1r2r)eagerflash_attention_2sdpacjeZdZddedeeffd Z ddejdeejdeejdee ejdee d ee d eejd eejd e ejee ejejfffd Z xZS)OPTDecoderLayerrHrIct||j|_t |j |||_|j|_|j|_t|j|_ tj|j|j|_tj |j|j"|j$|_tj |j"|j|j$|_tj|j|j|_y)N)rHrIelementwise_affinerK)r-r.rMrNOPT_ATTENTION_CLASSES_attn_implementation self_attndo_layer_norm_beforerRr activation_function activation_fnr LayerNormlayer_norm_elementwise_affineself_attn_layer_normrZffn_dimrSfc1fc2final_layer_norm)r/rHrIr0s r1r.zOPTDecoderLayer.__init__ss ++.v/J/JKSYeno$*$?$?!~~ #F$>$>?$&LL NNv/S/S% !99T^^V^^&BTBTU99V^^T^^&BTBTU " T^^PVPtPt ur2r`r3rbrarc use_cacher5rdrec |} |jr|j|}|j|||||||\}} } tjj ||j |j }| |z}|js|j|}|j} |jd|jd}|} |jr|j|}|j|}|j|}|j|}tjj ||j |j }| |zj| }|js|j|}|f} |r| | fz } |r| | fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`, *optional*): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence.. )r`rar5r3rbrcrdrkrh)rrrrrtrRrmrsryrnrrrrro)r/r`r3rbrarcrr5rdresidualself_attn_weightspresent_key_valuehidden_states_shapeoutputss r1r<zOPTDecoderLayer.forwards:!  $ $ 55mDM?Cnn')%)+/)?M? ; (*; --mt||VZVcVc-d  =0 (( 55mDM,11%--b-2D2DR2HI    $ $ 11-@M/ **=9 /  --mt||VZVcVc-d !M1778KL (( 11-@M "  )+ +G  )+ +Gr2r)NNNFFNN)r>r?r@r rrBr.r9rrrrC FloatTensorr<rDrEs@r1rrrsvyvXc]v(26268<,1$)3715Q||Q!.Q"%,,/ Q !u||!45 Q $D> QD>Qu//0Q!.Q u  (51B1BEDUDU1U+V"WW XQr2raH This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`OPTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zQThe bare OPT Model outputting raw hidden-states without any specific head on top.c8eZdZeZdZdZdgZdZdZ dZ dZ dZ dZ y)OPTPreTrainedModelmodelTrc|jj}t|tjrY|j j jd||j%|jj jyyt|tjrf|j j jd||j2|j j |jjyyy)Nr)meanstd) rHinit_std isinstancerrZrdatanormal_rLzero_ Embedding padding_idx)r/modulers r1 _init_weightsz OPTPreTrainedModel._init_weightsskk"" fbii ( MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>..r2N)r>r?r@r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_2_supports_sdpa_supports_cache_class_supports_quantized_cache_supports_static_cacherr2r1rrs@ L&*#*+!N  $! ?r2ra Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. for padding use -1. [What are position IDs?](../glossary#position-ids) cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. c<eZdZdZdeffd ZdZdZ ddejdejdejd e d e f d Z e dejd ed edejdej dejdefdZ ddeej&deejdeejd eeej*deej*dee d ee dee dee deej&deejdeeeffdZxZS) OPTDecoderz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`OPTDecoderLayer`] Args: config: OPTConfig rHc 8t|||j|_|j|_|j|_|j |_|j|_tj|j|j|j |_ t|j |j|_|j|jk7r2tj |j|jd|_nd|_|j|jk7r2tj |j|jd|_nd|_|j&r=|j(s1tj*|j|j,|_nd|_tj0t3|j4Dcgc]}t7||c}|_d|_|j=ycc}w)NFrKr)rI)r-r.rR layerdrop pad_token_idrmax_position_embeddingsmax_target_positions vocab_sizerrword_embed_proj_dim embed_tokensr&rMembed_positionsrZ project_out project_inr_remove_final_layer_normrrr ModuleListrangenum_hidden_layersrlayersgradient_checkpointing post_init)r/rHir0s r1r.zOPTDecoder.__init__Ss  ~~ ))!..$*$B$B! ++LL):):FPYj%>>*'7 MM  $**L,?,?v&,,Q/ +??AMnell;$$R(%7!; PP +')#))!,Q  KK , , 6*%%**o=%  E*..I0CCKQZ[Kr2rrrjrrc ||jdk(r|}|Stj|j} tj||f| ||}|dk7rtj |d}|tj |||jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|jz} | dk(} |ddddddd| fj| | |ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. device (`torch.device`): The device to place the 4D attention mask on. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuerjrr)diagonalrrhr)r8r9rrfulltriuarangeryexpandclonersrwr masked_fill) r3rrrjrrdrr_rr  mask_length padding_masks r1rz@OPTDecoder._prepare_4d_causal_attention_mask_with_cache_positionsyD  %.*<*<*>!*C(K*' E*..I** -0Ye\bK!##jjqA 5<< fEH^H^_acdHee eK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r2 input_ids head_maskrroutput_hidden_states return_dictr5rec F||n|jj}||n|jj}||n|jj}| | n|jj} |du|duz r t d|j r%|jr|rtjdd}||jd|jd}||j|}d} |r>t|ts.d} tj |}|tjd||j#nd} | 2t%j&| | |jd z|j( } |A| |jd z}t%j*|jd||j( }|j-||| ||}| 8t%j.|d } | |zd z j1} | dd| df} |j3|| | }|j4|j5|}||j7|j(z}|rd nd}|rd nd}d}t9|gdgD]j\}}| |j;dt=|j>k7s3t d|dt=|j>d|j;ddtA|j>D]\}}|r||fz }|jr%t%jBg}||jDkr?|j r7|jr+|jG|jH|||||ndd||| | }n|||| |||nd|||| }|d}|r ||rdnd }|s||d fz }|jJ|jK|}|jL|jM|}|r||fz }|r|nd}| r|jO}| stQd||||fDStS||||S)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(num_hidden_layers, num_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. for padding use -1. [What are position IDs?](../glossary#position-ids) cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FrhTzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.53.0. You should pass an instance of `DynamicCache` instead, e.g. `past_key_values=DynamicCache.from_legacy_cache(past_key_values)`.rrrr7)r5rrzThe `z` should be specified for z layers, but it is for r)r3r5rbrarcrrdr+c3&K|] }|| ywrr).0vs r1 z%OPTDecoder.forward..stqfgfstslast_hidden_staterr` attentions)*rHrcrruse_return_dictrXrrmrTrUrorsrrr rfrom_legacy_cacherr9rronesr r:r;rrrwziprnlenr enumeraterandr_gradient_checkpointing_func__call__rrto_legacy_cachetupler)r/rr3rrrrrcrrr5rdreturn_legacy_cacher  seq_lengthr pos_embedsr`all_hidden_statesall_self_attnsnext_decoder_cacher mask_nameidx decoder_layerdropout_probability layer_outputs next_caches r1r<zOPTDecoder.forwardsL2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK?I   --i8M# Z?"& *<<_MO&##Y @O?Z?99;`a  !"\\ "2]5H5H5K"KTaThThN  !)M,?,?,BBJ"ZZ (;(;A(> S`SgSghN.. M>?L]   <<A>L(>9A=CCEL'+;+<(<=L)).:JYe)f ?? & OOM:M% m6J6J(KK #7BD0d!%( k]$C  Iy$>>#A&3t{{+;<$ {*DSEUDVW%NN,Q/03 #,DKK"8( 6 C#!m%55!}}&+jjn#&7**t}} $ A A!**!&/&;IcN% " ! !.!#.!-7@7LYs^RV#2&7'#1 ! *!,M%28I1q%Q" =#3"55Q( 6T  , 11-@M    ' ,,];M -!1 1 +4'$ #335Jt]J@QSa$btt t&+&+%   r2)F NNNNNNNNNNN)r>r?r@rAr r.rrr9rr rr  staticmethodrBrjrrrrCrrrrrr<rDrEs@r1rrKs#y#J!"#( D DllD D  D  DL7 777{{ 7  7  777v1515,0=A59$(,0/3&*3715R E,,-R !.R ELL) R "$u'8'8"9: R   1 12 R D>R $D>R 'tnR d^R u//0R !.R  u-- .R r2rceZdZdeffd ZdZdZdZee e e e e e ddeej"deej$d eej$d eeej(d eej(d eed eedeedeedeej"deej$deee ffdZxZS)OPTModelrHcdt||t||_|j yr)r-r.rdecoderrr/rHr0s r1r.zOPTModel.__init__s&  !&)  r2c.|jjSrrCrrs r1rzOPTModel.get_input_embeddingss||(((r2c&||j_yrrFrs r1rzOPTModel.set_input_embeddingss$) !r2c|jSr)rCrs r1 get_decoderzOPTModel.get_decoder ||r2) checkpoint output_typerexpected_outputrr3rrrrrcrrr5rdrec ||n|jj}||n|jj}||n|jj}| | n|jj} |j ||| ||||||| |  } | s| St | j| j| j| jS)N rr3r5rrrrrcrrrdr$) rHrcrrr'rCrr%rr`r&) r/rr3rrrrrcrrr5rddecoder_outputss r1r<zOPTModel.forwards*2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK)r>r?r@r r.rrrIrOPT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErr9rCrrrrrrr<rDrEs@r1rArAsR y )*++?@&+$. 1515,0=A59$(,0/3&*3715, E,,-, !., ELL) , "$u'8'8"9: ,   1 12 , D>, $D>, 'tn, d^, u//0, !.,  u-- ., A, r2rAceZdZdgZfdZdZdZdZdZdZ dZ e e e  dd eej d eej"d eej"d eeej&deej&deej deedeedeedeedeej deej"deee ffdZedZxZS)OPTForCausalLMzlm_head.weightct||t||_t j |j |jd|_|jyNFrK) r-r.rArrrZrrlm_headrrDs r1r.zOPTForCausalLM.__init__"sK  f% yy!;!;V=N=NUZ[  r2cB|jjjSrrrCrrs r1rz#OPTForCausalLM.get_input_embeddings,zz!!...r2c:||jj_yrr[rs r1rz#OPTForCausalLM.set_input_embeddings/*/ 'r2c|jSrrYrs r1get_output_embeddingsz$OPTForCausalLM.get_output_embeddings2rJr2c||_yrr`)r/new_embeddingss r1set_output_embeddingsz$OPTForCausalLM.set_output_embeddings5s % r2c&||j_yrrrC)r/rCs r1 set_decoderzOPTForCausalLM.set_decoder8s$ r2c.|jjSrrfrs r1rIzOPTForCausalLM.get_decoder;szz!!!r2rLrrr3rrrlabelsrrcrrr5rdrec F||n|jj}| | n|jj} | | n|jj} |jj ||| |||||| | |  }|j |dj}d}|E|j|j}|j||fd|jji| }| s|f|ddz}||f|zS|St|||j|j|jS)a4 Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(num_hidden_layers, num_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. The two additional tensors are only required when the model is used as a decoder in a Sequence to Sequence model. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. for padding use -1. [What are position IDs?](../glossary#position-ids) cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. Returns: Example: ```python >>> from transformers import AutoTokenizer, OPTForCausalLM >>> model = OPTForCausalLM.from_pretrained("facebook/opt-350m") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious. I'm just a little bit of a weirdo." ```NrOrrrlosslogitsrr`r&)rHrcrr'rrCrYrxrwr loss_functionrrrr`r&)r/rr3rrrrjrrcrrr5rdr_rrnrmoutputs r1r<zOPTForCausalLM.forward>s[H2C1N-TXT_T_TqTq$8$D $++JjJj &1%.s.nU_j--aZ=N=N1OPns58)r1)rrureordered_past layer_pasts ` r1_reorder_cachezOPTForCausalLM._reorder_caches=) J ncmnn N r2 NNNNNNNNNNNN)r>r?r@_tied_weights_keysr.rrrardrgrIrrrSrr9rCrrrrrrr<r?rxrDrEs@r1rVrVs*+/0&%"+AP_`1515,0=A59-1$(,0/3&*3715O E,,-O !.O ELL) O "$u'8'8"9: O   1 12 O ))*O D>O $D>O 'tnO d^O u//0O !.O  u,, -O aO br2rVa The OPT Model transformer with a sequence classification head on top (linear layer). [`OPTForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ceZdZdeffd Zeeeee e e e  dde ejde ej de ej de eeej$de ej d e ejd e ed e ed e ed e ede ejdeee ffdZdZdZxZS)OPTForSequenceClassificationrHct|||j|_t||_t j |j|jd|_|jyrX) r-r. num_labelsrArrrZrscorerrDs r1r.z%OPTForSequenceClassification.__init__sT   ++f% YYv994??QVW  r2)rKrLrrM expected_lossrr3rrrrjrrcrrr5rec | | n|jj} |j|||| ||||| |  } | d} |j| }||jdd\}}n|jdd\}}|jj |dk7r t d|jj d}n|||jj k7j|jtj}tj|jd|jtj}||zjd}n.d}tj|jj d |tj||j |f}d}||jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}n|||}n|jj"d k(r=t1}||j3d|j$|j3d}n,|jj"d k(rt5}|||}| s|f| ddz}||f|zS|St7||| j8| j:| j<S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N rr3r5rrrrcrrrr+rz=Cannot handle batch sizes > 1 if no padding token is defined.rh)rrjz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationrl)rHr'rrrsrrXrwrr9int32rargmaxrTrUr0r> problem_typer~rjr;rBr squeezer rorrrr`r&)r/rr3rrrrjrrcrrr5transformer_outputsr`rnrrlast_non_pad_token non_pad_mask token_indices pooled_logitsrmloss_fctrps r1r<z$OPTForSequenceClassification.forwards8&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6#%(;AB(??F)-)9TGf$ Ev E/ /??-;;*55   r2cB|jjjSrr[rs r1rz1OPTForSequenceClassification.get_input_embeddingsYr\r2c:||jj_yrr[rs r1rz1OPTForSequenceClassification.set_input_embeddings\r^r2r>)r>r?r@r r.rrQr'_CHECKPOINT_FOR_SEQUENCE_CLASSIFICATIONrrS_SEQ_CLASS_EXPECTED_OUTPUT_SEQ_CLASS_EXPECTED_LOSSrr9rCrrrrrr<rrrDrEs@r1r|r|sh y++?@:4$2. 156:15@D59-1$(,0/3&*37\ E,,-\ !!2!23\ E--. \ "%ell(;"<= \   1 12 \ ))*\ D>\ $D>\ 'tn\ d^\ u//0\  u66 7\ A\ |/0r2r|z The OPT Model transformer with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZdeffd Zeeeee  dde e jde e jde e jde eee jde e jd e e jd e e jd e ed e ed e ede ede e jdeeeffdZdZdZxZS)OPTForQuestionAnsweringrHct||t||_t j |j d|_|jyr*) r-r.rArrrZr qa_outputsrrDs r1r.z OPTForQuestionAnswering.__init__hs@  f% ))F$>$>B r2rirr3rrrstart_positions end_positionsrrcrrr5rec | | n|jj} |j|||| |||| | |  } | d}|j|}|j dd\}}|j dj }|j dj }d}||t|jdkDr|j d}t|jdkDr|j d}|jd}|jd|j|j}|jd|j|j}t|}|||}|||}||zdz }| s||f| ddz}||f|zS|St|||| j| j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Returns: Example: ```python >>> from transformers import AutoTokenizer, OPTForQuestionAnswering >>> import torch >>> torch.manual_seed(4) # doctest: +IGNORE_RESULT >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> # note: we are loading a OPTForQuestionAnswering from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> model = OPTForQuestionAnswering.from_pretrained("facebook/opt-350m") >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" >>> inputs = tokenizer(question, text, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> answer_offset = len(tokenizer(question)[0]) >>> predict_answer_tokens = inputs.input_ids[ ... 0, answer_offset + answer_start_index : answer_offset + answer_end_index + 1 ... ] >>> predicted = tokenizer.decode(predict_answer_tokens) >>> predicted ' a nice puppet' ```Nrrrrhr7) ignore_indexr+)rm start_logits end_logitsr`r&)rHr'rrsplitrrxr+rnclamprwrr rr`r&)r/rr3rrrrrrrcrrr5rr`rnrr total_loss ignored_indexr start_lossend_lossrps r1r<zOPTForQuestionAnswering.forwardpsx&1%r?r@r r.rrQrrrSrr9rCrrrrrr<rrrDrEs@r1rr`swy++?@+GVef156:15@D596:48$(,0/3&*37j E,,-j !!2!23j E--. j "%ell(;"<= j   1 12 j "%"2"23j   0 01j D>j $D>j 'tnj d^j u//0j  u22 3j gAj X/0r2r)rVrArr|r)IrAtypingrrrrr9torch.utils.checkpointrtorch.nnrr r activationsr cache_utilsr rr generationrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrrmodeling_outputsrrrrmodeling_utilsrutilsrrrrrrconfiguration_optr !torch.nn.attention.flex_attentionr!integrations.flex_attentionr"r# get_loggerr>rTrRrSrTrrrrr&ModulerGrrrrOPT_START_DOCSTRINGrrQrrArVr|r__all__rr2r1rs// AA!;;)i .) !;JJ   H %)&+F'(;BLL;8]7299]7@[B[B|E1|E1R+ cbiicL"W?? ?.DNG #G T WC !C C Lx'xv t0#5t0t0n {00{0{0| r2