S^(hdZddlZddlmZmZmZmZmZmZddl Z ddl Z ddl m Z ddl mZmZmZddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZmZmZm Z m!Z!m"Z"dd l#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1erddlm2Z2e.jfe4Z5dZ6dZ7de8de8de jrfdZ:de8de8de jrfdZ;Gdde jxZ=Gdde jxZ>Gdde>Z?Gd d!e>Z@Gd"d#e jxZAe>e?e@d$ZBGd%d&e jxZCGd'd(e jxZDGd)d*e$ZEd+ZFd,ZGe,d-eFGd.d/eEZHe,d0eFGd1d2eEZIe,d3eFGd4d5eEZJe,d6eFGd7d8eEZKe,d9eFGd:d;eEZLe,deEZMgd?ZNy)@z PyTorch DistilBERT model adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) and in part from HuggingFace PyTorch version of Google AI Bert model (https://github.com/google-research/bert) N)DictListOptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)get_activation)PretrainedConfig)is_deepspeed_zero3_enabled)#_prepare_4d_attention_mask_for_sdpa)!flash_attn_supports_top_left_maskis_flash_attn_available)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indices"is_torch_greater_or_equal_than_2_2prune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings)DistilBertConfig)_flash_attention_forwardzdistilbert-base-uncasedr%n_posdimoutctrZddl}|jj|d5tj j dk(rt|||dddyt|||y#1swYyxYw)Nr) modifier_rankr'r(r))r deepspeedzeroGatheredParameterstorch distributedget_rank_create_sinusoidal_embeddings)r'r(r)r-s p/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/distilbert/modeling_distilbert.pycreate_sinusoidal_embeddingsr5Gsr!# ^^ . .s! . D M  ))+q0-EsL M M &EsD  M Ms 0A44A=ctjt|Dcgc];}t|Dcgc]$}|tjdd|dzz|z z &c}=c}}}d|_t j tj|dddddf|dddddf<t j tj|dddddf|dddddf<|jycc}wcc}}w)Ni'Frr$) nparrayrangepower requires_gradr0 FloatTensorsincosdetach_)r'r(r)posj position_encs r4r3r3Rs88hmnshtuadQVWZQ[\AcBHHUAaL34F$GG\uvLC$$RVVLADqD,A%BCC14a4L$$RVVLADqD,A%BCC14a4LKKM ]usC; )C6C; 6C; c|eZdZdeffd Zddej deej dej fdZxZ S) Embeddingsconfigc t|tj|j|j |j |_tj|j|j |_ tj|j d|_ tj|j|_ |jdtj|jj!ddy)N) padding_idx-q=eps position_ids)r$F) persistent)super__init__r Embedding vocab_sizer( pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormDropoutdropoutregister_bufferr0arangeexpandselfrF __class__s r4rPzEmbeddings.__init__[s !||F,=,=vzzW]WjWjk#%<<0N0NPVPZPZ#[ fjje<zz&..1   ELL)G)GHOOPWXej  input_ids input_embedsreturnc||j|}|jd}t|dr|jddd|f}nPt j |tj |j}|jdj|}|j|}||z}|j|}|j|}|S)a Parameters: input_ids (torch.Tensor): torch.tensor(bs, max_seq_length) The token ids to embed. input_embeds (*optional*, torch.Tensor): The pre-computed word embeddings. Can only be passed if the input ids are `None`. Returns: torch.tensor(bs, max_seq_length, dim) The embedded tokens (plus position embeddings, no token_type embeddings) Nr$rL)dtypedevicer) rTsizehasattrrLr0r[longrf unsqueeze expand_asrVrWrY)r^rarb seq_lengthrLrV embeddingss r4forwardzEmbeddings.forwardfs  // :L!&&q) 4 (,,Q  ^>yIL"66|D!$77 ^^J/ \\*- r`N) __name__ __module__ __qualname__rrPr0Tensorrrn __classcell__r_s@r4rErEZs>  /  Xell=S_d_k_kr`rEceZdZdeffd ZdeefdZ ddejdejdejdejd e ejd e d e ejd ffd Z xZS)MultiHeadSelfAttentionrFc*t|||_|j|_|j|_t j |j|_d|_ |j|jzdk7r&td|jd|jdt j|j|j|_ t j|j|j|_ t j|j|j|_t j|j|j|_t!|_|j|jz|_y)NpFrzself.n_heads: z must divide self.dim:  evenly in_features out_features)rOrPrFn_headsr(r rXattention_dropoutrY is_causal ValueErrorLinearq_link_linv_linout_linset pruned_headsattention_head_sizer]s r4rPzMultiHeadSelfAttention.__init__s  ~~ ::zzF$<$<=  88dll "a '~dll^;RSWS[S[R\\cde eYY6::FJJO YY6::FJJO YY6::FJJO yyVZZfjjQ &)e#'88t||#; r`headsct|dk(ryt||j|j|j\}}t |j ||_t |j||_t |j||_t |j|d|_ |jt|z |_|j|jz|_ |jj||_y)Nrr$r() lenrrrrrrrrrr(union)r^rindexs r4 prune_headsz"MultiHeadSelfAttention.prune_headss u:? 7 4<<!9!94;L;L u( E: ' E: ' E: )$,,1E ||c%j0 ++dll: --33E:r`querykeyvaluemask head_maskoutput_attentionsrc.c|j\}}|jd} jjzdd| f} dtjdtjffd } dtjdtjffd } | j |} | j |}| j|}| tjz } tj| |jdd}|dk(j| j|}|j|tjtj |j"j$}t&j(j+|d }j-|}|||z}tj||}| |}j/|}|r||fS|fS)  Parameters: query: torch.tensor(bs, seq_length, dim) key: torch.tensor(bs, seq_length, dim) value: torch.tensor(bs, seq_length, dim) mask: torch.tensor(bs, seq_length) Returns: weights: torch.tensor(bs, n_heads, seq_length, seq_length) Attention weights context: torch.tensor(bs, seq_length, dim) Contextualized layer. Optional: only if `output_attentions=True` r$xrcc`|jdjjddSseparate headsrMr$r7viewr transposerbs dim_per_headr^s r4shapez-MultiHeadSelfAttention.forward..shapes)66"b$,, =GG1M Mr`c|jddjjdjzSz group headsr$r7rMr contiguousrrrs r4unshapez/MultiHeadSelfAttention.forward..unshapes5;;q!$//166r2t||l?Z[ [r`r7r rrMr)rgr(rr0rsrrrmathsqrtmatmulrrrk masked_filltensorfinforeminr functionalsoftmaxrYr)r^rrrrrrq_lengthr(k_length mask_reshprrqkvscoresweightscontextrrs` @@r4rnzMultiHeadSelfAttention.forwards("JJLHc88A;xx4<</ !Q)  NU\\ Nell N \u|| \  \ $**U# $ $**S/ " $**U# $  ,' 'aQ!23  +55f=## %,,u{{6<<8<<= --''B'7,,w'   )G,,w*'",,w' W% %: r`NF)rprqrrrrPrintrr0rsrboolrrnrtrus@r4rwrws</<,;c;,-1"'>||>\\>|| > ll > ELL) > > u||S !>r`rwceZdZdZfdZ d dej dej dej dej deej ded e ej d ffd Z xZ S) DistilBertFlashAttention2aS DistilBert flash attention module. This module inherits from `MultiHeadSelfAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cBt||i|t|_yro)rOrPr_flash_attn_uses_top_left_mask)r^argskwargsr_s r4rPz"DistilBertFlashAttention2.__init__s# $)&) /P.Q+r`rrrrrrrc.c |j\}}jjzdtjdtjffd } | j |} | j |} | j|} jrjjnd} | jtjk(rtjrtj}nMtjdrjj }n j j"j}t$j'd|d| j)|} | j)|} | j)|} t+| | | ||| j,j.}|j1|jz}j3|}|r||fS|fS) rrrcc@|jdjS)rrM)rrr batch_sizerr^s r4reshapez2DistilBertFlashAttention2.forward..reshapes66*b$,, E Er`_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)rYuse_top_left_maskr)rgr(rr0rsrrrtrainingrFrrefloat32is_autocast_enabledget_autocast_gpu_dtyperhrweightlogger warning_oncetor&rrrr)r^rrrrrrrr(r query_states key_states value_states attn_dropout target_dtype attn_weightsattn_weights_reshaped attn_outputrrs` @@r4rnz!DistilBertFlashAttention2.forwards(%*JJL! Hcxx4<</  Fu|| F  F tzz%01 TZZ_- tzz%01 8< t{{443     .((*$;;= &?@#{{BB #zz0066    >$  (??<8L#|4J'??<8L/      "AAnn  !- 4 4Z4< !r`r) rprqrr__doc__rPr0rsrrrrnrtrus@r4rrs R-1"'N"||N"\\N"|| N" ll N" ELL) N" N" u||S !N"r`rceZdZdeffd Z d dej dej dej dej deej ded e ej d fffd Z xZ S) DistilBertSdpaAttentionrFc`t|||j|_t |_y)N)rF)rOrPr dropout_probrrequire_contiguous_qkvr]s r4rPz DistilBertSdpaAttention.__init__Qs, '"44*L&L#r`rrrrrrrc.c\|s|)tjdt ||||||S|j \}}j j zdtjdtjffd }dtjdtjffd } |j|} |j|} |j|} jrK| jjdk(r2|0| j} | j} | j} tj j"j%| | | |j&r j(ndd } | | } j+| } | fS) raDistilBertSdpaAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rrcc`|jdjjddSrrrs r4rz.DistilBertSdpaAttention.forward..shape}s+66*b$,, EOOPQSTU Ur`c|jddjjdjzSrrrs r4rz0DistilBertSdpaAttention.forward..unshapes6;;q!$//166z2t||VbGbc cr`cudarF) attn_mask dropout_pr)rrrOrnrgr(rr0rsrrrrrftyperr rscaled_dot_product_attentionrrr)r^rrrrrr_rrrrrrrrr_s` @@r4rnzDistilBertSdpaAttention.forwardVs(  5   d  7?!  !::< Aqxx4<</  VU\\ Vell V du|| d  d $**U# $ $**S/ " $**U# $  & &188==F+BtGW A A Ahh))FF +/==d''c G k* ll;/ ~r`r rprqrrrrPr0rsrrrrnrtrus@r4rrPsM/M-1"'G||G\\G|| G ll G ELL) G G u||S !GGr`rceZdZdeffd Zdej dej fdZdej dej fdZxZ S)FFNrFct|tj|j|_|j |_d|_tj|j|j|_ tj|j|j|_ t|j|_ y)Nryr$r|)rOrPr rXrYchunk_size_feed_forward seq_len_dimrr( hidden_dimlin1lin2r activationr]s r4rPz FFN.__init__s zzFNN3 '-'E'E$II&**6CTCTU II&*;*;&**U ():):;r`inputrccZt|j|j|j|Sro)rff_chunkrr)r^rs r4rnz FFN.forwards%(8T8TVZVfVfhmnnr`c|j|}|j|}|j|}|j|}|Sro)rrrrY)r^rrs r4rz FFN.ff_chunks= IIe  OOA  IIaL LLOr`) rprqrrrrPr0rsrnrrtrus@r4rrsI</<oU\\oelloellu||r`r)eagerflash_attention_2sdpac eZdZdeffd Z d dej deej deej dede ej dff d Z xZ S) TransformerBlockrFct||j|jzdk7r&t d|jd|jdt |j ||_tj|jd|_ t||_ tj|jd|_ y)Nrzconfig.n_heads z must divide config.dim r{rI)normalized_shaperK)rOrPr(rrDISTILBERT_ATTENTION_CLASSES_attn_implementation attentionr rW sa_layer_normrffnoutput_layer_normr]s r4rPzTransformerBlock.__init__s  :: &! +v~~.>>VW]WaWaVbbijk k5f6Q6QRSYZ\\6::5Qv;!#vzzu!Ur`rrrrrc.c(|j||||||}|r|\}}n.t|turtdt|d|d}|j ||z}|j |}|j ||z}|f}|rf|z}|S)ae Parameters: x: torch.tensor(bs, seq_length, dim) attn_mask: torch.tensor(bs, seq_length) Returns: sa_weights: torch.tensor(bs, n_heads, seq_length, seq_length) The attention weights ffn_output: torch.tensor(bs, seq_length, dim) The output of the transformer block contextualization. )rrrrrrz$sa_output must be a tuple but it is z typer)rrtuple TypeErrorrrr) r^rrrr sa_output sa_weights ffn_outputoutputs r4rnzTransformerBlock.forwards"NN/ #  $- !IzIe+"FtIFWW\ ]^^!! I&&y1}5 XXi( #'#9#9*y:P#Q   ]V+F r`)NNFrrus@r4rrsv V/ V -1,0"' ) <<)ELL))ELL) )  ) u||S ! )r`rceZdZdeffd Z d dej deej deej dededeed e e e ej d fffd Z xZ S) TransformerrFct||j|_tjt |jDcgc] }t |c}|_d|_ycc}wr) rOrPn_layersr ModuleListr:rlayergradient_checkpointing)r^rFrr_s r4rPzTransformer.__init__sQ  ]]eFOOF\#]$4V$<#]^ &+#$^sA-rrrroutput_hidden_states return_dictrc.c|rdnd}|rdnd}|} t|jD]\} } |r|| fz}|jr/|jr#|j | j | ||| |} n| | ||| |} | d} |r1t | dk7rtdt | | d} || fz}t | dk7stdt | |r|| fz}|std | ||fDSt| || S) a Parameters: x: torch.tensor(bs, seq_length, dim) Input sequence embedded. attn_mask: torch.tensor(bs, seq_length) Attention mask on the sequence. Returns: hidden_state: torch.tensor(bs, seq_length, dim) Sequence of hidden states in the last (top) layer all_hidden_states: Tuple[torch.tensor(bs, seq_length, dim)] Tuple of length n_layers with the hidden states from each layer. Optional: only if output_hidden_states=True all_attentions: Tuple[torch.tensor(bs, n_heads, seq_length, seq_length)] Tuple of length n_layers with the attention weights from each layer Optional: only if output_attentions=True NrMr7z7The length of the layer_outputs should be 2, but it is rr$z7The length of the layer_outputs should be 1, but it is c3&K|] }|| ywror).0rs r4 z&Transformer.forward..=sgqYZYfgs)last_hidden_state hidden_states attentions) enumeraterrr_gradient_checkpointing_func__call__rrr r)r^rrrrrrall_hidden_statesall_attentions hidden_statei layer_module layer_outputsrs r4rnzTransformer.forwardsX.#7BD0d (4 uOA|#$5$G!**t}} $ A A )) aL% ! !- aL% ! ),L }%*$'^_bcp_q^r%stt*1- !/:-!?}%*$'^_bcp_q^r%stt= uB 1\O C g\3Dn$Ugg g*:KXf  r`)NNFFN)rprqrrrrPr0rsrrrrrrnrtrus@r4rrs,/,-1,0"'%*&*C <<C ELL)C ELL) C  C # C d^C  ellC&7 88 9C r`rcHeZdZdZeZdZdZdZdZ dZ de jfdZ y)DistilBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. N distilbertTmodulect|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyt|tra|j jrJt!|j j"|j j$|j&jyyy)zInitialize the weights.r)meanstdNg?) isinstancer rrdatanormal_rFinitializer_rangebiaszero_rQrHrWfill_rEsinusoidal_pos_embdsr5rUr(rV)r^r,s r4 _init_weightsz'DistilBertPreTrainedModel._init_weightsQsM fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S )  + 0P0P ( 33T[[__fF`F`FgFg 1Q +r`)rprqrrrr% config_classload_tf_weightsbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_2_supports_sdpar Moduler8rr`r4r*r*Ds: $LO$&*#!NBIIr`r*aD This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`DistilBertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zfThe bare DistilBERT encoder/transformer outputting raw hidden-states without any specific head on top.ceZdZdeffd Zdej fdZdefdZ dej fdZ dej fd Z d e ee e effd Zeej#d eeee  ddeej0deej0deej0deej0deedeedeedeeeej0dfffdZxZS)DistilBertModelrFct||t||_t ||_|j dk(|_|j dk(|_|jy)Nrr) rOrPrErmr transformerr_use_flash_attention_2 _use_sdpa post_initr]s r4rPzDistilBertModel.__init__s[  $V,&v.&,&A&AEX&X#44> r`rcc.|jjSz1 Returns the position embeddings )rmrVr^s r4get_position_embeddingsz'DistilBertModel.get_position_embeddingss222r`new_num_position_embeddingsc||jjz }|dk(rytjd|d||j_|jj j j}tj|jj|jj|j_|jjrKt|jj|jj|j j ntj5|dkDr8tj||jj j d| n1tj|d||jj _ddd|jj j!|j"y#1swY9xYw) Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`. Arguments: new_num_position_embeddings (`int`): The number of new position embedding matrix. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. rNz(Setting `config.max_position_embeddings=z`...r,)rFrUrinformrVrcloner rQr(r7r5r0no_grad Parameterrrf)r^rKnum_position_embeds_diffold_position_embeddings_weights r4resize_position_embeddingsz*DistilBertModel.resize_position_embeddingssu$?AdAd#d  $q (  >?Z>[[_`a.I +)-)L)L)S)S)Y)Y)[&.0ll4;;;^;^`d`k`k`o`o.p+ ;; + + (kk99t{{TXTlTlTsTs  +a/]_]i]i6^DOO77>>?YAY@YZBD67P8PQBDOO77>   ++..t{{;  s :A/G!!G*c.|jjSrormrTrIs r4get_input_embeddingsz$DistilBertModel.get_input_embeddingss...r`new_embeddingsc&||j_yrorVr^rXs r4set_input_embeddingsz$DistilBertModel.set_input_embeddingss*8'r`heads_to_prunec|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrCrrr)r^r\rrs r4 _prune_headszDistilBertModel._prune_headssG +002 GLE5    " "5 ) 3 3 ? ? F Gr`batch_size, num_choices checkpoint output_typer9raattention_maskr inputs_embedsrrr.c||n|jj}||n|jj}||n|jj}| | t d|#|j |||j }n!||j dd}n t d| |jn |j} |du} |j||jj}|j||} |jr |d|vr|nd}nD|tj|| }|jr| r|st|| j |d}|j#| |||||S) NzDYou cannot specify both input_ids and inputs_embeds at the same timerMz5You have to specify either input_ids or inputs_embedsr)rfr$)tgt_len)rrrrrr)rFrruse_return_dictr%warn_if_padding_and_no_attention_maskrgrf get_head_masknum_hidden_layersrmrDr0onesrErrerC) r^rardrrerrr input_shaperfhead_mask_is_nonerms r4rnzDistilBertModel.forwards 2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U%.%:!! @T@T%-&&y$++2O2OP __Y >  & &0>0JqTbOb^imN%!&K!G~~"3 4 'tn4 d^4  ellC&7 88 94 j 4 r`rAz?DistilBert Model with a `masked language modeling` head on top.ceZdZdgZdeffd ZdejfdZde fdZ dejfdZ d ejfd Z eej!d eeee  dd eej.deej.deej.deej.deej0deedeedeedeeeej.dfffdZxZS)DistilBertForMaskedLMzvocab_projector.weightrFct||t|j|_t ||_t j|j|j|_ t j|jd|_ t j|j|j|_ |jt j|_y)NrIrJ)rOrPrrrAr+r rr(vocab_transformrWvocab_layer_normrRvocab_projectorrFr mlm_loss_fctr]s r4rPzDistilBertForMaskedLM.__init__-s  ():):;)&1!yyVZZ@ " VZZU C!yyV5F5FG //1r`rcc6|jjSrHr+rJrIs r4rJz-DistilBertForMaskedLM.get_position_embeddings<6688r`rKc:|jj|yrMNr+rTr^rKs r4rTz0DistilBertForMaskedLM.resize_position_embeddingsB 223NOr`c|jSrorxrIs r4get_output_embeddingsz+DistilBertForMaskedLM.get_output_embeddingsPs###r`rXc||_yrorrZs r4set_output_embeddingsz+DistilBertForMaskedLM.set_output_embeddingsSs -r`r`rarardrrelabelsrrr.c ||n|jj}|j|||||||} | d} |j| } |j | } |j | } |j | } d} |@|j| jd| jd|jd} |s| f| ddz} | | f| zS| St| | | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. NrardrrerrrrrMr$losslogitsrr) rFrhr+rvrrwrxryrrgrrr)r^rardrrerrrr dlbrt_outputrprediction_logitsmlm_lossrs r4rnzDistilBertForMaskedLM.forwardVs*.&1% 112CD 001BC  (():)?)?DUDZDZ[]D^)_agalalmoapqH')L,</ 'tn/ d^/  ~uU\\3%677 8/ j / r`rtz DistilBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZdeffd Zdej fdZdefdZ e e jde eee dd eej&d eej&d eej&d eej&d eej(deedeedeedeeeej&dfffdZxZS)#DistilBertForSequenceClassificationrFct|||j|_||_t ||_t j|j|j|_ t j|j|j|_ t j|j|_ |jyro)rOrP num_labelsrFrAr+r rr(pre_classifier classifierrXseq_classif_dropoutrYrFr]s r4rPz,DistilBertForSequenceClassification.__init__s   ++ )&1 ii FJJ?))FJJ0A0ABzz&"<"<=  r`rcc6|jjSrHr{rIs r4rJz;DistilBertForSequenceClassification.get_position_embeddingsr|r`rKc:|jj|yr~rrs r4rTz>DistilBertForSequenceClassification.resize_position_embeddingsrr`zbatch_size, sequence_lengthrarardrrerrrr.c ||n|jj}|j|||||||} | d} | dddf} |j| } t j | } |j | } |j| } d} ||jj|jdk(rd|j_nl|jdkDrL|jtjk(s|jtjk(rd|j_nd|j_|jjdk(rIt}|jdk(r&|| j|j} n|| |} n|jjdk(r=t!}|| j#d|j|j#d} n,|jjdk(rt%}|| |} |s| f| ddz}| | f|zS|St'| | | j(| j* S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrr$ regressionsingle_label_classificationmulti_label_classificationrMr)rFrhr+rr ReLUrYr problem_typerrer0rirr squeezer rr rrr)r^rardrrerrrrdistilbert_outputr% pooled_outputrrloss_fctrs r4rnz+DistilBertForSequenceClassification.forwards.&1%C 'tnC d^C  'u||S/@)AA BC n C r`rz DistilBert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZdeffd Zdej fdZdefdZ e e jde eee dd eej&d eej&d eej&d eej&d eej&deej&deedeedeedeeeej&dfffdZxZS)DistilBertForQuestionAnsweringrFcXt||t||_t j |j |j|_|jdk7rtd|jt j|j|_ |jy)Nr7z)config.num_labels should be 2, but it is )rOrPrAr+r rr(r qa_outputsrrX qa_dropoutrYrFr]s r4rPz'DistilBertForQuestionAnswering.__init__ s  )&1))FJJ0A0AB    !HIZIZH[\] ]zz&"3"34  r`rcc6|jjSrHr{rIs r4rJz6DistilBertForQuestionAnswering.get_position_embeddingsr|r`rKc:|jj|yr~rrs r4rTz9DistilBertForQuestionAnswering.resize_position_embeddingsrr`r`rarardrrestart_positions end_positionsrrr.c Z| | n|jj} |j||||||| } | d} |j| } |j | } | j dd\} }| j dj} |j dj}d}||t|jdkDr|j d}t|jdkDr|j d}| jd}|jd|}|jd|}tj|}|| |}|||}||zdz }| s| |f| ddz}||f|zS|St|| || j| j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrr$rMr) ignore_indexr7)r start_logits end_logitsrr)rFrhr+rYrsplitrrrrgclampr r rrr)r^rardrrerrrrrrrrrr total_loss ignored_indexr start_lossend_lossrs r4rnz&DistilBertForQuestionAnswering.forward,s8&1%D 'tnD d^D  +U5<<3D-EE FD j D r`rz DistilBert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. c|eZdZdeffd Zdej fdZdefdZ e e e e ee ddeej$d eej$d eej$d eej$d eej&d eedeedeedeeeej$dfffdZxZS) DistilBertForTokenClassificationrFc,t|||j|_t||_t j |j|_t j|j|j|_ |jyro) rOrPrrAr+r rXrYr hidden_sizerrFr]s r4rPz)DistilBertForTokenClassification.__init__sg   ++)&1zz&..1 ))F$6$68I8IJ r`rcc6|jjSrHr{rIs r4rJz8DistilBertForTokenClassification.get_position_embeddingsr|r`rKc:|jj|yr~rrs r4rTz;DistilBertForTokenClassification.resize_position_embeddingsrr`rarardrrerrrr.c ||n|jj}|j|||||||} | d} |j| } |j | } d} |. 'tn. d^.  $eELL#,=&>> ?. H . r`rz DistilBert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. ceZdZdeffd Zdej fdZdefdZ e e jde ee dd eej$d eej$d eej$d eej$d eej&deedeedeedeeeej$dfffdZxZS)DistilBertForMultipleChoicerFcTt||t||_t j |j |j |_t j |j d|_t j|j|_ |jy)Nr$) rOrPrAr+r rr(rrrXrrYrFr]s r4rPz$DistilBertForMultipleChoice.__init__so  )&1 ii FJJ?))FJJ2zz&"<"<=  r`rcc6|jjSrHr{rIs r4rJz3DistilBertForMultipleChoice.get_position_embeddingsr|r`rKc:|jj|y)a Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`. Arguments: new_num_position_embeddings (`int`) The number of new position embeddings. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. Nrrs r4rTz6DistilBertForMultipleChoice.resize_position_embeddingsrr`z(batch_size, num_choices, sequence_length)rcr9rardrrerrrr.c ||n|jj}||jdn|jd} |!|jd|j dnd}|!|jd|j dnd}|1|jd|j d|j dnd}|j |||||||} | d} | dddf} |j | } tj| } |j| } |j| } | jd| }d}|t}|||}|s|f| ddz}||f|zS|St||| j| jS)aW labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Returns: Examples: ```python >>> from transformers import AutoTokenizer, DistilBertForMultipleChoice >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased") >>> model = DistilBertForMultipleChoice.from_pretrained("distilbert-base-cased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> choice0 = "It is eaten with a fork and a knife." >>> choice1 = "It is eaten while held in the hand." >>> labels = torch.tensor(0).unsqueeze(0) # choice0 is correct (according to Wikipedia ;)), batch size 1 >>> encoding = tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors="pt", padding=True) >>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels) # batch size is 1 >>> # the linear classifier still needs to be trained >>> loss = outputs.loss >>> logits = outputs.logits ```Nr$rMrrr)rFrhrrrgr+rr rrYrr rrr)r^rardrrerrrr num_choicesrr%rrreshaped_logitsrrrs r4rnz#DistilBertForMultipleChoice.forwardsX&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImq(   r=#5#5b#9=;M;Mb;Q R  // )'/!5#" qz $QT* ++M: ! -0  ]3 / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   r`r)rprqrrrrPr rQrJrrTr!rorpr#rrrrr0rsrrrrrnrtrus@r4rrs; / 99 Pc P+#**+UV+DSbc-115,004-1,0/3&*T ELL)T !.T ELL) T   - T ))* T $D>T 'tnT d^T  (% c0A*BB CT dT r`r)rtrrrrrAr*)Orrtypingrrrrrrnumpyr8r0r torch.nnr r r activationsrconfiguration_utilsrintegrations.deepspeedrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrrmodeling_outputsrrrrrrmodeling_utilsr pytorch_utilsrrrrutilsrr r!r"r#configuration_distilbertr%r& get_loggerrprrqrrrrsr5r3r?rErwrrrrrrr*DISTILBERT_START_DOCSTRINGrorArtrrrr__all__rr`r4rs   :: AA)3@Kh. 7J   H %/$ EE#EELLE3U\\++\eRYYeP]" 6]"@M4M`")),$2 # 7ryy7tJ "))J \D "JlD /D D NIa 5a a H k *Ck k \ l %>l l ^ T '@T T n x ";x x v r`