S^(h dZddlmZmZmZddlZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZmZmZmZmZdd lmZdd lmZmZmZmZd d lmZej:eZd Z dZ!dZ"dZ#dZ$dZ%dZ&dZ'dZ(dZ)GddejTZ+GddejTZ,ejZj\dZ/ejZj\dZ0ejZj\dZ1ejZj\dZ2ejZj\dejfd e4fd!Z5ejZj\dejfd"ejffd#Z6ejZj\dejfd"ejfd$e4fd%Z7ejZj\dejfd"ejffd&Z8Gd'd(ejTZ9Gd)d*ejTZ:Gd+d,ejTZ;Gd-d.ejTZ<Gd/d0ejTZ=Gd1d2ejTZ>Gd3d4ejTZ?Gd5d6eZ@d7ZAd8ZBed9eAGd:d;e@ZCGd<d=ejTZDGd>d?ejTZEGd@dAejTZFGdBdCejTZGGdDdEejTZHedFeAGdGdHe@ZIGdIdJejTZJedKeAGdLdMe@ZKedNeAGdOdPe@ZLedQeAGdRdSe@ZMgdTZNy)UzPyTorch DeBERTa model.)OptionalTupleUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging) DebertaConfigrzmicrosoft/deberta-basez!lsanochkin/deberta-large-feedbackz' Paris'z0.54z#Palak/microsoft_deberta-large_squadz' a nice puppet'gQ? c*eZdZdZdfd ZdZxZS)DebertaLayerNormzBLayerNorm module in the TF style (epsilon inside the square root).ct|tjt j ||_tjt j||_||_ yN) super__init__r Parametertorchonesweightzerosbiasvariance_epsilon)selfsizeeps __class__s j/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/deberta/modeling_deberta.pyrzDebertaLayerNorm.__init__9sH ll5::d#34 LLT!23 #cX|j}|j}|jdd}||z jdjdd}||z t j ||j zz }|j|}|j|z|jz}|S)NT)keepdim) dtypefloatmeanpowr!sqrtr&tor#r%)r' hidden_states input_typer3varianceys r+forwardzDebertaLayerNorm.forward?s"(( %++- !!"d!3!D(--a055b$5G&-HtG\G\<\1]] %((4 KK- '$)) 3r,)g-q=__name__ __module__ __qualname____doc__rr; __classcell__r*s@r+rr6sL$ r,rc$eZdZfdZdZxZS)DebertaSelfOutputct|tj|j|j|_t |j|j|_tj|j|_ yr) rrrLinear hidden_sizedenserlayer_norm_eps LayerNormDropouthidden_dropout_probdropoutr'configr*s r+rzDebertaSelfOutput.__init__Ks\ YYv1163E3EF )&*<*S>STzz&"<"<= r,cr|j|}|j|}|j||z}|SrrHrMrJr'r7 input_tensors r+r;zDebertaSelfOutput.forwardQ7 =1  ]3 }|'CD r,r=r>r?rr;rArBs@r+rDrDJs > r,rDc|jd}|jd}tj|tj|j}tj|tj|j}|dddf|j ddj |dz }|d|ddf}|jd}|S)a Build relative position according to the query and key We assume the absolute position of query \(P_q\) is range from (0, query_size) and the absolute position of key \(P_k\) is range from (0, key_size), The relative positions from query to key is \(R_{q \rightarrow k} = P_q - P_k\) Args: query_size (int): the length of query key_size (int): the length of key Return: `torch.LongTensor`: A tensor with shape [1, query_size, key_size] r1deviceNrr.r)r(r!arangelongrYviewrepeat unsqueeze) query_layer key_layer query_sizekey_sizeq_idsk_ids rel_pos_idss r+build_relative_positionrfXs$!!"%J~~b!H LL5::k>P>P QE LLI>;++A. 0@0@0C[EUEUVWEXZfZkZklnZop qqr,c|j|jd|jd|jd|jdgS)NrrrWrh)rjr_r`s r+p2c_dynamic_expandrnzsG >>;++A. 0@0@0CY^^TVEWYbYgYghjYkl mmr,c|j|jdd|jd|jdfzS)Nr0rWrh) pos_indexp2c_attr`s r+pos_dynamic_expandrrs=   GLLN2A.)..2DinnUWFX1YY ZZr,r_ scale_factorctjtj|jdtj|zS)Nr.r1)r!r5tensorr(r2)r_rss r+scaled_size_sqrtrws0 ::ell;#3#3B#7u{{KlZ [[r,r`cd|jd|jdk7r t||S|SNrW)r(rf)r_r`rks r+ build_rposrzs1y~~b11&{I>>r,max_relative_positionsc tjtt|j d|j d|Sry)r!rvminmaxr()r_r`r{s r+compute_attention_spanrs4 <<C 0 0 4innR6HIKab ccr,c |jd|jdk7rA|dddddddfjd}tj|dt |||S|S)NrWrr.r0dimindex)r(r^r!gatherrr)rqr_r`rkrps r+uneven_size_correctedrs_y~~b11 Aq!,66r: ||G2DYPWYb2cddr,cpeZdZdZfdZdZ ddejdejdede ejde ejd e ejd e eje ejffd Z d ejd ejdejd ejde f dZ xZS)DisentangledSelfAttentiona Disentangled self-attention module Parameters: config (`str`): A model config class instance with the configuration to build a new model. The schema is similar to *BertConfig*, for more details, please refer [`DebertaConfig`] cZt||j|jzdk7r&t d|jd|jd|j|_t |j|jz |_|j|j z|_tj|j|jdzd|_ tjtj|jtj|_tjtj|jtj|_|j" |j"ng|_t%|d d|_t%|d d|_|j(rctj|j|jd|_tj|j|jd|_nd|_d|_|j&rt%|d d |_|j.d kr|j0|_tj2|j4|_d|j"vr1tj|j|jd|_d|j"vr/tj|j|j|_tj2|j<|_y)NrzThe hidden size (z6) is not a multiple of the number of attention heads ()r Fr%rurelative_attention talking_headr{r.rc2pp2c) rrrGnum_attention_heads ValueErrorintattention_head_size all_head_sizerrFin_projr r!r$r2q_biasv_bias pos_att_typegetattrrrhead_logits_projhead_weights_projr{max_position_embeddingsrKrL pos_dropoutpos_proj pos_q_projattention_probs_dropout_probrMrNs r+rz"DisentangledSelfAttention.__init__sm     : : :a ?#F$6$6#78 445Q8 $*#=#= #&v'9'9FP>PW\ ] )))"$))F,>,>@R@R"Szz&"E"EF r,c|jdd|jdfz}|j|}|jddddS)Nr.rr0rr )r(rr\permute)r'x new_x_shapes r+transpose_for_scoresz.DisentangledSelfAttention.transpose_for_scoressFffhsmt'?'?&DD FF; yyAq!$$r,r7attention_maskoutput_attentions query_statesrkrel_embeddingsreturnc L|9|j|}|j|jdd\}} } n|jjj|jdzd} t dD cgc]C} t jt |jD cgc] } | | dz| zc} dE}} } t j|d|jj|dj}t j|d|jj|dj} t j|d|jj|dj}|| |fDcgc]}|j|c}\}} } ||j|jddddfz}| |j|jddddfz} d}dt|jz}t!||}||j|jz }t j|| j#dd }|j$r*|(|&|j'|}|j)|| |||}|||z}|j*5|j+|j-ddddj-dddd}|j/}|j1|t j2|jj4}t6j8j;|d}|j=|}|j>5|j?|j-ddddj-dddd}t j|| }|j-ddddjA}|jCdd d z}|jE|}|s|dfS||fScc} wcc} } wcc}w) a Call the module Args: hidden_states (`torch.FloatTensor`): Input states to the module usually the output from previous layer, it will be the Q,K and V in *Attention(Q,K,V)* attention_mask (`torch.BoolTensor`): An attention mask matrix of shape [*B*, *N*, *N*] where *B* is the batch size, *N* is the maximum sequence length in which element [i,j] = *1* means the *i* th token in the input can attend to the *j* th token. output_attentions (`bool`, *optional*): Whether return the attention matrix. query_states (`torch.FloatTensor`, *optional*): The *Q* state in *Attention(Q,K,V)*. relative_pos (`torch.LongTensor`): The relative position encoding between the tokens in the sequence. It's of shape [*B*, *N*, *N*] with values ranging in [*-max_relative_positions*, *max_relative_positions*]. rel_embeddings (`torch.FloatTensor`): The embedding of relative distances. It's a tensor of shape [\(2 \times \text{max_relative_positions}\), *hidden_size*]. Nr r.rrrurr0rW)r.)#rrchunkr#rranger!catmatmultr6r1rrlenrrw transposerrdisentangled_att_biasrrbool masked_fillfinfor}r functionalsoftmaxrMr contiguousr(r\)r'r7rrrrkrqpr_r` value_layerwskiqkvwqvrrel_attrsscaleattention_scoresattention_probs context_layernew_context_layer_shapes r+r;z!DisentangledSelfAttention.forwardsL  m,B262K2KB2O2U2UVW]_2U2` /KK$$**4+C+Ca+GQ*OBhmnohpqcdEIIeDX$YY !D$=$=dkk$PTVW->X$YY 3t0011  l;!EHH;3D3DH$EE  << Y5H5HR5PQ  " "~'AlF^!--n=N00iWegstG  /'9   ,#445E5M5MaQRTUWX5YZbbcdfgijlmn ',,.+77.8I5;;WbWhWhKiKmKmn--//0@b/I,,7  ! ! -"44_5L5LQPQSTVW5XYaabcefhiklmO _kB %--aAq9DDF "/"4"4"6s";e"C%**+BC  !4( (//UWq3ds>+P)P; PP!Pr_r`rsc $|t|||j}|jdk(r!|jdjd}nT|jdk(r|jd}n/|jdk7rt d|jt |||j }|j}||j |z |j |zddfjd}d}d|jvr|j|}|j|}tj||jdd } tj||zd|dzdz } tj| dt!| || } || z }d |jvr|j#|} |j| } | t%| |z} t'|||} tj| |zd|dzdz } tj|| jdd j)|j* }tj|dt-| || jdd }t/||||}||z }|S) Nr0rr rz2Relative position ids must be of dim 2 or 3 or 4. rr.rWrrru)rfrYrr^rrr{r[rrrr!rrclamprrlrrwrzr6r1rnr)r'r_r`rkrrsatt_spanscore pos_key_layerc2p_attrjpos_query_layerr_posp2c_posrqs r+rz/DisentangledSelfAttention.disentangled_att_bias2s  2; ;K]K]^L     "'11!4>>qAL    1 $'11!4L    1 $QR^RbRbRdQefg g)+y$B]B]^#((* '  ' '( 2T5P5PS[5[ []^ ^ )A,  D%% % MM.9M 55mDMll; 0G0GB0OPGkk,"91hlQ>NOGll7:LWVaco:pqG W E D%% %"oon=O"77HO /N NOE kk5&8"3Q1 q8HIGll9o.G.GB.O.R.RYbYhYh.R.ijGllR'9';PY'ZiB ,G[)\ZG W E r,FNNN)r=r>r?r@rrr!Tensorrrrr;rrrArBs@r+rrs$GL%#(/3/315U0||U0 U0 U0 u||, U0 u||, U0!.U0 u||Xell33 4U0n6\\6<<6ll 6  6  6r,rc*eZdZdZfdZddZxZS)DebertaEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|t|dd}t|d|j|_t j |j|j||_t|dd|_ |jsd|_ n/t j |j|j|_ |jdkDr0t j |j|j|_ nd|_ |j|jk7r2t j|j|jd|_nd|_t!|j|j"|_t j&|j(|_||_|j/d t1j2|jj5d d y) N pad_token_idrembedding_size) padding_idxposition_biased_inputTFr position_ids)rr.) persistent)rrrrGrr Embedding vocab_sizeword_embeddingsrposition_embeddingsrtype_vocab_sizetoken_type_embeddingsrF embed_projrrIrJrKrLrMrOregister_bufferr!rZri)r'rOrr*s r+rzDebertaEmbeddings.__init__nsy v~q9 %f.>@R@RS!||F,=,=t?R?R`lm%,V5Ld%S"))'+D $')||F4R4RTXTgTg'hD $  ! !A %)+f6L6LdNaNa)bD &)-D &   &"4"4 4 ii(;(;V=O=OV[\DO"DO)&*<*S>STzz&"<"<=    ELL)G)GHOOPWXej  r,c||j}n|jdd}|d}||jddd|f}|:tj|tj|jj }||j |}|j |j|j }ntj|}|} |jr| |z} |j|j|} | | z} |j|j| } |j| } ||j| jk7rD|jdk(r |jdjd}|jd}|j!| j"}| |z} |j%| } | S)Nr.rrXrr0)r(rr!r$r[rYrr zeros_likerrrrJrsqueezer^r6r1rM) r' input_idstoken_type_idsrmask inputs_embeds input_shape seq_lengthr embeddingsrs r+r;zDebertaEmbeddings.forwards  #..*K',,.s3K ^  ,,Q  ^$>~$N !#&;;J ?? &4J^^J/  xxzZ^^--88:?<<?2215D~~a(77:++,D#d*J\\*- r,)NNNNNr<rBs@r+rrksQ >,r,rcpeZdZfdZ ddedeejeejffdZ xZ S)DebertaAttentioncpt|t||_t ||_||_yr)rrrr'rDoutputrOrNs r+rzDebertaAttention.__init__s- -f5 '/  r,rrcv|j||||||\}}||}|j||} |r| |fS| dfS)N)rrkr)r'r) r'r7rrrrkr self_output att_matrixattention_outputs r+r;zDebertaAttention.forwardse#'))   %%) #,#  Z  (L;;{LA $j1 1$d+ +r,r r=r>r?rrrr!rrr;rArBs@r+rrsF#(, , u||Xell33 4,r,rcVeZdZfdZdej dej fdZxZS)DebertaIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) rrrrFrGintermediate_sizerH isinstance hidden_actstrr intermediate_act_fnrNs r+rzDebertaIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r,r7rcJ|j|}|j|}|Sr)rHrr'r7s r+r;zDebertaIntermediate.forwards& =1 00? r,r=r>r?rr!rr;rArBs@r+rrs#9U\\ellr,rc$eZdZfdZdZxZS) DebertaOutputc t|tj|j|j |_t|j |j|_ tj|j|_ ||_ yr)rrrrFrrGrHrrIrJrKrLrMrOrNs r+rzDebertaOutput.__init__sc YYv779K9KL )&*<*S>STzz&"<"<=  r,cr|j|}|j|}|j||z}|SrrQrRs r+r;zDebertaOutput.forwardrTr,rUrBs@r+rrs r,rcpeZdZfdZ ddedeejeejffdZ xZ S) DebertaLayerct|t||_t ||_t ||_yr)rrr attentionr intermediaterrrNs r+rzDebertaLayer.__init__s3 )&1/7#F+ r,rrc|j||||||\}}|j|} |j| |} |r| |fS| dfS)Nrrrkr)r r r) r'r7rrrkrrrrintermediate_output layer_outputs r+r;zDebertaLayer.forwardsn(,~~  /%%) (6( $*#//0@A{{#68HI  *- - $' 'r,)NNNFrrBs@r+rrsF,"'( ( u||Xell33 4(r,rc eZdZdZfdZdZdZd dZ d dejdejde d e d e f d Z xZ S)DebertaEncoderz8Modified BertEncoder with relative position bias supportct|tjt |j Dcgc] }t |c}|_t|dd|_ |jrdt|dd|_ |jdkr|j|_ tj|jdz|j|_d|_ycc}w)NrFr{r.rr0)rrr ModuleListrnum_hidden_layersrlayerrrr{rrrGrgradient_checkpointing)r'rO_r*s r+rzDebertaEncoder.__init__!s ]]%H`H`Ba#bQL$8#bc ")&2F"N  " "*1&:RTV*WD '**Q..4.L.L+"$,,t/J/JQ/NPVPbPb"cD &+#$csCcR|jr|jj}|Sd}|Sr)rrr#)r'rs r+get_rel_embeddingz DebertaEncoder.get_rel_embedding,s07;7N7N,,33UYr,c|jdkrE|jdjd}||jdjdz}|S|jdk(r|jd}|S)Nr0rrWr.r )rr^r)r'rextended_attention_masks r+get_attention_maskz!DebertaEncoder.get_attention_mask0s    1 $&4&>&>q&A&K&KA&N #47N7V7VWY7Z7d7deg7hhN   !Q &+55a8Nr,cZ|jr||t||}|St||}|Sr)rrf)r'r7rrks r+ get_rel_poszDebertaEncoder.get_rel_pos9s>  " "|';'6|]S  7}mT r,r7routput_hidden_statesr return_dictc |j|}|j|||}|r|fnd}|rdnd} |} |j} t|jD]k\} } |j r1|j r%|j| j| |||| |\}}n| | |||| |\}}|r||fz}||}n|} |sf| |fz} m|std||| fDSt||| S)N)rrkrrc3&K|] }|| ywrr").0rs r+ z)DebertaEncoder.forward..ushqZ[Zghslast_hidden_stater7 attentions) rrr enumeraterrtraining_gradient_checkpointing_func__call__tupler )r'r7rrrrrkr all_hidden_statesall_attentionsnext_kvrr layer_moduleatt_ms r+r;zDebertaEncoder.forwardAs700@'' |\R OcM;Kim0d//1(4 ;OA|**t}}'+'H'H ))"  "%($ u(4"!-!-#1&7 ($ u$$58H$H!', ' !/5(!:= ;@h]4E~$Vhh h+;LYg  r,)NN)TFNNT) r=r>r?r@rrrrr!rrr;rArBs@r+rrshB ,&*"' 7 ||7  7 # 7  7 7 r,rc(eZdZdZeZdZdgZdZdZ y)DebertaPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. debertarTc6t|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjtfrJ|jjjd|jjjyt|trI|j jj|j"jjyt|t$t&fr%|jjjyy)zInitialize the weights.g)r3stdNg?)rrrFr#datanormal_rOinitializer_ranger%zero_rrrJrfill_rrrLegacyDebertaLMPredictionHeadDebertaLMPredictionHead)r'modules r+ _init_weightsz$DebertaPreTrainedModel._init_weightssk fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>. /? @ A MM   $ $S ) KK   " " $  9 : MM   $ $ & MM   $ $ & !>@W X Y KK   " " $Zr,N) r=r>r?r@r config_classbase_model_prefix"_keys_to_ignore_on_load_unexpectedsupports_gradient_checkpointingr@r"r,r+r4r4{s( !L!*?)@&&*#%r,r4a The DeBERTa model was proposed in [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://arxiv.org/abs/2006.03654) by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen. It's build on top of BERT/RoBERTa with two improvements, i.e. disentangled attention and enhanced mask decoder. With those two improvements, it out perform BERT/RoBERTa on a majority of tasks with 80GB pretraining data. This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`DebertaConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zaThe bare DeBERTa Model transformer outputting raw hidden-states without any specific head on top.c\eZdZfdZdZdZdZeejde e e e  ddeej deej d eej d eej d eej d eed eedeedeee ffdZxZS) DebertaModelct||t||_t ||_d|_||_|jyNr) rrrrrencoderz_stepsrO post_initrNs r+rzDebertaModel.__init__s@  +F3%f-    r,c.|jjSrrrr's r+get_input_embeddingsz!DebertaModel.get_input_embeddingss...r,c&||j_yrrMr'new_embeddingss r+set_input_embeddingsz!DebertaModel.set_input_embeddingss*8'r,ctd)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel z7The prune function is not implemented in DeBERTa model.)NotImplementedError)r'heads_to_prunes r+ _prune_headszDebertaModel._prune_headss ""[\\r,batch_size, sequence_length checkpoint output_typerArrrrrrrr rc |||n|jj}||n|jj}||n|jj}| | t d|#|j |||j } n!||j dd} n t d| |jn |j} |tj| | }|&tj| tj| }|j|||||} |j| |d||} | d } |jd kDr| d }t|jDcgc]}|jj d}}| d}|jj#}|jj%|}|jj'| }|d dD]!}|||d ||| }| j)|#| d}|s|f| |rd dzSd dzSt+||r | j,nd| j.Scc}w)NzDYou cannot specify both input_ids and inputs_embeds at the same timer.z5You have to specify either input_ids or inputs_embeds)rYrX)rrrrrT)rrr rrWFr r0r&)rOrruse_return_dictr%warn_if_padding_and_no_attention_maskr(rYr!r"r$r[rrIrJrrrrrappendr r7r()r'rrrrrrrr rrYembedding_outputencoder_outputsencoded_layersr7rlayersrrrel_posrsequence_outputs r+r;zDebertaModel.forwardsz"2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U%.%:!! @T@T  !"ZZ FCN  !"[[EJJvVN??)%' + ,,  !%/# ' )+ <R8\(]] ]XY8\(]] ]-;O/77UY&11  +Ks H9)NNNNNNNN)r=r>r?rrOrSrWrDEBERTA_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrr!rrrrr;rArBs@r+rFrFs /9]++C+J+JKh+ij&#$-11515/304,0/3&*N ELL)N !.N !. N u||, N   - N $D>N 'tnN d^N  uo% &N k N r,rFc$eZdZfdZdZxZS)$LegacyDebertaPredictionHeadTransformct|t|d|j|_t j |j|j|_t|jtrt|j|_ n|j|_ t j|j|j|_ y)Nr)r))rrrrGrrrFrHrrrr transform_act_fnrJrIrNs r+rz-LegacyDebertaPredictionHeadTransform.__init__Ms %f.>@R@RSYYv1143F3FG f'' -$*6+<+<$=D !$*$5$5D !d&9&9v?T?TUr,cl|j|}|j|}|j|}|Sr)rHrmrJrs r+r;z,LegacyDebertaPredictionHeadTransform.forwardXs4 =1 --m< }5 r,rUrBs@r+rkrkLs Vr,rkc*eZdZfdZdZdZxZS)r=ct|t||_t |d|j |_tj|j |jd|_ tjtj|j|_|j|j_y)NrFr)rrrk transformrrGrrrFrdecoderr r!r$r%rNs r+rz&LegacyDebertaLMPredictionHead.__init__`s =fE%f.>@R@RSyy!4!4f6G6GeT LLV->->!?@ !II r,c:|j|j_yr)r%rrrNs r+ _tie_weightsz*LegacyDebertaLMPredictionHead._tie_weightsns II r,cJ|j|}|j|}|Sr)rqrrrs r+r;z%LegacyDebertaLMPredictionHead.forwardqs$}5  ]3 r,)r=r>r?rrtr;rArBs@r+r=r=_s &&r,r=cVeZdZfdZdej dej fdZxZS)LegacyDebertaOnlyMLMHeadcBt|t||_yr)rrr= predictionsrNs r+rz!LegacyDebertaOnlyMLMHead.__init__ys 8@r,rerc(|j|}|Sr)ry)r'reprediction_scoress r+r;z LegacyDebertaOnlyMLMHead.forward}s ,,_=  r,rrBs@r+rwrwxs$A!u||! !r,rwc(eZdZdZfdZdZxZS)r>zMhttps://github.com/microsoft/DeBERTa/blob/master/DeBERTa/deberta/bert.py#L270ct|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|jd|_ tjtj|j |_y)NT)r)elementwise_affine)rrrrFrGrHrrrr rmrJrIr r!r$rr%rNs r+rz DebertaLMPredictionHead.__init__s YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>ShlmLLV->->!?@ r,c|j|}|j|}|j|}tj||j j |jz}|Sr)rHrmrJr!rr#rr%)r'r7rs r+r;zDebertaLMPredictionHead.forwardsd =1 --m<    ]O4J4J4L4L4NORVR[R[[ r,r<rBs@r+r>r>sW Ar,r>c$eZdZfdZdZxZS)DebertaOnlyMLMHeadcBt|t||_yr)rrr>lm_headrNs r+rzDebertaOnlyMLMHead.__init__s .v6 r,c*|j||}|Sr)r)r'rerr{s r+r;zDebertaOnlyMLMHead.forwards LL/J  r,rUrBs@r+rrs 7 !r,rz5DeBERTa Model with a `language modeling` head on top.ceZdZddgZfdZdZdZeejde e e e dee dd eej$d eej$d eej$d eej$d eej$deej$deedeedeedeee ffdZxZS)DebertaForMaskedLMzcls.predictions.decoder.weightzcls.predictions.decoder.biasct|||j|_t||_|jrt ||_nddg|_t||_ |jy)Nzlm_predictions.lm_head.weightz)deberta.embeddings.word_embeddings.weight) rrlegacyrFr5rwcls_tied_weights_keysrlm_predictionsrKrNs r+rzDebertaForMaskedLM.__init__sa  mm #F+ ;;/7DH'FHs&tD #"4V"D   ' ' -/=/B/BD   ' ' ,r,rXz[MASK])rZr[rArexpected_output expected_lossrrrrrlabelsrrr rc | | n|jj} |j|||||||| } | d} |jr|j | } n0|j | |jj j} d} |Ft}|| jd|jj|jd} | s| f| ddz}| | f|zS|St| | | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrrrrrrr rr.rlosslogitsr7r()rOr]r5rrrrrrr\rr r7r()r'rrrrrrrrr outputsrer{masked_lm_lossloss_fctrs r+r;zDebertaForMaskedLM.forwards!8&1%r?rrrrrrfrgr_CHECKPOINT_FOR_MASKED_LMr ri_MASKED_LM_EXPECTED_OUTPUT_MASKED_LM_EXPECTED_LOSSrr!rrrrr;rArBs@r+rrs0:4 'tn4 d^4  un$ %4 k4 r,rc4eZdZfdZdZedZxZS) ContextPoolerct|tj|j|j|_tj |j|_||_ yr) rrrrFpooler_hidden_sizerHrKpooler_dropoutrMrOrNs r+rzContextPooler.__init__sI YYv88&:S:ST zz&"7"78  r,c|dddf}|j|}|j|}t|jj|}|SrH)rMrHr rOpooler_hidden_act)r'r7 context_token pooled_outputs r+r;zContextPooler.forwardsM&ad+  ]3  =1 t{{<<=mL r,c.|jjSr)rOrGrNs r+ output_dimzContextPooler.output_dims{{&&&r,)r=r>r?rr;propertyrrArBs@r+rrs! ''r,rz DeBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cveZdZfdZdZdZeejde e e e  dde ejde ejde ejd e ejd e ejd e ejd e ed e ede edeee ffdZxZS) DebertaForSequenceClassificationct||t|dd}||_t ||_t ||_|jj}tj|||_ t|dd}||jjn|}tj||_|j!y)N num_labelsr0 cls_dropout)rrrrrFr5rpoolerrrrF classifierrOrLrKrMrK)r'rOrrdrop_outr*s r+rz)DebertaForSequenceClassification.__init__%s  V\15 $#F+ #F+ [[++ ))J ;6=$76>6F4;;22Hzz(+  r,c6|jjSr)r5rOrNs r+rOz5DebertaForSequenceClassification.get_input_embeddings7s||0022r,c:|jj|yr)r5rSrQs r+rSz5DebertaForSequenceClassification.set_input_embeddings:s )).9r,rXrYrrrrrrrrr rc  | | n|jj} |j|||||||| } | d} |j| } |j | } |j | } d}||jj |jdk(rXtj}| jdj|j} || |jd}n_|jdk(s|jddk(r|dk\j}|j!}|jddkDrt#j$| d|j'|jd| jd}t#j$|d|jd}t)}||jd|jj+|jd}nIt#j,dj| }n#tj.d}|| |zj1dj3 }n|jj dk(rIt}|jdk(r&|| j5|j5}n|| |}n|jj dk(r=t)}|| jd|j|jd}n,|jj dk(rt7}|| |}| s| f| ddz}||f|zS|St9|| | j:| j< S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrrrrr rrr. regressionsingle_label_classificationmulti_label_classificationr)rOr]r5rrMr problem_typerrr r\r6r1rr(nonzeror[r!rrirr2rv LogSoftmaxsumr3rrrr7r()r'rrrrrrrrr r encoder_layerrrrloss_fn label_indexlabeled_logitsr log_softmaxrs r+r;z(DebertaForSequenceClassification.forward=s0&1%#)Q;"7"7"9K#[[]F"''*Q.)."A{'9'9+:J:J1:Mv{{[\~'^*"'fa9I9I"9M!N#3#5'(;(;B(P(V(V(XZ`ZeZefhZij$||A11&9"$--"3K)&1F:??CIIKKD))\9"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./Y,F)-)9TGf$ Ev E'fG4I4IV]VhVh  r,r)r=r>r?rrOrSrrfrgrrhrrirr!rrrrr;rArBs@r+rrs'$3:++C+J+JKh+ij&,$-11515/304)-,0/3&*M ELL)M !.M !. M u||, M   - M &M $D>M 'tnM d^M  u.. /M k M r,rz DeBERTa Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cjeZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e ed e ed e ed eee ffdZxZS)DebertaForTokenClassificationc,t|||j|_t||_t j |j|_t j|j|j|_ |jyr) rrrrFr5rrKrLrMrFrGrrKrNs r+rz&DebertaForTokenClassification.__init__si   ++#F+ zz&"<"<= ))F$6$68I8IJ r,rXrYrrrrrrrrr rc | | n|jj} |j|||||||| } | d} |j| } |j | } d} |r?rrrfrgrrhrrirr!rrrrr;rArBs@r+rrs ++C+J+JKh+ij&)$-11515/304)-,0/3&*- ELL)- !.- !. - u||, -   - - &- $D>- 'tn- d^-  u++ ,- k - r,rz DeBERTa Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZfdZeej deee e e e e e ddeej"deej"deej"deej"deej"d eej"d eej"d eed eed eedeee ffdZxZS)DebertaForQuestionAnsweringct|||j|_t||_t j |j|j|_|jyr) rrrrFr5rrFrG qa_outputsrKrNs r+rz$DebertaForQuestionAnswering.__init__sS   ++#F+ ))F$6$68I8IJ r,rX)rZr[rArrqa_target_start_indexqa_target_end_indexrrrrrstart_positions end_positionsrrr rc &| | n|jj} |j||||||| | } | d} |j| } | j dd\}}|j dj }|j dj }d}||t|jdkDr|j d}t|jdkDr|j d}|jd}|jd|}|jd|}t|}|||}|||}||zdz }| s||f| ddz}||f|zS|St|||| j| j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrrr.r) ignore_indexr0)r start_logits end_logitsr7r()rOr]r5rsplitrrrr(rrrr7r()r'rrrrrrrrrr rrerrr total_loss ignored_indexr start_lossend_lossrs r+r;z#DebertaForQuestionAnswering.forwardsB&1%r?rrrfrgr_CHECKPOINT_FOR_QArri_QA_EXPECTED_OUTPUT_QA_EXPECTED_LOSS_QA_TARGET_START_INDEX_QA_TARGET_END_INDEXrr!rrrrr;rArBs@r+rrs@++C+J+JKh+ij%0$+'40-11515/3042604,0/3&*F ELL)F !.F !. F u||, F   - F "%,,/F   -F $D>F 'tnF d^F  u22 3F kF r,r)rrrrrFr4)Or@typingrrrr!torch.utils.checkpointrtorch.nnrrr activationsr modeling_outputsr r rrrmodeling_utilsrutilsrrrrconfiguration_debertar get_loggerr=loggerrirhrrrrrrrrModulerrDjitscriptrfrlrnrrrrrwrzrrrrrrrrrr4DEBERTA_START_DOCSTRINGrfrFrkr=rwr>rrrrrr__all__r"r,r+rs)) AA!.uu0   H %!.@'!;(ryy(  8rrnn[[\%,,\c\\ELLU\\d ddgjdd C CLN Nb,ryy,F "))  BII (299(BZ RYYZ z%_%B")Xgl )l l ^299&BII2!ryy!bii6!!QSjk[ /[ l[ |'BII', l '=l l ^ ? $:? ? D [ "8[ [ | r,