T^(h0dZddlZddlmZmZmZmZddlZddlZddl m Z ddlm Z ddl m Z mZmZddlmZmZdd lmZdd lmZmZdd lmZmZmZmZmZmZmZm Z dd l!m"Z"dd l#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/e,j`e1Z2dZ3dZ4Gdde jjZ6Gdde jjZ7Gdde7Z8Gdde jjZ9e7e8dZ:Gdde jjZ;Gdde jjZ<Gd d!e jjZ=Gd"d#e jjZ>Gd$d%e jjZ?Gd&d'e jjZ@Gd(d)e"ZAd*ZBd+ZCe)d,eBGd-d.eAZDe)d/eBGd0d1eAeZEe)d2eBGd3d4eAZFGd5d6e jjZGe)d7eBGd8d9eAZHe)d:eBGd;dd?eAZJGd@dAe jjZKe)dBeBGdCdDeAZLdGdEZMgdFZNy)HzPyTorch RoBERTa model.N)ListOptionalTupleUnion)version)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNgelu)GenerationMixin)#_prepare_4d_attention_mask_for_sdpa*_prepare_4d_causal_attention_mask_for_sdpa))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardget_torch_versionloggingreplace_return_docstrings) RobertaConfigzFacebookAI/roberta-baser%c2eZdZdZfdZ ddZdZxZS)RobertaEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_t#|dd|_|j'dt)j*|jj-dd|j'd t)j.|j0j3t(j4 d|j |_tj|j|j |j6|_ y) N) padding_idxepsposition_embedding_typeabsolute position_ids)r$F) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr,register_buffertorcharangeexpandzerosr.sizelongr)selfconfig __class__s j/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/roberta/modeling_roberta.pyr5zRobertaEmbeddings.__init__Dsi !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  "..#%<<  * *F,>,>DL\L\$  c|+|t||j|}n|j|}||j}n|jdd}|d}|st |dr-|j ddd|f}|j |d|} | }n:tj|tj|jj}||j|}|j|} || z} |jdk(r|j|} | | z } |j!| } |j#| } | S)Nr/r$r1rr3devicer-)"create_position_ids_from_input_idsr)&create_position_ids_from_inputs_embedsrJhasattrr1rHrFrIrKr.rTr:r>r,r<r?rC) rM input_idsr1r. inputs_embedspast_key_values_length input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr> embeddingsr<s rPforwardzRobertaEmbeddings.forward]sR  $A)TM]M]_uv #JJ=Y  #..*K',,.s3K ^  !t-.*.*=*=a*n*M'3J3Q3QR]^_R`bl3m0!A!&[ SWSdSdSkSk!l   00;M $ : :> J"%::  ' ': 5"&":":<"H  - -J^^J/ \\*- rQc|jdd}|d}tj|jdz||jzdztj|j }|j dj|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr/r$rSr)rJrFrGr)rKrT unsqueezerH)rMrYr[sequence_lengthr.s rPrVz8RobertaEmbeddings.create_position_ids_from_inputs_embedss$((*3B/ %a.||   q /D4D4D"Dq"HPUPZPZcpcwcw %%a(// <s  4rs&P=rQr'cPeZdZd fd Zdej dej fdZ ddej deejdeejdeejd eejd ee e ejd ee de ej fd Z xZ S)RobertaSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|_|xs t#|dd|_|j$dk(s|j$d k(rF|j&|_tj(d |j&zd z |j|_|j,|_y) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r,r- relative_keyrelative_key_queryr$)r4r5r8num_attention_headsrW ValueErrorintattention_head_size all_head_sizerLinearquerykeyvaluerAattention_probs_dropout_probrCrDr,r;r6distance_embedding is_decoderrMrNr,rOs rPr5zRobertaSelfAttention.__init__s     : : :a ?PVXhHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD # ++rQxreturnc|jdd|j|jfz}|j|}|j ddddS)Nr/rrqr$r )rJrrruviewpermute)rMr new_x_shapes rPtranspose_for_scoresz)RobertaSelfAttention.transpose_for_scoressLffhsmt'?'?AYAY&ZZ FF; yyAq!$$rQ hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsc$|j|}|du} | r||d} |d} |}n| rC|j|j|} |j|j|} |}n|y|j|j|} |j|j|} t j |d| gd} t j |d| gd} n@|j|j|} |j|j|} |j|} |du} |j r| | f}t j| | jdd}|jdk(s|jdk(r| jd| jd}}| rDt j|dz tj|j jdd}n@t j|tj|j jdd}t j|tj|j jdd}||z }|j!||j"zdz }|j%| j& }|jdk(rt j(d | |}||z}nE|jdk(r6t j(d | |}t j(d | |}||z|z}|t+j,|j.z }|||z}t0j2j5|d}|j7|}|||z}t j|| }|j9dddd j;}|j=dd|j>fz}|j|}|r||fn|f}|j r||fz}|S)Nrr$rqdimr/rorprSr2zbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rxrryrzrFcatr}matmul transposer,shapetensorrKrTrrGr|r;tor3einsummathsqrtrur functionalsoftmaxrCr contiguousrJrv)rMrrrrrrrmixed_query_layeris_cross_attention key_layer value_layer query_layer use_cacheattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputss rPr`zRobertaSelfAttention.forwards!JJ}5 3$> ."<&q)I(+K3N 11$((;P2QRI33DJJ?T4UVK3N  '11$((=2IJI33DJJ}4MNK >!#4i"@aHI))^A%6 $D!LK11$((=2IJI33DJJ}4MNK//0AB "$. ??(5N!<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC 6G=/2mM] ?? 11GrQNNNNNNF) rdrerfr5rFTensorrr FloatTensorrboolr`rhris@rPrkrks,4%ell%u||%7;15=A>BDH,1c||c!!2!23cE--. c ((9(9: c !)):): ; c!uU->->'?!@Ac$D>c u|| crQrkceZdZd fd Z d dej deej deejdeejdeejdeeeejdee d eej ffd Z xZ S) RobertaSdpaSelfAttentionct||||j|_t j t t j dk|_y)Nr,z2.2.0)r4r5r{ dropout_probrparser!require_contiguous_qkvr~s rPr5z!RobertaSdpaSelfAttention.__init__ sH 9PQ"??&-mm4E4G&H7==Y`Ka&a#rQrrrrrrrrc p|jdk7s|s|*tjdt||||||||S|j \}} } |j |j|} |du} | r|n|} | r|n|}| r*|r(|djd| jdk(r|\}}n||j |j| }|j |j| }|:| s8tj|d|gd}tj|d|gd}|jr||f}|jrK| jj dk(r2|0| j#} |j#}|j#}|jr | s || dkDrdnd }tj$j&j)| ||||j*r |j,nd | }|j/dd}|j1|| |j2}|f}|jr||fz}|S) Nr-aRobertaSdpaSelfAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support non-absolute `position_embedding_type` or `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rrqr$rcudaTF) attn_mask dropout_p is_causal)r,logger warning_oncer4r`rJrrxrryrzrFrr}rrTtyperrrscaled_dot_product_attentiontrainingrrreshaperv)rMrrrrrrrbsztgt_len_rrcurrent_statesrrr attn_outputrrOs rPr`z RobertaSdpaSelfAttention.forward&s`  ' ': 59JiNc   H 7?%&! (,,.Wa// =0IJ 3$>2D.-3E/> .^A5F5L5LQ5OSaSgSghiSj5j%3 "I{11$((>2JKI33DJJ~4NOK)2D!II~a'8)&D!L #ii):K(HaP ??(5N  & &;+=+=+B+Bf+LQ_Qk%002K!,,.I%002KOO,>>CY^ehi^iDot hh))FF   $+/==d''c G "++Aq1 !))#w8J8JK . ?? 11GrQrr) rdrerfr5rFrrrrrr`rhris@rPrrsb2615=A>BDH,1[||[!.[E--. [ ((9(9: [ !)):): ; [!uU->->'?!@A[$D>[ u|| [[rQrcneZdZfdZdej dej dej fdZxZS)RobertaSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr*) r4r5rrwr8denser?r@rArBrCrLs rPr5zRobertaSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rQr input_tensorrcr|j|}|j|}|j||z}|SrrrCr?rMrrs rPr`zRobertaSelfOutput.forward7 =1  ]3 }|'CD rQrdrerfr5rFrr`rhris@rPrr1> U\\RWR^R^rQr)eagersdpaceZdZd fd ZdZ d dej deejdeejdeejdeejdee e ejd ee d e ej fd Z xZ S)RobertaAttentionct|t|j|||_t ||_t|_y)Nr) r4r5ROBERTA_SELF_ATTENTION_CLASSES_attn_implementationrMroutputset pruned_headsr~s rPr5zRobertaAttention.__init__sC 263N3NO ,C  (/ ErQc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrr$r)lenrrMrrrurrrxryrzrrrvunion)rMheadsindexs rP prune_headszRobertaAttention.prune_headss u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rQrrrrrrrrc p|j|||||||}|j|d|} | f|ddz} | S)Nrr$)rMr) rMrrrrrrr self_outputsattention_outputrs rPr`zRobertaAttention.forwardsWyy    ! "    ;;|A F#% QR(88rQrr) rdrerfr5rrFrrrrrr`rhris@rPrrs";*7;15=A>BDH,1||!!2!23E--.  ((9(9:  !)):): ; !uU->->'?!@A$D> u|| rQrcVeZdZfdZdej dej fdZxZS)RobertaIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r4r5rrwr8intermediate_sizer isinstance hidden_actstrr intermediate_act_fnrLs rPr5zRobertaIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rQrrcJ|j|}|j|}|Sr)rr)rMrs rPr`zRobertaIntermediate.forwards& =1 00? rQrris@rPrrs#9U\\ellrQrcneZdZfdZdej dej dej fdZxZS) RobertaOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r4r5rrwrr8rr?r@rArBrCrLs rPr5zRobertaOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rQrrrcr|j|}|j|}|j||z}|Srrrs rPr`zRobertaOutput.forwardrrQrris@rPrrrrQrceZdZfdZ d dej deejdeejdeejdeejdeeeejdee d eej fd Z d Z xZ S) RobertaLayercft||j|_d|_t ||_|j |_|j|_|jr,|j st|dt |d|_ t||_ t||_ y)Nr$z> should be used as a decoder model if cross attention is addedr-r)r4r5chunk_size_feed_forward seq_len_dimr attentionr}add_cross_attentionrscrossattentionr intermediaterrrLs rPr5zRobertaLayer.__init__s '-'E'E$)&1 ++#)#=#=  # #?? D6)g!hii"26S]"^D /7#F+ rQrrrrrrrrc ||ddnd}|j|||||} | d} |jr | dd} | d} n| dd} d} |jrT|Rt|dstd|d||d dnd}|j | ||||||}|d} | |ddz} |d}  | z} t |j |j|j| }|f| z} |jr|  fz} | S) Nrq)rrrr$r/rz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rr}rWrsrrfeed_forward_chunkrr)rMrrrrrrrself_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputs rPr`zRobertaLayer.forwards}:H9S>"1#5Y] !%   /3 "0" 2!4 ??,Qr2G 6r : ,QR0G'+$ ??4@4!12 =dVDDD @N?Yrs(;_c %&*&9&9 %&)!' # 7q9  7" ==G,C2+F ( 14P P 0  # #T%A%A4CSCSUe  /G+ ??!2 44GrQcL|j|}|j||}|Sr)rr)rMrintermediate_outputr s rPrzRobertaLayer.feed_forward_chunk=s,"//0@A{{#68HI rQr) rdrerfr5rFrrrrrr`rrhris@rPrrs ,"7;15=A>BDH,1?||?!!2!23?E--. ? ((9(9: ? !)):): ; ?!uU->->'?!@A?$D>? u|| ?BrQrcDeZdZfdZ ddej deejdeejdeejdeejdeeeejdee d ee d ee d ee d e eej e ffd Z xZ S)RobertaEncoderct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) r4r5rNr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)rMrNrrOs rPr5zRobertaEncoder.__init__EsN  ]]%H`H`Ba#bQL$8#bc &+#$csA#rrrrrpast_key_valuesrroutput_hidden_states return_dictrc | rdnd} |rdnd} |r|jjrdnd} |jr%|jr|rtj dd}|rdnd}t |jD]\}}| r| |fz} |||nd}|||nd}|jr/|jr#|j|j|||||||}n||||||||}|d}|r ||dfz }|s|| |dfz} |jjs| |dfz} | r| |fz} | std||| | | fDSt||| | | S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frr/r$rqc3$K|]}|| ywrr).0vs rP z)RobertaEncoder.forward..s = s)last_hidden_staterr attentionscross_attentions) rNrrrrr enumerater_gradient_checkpointing_func__call__tupler)rMrrrrrrrrrrall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_maskr layer_outputss rPr`zRobertaEncoder.forwardKs#7BD$5b4%64;;;Z;Zr`d  & &4==##p" #,R$(4# VOA|#$58H$H!.7.CilO3B3N_Q/TXN**t}} $ A A ))!"#)*"% ! !-!"#)*"%! *!,M"}R'8&::" &9]1=MBEI$(,1/4&*S ||S !!2!23S E--. S ((9(9: S !)):): ; S "%e.?.?(@"ABS D>S $D>S 'tnS d^S  uU\\"$MM NS rQrcVeZdZfdZdej dej fdZxZS) RobertaPoolerct|tj|j|j|_tj |_yr)r4r5rrwr8rTanh activationrLs rPr5zRobertaPooler.__init__s9 YYv1163E3EF '')rQrrc\|dddf}|j|}|j|}|SNr)rr2)rMrfirst_token_tensor pooled_outputs rPr`zRobertaPooler.forwards6+1a40 #56  6 rQrris@rPr/r/s#$ U\\ellrQr/c.eZdZdZeZdZdZgdZdZ dZ y)RobertaPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. robertaT)r'rkrclt|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyt|tr%|jjjyy)zInitialize the weightsr)meanstdNg?)rrrwweightdatanormal_rNinitializer_rangebiaszero_r6r)r?fill_ RobertaLMHead)rMmodules rP _init_weightsz$RobertaPreTrainedModel._init_weightss& fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S )  . KK   " " $/rQN) rdrerfrgr% config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_sdparFrrQrPr8r8s* !L!&*#aN%rQr8aA This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`RobertaConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zaThe bare RoBERTa Model transformer outputting raw hidden-states without any specific head on top.c!eZdZdZddgZdfd ZdZdZdZe e jde e ee  dd eej$d eej$d eej$d eej$deej$deej$deej$deej$deeej(deedeedeedeedeeej$effdZxZS) RobertaModela The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. r'rct||||_t||_t ||_|r t|nd|_|j|_ |j|_ |jyr) r4r5rNr'r_rencoderr/poolerrattn_implementationr, post_init)rMrNadd_pooling_layerrOs rPr5zRobertaModel.__init__(sg   +F3%f- /@mF+d #)#>#> '-'E'E$ rQc.|jjSrr_r:rMs rPget_input_embeddingsz!RobertaModel.get_input_embeddings7s...rQc&||j_yrrU)rMrzs rPset_input_embeddingsz!RobertaModel.set_input_embeddings:s*/'rQc|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrOrrr)rMheads_to_prunerrs rP _prune_headszRobertaModel._prune_heads=sE +002 CLE5 LL  u % / / ; ;E B CrQbatch_size, sequence_length checkpoint output_typerGrXrr1r.rrYrrrrrrrrc | | n|jj} | | n|jj} | | n|jj} |jjr| | n|jj } nd} | | t d|#|j|||j}n!||jdd}n t d|\}}| |jn |j}| | ddjdnd}|pt|jdr4|jjddd|f}|j||}|}n&tj |tj"| }|j||||| }|tj$|||zf| }|j&d k(xr|j(d k(xr |duxr| }|rQ|j+dk(r>|jjrt-||||}n+t/||j0|}n|j3||}|jjrs|q|j\}}}||f}|tj$|| }|r,|j+dk(rt/||j0|}n|j5|}nd}|j7||jj8}|j;|||||| | | | |  }|d}|j<|j=|nd}| s ||f|ddzSt?|||j@|jB|jD|jFS)a? encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)` or `(batch_size, sequence_length, target_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timer/z5You have to specify either input_ids or inputs_embedsrrqr1rS)rXr.r1rYrZ)rTrr-)r) rrrrrrrrrr$)r pooler_outputrrr r!)$rNrruse_return_dictr}rrs%warn_if_padding_and_no_attention_maskrJrTrrWr_r1rHrFrIrKonesrQr,rrrr3get_extended_attention_maskinvert_attention_mask get_head_maskrrOrPrrrr r!) rMrXrr1r.rrYrrrrrrrr[ batch_sizer\rTrZr]r^embedding_outputuse_sdpa_attention_masksextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskencoder_outputssequence_outputr6s rPr`zRobertaModel.forwardEsT2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@TDSC^!3A!6!` $D>` 'tn` d^`  uU\\"$PP Q` k ` rQrMzIRoBERTa Model with a `language modeling` head on top for CLM fine-tuning.c#.eZdZddgZfdZdZdZeejde e e  dde ejd e ej d e ejd e ejd e ej d e ej de ej de ej de ejdeeej de ede ede ede edeeej(e ffdZdZxZS)RobertaForCausalLMlm_head.decoder.weightlm_head.decoder.biasct|||jstj dt |d|_t||_|jy)NzOIf you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`FrS r4r5r}rwarningrMr9rDlm_headrRrLs rPr5zRobertaForCausalLM.__init__sL    NNl m#FeD $V,  rQc.|jjSrrdecoderrVs rPget_output_embeddingsz(RobertaForCausalLM.get_output_embeddings||###rQc&||j_yrrrMnew_embeddingss rPset_output_embeddingsz(RobertaForCausalLM.set_output_embeddings- rQr^)rarGrXrr1r.rrYrrlabelsrrrrrrc ||n|jj}| d} |j||||||||| | | | | }|d}|j|}d}| E| j |j } |j || fd|jji|}|s|f|ddz}||f|zS|St|||j|j|j|jS)a2 encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Returns: Example: ```python >>> from transformers import AutoTokenizer, RobertaForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base") >>> config = AutoConfig.from_pretrained("FacebookAI/roberta-base") >>> config.is_decoder = True >>> model = RobertaForCausalLM.from_pretrained("FacebookAI/roberta-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NF) rr1r.rrYrrrrrrrrr7rq)losslogitsrrr r!) rNrdr9rrrT loss_functionr7rrrr r!)rMrXrr1r.rrYrrrrrrrrkwargsrrsprediction_scoreslm_lossrs rPr`zRobertaForCausalLM.forwards6~&1%.xs.nU_j--aZ=N=N1OPns58)r%)rMrrreordered_past layer_pasts ` rP_reorder_cachez!RobertaForCausalLM._reorder_cachets=) J ncmnn N rQ)NNNNNNNNNNNNNN)rdrerf_tied_weights_keysr5rrr rtrur#rrwrrF LongTensorrrrrrr`rrhris@rPryrys34JK $.++C+J+JKh+ij+L[jk156:59371559=A>B-1;?$(,0/3&*j E,,-j !!2!23j !!1!12 j u//0 j E--. j   1 12j  ((9(9:j !)):): ;j ))*j uU%6%678j D>j $D>j 'tnj d^j " uU\\"$EE F#j lkj XrQryz5RoBERTa Model with a `language modeling` head on top.ceZdZddgZfdZdZdZeejde e e e ddd  dd eej d eej"d eej deej deej"deej"deej"deej"deej deedeedeedeeej*e ffdZxZS)RobertaForMaskedLMrzr{ct|||jrtj dt |d|_t||_|jy)NznIf you want to use `RobertaForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Fr}r~rLs rPr5zRobertaForMaskedLM.__init__sR     NN1  $FeD $V,  rQc.|jjSrrrVs rPrz(RobertaForMaskedLM.get_output_embeddingsrrQc&||j_yrrrs rPrz(RobertaForMaskedLM.set_output_embeddingsrrQr^zz' Paris'g?)r`rarGmaskexpected_output expected_lossrXrr1r.rrYrrrrrrrc | | n|jj} |j||||||||| | |  } | d}|j|}d}| a| j |j } t }||jd|jj| jd}| s|f| ddz}||f|zS|St||| j| jS)a( labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` kwargs (`Dict[str, any]`, *optional*, defaults to `{}`): Used to hide legacy arguments that have been deprecated. N) rr1r.rrYrrrrrrr/rqrrrr ) rNrdr9rrrTr rr7rrr )rMrXrr1r.rrYrrrrrrrrsrmasked_lm_lossloss_fctrs rPr`zRobertaForMaskedLM.forwards@&1%B-1,0/3&*9 E,,-9 !!2!239 !!1!12 9 u//0 9 E--. 9   1 129  ((9(9:9 !)):): ;9 ))*9 $D>9 'tn9 d^9  uU\\"N2 39 k9 rQrc.eZdZdZfdZdZdZxZS)rDz*Roberta Head for masked language modeling.ct|tj|j|j|_tj |j|j|_tj|j|j|_ tjtj|j|_|j|j_yr)r4r5rrwr8rr?r@ layer_normr7r ParameterrFrIrArLs rPr5zRobertaLMHead.__init__s YYv1163E3EF ,,v'9'9v?T?TUyy!3!3V5F5FG LLV->->!?@  II rQc |j|}t|}|j|}|j|}|Sr)rrrrrMfeaturesrrs rPr`zRobertaLMHead.forwards; JJx  G OOA  LLOrQc|jjjjdk(r|j|j_y|jj|_y)Nmeta)rrArTrrVs rP _tie_weightszRobertaLMHead._tie_weightssC <<   # # ( (F 2 $ DLL  ))DIrQ)rdrerfrgr5r`rrhris@rPrDrDs4&*rQrDz RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZeej dedee dd dde e jde e jd e e jd e e jd e e jd e e jd e e jde ede ede edeee j"effdZxZS) RobertaForSequenceClassificationct|||j|_||_t |d|_t ||_|jyNFr}) r4r5 num_labelsrNrMr9RobertaClassificationHead classifierrRrLs rPr5z)RobertaForSequenceClassification.__init__sJ   ++ #FeD 3F; rQr^z'cardiffnlp/twitter-roberta-base-emotionz 'optimism'g{Gz?r`rarGrrrXrr1r.rrYrrrrrc T| | n|jj} |j|||||||| |  } | d} |j| } d}||j | j }|jj |jdk(rd|j_nl|jdkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt}|jdk(r&|| j|j}n|| |}n|jj dk(r=t}|| jd|j|jd}n,|jj dk(rt!}|| |}| s| f| d dz}||f|zS|St#|| | j$| j& S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr1r.rrYrrrrr$ regressionsingle_label_classificationmulti_label_classificationr/rqr)rNrdr9rrrT problem_typerr3rFrKrtr squeezer rr rrr rMrXrr1r.rrYrrrrrrsrrrrs rPr`z(RobertaForSequenceClassification.forward s6&1%E 'tnE d^E  uU\\"$<< =E kE rQrz Roberta Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. ceZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e ed e ed e edeee j$e ffdZxZS)RobertaForMultipleChoicect||t||_t j |j |_t j|jd|_ |jy)Nr$) r4r5rMr9rrArBrCrwr8rrRrLs rPr5z!RobertaForMultipleChoice.__init__esV  #F+ zz&"<"<= ))F$6$6: rQz(batch_size, num_choices, sequence_lengthr_rXr1rrr.rrYrrrrc | | n|jj} ||jdn|jd} |!|jd|j dnd} |!|jd|j dnd} |!|jd|j dnd}|!|jd|j dnd}|1|jd|j d|j dnd}|j | | |||||| |  }|d}|j |}|j|}|jd| }d}|.|j|j}t}|||}| s|f|ddz}||f|zS|St|||j|jS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr$r/r)r.r1rrrYrrrrqr)rNrdrrrJr9rCrrrTr rrr )rMrXr1rrr.rrYrrr num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsrr6rreshaped_logitsrrrs rPr`z RobertaForMultipleChoice.forwardos2&1%?^bLXLdL--b,2C2CB2GHjnR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrv(   r=#5#5b#9=;M;Mb;Q R  ,, *..,/!5#     ]3 / ++b+6  YY556F')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   rQr)rdrerfr5r rtrurrvrrwrrFrrrrrrr`rhris@rPrr]sK++C+J+JKu+vw&-$15596:-1371559,0/3&*A E,,-A !!1!12A !!2!23 A ))* A u//0 A E--.A   1 12A $D>A 'tnA d^A  uU\\"$== >A x A rQrz Roberta Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. ceZdZfdZeej dedee dd dde e jde e jd e e jd e e jd e e jd e e jd e e jde ede ede edeee j"effdZxZS)RobertaForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyr)r4r5rrMr9classifier_dropoutrBrrArCrwr8rrRrMrNrrOs rPr5z&RobertaForTokenClassification.__init__s   ++#FeD )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rQr^z'Jean-Baptiste/roberta-large-ner-englishzF['O', 'ORG', 'ORG', 'O', 'O', 'O', 'O', 'O', 'LOC', 'O', 'LOC', 'LOC']g{Gz?rrXrr1r.rrYrrrrrc | | n|jj} |j|||||||| |  } | d} |j| } |j | } d}|W|j | j }t}|| jd|j|jd}| s| f| ddz}||f|zS|St|| | j| jS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr/rqr) rNrdr9rCrrrTr rrrrr rs rPr`z%RobertaForTokenClassification.forwards2&1%4 'tn4 d^4  uU\\"$99 :4 k4 rQrc(eZdZdZfdZdZxZS)rz-Head for sentence-level classification tasks.cZt|tj|j|j|_|j |j n |j}tj||_ tj|j|j|_ yr) r4r5rrwr8rrrBrArCrout_projrs rPr5z"RobertaClassificationHead.__init__s YYv1163E3EF )/)B)B)NF % %TZTnTn zz"45  &"4"4f6G6GH rQc |dddddf}|j|}|j|}tj|}|j|}|j |}|Sr4)rCrrFtanhrrs rPr`z!RobertaClassificationHead.forwardsY Q1W  LLO JJqM JJqM LLO MM! rQ)rdrerfrgr5r`rhris@rPrrs7IrQrz Roberta Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZfdZeej dedee dd dde e jde e jd e e jd e e jd e e jd e e jd e e jde e jde ede ede edeee j"effdZxZS)RobertaForQuestionAnsweringct|||j|_t|d|_t j |j|j|_|jyr) r4r5rrMr9rrwr8 qa_outputsrRrLs rPr5z$RobertaForQuestionAnswering.__init__,sU   ++#FeD ))F$6$68I8IJ rQr^zdeepset/roberta-base-squad2z ' puppet'gQ?rrXrr1r.rrYstart_positions end_positionsrrrrc (| | n|jj} |j||||||| | |  } | d} |j| }|j dd\}}|j dj }|j dj }d}||t|jdkDr|j d}t|jdkDr|j d}|jd}|jd|}|jd|}t|}|||}|||}||zdz }| s||f| ddz}||f|zS|St|||| j| j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrr$r/r) ignore_indexrq)r start_logits end_logitsrr )rNrdr9rsplitrrrrJclampr rrr )rMrXrr1r.rrYrrrrrrrsrrr total_loss ignored_indexr start_lossend_lossrs rPr`z#RobertaForQuestionAnswering.forward6s@&1%H 'tnH d^H  uU\\"$@@ AH kH rQrc|j|j}tj|dj ||z|z}|j |zS)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor r$r)nertrFcumsumtype_asrK)rXr)rZrincremental_indicess rPrUrUsW << $ ( ( *D <<!4<rs  // AA')   .ll1   H %/!V= V=tC299CNb3bL  " $"0ryy0h "))  BII S299SnZ RYYZ | BII %_%@ 0fg R )R  R jSUlI/IIXQSjkZ /Z lZ z*BII*> Y '=Y Y x R 5R R j K $:K K \ , [ "8[ [ | 4 rQ