T^(hQxdZddlZddlZddlmZmZmZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZdd lmZmZmZmZmZmZmZmZdd lmZdd lmZmZm Z dd l!m"Z"m#Z#m$Z$m%Z%m&Z&d dl'm(Z(e%jRe*Z+dZ,dZ-dZ.Gdde j^Z0Gdde j^Z1Gdde j^Z2Gdde j^Z3Gdde j^Z4Gdde j^Z5Gdde j^Z6Gd d!e j^Z7Gd"d#e j^Z8Gd$d%e j^Z9Gd&d'e j^Z:Gd(d)e j^Z;Gd*d+eZe#d.e=Gd/d0e<Z?e#d1e=Gd2d3e<Z@e#d4e=Gd5d6ed?e<ZDe#d@e=GdAdBe<ZEgdCZFy)DzPyTorch RemBERT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GenerationMixin))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) RemBertConfigrgoogle/rembertc  ddl}ddl}ddl}t jj|}tjd||jj|}g}g} |D]s\} tfddDrtjdd| |jj|} |j| j| ut|| D]\} j!d d j#d td Dr(tjd d j%d|} D]} |j'd| r|j#d| }n| g}|ddk(s|ddk(r t)| d} nW|ddk(s|ddk(r t)| d} n:|ddk(r t)| d} n%|ddk(r t)| d} n t)| |d} t/|dk\st1|d}| |}  dddk(r t)| d} n| dk(r|j3| } | j4| j4k7r&t7d| j4d| j4d  tjd!t=j>| | _ |S#t$rtj dwxYw#t*$r7tjdj-d j%YwxYw#t8$r1}|xj:| j4| j4fz c_d}~wwxYw)"z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from c3&K|]}|v ywN).0denynames j/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/rembert/modeling_rembert.py z-load_tf_weights_in_rembert..NsXtt|Xs)adam_vadam_moutput_embeddingclszLoading TF weight z with shape zbert/zrembert//c3$K|]}|dv yw))r*r+AdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepNr$)r%ns r(r)z-load_tf_weights_in_rembert.._s  n n z Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifierz Skipping {}ri _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )!renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variablesany load_variableappendzipreplacesplitjoin fullmatchgetattrAttributeErrorformatlenint transposeshape ValueErrorAssertionErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathr@nptftf_path init_varsnamesarraysrZarraypointerm_name scope_namesnumer's @r(load_tf_weights_in_rembertrp8sV ggoo01G KK8 BC''0I E F  e X(WX X  (l5'BC&&w5 T e 5&)1/ e||GZ0zz#     KK)CHHTN#34 5  'F||,f5 hhy&9 %h 1~)[^w-F!'84Q=0KNf4L!'62Q#33!'84Q7*!'<8%g{1~>G;1$+a.)!#,+ ', #$<= (gx0G x LL'E }} + >'--@QRWR]R]Q^^i!jkk,  078''. c1/d LS   Q   n&KK 4 4SXXd^ DE  FFw}}ekk2 2F  s5 J7!K?L7 K J"%:: "66|D)) ^^J/ \\*- r)NNNNr) __name__ __module__ __qualname____doc__r{rr^ LongTensor FloatTensorrXTensorr __classcell__rs@r(rrrrsQ (15593759&' E,,-!!1!12u//0    1 12  !$  rrrcVeZdZfdZdej dej fdZxZS) RemBertPoolerct|tj|j|j|_tj |_yr#)rzr{rLinear hidden_sizedenseTanh activationrs r(r{zRemBertPooler.__init__s9 YYv1163E3EF '')r hidden_statesrc\|dddf}|j|}|j|}|S)Nr)rr)rrfirst_token_tensor pooled_outputs r(rzRemBertPooler.forwards6+1a40 #56  6 rrrrr{r^rrrrs@r(rrs#$ U\\ellrrceZdZfdZdZ d dej deejdeejdeejdeejde e ejd e d e fd Z xZ S) RemBertSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|_|j"|_y)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())rzr{rnum_attention_headshasattrr[rXattention_head_size all_head_sizerrquerykeyvaluerattention_probs_dropout_probr is_decoderrs r(r{zRemBertSelfAttention.__init__s)     : : :a ?PVXhHi#F$6$6#78 445Q8  $*#=#= #&v'9'9Frr )rrrviewpermute)rx new_x_shapes r(transpose_for_scoresz)RemBertSelfAttention.transpose_for_scoressNffhsmt'?'?AYAY&ZZ AFFK yyAq!$$rrattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsrc|j|}|du} | r||d} |d} |}n| rC|j|j|} |j|j|} |}n|y|j|j|} |j|j|} t j |d| gd} t j |d| gd} n@|j|j|} |j|j|} |j|} |j r| | f}t j| | jdd} | tj|jz } || |z} tjj| d}|j|}|||z}t j|| }|j!ddddj#}|j%dd|j&fz}|j(|}|r||fn|f}|j r||fz}|S)Nrrr>dimrxr )rrrrr^catrmatmulrYmathsqrtrr functionalsoftmaxrr contiguousrrr)rrrrrrrrmixed_query_layeris_cross_attention key_layer value_layer query_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputss r(rzRemBertSelfAttention.forwardsX!JJ}5 3$> ."<&q)I(+K3N 11$((;P2QRI33DJJ?T4UVK3N  '11$((=2IJI33DJJ}4MNK >!#4i"@aHI))^A%6 $D!LK11$((=2IJI33DJJ}4MNK//0AB ??(5N!<< Y5H5HR5PQ+dii8P8P.QQ  %/.@ --//0@b/I,,7  - 9O _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S* **,CD 6G=/2mM] ?? 11GrNNNNNF) rrrr{rr^rrrrboolrrrs@r(rrs,(%7;15=A>B:>"'L||L!!2!23LE--. L ((9(9: L !)):): ; LeE$5$567L L LrrcneZdZfdZdej dej dej fdZxZS)RemBertSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNru) rzr{rrrrrrrrrrs r(r{zRemBertSelfOutput.__init__<s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rr input_tensorrcr|j|}|j|}|j||z}|Sr#rrrrrrs r(rzRemBertSelfOutput.forwardB7 =1  ]3 }|'CD rrrs@r(rr;1> U\\RWR^R^rrceZdZfdZdZ d dej deejdeejdeejdeejdee e ejd ee d e ej fd Z xZ S) RemBertAttentionct|t||_t ||_t |_yr#)rzr{rrroutputset pruned_headsrs r(r{zRemBertAttention.__init__Js0 (0 '/ Erc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr)rWrrrrrrrrrrrrunion)rheadsindexs r( prune_headszRemBertAttention.prune_headsQs u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rrrrrrrrrc p|j|||||||}|j|d|} | f|ddz} | S)Nrr)rr) rrrrrrrr self_outputsattention_outputrs r(rzRemBertAttention.forwarddsWyy    ! "    ;;|A F#% QR(88rr) rrrr{rr^rrrrrrrrs@r(rrIs";,7;15=A>BDH,1||!!2!23E--.  ((9(9:  !)):): ; !uU->->'?!@A$D> u|| rrcVeZdZfdZdej dej fdZxZS)RemBertIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr#) rzr{rrrintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrs r(r{zRemBertIntermediate.__init__~s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rrrcJ|j|}|j|}|Sr#)rrrrs r(rzRemBertIntermediate.forwards& =1 00? rrrs@r(rr}s#9U\\ellrrcneZdZfdZdej dej dej fdZxZS) RemBertOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) rzr{rrrrrrrrrrrs r(r{zRemBertOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rrrrcr|j|}|j|}|j||z}|Sr#rrs r(rzRemBertOutput.forwardrrrrs@r(r r rrr ceZdZfdZ d dej deejdeejdeejdeejdeeeejdee d eej fd Z d Z xZ S) RemBertLayercbt||j|_d|_t ||_|j |_|j|_|jr*|j st|dt ||_ t||_ t||_ y)Nrz> should be used as a decoder model if cross attention is added)rzr{chunk_size_feed_forward seq_len_dimr attentionradd_cross_attentionr[crossattentionr intermediater rrs r(r{zRemBertLayer.__init__s '-'E'E$)&1 ++#)#=#=  # #?? D6)g!hii"26":D /7#F+ rrrrrrrrrc ||ddnd}|j|||||} | d} |jr | dd} | d} n| dd} d} |jrT|Rt|dstd|d||d dnd}|j | ||||||}|d} | |ddz} |d}  | z} t |j |j|j| }|f| z} |jr|  fz} | S) Nr>)rrrrrxrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrr[rrfeed_forward_chunkrr)rrrrrrrrself_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputs r(rzRemBertLayer.forwards}:H9S>"1#5Y] !%   /3 "0" 2!4 ??,Qr2G 6r : ,QR0G'+$ ??4@4!12 =dVDDD @N?Yrs(;_c %&*&9&9 %&)!' # 7q9  7" ==G,C2+F ( 14P P 0  # #T%A%A4CSCSUe  /G+ ??!2 44GrcL|j|}|j||}|Sr#)rr)rrintermediate_outputrs r(rzRemBertLayer.feed_forward_chunks,"//0@A{{#68HI rr) rrrr{r^rrrrrrrrrs@r(rrs ,$7;15=A>BDH,1?||?!!2!23?E--. ? ((9(9: ? !)):): ; ?!uU->->'?!@A?$D>? u|| ?DrrceZdZfdZ ddej deejdeejdeejdeejdeeeejdee d e d e d e d e ee ffd Z xZ S)RemBertEncoderc.t|||_tj|j |j |_tjt|jDcgc] }t|c}|_ d|_ ycc}w)NF)rzr{rbrrr~rembedding_hidden_mapping_in ModuleListrangenum_hidden_layersrlayergradient_checkpointing)rrb_rs r(r{zRemBertEncoder.__init__sn  +-99V5P5PRXRdRd+e(]]%H`H`Ba#bQL$8#bc &+#$cs,Brrrrrpast_key_values use_cacheroutput_hidden_states return_dictrc |jr%|jr|rtjdd}|j |}| rdnd} |rdnd} |r|j j rdnd} |rdnd}t|jD]\}}| r| |fz} |||nd}|||nd}|jr/|jr#|j|j|||||||}n||||||||}|d}|r ||dfz }|s|| |dfz} |j j s| |dfz} | r| |fz} | std||| | | fDSt||| | | S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr$rrxrr>c3$K|]}|| ywr#r$)r%vs r(r)z)RemBertEncoder.forward..>s = r4)last_hidden_stater,r attentionscross_attentions) r*trainingrD warning_oncer%rbr enumerater)_gradient_checkpointing_func__call__tupler )rrrrrrr,r-rr.r/all_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_maskr layer_outputss r(rzRemBertEncoder.forwards  & &4==##p" 88G "6BD$5b4%64;;;Z;Zr`d#,R$(4# VOA|#$58H$H!.7.CilO3B3N_Q/TXN**t}} $ A A ))!"#)*"% ! !-!"#)*"%! *!,M"}R'8&::" &9]1=MBEI$("'%* S ||S !!2!23S E--. S ((9(9: S !)):): ; S "%e.?.?(@"ABS D>S  S #S S  u?? @S rr#cVeZdZfdZdej dej fdZxZS)RemBertPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) rzr{rrrrrrrr transform_act_fnrrrs r(r{z'RemBertPredictionHeadTransform.__init__Ts{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrrrcl|j|}|j|}|j|}|Sr#)rrGrr s r(rz&RemBertPredictionHeadTransform.forward]s4 =1 --m< }5 rrrs@r(rErESs$UU\\ellrrEcVeZdZfdZdej dej fdZxZS)RemBertLMPredictionHeadcnt|tj|j|j |_tj|j |j|_t|j|_ tj|j |j|_ yr)rzr{rrroutput_embedding_sizerr}decoderr rrrrrs r(r{z RemBertLMPredictionHead.__init__esz YYv1163O3OP yy!=!=v?P?PQ  !2!23f&B&BH]H]^rrrc|j|}|j|}|j|}|j|}|Sr#)rrrrMr s r(rzRemBertLMPredictionHead.forwardls@ =1  6 }5  ]3 rrrs@r(rJrJds$_U\\ellrrJcVeZdZfdZdej dej fdZxZS)RemBertOnlyMLMHeadcBt|t||_yr#)rzr{rJ predictionsrs r(r{zRemBertOnlyMLMHead.__init__vs 26:rsequence_outputrc(|j|}|Sr#)rR)rrSprediction_scoress r(rzRemBertOnlyMLMHead.forwardzs ,,_=  rrrs@r(rPrPus#;!u||! !rrPc&eZdZdZeZeZdZdZ dZ y)RemBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. rembertTct|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyy)zInitialize the weightsg)meanstdNg?)rrrr7r`normal_rbinitializer_ranger:zero_r|rtrfill_)rmodules r( _init_weightsz$RemBertPreTrainedModel._init_weightss fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).rN) rrrrr config_classrpload_tf_weightsbase_model_prefixsupports_gradient_checkpointingrar$rr(rWrWs$ !L0O!&*#*rrWaJ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`RemBertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a5 Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zaThe bare RemBERT Model transformer outputting raw hidden-states without any specific head on top.c!eZdZdZdfd ZdZdZdZee jde de e  dd eej d eej d eej d eej d eej"deej"deej"deej"deeeej"deedeedeedeedeeeffdZxZS) RemBertModela The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ct||||_t||_t ||_|r t|nd|_|jyr#) rzr{rbrrrr#encoderrpooler post_init)rrbadd_pooling_layerrs r(r{zRemBertModel.__init__sK   +F3%f- /@mF+d  rc.|jjSr#rrrs r(get_input_embeddingsz!RemBertModel.get_input_embeddingss...rc&||j_yr#rn)rrs r(set_input_embeddingsz!RemBertModel.set_input_embeddingss*/'rc|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrir)rr)rheads_to_pruner)rs r( _prune_headszRemBertModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B Crbatch_size, sequence_lengthr  checkpoint output_typerbrrrrwrrrrr,r-rr.r/rc | | n|jj} | | n|jj} | | n|jj} |jjr| | n|jj } nd} | | t d|#|j|||j}n!||jdd}n t d|\}}| |jn |j}| | ddjdnd}|tj|||zf|}|&tj|tj| }|j||}|jjrE|C|j\}}}||f}|tj||}|j!|}nd}|j#||jj$}|j'||||| }|j)|||||| | | | | }|d}|j*|j+|nd}| s ||f|d dzSt-|||j.|j0|j2|j4 S)a  encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timerxz5You have to specify either input_ids or inputs_embedsrr>)rr)rrwrrr) rrrrr,r-rr.r/r)r3 pooler_outputr,rr4r5)rbrr.use_return_dictrr-r[%warn_if_padding_and_no_attention_maskrrrZr^onesrrget_extended_attention_maskinvert_attention_mask get_head_maskr(rrirjrr,rr4r5)rrrrrwrrrrr,r-rr.r/r batch_sizerrrextended_attention_maskencoder_batch_sizeencoder_sequence_lengthr+encoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputsrSrs r(rzRemBertModel.forwardsR2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@TDSC^!3A!6!BEI$(,0/3&*y E,,-y !!1!12y !!1!12 y u//0 y E--. y   1 12y  ((9(9:y !)):): ;y "%e.?.?(@"ABy D>y $D>y 'tny d^y  uBB Cy k y rrgz5RemBERT Model with a `language modeling` head on top.ceZdZdgZfdZdZdZeejde de e  dde ejd e ejd e ejd e ejd e ej d e ej de ej de ej de ejde ede ede edeee ffdZddZxZS)RemBertForMaskedLMcls.predictions.decoder.weightct|||jrtj dt |d|_t||_|jy)NznIf you want to use `RemBertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Frl rzr{rrDwarningrgrXrPr-rkrs r(r{zRemBertForMaskedLM.__init__sR     NN1  $FeD %f- rcB|jjjSr#r-rRrMros r(get_output_embeddingsz(RemBertForMaskedLM.get_output_embeddingsxx##+++rc:||jj_yr#rrnew_embeddingss r(set_output_embeddingsz(RemBertForMaskedLM.set_output_embeddings'5$rrwr rxrrrrwrrrrlabelsrr.r/rc | | n|jj} |j||||||||| | |  } | d}|j|}d}| Ft }||j d|jj | j d}| s|f| ddz}||f|zS|St||| j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. N) rrrwrrrrrr.r/rrxr>losslogitsrr4) rbr}rXr-rrr}rrr4)rrrrrwrrrrrrr.r/rrSrUmasked_lm_lossloss_fctrs r(rzRemBertForMaskedLM.forwards6&1% &.IIr) NNNNNNNNNNNNr#)rrr_tied_weights_keysr{rrrrrVrrrrr^rrrrrrrrrs@r(rrs~:; ,6++C+J+JKh+ij#"$155959371559=A>B-1,0/3&*6 E,,-6 !!1!126 !!1!12 6 u//0 6 E--. 6   1 126  ((9(9:6 !)):): ;6 ))*6 $D>6 'tn6 d^6  un$ %6 k 6 p JrrzIRemBERT Model with a `language modeling` head on top for CLM fine-tuning.c#eZdZdgZfdZdZdZeejde e e  dde ejde ejd e ejd e ejd e ej d e ej d e ej de ej de eeej de ejde ede ede ede edeee ffdZdZxZS)RemBertForCausalLMrct|||jstj dt |d|_t||_|jy)NzOIf you want to use `RemBertForCausalLM` as a standalone, add `is_decoder=True.`Frrrs r(r{zRemBertForCausalLM.__init__sL    NNl m#FeD %f- rcB|jjjSr#rros r(rz(RemBertForCausalLM.get_output_embeddingsrrc:||jj_yr#rrs r(rz(RemBertForCausalLM.set_output_embeddingsrrrw)rzrbrrrrwrrrrr,rr-rr.r/rc ||n|jj}|j||||||||| | | | | }|d}|j|}d}| *|j|| fd|jj i|}|s|f|ddz}||f|zS|St |||j|j|j|jS)a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Returns: Example: ```python >>> from transformers import AutoTokenizer, RemBertForCausalLM, RemBertConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/rembert") >>> config = RemBertConfig.from_pretrained("google/rembert") >>> config.is_decoder = True >>> model = RemBertForCausalLM.from_pretrained("google/rembert", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```N) rrrwrrrrr,r-rr.r/rr}r>)rrr,rr4r5) rbr}rXr- loss_functionr}rr,rr4r5)rrrrrwrrrrr,rr-rr.r/kwargsrrSrUlm_lossrs r(rzRemBertForCausalLM.forwardsz&1%.os.rU_j--aZ=N=N1OPrs58r>)r;)rr,rreordered_past layer_pasts ` r(_reorder_cachez!RemBertForCausalLM._reorder_cacheksT) J rcmnpopcqrrQR.! N  r)NNNNNNNNNNNNNN)rrrrr{rrrrrVrrrrr^rrrrrrrrrs@r(rrs;; ,6++C+J+JKh+ij+L[jk155959371559=A>BEI-1$(,0/3&*d E,,-d !!1!12d !!1!12 d u//0 d E--. d   1 12d  ((9(9:d !)):): ;d "%e.?.?(@"ABd ))*d D>d $D>d 'tnd d^d " u77 8#d lkd Lrrz RemBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZeej dedee  dde e jde e jde e jde e jd e e jd e e jd e e jd e ed e ede edeeeffdZxZS) RemBertForSequenceClassificationc,t|||j|_t||_t j |j|_t j|j|j|_ |jyr# rzr{ num_labelsrgrXrrclassifier_dropout_probrrrr=rkrs r(r{z)RemBertForSequenceClassification.__init__}si   ++#F+ zz&"@"@A ))F$6$68I8IJ rrwr rxrrrrwrrrrr.r/rc @| | n|jj} |j|||||||| |  } | d} |j| } |j | } d}||jj |j dk(rd|j_nl|j dkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt}|j dk(r&|| j|j}n|| |}n|jj dk(r=t}|| jd|j |jd}n,|jj dk(rt}|| |}| s| f| ddz}||f|zS|St!|| | j"| j$ S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrrwrrrr.r/r regressionsingle_label_classificationmulti_label_classificationrxr>r)rbr}rXrr= problem_typerrr^rrXr squeezerrrrrr4)rrrrrwrrrrr.r/rrrrrrs r(rz(RemBertForSequenceClassification.forwards2&1%E 'tnE d^E  u.. /E k E rrz RemBERT Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. ceZdZfdZeej dedee  dde e jde e jde e jde e jd e e jd e e jd e e jd e ed e ede edeeeffdZxZS)RemBertForMultipleChoicect||t||_t j |j |_t j|jd|_ |jy)Nr) rzr{rgrXrrrrrrr=rkrs r(r{z!RemBertForMultipleChoice.__init__sV  #F+ zz&"@"@A ))F$6$6: rz(batch_size, num_choices, sequence_lengthr rxrrrrwrrrrr.r/rc L| | n|jj} ||jdn|jd} |!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|1|jd|j d|j dnd}|j |||||||| |  } | d} |j | } |j| }|jd| }d}|t}|||}| s|f| ddz}||f|zS|St||| j| jS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrrxrrr>r) rbr}rZrrrXrr=rrrr4)rrrrrwrrrrr.r/ num_choicesrrrreshaped_logitsrrrs r(rz RemBertForMultipleChoice.forwards2&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  ,, ))%'/!5#     ]3 / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   rr)rrrr{rrrVrrrrr^rrrrrrrrs@r(rrsB++C+J+JKu+vw#-$266:59481559-1,0/3&*@ E--.@ !!2!23@ !!1!12 @ u001 @ E--. @   1 12@ ))*@ $D>@ 'tn@ d^@  u// 0@ x @ rrz RemBERT Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. ceZdZfdZeej dedee  dde e jde e jde e jde e jd e e jd e e jd e e jd e ed e ede edeeeffdZxZS)RemBertForTokenClassificationc0t|||j|_t|d|_t j |j|_t j|j|j|_ |jyNFrrrs r(r{z&RemBertForTokenClassification.__init__8sk   ++#FeD zz&"@"@A ))F$6$68I8IJ rrwr rxrrrrwrrrrr.r/rc | | n|jj} |j|||||||| |  } | d} |j| } |j | } d}|r) rbr}rXrr=rrrrrr4)rrrrrwrrrrr.r/rrSrrrrs r(rz%RemBertForTokenClassification.forwardCs.&1%2 'tn2 d^2  u++ ,2 k 2 rrz RemBERT Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZfdZeej dedee  dde e jde e jde e jde e jd e e jd e e jd e e jd e e jd e ede ede edeeeffdZxZS)RemBertForQuestionAnsweringct|||j|_t|d|_t j |j|j|_|jyr) rzr{rrgrXrrr qa_outputsrkrs r(r{z$RemBertForQuestionAnswering.__init__sU   ++#FeD ))F$6$68I8IJ rrwr rxrrrrwrrstart_positions end_positionsrr.r/rc | | n|jj} |j||||||| | |  } | d} |j| }|j dd\}}|j d}|j d}d}||t |jdkDr|j d}t |jdkDr|j d}|jd}|jd||jd|t|}|||}|||}||zdz }| s||f| ddz}||f|zS|St|||| j| j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrrrxr) ignore_indexr>)r start_logits end_logitsrr4) rbr}rXrrQrrWrclamp_rrrr4)rrrrrwrrrrrr.r/rrSrrr total_loss ignored_indexr start_lossend_lossrs r(rz#RemBertForQuestionAnswering.forwards<&1%H 'tnH d^H  u22 3H k H rr) rrrrrrrrgrWrp)GrrrFtypingrrrr^torch.utils.checkpointrtorch.nnrrr activationsr generationr modeling_outputsr rrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrrrrconfiguration_rembertr get_loggerrrDr_CHECKPOINT_FOR_DOCrpModulerrrrrrrr rr#rErJrPrWREMBERT_START_DOCSTRINGrrgrrrrrr__all__r$rr(rs )) AA!)   .ll1   H %!&Pf3 3n BII f299fT  0ryy0h "))  BII U299Up\ RYY\ @RYY" bii "!!*_*8 /dgf )f f RQSjkbJ/bJlbJJSUlD/DDN V '=V V r Q 5Q Q h D $:D D N Z "8Z Z z r