S^(hH3$dZddlmZddlZddlZddlmZddlmZm Z m Z ddl Z ddl ZddlmZddlmZmZmZmZmZmZdd lmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$dd l%m&Z&m'Z'm(Z(dd l)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/d d l0m1Z1e.jde3Z4dZ5dZ6Gdde"jnjpZ9Gdde"jnjpZ:Gdde"jnjpZ;Gdde"jnjpZ<Gdde"jnjpZ=Gdde"jnjpZ>Gdde"jnjpZ?Gdde"jnjpZ@Gd d!e"jnjpZAGd"d#e"jnjpZBGd$d%e"jnjpZCGd&d'eZDe#Gd(d)e"jnjpZEeGd*d+e*ZFd,ZGd-ZHe,d.eGGd/d0eDZIe,d1eGGd2d3eDZJGd4d5e"jnjpZKe,d6eGGd7d8eDeZLGd9d:e"jnjpZMe,d;eGGd<d=eDeZNe,d>eGGd?d@eDeZOe,dAeGGdBdCeDe ZPe,dDeGGdEdFeDeZQgdGZRy)HzTF Electra model.) annotationsN) dataclass)OptionalTupleUnion)get_tf_activation)+TFBaseModelOutputWithPastAndCrossAttentionsTFMaskedLMOutputTFMultipleChoiceModelOutputTFQuestionAnsweringModelOutputTFSequenceClassifierOutputTFTokenClassifierOutput) TFMaskedLanguageModelingLossTFModelInputTypeTFMultipleChoiceLossTFPreTrainedModelTFQuestionAnsweringLossTFSequenceClassificationLossTFSequenceSummaryTFTokenClassificationLossget_initializerkeraskeras_serializable unpack_inputs)check_embeddings_within_bounds shape_liststable_softmax) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) ElectraConfigz"google/electra-small-discriminatorr&c^eZdZdfd ZddZ d ddZd dZxZS) TFElectraSelfAttentionc t|d i||j|jzdk7r&t d|jd|jd|j|_t |j|jz |_|j|j z|_tj|j |_ tjj|jt|jd|_tjj|jt|jd|_tjj|jt|jd|_tjj'|j( |_|j,|_||_y) NrzThe hidden size (z6) is not a multiple of the number of attention heads ()queryunitskernel_initializernamekeyvaluerate)super__init__ hidden_sizenum_attention_heads ValueErrorintattention_head_size all_head_sizemathsqrtsqrt_att_head_sizerlayersDenserinitializer_ranger+r0r1Dropoutattention_probs_dropout_probdropout is_decoderconfigselfrGkwargs __class__s m/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/electra/modeling_tf_electra.pyr6zTFElectraSelfAttention.__init__Fs "6"    : : :a ?#F$6$6#78''-'A'A&B!E  $*#=#= #&v'9'9F ."<&q)I(+K3N 11$((BW(2XZdeI33DJJF[J4\^hiK3N  '11$((-(2PR\]I33DJJmJ4TV`aK >!#4i"@qII))^A%6 $D1MK11$((-(2PR\]I33DJJmJ4TV`aK//0A:N ??(5N99[)N WWT,,4D4J4J K99%5r:  %!vv&6G )0@rJ,,o,Q   kk/9EO99_kB<<(8|L::-=jRTVZVhVhEij9J#_5QaPc ?? 11GrMc|jryd|_t|dddtj|jj 5|jj dd|jjgdddt|dddtj|jj 5|jj dd|jjgdddt|ddetj|jj 5|jj dd|jjgdddyy#1swYxYw#1swYxYw#1swYyxYw)NTr+r0r1) builtgetattrrW name_scoper+r/buildrGr7r0r1rI input_shapes rLrzTFElectraSelfAttention.builds9 ::  4$ ' 3tzz/ H   $dkk.E.E!FG H 4 % 1txx}}- FdDKK,C,CDE F 4$ ' 3tzz/ H   $dkk.E.E!FG H H 4  H H F F H Hs$ 3E*<3E6-3F*E36E?F rGr&)rQ tf.TensorrZr:returnrF)rmrrnrrorrprrqrrrTuple[tf.Tensor]rsboolrfrrrN)__name__ __module__ __qualname__r6r[r~r __classcell__rKs@rLr(r(Es87 O O"O O ) O !* O)O OO Ob HrMr(c2eZdZdfd ZdddZddZxZS)TFElectraSelfOutputc xt|di|tjj |j t |jd|_tjj|jd|_ tjj|j|_||_yNdenser, LayerNormepsilonr/r2r4r5r6rr@rAr7rrBrLayerNormalizationlayer_norm_epsrrChidden_dropout_probrErGrHs rLr6zTFElectraSelfOutput.__init__ "6"\\''$$IaIa9bip(  88AVAV]h8i||++1K1K+L  rMcz|j|}|j||}|j||z}|SNr]rerrErrIrm input_tensorrfs rLr~zTFElectraSelfOutput.call? - 8  MH M ml.JK rMc"|jryd|_t|dddtj|jj 5|jj dd|jjgdddt|ddetj|jj 5|jj dd|jjgdddyy#1swY|xYw#1swYyxYwNTrr) rrrWrrr/rrGr7rrs rLrzTFElectraSelfOutput.builds ::  4$ ' 3tzz/ H   $dkk.E.E!FG H 4d + 7t~~223 L$$dD$++2I2I%JK L L 8 H H L L 3C9<3D9DDrrrmrrrrfrrrrrrrr6r~rrrs@rLrr LrMrc\eZdZdfd ZdZ d ddZddZxZS) TFElectraAttentionc lt|di|t|d|_t |d|_y)NrIr/outputr4)r5r6r(self_attentionr dense_outputrHs rLr6zTFElectraAttention.__init__s1 "6"4V&I/XFrMctrNotImplementedError)rIheadss rL prune_headszTFElectraAttention.prune_headss!!rMc x|j||||||||} |j| d||} | f| ddz} | S)Nrmrnrorprqrrrsrfrrmrrfr%)rr) rIrrnrorprqrrrsrf self_outputsr|r}s rLr~zTFElectraAttention.callso**&)"7#9)/+   ,,&q/ x- $% QR(88rMc|jryd|_t|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swYexYw#1swYyxYw)NTrr)rrrWrrr/rrrs rLrzTFElectraAttention.builds ::  4)4 0 <t22778 0##))$/ 0 4 . :t00556 .!!''- . . ; 0 0 . . C %C CC rr)rrrnrrorrprrqrrrrrsrrfrrrr)rrrr6rr~rrrs@rLrrsuG ""  )  !* )  : .rMrc0eZdZdfd ZddZddZxZS)TFElectraIntermediatec Tt|di|tjj |j t |jd|_t|jtr"t|j|_ ||_y|j|_ ||_y)Nrr,r4)r5r6rr@rAintermediate_sizerrBr isinstance hidden_actstrr intermediate_act_fnrGrHs rLr6zTFElectraIntermediate.__init__s "6"\\''**vOgOg?hov(   f'' -'89J9J'KD $ (.'8'8D $ rMcL|j|}|j|}|S)Nr])rr)rIrms rLr~zTFElectraIntermediate.call+s( - 8 00? rMc(|jryd|_t|ddetj|jj 5|jj dd|jjgdddyy#1swYyxYwNTr rrrWrrr/rrGr7rs rLrzTFElectraIntermediate.build1} ::  4$ ' 3tzz/ H   $dkk.E.E!FG H H 4 H H 3BBrrmrrrrrrs@rLrrs  HrMrc2eZdZdfd ZdddZddZxZS)TFElectraOutputc xt|di|tjj |j t |jd|_tjj|jd|_ tjj|j|_||_yrrrHs rLr6zTFElectraOutput.__init__<rrMcz|j|}|j||}|j||z}|Srrrs rLr~zTFElectraOutput.callFrrMc"|jryd|_t|dddtj|jj 5|jj dd|jjgdddt|ddetj|jj 5|jj dd|jjgdddyy#1swY|xYw#1swYyxYwr) rrrWrrr/rrGrrr7rs rLrzTFElectraOutput.buildMs ::  4$ ' 3tzz/ N   $dkk.K.K!LM N 4d + 7t~~223 L$$dD$++2I2I%JK L L 8 N N L Lrrrrrrrs@rLrr;rrMrcVeZdZdfd Z d ddZddZxZS)TFElectraLayerc Dt|di|t|d|_|j|_|j |_|j r,|jst |dt|d|_t|d|_ t|d|_ y)N attentionrz> should be used as a decoder model if cross attention is addedcrossattention intermediaterr4) r5r6rrrFadd_cross_attentionr9rrrr bert_outputrHs rLr6zTFElectraLayer.__init__[s "6"+FE ++#)#=#=  # #?? D6)g!hii"4VBR"SD 1&~N*6ArMc ||ddnd} |j|||dd| ||} | d} |jr | dd} | d} n| dd} d}|jrV|Tt|dstd|d||d dnd}|j | |||||||}|d} | |ddz} |d} |z} |j | }|j || | }|f| z} |jr|  fz} | S) NrT)rrnrorprqrrrsrfrr%rOrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`)rmr)rrFhasattrr9rrr)rIrmrnrorprqrrrsrfself_attn_past_key_valueself_attention_outputsr|r}present_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputsintermediate_output layer_outputs rLr~zTFElectraLayer.callhs:H9S>"1#5Y] !%&)"&#'3/"0 " 2!4 ??,Qr2G 6r : ,QR0G'+$ ??4@4!12 =dVDDD @N?Yrs(;_c %&*&9&9--#&;'=8"3!': ' # 7q9  7" ==G,C2+F ( 14P P "//>N/O''-z(TFElectraEncoder.call..sghgts)last_hidden_statepast_key_valuesrm attentionscross_attentions)rGr enumeratertupler )rIrmrnrorprqr use_cachersoutput_hidden_states return_dictrfall_hidden_statesall_attentionsall_cross_attentionsnext_decoder_cacher layer_modulerr layer_outputss rLr~zTFElectraEncoder.callsV#7BD0d%64;;;Z;Zr`d#,R$(4 VOA|#$58H$H!3B3N_Q/TXN(+-#A,&;'=-"3! M*!,M"}R'8&::" !/=3C2E!E;;227L7X+?=QRCSBU+U(1 V6 1]4D D )+| | td|At||jjt j |j |}t|dd}|t j|d}|2t jt j||d|zd }t j |j|}t j |j|} ||z| z} |j| } |j| | } | S) z Applies embedding based on inputs tensor. Returns: final_embeddings (`tf.Tensor`): output embedding tensor. Nz5Need to provide either `input_ids` or `input_embeds`.)paramsindicesrOrdimsr1r%)startlimitr_r]re)r9rrGrrWgatherrrfill expand_dimsrrrrrE) rI input_ids position_idstoken_type_ids inputs_embedspast_key_values_lengthrfrposition_embedstoken_type_embedsfinal_embeddingss rLr~zTFElectraEmbeddings.callZs   !6TU U  *9dkk6L6L MIIT[[)LM /4  !WW+Q?N  >>5[^Nd=delmL))4+C+C\ZIIT-G-GQ_`(?:=NN>>1A>B<? A)$++*@*@A-P D11-@"E rMc"|jryd|_t|dddtj|jj 5|jj dd|jjgdddt|ddetj|jj 5|jj dd|jjgdddyy#1swY|xYw#1swYyxYw)NTrr4) rrrWrrr/rrGr7r4rs rLrz'TFElectraDiscriminatorPredictions.builds ::  4$ ' 3tzz/ H   $dkk.E.E!FG H 4+T 2 >t4499: S%%++T49P9P,QR S S ? H H S Srrrrrs@rLr2r2s SrMr2c.eZdZfdZddZddZxZS)TFElectraGeneratorPredictionsc t|di|tjj |j d|_tjj|jd|_ ||_ y)Nrrrrr4) r5r6rr@rrrrArrrGrHs rLr6z&TFElectraGeneratorPredictions.__init__s] "6"88AVAV]h8i\\''(=(=G'L  rMcl|j|}td|}|j|}|S)Ngelu)rr r)rIgenerator_hidden_statesrfrms rLr~z"TFElectraGeneratorPredictions.calls7 #:; 1)&1-@ }5 rMc"|jryd|_t|dddtj|jj 5|jj dd|jjgdddt|ddetj|jj 5|jj dd|jjgdddyy#1swY|xYw#1swYyxYw)NTrr) rrrWrrr/rrGrrr7rs rLrz#TFElectraGeneratorPredictions.builds ::  4d + 7t~~223 O$$dD$++2L2L%MN O 4$ ' 3tzz/ H   $dkk.E.E!FG H H 4 O O H Hrrrrrs@rLr:r:s HrMr:c$eZdZdZeZdZdgZdgZy)TFElectraPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. electrazgenerator_lm_head.weightrEN) rrrr0r& config_classbase_model_prefix"_keys_to_ignore_on_load_unexpected_keys_to_ignore_on_load_missingr4rMrLrArAs% !L!*E)F&'1l#rMrAceZdZeZfdZdZdZdZd dZ dZ e d d dZ d dZ xZS) TFElectraMainLayerc .t|di|||_|j|_t |d|_|j |jk7r0tjj|jd|_ t|d|_ y)Nrrembeddings_projectencoderr4)r5r6rGrFrrrr7rr@rArJrrKrHs rLr6zTFElectraMainLayer.__init__sz "6"  ++-f<H  F$6$6 6&+ll&8&89K9KRf&8&gD #'Y? rMc|jSr)rrIs rLget_input_embeddingsz'TFElectraMainLayer.get_input_embeddingss rMc`||j_t|d|j_yNr)rrrrrIr1s rLset_input_embeddingsz'TFElectraMainLayer.set_input_embeddingss$!&%/%6q%9"rMct)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel r)rIheads_to_prunes rL _prune_headszTFElectraMainLayer._prune_headss "!rMc 2|\}}|tj|||zfd}t|}||z}|jrtj|} tj tj | ddddf||df| ddddf} tj| |j} | |dddddfz} t| }tj| |dd|d|df} |dkDr3| dddd| dddf} n tj||ddd|df} tj| |} tjd|} tjd|} tjtj| | | } | S)Nr%r rbrrT?) rWr%rrFr less_equaltilerircrXconstantrlsubtract)rIrnrrcr+rZ seq_lengthattention_mask_shapemask_seq_lengthseq_ids causal_maskextended_attention_maskone_cstten_thousand_csts rLget_extended_attention_maskz.TFElectraMainLayer.get_extended_attention_masks!, J  !WW:zDZ7Z*[cdeN *.9$'== ??hh/G--dA ._a0PQa &K''+^5I5IJK&1N1dA:4N&N ##-.E#F &(jj'*>q*A1FZ[\F]_stu_v)w' #&)*A!Q VWBW*X'&(jj!5a!8!Q@TUV@W X' ##%''*A"O++c/;;xu="$++bkk'CZ.[]m"n&&rMcJ|tdg|jjz}|Sr)rrGr)rIros rL get_head_maskz TFElectraMainLayer.get_head_masks*  % %!>!>>IrMc |jjsd} | | td| t|}n|t|dd}n td|\}}| &d}dgt |j j z} nt| ddd}|tj|||zfd}|tj|d}|j|||||| }|j|||j|}|jrf|dtj||j }t t|}|d k(r|dddddddf}|d k(r |ddddddf}d z dz}nd}|j|}t|dr|j||}|j |||||| | | | | | }|S)NFzDYou cannot specify both input_ids and inputs_embeds at the same timerOz5You have to specify either input_ids or inputs_embedsrrr%r )r'r(r)r*r+rfrbrrTrWrXrJrf) rmrnrorprqrrrsrrrf)rGrFr9rlenrKrrWr%rrercrirgrrJ)rIr'rnr)r(ror*rprqrrrsrrrfrrZr]r+rmrbnum_dims_encoder_attention_maskencoder_extended_attention_masks rLr~zTFElectraMainLayer.calls'${{%%I  ]%>cd d  "$Y/K  &$]3CR8KTU U!, J  "%& "#fs4<<+=+='>>O%/0B10E%Fr%J "  !WW:zDZ7Z*[cdeN  !WW+Q?N%)'#9 ( #'"B"B K)<)<>T#  ??5A&(WW-CKbKhKh%i ".1*=S2T.U +.!32HDRSUV2W/.!32HDRVXYIY2Z/035T/TX`.` +.2 +&&y1 4- . 33MH3UM '2"7#B+/!5#%  rMc|jryd|_t|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddetj|jj 5|jj dd|jjgdddyy#1swYxYw#1swYxYw#1swYyxYw)NTrrKrJ) rrrWrrr/rrKrJrGrrs rLrzTFElectraMainLayer.builds ::  4t , 8t334 ,%%d+ , 4D ) 5t||001 ) ""4( ) 4-t 4 @t66;;< X''--tT4;;;U;U.VW X X A  , , ) ) X X$ D<%E?3E<EEE)rNNNNNNNNNNNNNFr'TFModelInputType | Nonernnp.ndarray | tf.Tensor | Noner)rrr(rrrorrr*rrrprrrqrrrz4Optional[Tuple[Tuple[Union[np.ndarray, tf.Tensor]]]]rrrsrrrrrrfrrrr)rrrr&rCr6rNrRrUrergrr~rrrs@rLrHrHs L @:"/'b.28<8<6:377;?C@DPT$(,0/3&*#(_*_6_6 _ 4 _ 1 _5_ =_!>_N_"_*_-_$_!_ N!__B XrMrHc<eZdZUdZdZded<dZded<dZded<y)TFElectraForPreTrainingOutputa Output type of [`TFElectraForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `tf.Tensor` of shape `(1,)`): Total loss of the ELECTRA objective. logits (`tf.Tensor` of shape `(batch_size, sequence_length)`): Prediction scores of the head (scores for each token before SoftMax). hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr/rdrrmr)rrrr0rd__annotations__rmrr4rMrLrtrts**#'F &-1M*1*.J'.rMrta} This model inherits from [`TFPreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a [keras.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. TensorFlow models and layers in `transformers` accept two formats as input: - having all inputs as keyword arguments (like PyTorch models), or - having all inputs as a list, tuple or dict in the first positional argument. The reason the second format is supported is that Keras methods prefer this format when passing inputs to models and layers. Because of this support, when using methods like `model.fit()` things should "just work" for you - just pass your inputs and labels in any format that `model.fit()` supports! If, however, you want to use the second format outside of Keras methods like `fit()` and `predict()`, such as when creating your own layers or models with the Keras `Functional` API, there are three possibilities you can use to gather all the input Tensors in the first positional argument: - a single Tensor with `input_ids` only and nothing else: `model(input_ids)` - a list of varying length with one or several input Tensors IN THE ORDER given in the docstring: `model([input_ids, attention_mask])` or `model([input_ids, attention_mask, token_type_ids])` - a dictionary with one or several input Tensors associated to the input names given in the docstring: `model({"input_ids": input_ids, "token_type_ids": token_type_ids})` Note that when creating models and layers with [subclassing](https://keras.io/guides/making_new_layers_and_models_via_subclassing/) then you don't need to worry about any of this, as you can just pass inputs like you would to any other Python function! Parameters: config ([`ElectraConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`Numpy array` or `tf.Tensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`Numpy array` or `tf.Tensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`Numpy array` or `tf.Tensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`Numpy array` or `tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`tf.Tensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. This argument can be used only in eager mode, in graph mode the value in the config will be used instead. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. This argument can be used in eager mode, in graph mode the value will always be set to True. training (`bool`, *optional*, defaults to `False`): Whether or not to use the model in training mode (some modules like dropout modules have different behaviors between training and evaluation). a]The bare Electra Model transformer outputting raw hidden-states without any specific head on top. Identical to the BERT model except that it uses an additional linear layer between the embedding layer and the encoder if the hidden size and embedding size are different. Both the generator and discriminator checkpoints may be loaded into this model.ceZdZfdZeeejdee e e  d ddZ ddZ xZS) TFElectraModelcPt||g|i|t|d|_y)NrBr)r5r6rHrBrIrGr^rJrKs rLr6zTFElectraModel.__init__s( 3&3F3)&yA rMbatch_size, sequence_length checkpoint output_typerCcD|j|||||||| | || | | |}|S)a encoder_hidden_states (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`Tuple[Tuple[tf.Tensor]]` of length `config.n_layers`) contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*, defaults to `True`): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Set to `False` during training, `True` during generation )r'rnr)r(rorprqrrr*rsrrrf)rB)rIr'rnr)r(ror*rprqrrrsrrrfr}s rLr~zTFElectraModel.callsIX,,))%"7#9+'/!5# "rMc|jryd|_t|ddNtj|jj 5|jj ddddyy#1swYyxYw)NTrB)rrrWrrBr/rrs rLrzTFElectraModel.buildRsi ::  4D ) 5t||001 ) ""4( ) ) 6 ) )s A11A:rorpr)rrrr6rr"ELECTRA_INPUTS_DOCSTRINGformatr _CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCr~rrrs@rLrwrwsB *+C+J+JKh+ij&?$.28<8<6:377;?C@DPT$(,0/3&*#(6*6666 6 4 6 1 656 =6!>6N6"6*6-6$6!6 N!6 k6p)rMrwaH Electra model with a binary classification head on top as used during pretraining for identifying generated tokens. Even though both the discriminator and generator may be loaded into this model, the discriminator is the only model of the two to have the correct classification head to be used for this model. ceZdZfdZeeejdee e  d ddZ ddZ xZ S) TFElectraForPreTrainingc nt||fi|t|d|_t |d|_y)NrBrdiscriminator_predictions)r5r6rHrBr2rrHs rLr6z TFElectraForPreTraining.__init__es3 *6*)&yA )J6Xs)t&rMrz)r}rCc |j||||||||| |  } | d} |j| } | s | f| ddzSt| | j| jS)a! Returns: Examples: ```python >>> import tensorflow as tf >>> from transformers import AutoTokenizer, TFElectraForPreTraining >>> tokenizer = AutoTokenizer.from_pretrained("google/electra-small-discriminator") >>> model = TFElectraForPreTraining.from_pretrained("google/electra-small-discriminator") >>> input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :] # Batch size 1 >>> outputs = model(input_ids) >>> scores = outputs[0] ``` r'rnr)r(ror*rsrrrfrr%N)rdrmr)rBrrtrmr)rIr'rnr)r(ror*rsrrrfr7discriminator_sequence_outputrds rLr~zTFElectraForPreTraining.callks>'+ll))%'/!5#'3 ' #)DA(F%//0MN9:12>> >,5CC2==  rMc|jryd|_t|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swYexYw#1swYyxYw)NTrBr)rrrWrrBr/rrrs rLrzTFElectraForPreTraining.builds ::  4D ) 5t||001 ) ""4( ) 44d ; Gt==BBC ;..44T: ; ; H ) ) ; ;r) NNNNNNNNNF)r'rqrnrrr)rrr(rrrorrr*rrrsrrrrrrfrrz6Union[TFElectraForPreTrainingOutput, Tuple[tf.Tensor]]r)rrrr6rr"rrr$rtrr~rrrs@rLrr[su *+C+J+JKh+ij+HWfg.28<8<6:377;,0/3&*#(2 *2 62 6 2 4 2 1 2 52 *2 -2 $2 !2  @2 hk2 h ;rMrcFeZdZfdZfdZdZdZdZdZdZ xZ S)TFElectraMaskedLMHeadc bt|di|||_|j|_||_y)Nr4)r5r6rGrinput_embeddings)rIrGrrJrKs rLr6zTFElectraMaskedLMHead.__init__s0 "6" $33 0rMc|j|jjfddd|_t||y)NzerosTbias)rRr trainabler/)rrGrrr5r)rIrrKs rLrzTFElectraMaskedLMHead.builds7OO4;;+A+A*CQXdhouOv   k"rMc|jSr)rrMs rLget_output_embeddingsz+TFElectraMaskedLMHead.get_output_embeddingss$$$rMc`||j_t|d|j_yrP)rrrrrQs rLset_output_embeddingsz+TFElectraMaskedLMHead.set_output_embeddingss(',$+5e+rMrz Electra model with a language modeling head on top. Even though both the discriminator and generator may be loaded into this model, the generator is the only model of the two to have been trained for the masked language modeling task. c eZdZfdZdZdZeeejde de e ddd  d d d Z dd ZxZS)TFElectraForMaskedLMc Vt||fi|||_t|d|_t |d|_t|jtrt|j|_ n|j|_ t||jjd|_y)NrBrgenerator_predictionsgenerator_lm_head)r5r6rGrHrBr:rrrrr r rrrrHs rLr6zTFElectraForMaskedLM.__init__s *6* )&yA %B6Pg%h" f'' -/0A0ABDO$//DO!6vt||?V?V]p!qrMc|jSr)rrMs rL get_lm_headz TFElectraForMaskedLM.get_lm_heads%%%rMctjdt|jdz|jjzS)NzMThe method get_prefix_bias_name is deprecated. Please use `get_bias` instead./)warningswarn FutureWarningr/rrMs rLget_prefix_bias_namez)TFElectraForMaskedLM.get_prefix_bias_names1 egtuyy3!7!7!generator_sequence_outputprediction_scoresrrs rLr~zTFElectraForMaskedLM.calls<#',,))%'/!5##/ # %! 667P[c6d 223Dx2X~t4+?+?HY+Z'),CAB,GGF)-)9TGf$ Ev E$1??.99   rMc|jryd|_t|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swYxYw#1swYqxYw#1swYyxYw)NTrBrr) rrrWrrBr/rrrrs rLrzTFElectraForMaskedLM.build-s  ::  4D ) 5t||001 ) ""4( ) 40$ 7 Ct99>>? 7**006 7 4,d 3 ?t55::; 3&&,,T2 3 3 @  ) ) 7 7 3 3s$ D%%D1?D=%D.1D:=E NNNNNNNNNNF)r'rqrnrrr)rrr(rrrorrr*rrrsrrrrrrrrrfrrz)Union[TFMaskedLMOutput, Tuple[tf.Tensor]]r)rrrr6rrrr"rrr r rr~rrrs@rLrrs r&=*+C+J+JKh+ij3$$ ! .28<8<6:377;,0/3&*04#(/ */ 6/ 6 / 4 / 1 / 5/ */ -/ $/ ./ !/  3/ k/ b 3rMrc0eZdZdZfdZdZddZxZS)TFElectraClassificationHeadz-Head for sentence-level classification tasks.c t|di|tjj |j t |jd|_|j |jn |j}tjj||_ tjj |jt |jd|_||_y)Nrr.r/out_projr4)r5r6rr@rAr7rrBrclassifier_dropout%classifhidden_dropout_probier_dropoutrrCrE num_labelsrrGrIrGrJrrKs rLr6z$TFElectraClassificationHead.__init__?s "6"\\''   ?6C[C[3\cj(  ((4  8 8++  ||++,>?  **   /&BZBZ2[bl+   rMc |dddddf}|j|}|j|}td|}|j|}|j|}|S)Nrr=)rErr r)rIr^rJxs rLr~z TFElectraClassificationHead.callPs] 1a7O LLO JJqM % f %a ( LLO MM! rMc"|jryd|_t|dddtj|jj 5|jj dd|jjgdddt|ddetj|jj 5|jj dd|jjgdddyy#1swY|xYw#1swYyxYw)NTrr) rrrWrrr/rrGr7rrs rLrz!TFElectraClassificationHead.buildZs ::  4$ ' 3tzz/ H   $dkk.E.E!FG H 4T * 6t}}112 K ##T41H1H$IJ K K 7 H H K Krr)rrrr0r6r~rrrs@rLrr<s7" KrMrz ELECTRA Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c eZdZfdZeeejdede e dd d d dZ d dZ xZ S) "TFElectraForSequenceClassificationct||g|i||j|_t|d|_t |d|_y)NrBr classifier)r5r6rrHrBrrrys rLr6z+TFElectraForSequenceClassification.__init__nsC 3&3F3 ++)&yA 5f<PrMrzz$bhadresh-savani/electra-base-emotionz'joy'gQ?r|r}rCrrc |j||||||||| |  } |j| d} | dn|j| | }| s| f| ddz}||f|zS|St|| | j| j S)a labels (`tf.Tensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rrNr%r)rBrrrrmr)rIr'rnr)r(ror*rsrrrrfr}rdrrs rLr~z'TFElectraForSequenceClassification.callts:,,))%'/!5#  ,~t4+?+?+OY,F)-)9TGf$ Ev E)!//))   rMc|jryd|_t|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swYexYw#1swYyxYwNTrBr)rrrWrrBr/rrrs rLrz(TFElectraForSequenceClassification.builds ::  4D ) 5t||001 ) ""4( ) 4t , 8t334 ,%%d+ , , 9 ) ) , ,rr)r'rqrnrrr)rrr(rrrorrr*rrrsrrrrrrrrrfrrz3Union[TFSequenceClassifierOutput, Tuple[tf.Tensor]]r)rrrr6rr"rrr rrr~rrrs@rLrrfsQ *+C+J+JKh+ij9.$ .28<8<6:377;,0/3&*04#(- *- 6- 6 - 4 - 1 - 5- *- -- $- .- !-  =- k- ^ ,rMrz ELECTRA Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. ceZdZfdZeeejdee e e  d ddZ ddZ xZS) TFElectraForMultipleChoicec t||g|i|t|d|_t ||j d|_tjjdt|j d|_ ||_ y)NrBrsequence_summary)rBr/r%rr) r5r6rHrBrrBrrr@rArrrGrys rLr6z#TFElectraForMultipleChoice.__init__s| 3&3F3)&yA 1 f&>&>EW!  ,,,, /&2J2J"KR^-  rMz(batch_size, num_choices, sequence_lengthr{c |t|d} t|d} nt|d} t|d} |tj|d| fnd}|tj|d| fnd}|tj|d| fnd}|tj|d| fnd}|%tj|d| t|dfnd}|j||||||||| |  }|j |d}|j |}tj|d| f}| dn|j | |}| s|f|ddz}||f|zS|St|||j|jS) a5 labels (`tf.Tensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr%rTrOrrrr) rrWrXrBrrrr rmr)rIr'rnr)r(ror*rsrrrrf num_choicesr]flat_input_idsflat_attention_maskflat_token_type_idsflat_position_idsflat_inputs_embedsr}rdreshaped_logitsrrs rLr~zTFElectraForMultipleChoice.calls6  $Y/2K#I.q1J$]3A6K#M215JDMDYIJ/?@_cN\Nhbjj"j9IJnrN\Nhbjj"j9IJnrJVJbBJJ|b*5EFhl( JJ}r:z-7PQR7S&T U  ,,$..*,/!5#  &&wqz2(**Vb+->?~t4+?+?+X%''!"+5F)-)9TGf$ Ev E*"!//))   rMc|jryd|_t|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddetj|jj 5|jj dd|jjgdddyy#1swYxYw#1swYxYw#1swYyxYw)NTrBrr) rrrWrrBr/rrrrGr7rs rLrz TFElectraForMultipleChoice.builds ::  4D ) 5t||001 ) ""4( ) 4+T 2 >t4499: 2%%++D1 2 4t , 8t334 M%%tT4;;3J3J&KL M M 9  ) ) 2 2 M Mrnr)r'rqrnrrr)rrr(rrrorrr*rrrsrrrrrrrrrfrrz4Union[TFMultipleChoiceModelOutput, Tuple[tf.Tensor]]r)rrrr6rr"rrr rr rr~rrrs@rLrrs *+C+J+JKu+vw&/$.28<8<6:377;,0/3&*04#(? *? 6? 6 ? 4 ? 1 ? 5? *? -? $? .? !?  >? x? B MrMrz Electra model with a token classification head on top. Both the discriminator and generator may be loaded into this model. c eZdZfdZeeejdede e dd d d dZ d dZ xZ S) TFElectraForTokenClassificationc pt||fi|t|d|_|j |jn |j }t jj||_ t jj|jt|jd|_||_y)NrBrrr)r5r6rHrBrrrr@rCrErArrrBrrGrs rLr6z(TFElectraForTokenClassification.__init__,s *6*)&yA )/)B)B)NF % %TZTnTn ||++,>? ,,,,   /&BZBZ2[bn-  rMrzzDbhadresh-savani/electra-base-discriminator-finetuned-conll03-englishzK['B-LOC', 'B-ORG', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'O', 'B-LOC', 'I-LOC']g)\(?rc "|j||||||||| |  } | d} |j| } |j| }| dn|j| |}| s|f| ddz}||f|zS|St ||| j | j S)z labels (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. rrNr%r)rBrErrrrmr)rIr'rnr)r(ror*rsrrrrfr7rrdrrs rLr~z$TFElectraForTokenClassification.call9s6'+ll))%'/!5#'3 ' #)DA(F%(, 5R(S%!>?~t4+?+?+OY!?#%88FAB#? jzz,R8 ZZ 4   &=+D&8F%2F> "''z0JKD,AB/0F *.)9TGf$ Ev E-%!5CC2==   rMc|jryd|_t|ddMtj|jj 5|jj ddddt|ddetj|jj 5|jj dd|jjgdddyy#1swY|xYw#1swYyxYw)NTrBr) rrrWrrBr/rrrGr7rs rLrz#TFElectraForQuestionAnswering.buildrr) NNNNNNNNNNNF)r'rqrnrrr)rrr(rrrorrr*rrrsrrrrrrrrrrrrfrrz7Union[TFQuestionAnsweringModelOutput, Tuple[tf.Tensor]]r)rrrr6rr"rrr r rr~rrrs@rLrr}s*+C+J+JKh+ij82$ ).28<8<6:377;,0/3&*9=7;#(? *? 6? 6 ? 4 ? 1 ? 5? *? -? $? 7? 5? !?  A? k? B MrMr)rrrrrrrwrA)Sr0 __future__rr=r dataclassesrtypingrrrnumpynp tensorflowrWactivations_tfr modeling_tf_outputsr r r r rrmodeling_tf_utilsrrrrrrrrrrrrtf_utilsrrrutilsrr r!r"r#r$configuration_electrar& get_loggerrloggerrrr@Layerr(rrrrrrrrr2r:rArHrtELECTRA_START_DOCSTRINGrrwrrrrrrrr__all__r4rMrLrs" !))/    SR1   H %:!AHU\\//AHJL%,,,,L>0.++0.hHELL..H<Lell((L>d0U\\''d0PK&u||))K&^Hell((H@YOMOMd `M$<>U`M`MF rM