T^(hd dZddlZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZmZmZmZmZmZmZdd lmZdd lmZmZdd lmZmZmZm Z d d l!m"Z"d dl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)e jTe+Z,dZ-dZ.Gddej^Z0Gddej^Z1Gddej^Z2Gddej^Z3Gddej^Z4Gddej^Z5Gddej^Z6Gdd ej^Z7Gd!d"ej^Z8Gd#d$eZ9d%Z:d&Z;ed'e:Gd(d)e9Z<ed*e:Gd+d,e9Z=Gd-d.ej^Z>ed/e:Gd0d1e9Z?ed2e:Gd3d4e9Z@ed5e:Gd6d7e9ZAGd8d9ej^ZBed:e:Gd;dZEy)@zPyTorch I-BERT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)gelu))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging) IBertConfig)IntGELU IntLayerNorm IntSoftmaxQuantActQuantEmbedding QuantLinearzkssteven/ibert-roberta-baserc2eZdZdZfdZ ddZdZxZS)IBertEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. ct||j|_d|_d|_d|_d|_d|_t|j|j|j|j|j|_ t|j|j|j|j|_|jdt!j"|j$j'dd t)|d d |_|j|_t|j$|j|j,|j|j|_t1|j|j |_t1|j|j |_t7|j|j8|j|j|j:|_t1|j |j |_tAjB|jD|_#y)N ) padding_idx weight_bit quant_mode)r*r+ position_ids)rF) persistentposition_embedding_typeabsoluter+eps output_bitr+ force_dequant)$super__init__r+ embedding_bitembedding_act_bitact_bit ln_input_bit ln_output_bitr vocab_size hidden_size pad_token_idword_embeddingstype_vocab_sizetoken_type_embeddingsregister_buffertorcharangemax_position_embeddingsexpandgetattrr/r)position_embeddingsrembeddings_act1embeddings_act2rlayer_norm_epsr5 LayerNormoutput_activationrDropouthidden_dropout_probdropoutselfconfig __class__s f/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/ibert/modeling_ibert.pyr7zIBertEmbeddings.__init__8s  ++!# -      ++))  &4  " "F$6$64CUCUbfbqbq& "  ELL)G)GHOOPWXej  (/v7PR\']$"..#1  * *   (()) $   ((>(>4??['(>(>4??[&   %%)) ..  "*$,,4??!Szz&"<"<= c|D|1t||j|j|j}n|j |}||j }n|j dd}|:t j|t j|jj}||j|\}}nd}|j|\}} |j|||| \} } |jdk(r,|j|\} } |j| | | | \} } |j| | \} } |j!| } |j#| | \} } | | fS)Nr-dtypedeviceidentityidentity_scaling_factorr0)"create_position_ids_from_input_idsr)tor[&create_position_ids_from_inputs_embedssizerDzeroslongr,r@rBrJr/rIrMrQrN)rS input_idstoken_type_idsr, inputs_embedspast_key_values_length input_shapeinputs_embeds_scaling_factorrB$token_type_embeddings_scaling_factor embeddingsembeddings_scaling_factorrI"position_embeddings_scaling_factors rVforwardzIBertEmbeddings.forwardls  $At//1G "Y%%& $JJ=Y  #..*K',,.s3K  !"[[EJJtO`O`OgOghN  :>:N:Ny:Y 7M7+/ (FJF`F`aoFpCC040D0D  (*$H 1E1 - -  ' ': 5FJF^F^_kFl C !C484H4H),(J 5I5 1J115zKd0e- -\\*- 040F0FzSl0m- -444rWc|jdd}|d}tj|jdz||jzdztj|j }|j dj|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr-rrYr)rbrDrEr)rdr[ unsqueezerG)rSrgrisequence_lengthr,s rVraz6IBertEmbeddings.create_position_ids_from_inputs_embedss$((*3B/ %a.||   q /D4D4D"Dq"HPUPZPZcpcwcw %%a(// <jrs-5^=rWr#c2eZdZfdZdZ ddZxZS)IBertSelfAttentionc t||j|jzdk7r2t |ds&t d|jd|jd|j |_d|_d|_d|_ |j|_t|j|jz |_ |j|jz|_ t|j|jd|j|j|j d |_t|j|jd|j|j|j d |_t|j|jd|j|j|j d |_t#|j|j |_t#|j|j |_t#|j|j |_t#|j|j |_t-j.|j0|_t5|d d |_|j6d k7r t d t9|j|j |j:|_y)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r%r(Tbiasr*bias_bitr+ per_channelr1r/r0zDI-BERT only supports 'absolute' for `config.position_embedding_type`r+r5)r6r7r>num_attention_headshasattr ValueErrorr+r*rr:intattention_head_size all_head_sizer!querykeyvaluerquery_activationkey_activationvalue_activationrNrrOattention_probs_dropout_probrQrHr/rr5softmaxrRs rVr7zIBertSelfAttention.__init__s?     : : :a ?PVXhHi#F$6$6#78 445Q8 !++  #)#=#= #&v'9'9Fu;;:>((=Rn:o77>BjjXt>u;;372G2G ?3 / //3.A.A/Sq.r+ +262G2G ?3 / / // < --i8 // < !<< Y5H5HR5PQ $223+e3 ??.HKc.cfk.k +.2 +  %/.@ ;?,, =; 77 ,,7  - 9O _kB ) 5+ILf+f (+/ (%--aAq9DDF "/"4"4"6s";t?Q?Q>S"S* **,CD 7;6L6L 77 3 37H=/2mM]!*+I J.0  ---rWNNF)rsrtrur7rrorwrxs@rVrzrzs 8pt% K.rWrzc$eZdZfdZdZxZS)IBertSelfOutputc t||j|_d|_d|_d|_d|_d|_t|j|jd|j|j |jd|_ t|j |j|_ t|j|j|j|j|j|_t|j|j|_t%j&|j(|_yNr%r(r'Tr~r1r2)r6r7r+r:r*rr;r<r!r>denser ln_input_actrrLr5rMrNrrOrPrQrRs rVr7zIBertSelfOutput.__init__<s  ++        ]]  %T%6%64??S%   %%)) ..  "*$,,4??!Szz&"<"<= rWc|j||\}}|j|}|j||||\}}|j||\}}|j ||\}}||fSNr\rrQrrMrNrSrr input_tensorinput_tensor_scaling_factors rVrozIBertSelfOutput.forwardY6:jjPl6m3 3 ]3 6:6G6G  (!$? 7H7 3 3 7;nn]Tp6q3 36:6L6L 77 3 3:::rWrsrtrur7rorwrxs@rVrr; >:;rWrc2eZdZfdZdZ ddZxZS)IBertAttentionct||j|_t||_t ||_t|_yN) r6r7r+rzrSroutputset pruned_headsrRs rVr7zIBertAttention.__init__ks=  ++&v. %f- ErWc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrdim)lenrrSrrrrrrrrrrunion)rSheadsindexs rV prune_headszIBertAttention.prune_headsrs u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rWc|j|||||\}}|j|d|d||\}} |f|ddz} | f|ddz} | | fS)Nrr)rSr) rSrrrrr self_outputsself_outputs_scaling_factorattention_outputattention_output_scaling_factorroutputs_scaling_factors rVrozIBertAttention.forwards59II  (    5 1 1=AKK O8;]Lh= 99$% QR(88"A!CFabcbdFe!e...rWr)rsrtrur7rrorwrxs@rVrrjs";, /rWrc$eZdZfdZdZxZS)IBertIntermediatec t||j|_d|_d|_d|_t |j|jd|j|j |jd|_ |jdk7r tdt|j|j|_t|j|j|_y) Nr%r(Tr~r z3I-BERT only supports 'gelu' for `config.hidden_act`rr1)r6r7r+r:r*rr!r>intermediate_sizer hidden_actrrr5intermediate_act_fnrrNrRs rVr7zIBertIntermediate.__init__s  ++       $ $]]      &RS S#*dooU[UiUi#j !)$,,4??!SrWc|j||\}}|j||\}}|j||\}}||fSr)rrrN)rSrrs rVrozIBertIntermediate.forwardsa6:jjPl6m3 36:6N6N 77 3 3 7;6L6L 77 3 3:::rWrrxs@rVrrsT( ;rWrc$eZdZfdZdZxZS) IBertOutputc t||j|_d|_d|_d|_d|_d|_t|j|jd|j|j |jd|_ t|j |j|_ t|j|j|j|j|j |_t|j|j|_t'j(|j*|_yr)r6r7r+r:r*rr;r<r!rr>rrrrrLr5rMrNrrOrPrQrRs rVr7zIBertOutput.__init__s  ++    $ $   ]]  %T%6%64??S%   %%)) ..  "*$,,4??!Szz&"<"<= rWc|j||\}}|j|}|j||||\}}|j||\}}|j ||\}}||fSrrrs rVrozIBertOutput.forwardrrWrrxs@rVrrrrWrc2eZdZfdZ ddZdZxZS) IBertLayercXt||j|_d|_d|_t ||_t||_t||_ t|j|j|_ t|j|j|_ y)Nr%rr1)r6r7r+r: seq_len_dimr attentionr intermediaterrrpre_intermediate_actpre_output_actrRs rVr7zIBertLayer.__init__s}  ++ '/-f5!&) $,T\\doo$V!&t||PrWc|j|||||\}}|d}|d} |dd} |j|| \} } | f| z} | S)N)rrr)rfeed_forward_chunk) rSrrrrrself_attention_outputs%self_attention_outputs_scaling_factorrrr layer_outputlayer_output_scaling_factors rVrozIBertLayer.forwardsIM  (  / IWI E E2!4*OPQ*R'(,484K4K =5 1 1 /G+rWc|j||\}}|j||\}}|j||\}}|j||||\}}||fSr)rrrr)rSrrintermediate_output"intermediate_output_scaling_factorrrs rVrzIBertLayer.feed_forward_chunks<@z'IBertEncoder.forward..Rs = s)last_hidden_statepast_key_valuesr attentionscross_attentions) enumeratertupler )rSrrrrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_mask layer_outputss rVrozIBertEncoder.forward-s#7BD$5b4#!(4 POA|#$58H$H!.7.CilO(,! M*!,M &9]1=MrTanh activationrRs rVr7zIBertPooler.__init__gsF  ++YYv1163E3EF '')rWc\|dddf}|j|}|j|}|SNr)rr)rSrfirst_token_tensor pooled_outputs rVrozIBertPooler.forwardms6+1a40 #56  6 rWrrxs@rVrrfs $ rWrc&eZdZdZeZdZdZddZy)IBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. ibertct|ttjfrm|jj j d|jj|j%|jj jyyt|ttjfrz|jj j d|jj|j2|jj |jjyyt|ttjfrJ|jj j|jj j!dyt|t"r%|jj jyy)zInitialize the weightsg)meanstdNg?) isinstancer!rrweightdatanormal_rTinitializer_rangerzero_r Embeddingr)rrMfill_ IBertLMHead)rSmodules rV _init_weightsz"IBertPreTrainedModel._init_weightss5 f{BII6 7 MM   & &CT[[5R5R & S{{&   &&('  > ? MM   & &CT[[5R5R & S!!- ""6#5#56<<>. r|| < = KK   " " $ MM   $ $S )  , KK   " " $-rWNctd)Nz6`resize_token_embeddings` is not supported for I-BERT.)NotImplementedError)rSnew_num_tokenss rVresize_token_embeddingsz,IBertPreTrainedModel.resize_token_embeddingss!"Z[[rWr) rsrtrurvr config_classbase_model_prefixr3r7rrWrVr$r$vs L%$\rWr$a? This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`IBertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a5 Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z`The bare I-BERT Model transformer outputting raw hidden-states without any specific head on top.ceZdZdZdfd ZdZdZdZee jde e e e ddeej"d eej$d eej"d eej"d eej$d eej$deedeedeedee eej$ffdZxZS) IBertModela The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. ct||||_|j|_t ||_t ||_|r t|nd|_ |jyr) r6r7rTr+r#rlrencoderrpooler post_init)rSrTadd_pooling_layerrUs rVr7zIBertModel.__init__sX    ++)&1#F+ ->k&)D  rWc.|jjSrrlr@rSs rVget_input_embeddingszIBertModel.get_input_embeddingss...rWc&||j_yrrB)rSrs rVset_input_embeddingszIBertModel.set_input_embeddingss*/'rWc|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr=rrr)rSheads_to_prunerrs rV _prune_headszIBertModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B CrWbatch_size, sequence_length checkpoint output_typer8rerrfr,rrgrrrreturnc ||n|jj}||n|jj}| | n|jj} | | t d|#|j |||j } n!||j dd} n t d| \} } | |jn |j} |tj| | f| }|&tj| tj| }|j|| }|j||jj}|j||||\}}|j!||||||| }|d}|j"|j#|nd}| s ||f|d dzSt%|||j&|j(|j*|j, S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer-z5You have to specify either input_ids or inputs_embeds)r[rY)rer,rfrg)rrrrrrr)r  pooler_outputr rr r )rTrruse_return_dictr%warn_if_padding_and_no_attention_maskrbr[rDonesrcrdget_extended_attention_mask get_head_maskrrlr=r>r r rr r )rSrerrfr,rrgrrrri batch_size seq_lengthr[extended_attention_maskembedding_outputembedding_output_scaling_factorencoder_outputssequence_outputr"s rVrozIBertModel.forwards$2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T  !"ZZ*j)A6RN  !"[[EJJvVN150P0PQ_al0m&&y$++2O2OP <@OO%)' =L= 99 ,,  +2/!5#' *!,8< 8O O4UY #]3oab6II I;-'+;;)77&11,==   rW)T) NNNNNNNNN)rsrtrurvr7rDrFrJrIBERT_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrrD LongTensor FloatTensorboolrrrorwrxs@rVr;r;sI  /0C++A+H+HIf+gh&@$156:59371559,0/3&*K E,,-K !!2!23K !!1!12 K u//0 K E--. K   1 12K $D>K 'tnK d^K  ;U5CTCT=UU VK i K rWr;z4I-BERT Model with a `language modeling` head on top.ceZdZddgZfdZdZdZeejde e e e d dd eej d eej"d eej d eej d eej"deej"deej deedeedeedee eej"ffdZxZS)IBertForMaskedLMzlm_head.decoder.biaszlm_head.decoder.weightct||t|d|_t ||_|j yNF)r@)r6r7r;r%r1lm_headr?rRs rVr7zIBertForMaskedLM.__init__Xs6  %@ "6*  rWc.|jjSr)ridecoderrCs rVget_output_embeddingsz&IBertForMaskedLM.get_output_embeddingsas||###rWc\||j_|j|j_yr)rirkr)rSnew_embeddingss rVset_output_embeddingsz&IBertForMaskedLM.set_output_embeddingsds - *// rWrKz)rMrNr8maskrerrfr,rrglabelsrrrrOc | | n|jj} |j|||||||| |  } | d} |j| } d}|Ft }|| j d|jj |j d}| s| f| ddz}||f|zS|St|| | j| jS)a( labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` kwargs (`Dict[str, any]`, *optional*, defaults to `{}`): Used to hide legacy arguments that have been deprecated. Nrrfr,rrgrrrrr-rlosslogitsrr ) rTrRr%rirrr=rrr )rSrerrfr,rrgrqrrrrr]prediction_scoresmasked_lm_lossloss_fctrs rVrozIBertForMaskedLM.forwardhs8&1%3 'tn3 d^3  ~uU%6%677 83 i3 rWrfc0eZdZdZfdZdZddZxZS)r1z)I-BERT Head for masked language modeling.ct|tj|j|j|_tj |j|j|_tj|j|j|_ tjtj|j|_|j|j_y)N)r3)r6r7rrr>rrMrL layer_normr=rk ParameterrDrcrrRs rVr7zIBertLMHead.__init__s YYv1163E3EF ,,v'9'9v?T?TUyy!3!3V5F5FG LLV->->!?@  II rWc |j|}t|}|j|}|j|}|Sr)rr r~rk)rSfeatureskwargsrs rVrozIBertLMHead.forwards; JJx  G OOA  LLOrWc|jjjjdk(r|j|j_y|jj|_y)Nmeta)rkrr[typerCs rV _tie_weightszIBertLMHead._tie_weightssC <<   # # ( (F 2 $ DLL  ))DIrW)rON)rsrtrurvr7rorrwrxs@rVr1r1s3&*rWr1z I-BERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e ed e ed e edee ee jffdZxZS)IBertForSequenceClassificationct|||j|_t|d|_t ||_|jyrh)r6r7 num_labelsr;r%IBertClassificationHead classifierr?rRs rVr7z'IBertForSequenceClassification.__init__sC   ++%@ 1&9 rWrKrLrerrfr,rrgrqrrrrOc | | n|jj} |j|||||||| |  } | d} |j| } d}||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt}|j dk(r&|| j|j}n|| |}n|jjdk(r=t}|| jd|j |jd}n,|jjdk(rt}|| |}| s| f| d dz}||f|zS|St|| | j | j" S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrsrr regressionsingle_label_classificationmulti_label_classificationr-rrt)rTrRr%r problem_typerrZrDrdrr squeezerrrrrr rSrerrfr,rrgrqrrrrr]rvruryrs rVroz&IBertForSequenceClassification.forwards2&1%B 'tnB d^B  'u/@/@)AA BB i B rWrz I-BERT Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. ceZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e ed e ed e edee ee jffdZxZS)IBertForMultipleChoicect||t||_t j |j |_t j|jd|_ |jy)Nr) r6r7r;r%rrOrPrQrr>rr?rRs rVr7zIBertForMultipleChoice.__init__)sV  ' zz&"<"<= ))F$6$6: rWz(batch_size, num_choices, sequence_lengthrLrerfrrqr,rrgrrrrOc L| | n|jj} ||jdn|jd} |!|jd|j dnd} |!|jd|j dnd} |!|jd|j dnd}|!|jd|j dnd}|1|jd|j d|j dnd}|j | | |||||| |  }|d}|j |}|j|}|jd| }d}|t}|||}| s|f|ddz}||f|zS|St|||j|jS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr-r)r,rfrrrgrrrrrt) rTrRshaperrbr%rQrrrrr )rSrerfrrqr,rrgrrr num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsrr"rvreshaped_logitsruryrs rVrozIBertForMultipleChoice.forward3s2&1%?^bLXLdL--b,2C2CB2GHjnR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrv(   r=#5#5b#9=;M;Mb;Q R  ** *..,/!5#     ]3 / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   rWrz)rsrtrur7rr^r_rr`rrarrDrbrcrdrrrorwrxs@rVrr!s@++A+H+HIs+tu&-$15596:-1371559,0/3&*? E,,-? !!1!12? !!2!23 ? ))* ? u//0 ? E--.?   1 12? $D>? 'tn? d^?  (%0A0A*BB C? v ? rWrz I-BERT Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. ceZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e ed e ed e edee ee jffdZxZS)IBertForTokenClassificationc0t|||j|_t|d|_t j |j|_t j|j|j|_ |jyrh) r6r7rr;r%rrOrPrQrr>rr?rRs rVr7z$IBertForTokenClassification.__init__sk   ++%@ zz&"<"<= ))F$6$68I8IJ rWrKrLrerrfr,rrgrqrrrrOc | | n|jj} |j|||||||| |  } | d} |j| } |j | } d}|2 'tn2 d^2  $eE,=,=&>> ?2 i 2 rWrc(eZdZdZfdZdZxZS)rz-Head for sentence-level classification tasks.c&t|tj|j|j|_tj |j|_tj|j|j|_ yr) r6r7rrr>rrOrPrQrout_projrRs rVr7z IBertClassificationHead.__init__s` YYv1163E3EF zz&"<"<=  &"4"4f6G6GH rWc |dddddf}|j|}|j|}tj|}|j|}|j |}|Sr )rQrrDtanhr)rSrrrs rVrozIBertClassificationHead.forwards^ Aq)  ]3  =1  =1  ]3  m4 rW)rsrtrurvr7rorwrxs@rVrrs7I rWrz I-BERT Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e e jd e ed e ede edee ee jffdZxZS)IBertForQuestionAnsweringct|||j|_t|d|_t j |j|j|_|jyrh) r6r7rr;r%rrr> qa_outputsr?rRs rVr7z"IBertForQuestionAnswering.__init__sU   ++%@ ))F$6$68I8IJ rWrKrLrerrfr,rrgstart_positions end_positionsrrrrOc (| | n|jj} |j||||||| | |  } | d} |j| }|j dd\}}|j dj }|j dj }d}||t|jdkDr|j d}t|jdkDr|j d}|jd}|jd|}|jd|}t|}|||}|||}||zdz }| s||f| ddz}||f|zS|St|||| j| j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrsrrr-r) ignore_indexr)ru start_logits end_logitsrr )rTrRr%rsplitrrrrbclamprrrr )rSrerrfr,rrgrrrrrrr]rvrr total_loss ignored_indexry start_lossend_lossrs rVroz!IBertForQuestionAnswering.forwards<&1%H 'tnH d^H  +U53D3D-EE FH i H rWrc|j|j}tj|dj ||z|z}|j |zS)aM Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's *utils.make_positions*. Args: input_ids (`torch.LongTensor`): Indices of input sequence tokens in the vocabulary. Returns: torch.Tensor rr)nerrDcumsumtype_asrd)rer)rhrpincremental_indicess rVr_r_?sW << $ ( ( *D <<!4<rsb$ )) AA.Quu,cc   H %3w=biiw=tK.K.\,;bii,;^./RYY./b; ;D,;")),;^7979t= 299= @ "))  \?\> /dfv %v v rPRghM +M iM `*"))*> S %9S S l P 1P P f D "6D D Nbii& Y 4Y Y x4" rW