S^(h#:dZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZdd lmZdd lmZmZmZmZmZmZdd lmZmZm Z dd l!m"Z"m#Z#m$Z$dd l%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-e*j\e/Z0dZ1dZ2dZ3d6dZ4Gdde jjZ6Gdde jjZ7dZ8dZ9e(de8Gdde jjZ:GddeZ;Gdd e;Z<e(d!e8Gd"d#e;eZ=e(d$e8Gd%d&e;Z>e(d'e8Gd(d)e;Z?e(d*e8Gd+d,e;Z@e(d-e8eGd.d/e&ZAGd0d1e;ZBe(d2e8Gd3d4e;ZCgd5ZDy)7z%PyTorch Flaubert model, based on XLM.N) dataclass)DictOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)gelu)GenerationMixin)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModelSequenceSummary SQuADHead)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings)FlaubertConfigzflaubert/flaubert_base_casedr"ctjt|Dcgc];}t|Dcgc]$}|tjdd|dzz|z z &c}=c}}}d|_t j tj|dddddf|dddddf<t j tj|dddddf|dddddf<|jycc}wcc}}w)Ni'Frr!) nparrayrangepower requires_gradtorch FloatTensorsincosdetach_)n_posdimoutposj position_encs l/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/flaubert/modeling_flaubert.pycreate_sinusoidal_embeddingsr69s88hmnshtuadQVWZQ[\AcBHHUAaL34F$GG\uvLC$$RVVLADqD,A%BCC14a4L$$RVVLADqD,A%BCC14a4LKKM ]usC; )C6C; 6C; ctj|tj|j}||}n/|j j |ksJ||dddfk}|j d}|r'|ddddfj||d|ddddfk}n|}|j ||fk(sJ|dus|j |||fk(sJ||fS)zH Generate hidden states mask, and optionally an attention mask. dtypedeviceNrr!F)r*arangelongr:maxitemsizerepeat)slenlengthscausal padding_maskalenmaskbs attn_masks r5 get_masksrIBs <<EJJw~~ FD{{}!!#t+++gag&& aB tQ'..r4;tD!TM?RR   99;2t* $$ $ U?inn.2tT2BBB B ?cNeZdZejZfdZdZddZxZ S)MultiHeadAttentionct|ttj|_||_||_|j|_ |j |jzdk(sJtj|||_ tj|||_ tj|||_tj|||_t!|_y)Nr)super__init__nextrLNEW_IDlayer_idr0n_headsattention_dropoutdropoutrLinearq_link_linv_linout_linset pruned_heads)selfrSr0config __class__s r5rOzMultiHeadAttention.__init___s /667  // xx$,,&!+++YYsC( YYsC( YYsC( yyc* ErJc|j|jz}t|dk(ryt||j||j\}}t |j ||_t |j||_t |j||_t |j|d|_ |jt|z |_||jz|_|jj||_y)Nrr!r0) r0rSlenrr\rrWrXrYrZunion)r]headsattention_head_sizeindexs r5 prune_headszMultiHeadAttention.prune_headsms"hh$,,6 u:? 7t||M`bfbsbst u' E: ' E: ' E: )$,,1E ||c%j0 &5 --33E:rJc|j\}}| ||n|d|z} n|jd} j} j| z|jdk(rd|| fndd| f} fd} fd} | j|}|/| j |}| j |}nB|j |vr2|x}}| j |}| j |}|xj |vrY|E|j \}}tj|gd}tj|gd}n|j \}}f|j <|tjz }tj|jdd}|dk(j| j|}|j|tj |j"j$t&j(j+|j-d j/|}t&j(j1|j0j2 }|||z}tj|}| |}j5|f}|r||fz}|S) zd Self-attention (if kv is None) or attention over source sentence (provided by kv). rAr!r c`|jdjjddS) projectionr!r$)viewrS transposexrG dim_per_headr]s r5shapez)MultiHeadAttention.forward..shapes)66"b$,, =GG1M MrJc|jddjjdjzS)zcompute contextr!r$rk)rm contiguousrlrSrns r5unshapez+MultiHeadAttention.forward..unshapes5;;q!$//166r2t||l?Z[ [rJr$rarrkptraining)r?rSr0rWrXrYrRr*catmathsqrtmatmulrmrl expand_as masked_fill_finfor9minr functionalsoftmaxfloattype_asrUrwrZ)r]inputrFkvcache head_maskoutput_attentionsqlenr0klenrS mask_reshaperqrtqkvk_v_scoresweightscontextoutputsrGrps` @@r5forwardzMultiHeadAttention.forward|s  D# : =4eFmd.BD771:D,,xx7* .2hhjAoAtT*B1dCS  N \ $**U# $ :djj'(Adjj'(A ]dmm58JAdjjm$Adjjm$A  }}%:"4==1FB 2q'q1A 2q'q1A /DAq$%q6E$--  ,' 'aQ!23  -77?D%++fll";"?"?@--'' B'?GGO--''4<<$--'X   )G,,w*'"<<(*  *GrJ)NNNF) __name__ __module__ __qualname__ itertoolscountrQrOrgr __classcell__r_s@r5rLrL\s Y__ F " ;=rJrLc*eZdZfdZdZdZxZS)TransformerFFNcBt||j|_tj|||_tj|||_|jrtntjj|_ |j|_ d|_ yNr!)rNrOrUrrVlin1lin2gelu_activationr rreluactchunk_size_feed_forward seq_len_dim)r]in_dim dim_hiddenout_dimr^r_s r5rOzTransformerFFN.__init__sn ~~ IIfj1 IIj'2 !114r}}7I7I'-'E'E$rJcZt|j|j|j|SN)rff_chunkrr)r]rs r5rzTransformerFFN.forwards%(8T8TVZVfVfhmnnrJc|j|}|j|}|j|}tjj ||j |j }|S)Nru)rrrrrrUrw)r]rros r5rzTransformerFFN.ff_chunksR IIe  HHQK IIaL MM ! !!t||dmm ! LrJ)rrrrOrrrrs@r5rrsorJraB This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`FlaubertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) lengths (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Length of each sentence that can be used to avoid performing attention on padding token indices. You can also use `attention_mask` for the same result (see above), kept here for compatibility. Indices selected in `[0, ..., input_ids.size(-1)]`: cache (`Dict[str, torch.FloatTensor]`, *optional*): Dictionary strings to `torch.FloatTensor` that contains precomputed hidden-states (key and values in the attention blocks) as computed by the model (see `cache` output below). Can be used to speed up sequential decoding. The dictionary object will be modified in-place during the forward pass to add newly computed hidden-states. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zbThe bare Flaubert Model transformer outputting raw hidden-states without any specific head on top.c*eZdZdZfdZddZxZS)FlaubertPredLayerz? Prediction layer (cross_entropy or adaptive_softmax). ct||j|_|j|_|j|_|j }|jdur(t j||jd|_yt j||j|j|jd|_y)NFT)bias) in_features n_classescutoffs div_value head_bias) rNrOasmn_words pad_indexemb_dimrrVprojAdaptiveLogSoftmaxWithLoss asm_cutoffs asm_div_value)r]r^r0r_s r5rOzFlaubertPredLayer.__init__'s ::~~ ))nn ::  #v~~DADI55 ..** .. DIrJczd}|jdurl|j|}|f|z}|Qtjj |j d|j |j dd}|f|z}|S|jj|}|f|z}||j||\}}|f|z}|S)z,Compute the loss, and optionally the scores.Frkmean) reduction)rrrr cross_entropyrlrlog_prob)r]royrrloss_s r5rzFlaubertPredLayer.forward9s 88u YYq\Fi')G}}}226;;r4<<3PRSRXRXY[R\hn2o'G+ YY''*Fi')G}))Aq/4'G+rJr)rrr__doc__rOrrrs@r5rrs $rJrcDeZdZdZeZdZdZfdZe dZ dZ xZ S)FlaubertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. N transformerc$t||i|yr)rNrO)r]inputskwargsr_s r5rOz FlaubertPreTrainedModel.__init__Ws &+F+rJc&tjgdgdgdg}tjgdgdgdg}|jjr8|jjdkDrtjgdgdgdg}nd}|||dS) N)rrr!)r!r$r rr)rrr)r!r!rrr!)r!r!r!rr)r!rrr!r!r!) input_idsattention_masklangs)r*tensorr^ use_lang_embn_langs)r] inputs_list attns_list langs_lists r5 dummy_inputsz$FlaubertPreTrainedModel.dummy_inputsZsrllO_o#VW \\?O_"UV ;; # # (;(;a(?&YZJJ(JQ[\\rJct|tjr|jV|jj@tj j |jd|jj|j1|jj|jjt|tjr|j|jjvtj j |jd|jj|j*tj j|jdt|tjrI|jjj|jjj!dt|t"rb|jj$rKt'|jj(|jj*|j,jyyy)zInitialize the weights.Nr)rstdg?)r1) isinstancer Embeddingr^embed_init_stdinitnormal_weight padding_idxdatazero_rVinit_stdr constant_ LayerNormfill_ FlaubertModelsinusoidal_embeddingsr6max_position_embeddingsrposition_embeddings)r]modules r5 _init_weightsz%FlaubertPreTrainedModel._init_weightsdsq fbll +{{&4;;+E+E+Q A4;;;U;UV!!- ""6#5#56<<> fbii ({{&4;;+?+?+K A4;;;O;OP;;*GG%%fkk37 fbll + KK   " " $ MM   $ $S ) fm ,1R1R ( 33T[[5H5HfNhNhNoNo 2S ,rJ) rrrrr" config_classload_tf_weightsbase_model_prefixrOpropertyrrrrs@r5rrMs: "LO%,]]rJrceZdZfdZdZdZdZeee e e e  dde ejde ej de ej"d e ejd e ejd e ejd e eeej fd e ej de ej de ede ede edeee ffdZxZS)rc  t|||j|_|j |_|jr t d|j |_|j |_|j|_|j|_|j|_ |j|_ |j|_ |jdz|_ |j|_|j|_|j |_|j"|_|j|jzdk(sJdt%j&|j(|j|_|j dkDr;|jr/t%j&|j |j|_t%j&|j|j|j|_t%j0|j|j2|_t%j6|_t%j6|_t%j6|_t%j6|_tA|jD] }|j8jCtE|j|j||j:jCt%j0|j|j2|j<jCtG|j|j|j||j>jCt%j0|j|j2tI|d r|jJjMjO}i|_%|D]h\}}|j8tQ|j|jk(s6|jStQ|tUtWtP|ij|jYt[|d d |_.t[|d d |_/|jadtcjd|j(jgdd y)Nz1Currently Flaubert can only be used as an encoderrrz-transformer dim must be a multiple of n_headsr!)r)eps)r^r\ layerdroprpre_normF position_ids)r!rk) persistent)4rNrO is_encoder is_decoderNotImplementedErrorrCrrr eos_indexrrr0 hidden_dimrSn_layersrUrTrrrrlang_embeddings embeddingsrlayer_norm_epslayer_norm_emb ModuleList attentions layer_norm1ffns layer_norm2r'appendrLrhasattrr\copyitemsintrglistmap post_initgetattrrrregister_bufferr*r;expand)r]r^rr\layerrdr_s r5rOzFlaubertModel.__init__zsg  !++$/// ??%&YZ Zmm ~~ "//~~ ))))>>((Q,~~  ~~ !'!9!9xx$,,&!+\-\\+$&<<0N0NPTPXPX#Y >>A &"5"5#%<< dhh#GD ,,t||TXX4>>Z ll4889N9NO--/==?MMO ==? t}}% WA OO " "#5dllDHHU[#\ ]    # #BLLv?T?T$U V II  ^DHHdootxxX^_ `    # #BLLv?T?T$U V W 6> *!..335;;=L"$F  , J u??3u:.66&..H$$c%j$s32G%HI J  c: E:   ELL)G)GHOOPWXej  rJc|jSrrr]s r5get_input_embeddingsz"FlaubertModel.get_input_embeddingss rJc||_yrrr]new_embeddingss r5set_input_embeddingsz"FlaubertModel.set_input_embeddingss (rJcp|jD]#\}}|j|j|%y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)rr rg)r]heads_to_prunerrds r5 _prune_headszFlaubertModel._prune_headss7 +002 6LE5 OOE " . .u 5 6rJ checkpoint output_typerrrrtoken_type_idsrrBrr inputs_embedsroutput_hidden_states return_dictreturnc 0 | | n|jj} | | n|jj} | | n|jj} ||j \} }n| j dd\} }| |j n | j }|K|.||j k7jdj}ntj|g| z|}|j d| k(sJ|jj|ksJt|||j|\}}|~t|dr)|j ddd|f}|j#| |f}n`tj$|tj|}|j'dj#| |f}n|j | |fk(sJ||j | |fk(sJ|j)||jj*}|H|F||d z }|dd| df}|dd| df}| |dd| df}|dd| df}|dd| df}| |j-|} | |j/|j1| z}|9|j2r-|jj4dkDr||j7|z}|||j-|z}|j9|}t:j<j?||j>|j@ }||j'djC|jDz}| rd nd}| rd nd}tG|j*D]}|j@r%tjHg}||jJkr5| r||fz}|jLs~|jN||||||| }|d}| r ||dfz}t:j<j?||j>|j@ }||z}|jP||}n||jP||}|jN|||||| }|d}| r ||dfz}t:j<j?||j>|j@ }||z}|jLs.||jR||z}|jT||}n-|jT||}||jR||z}||j'djC|jDz}| r||fz}||d xx|j dz cc<| stWd|||fDStY|||S)Nrkr!ra)r:r)rDrr8rArur)rrr)rrc3&K|] }|| ywrr).0rs r5 z(FlaubertModel.forward..{sYq1=Ys)last_hidden_state hidden_statesr )-r^rr+use_return_dictr?r:rsumr<r*rr=r>rIrCrrrr; unsqueeze get_head_maskrrrr|rrrr rrrUrwtor9r'randrrr r r rtupler)r]rrrr)rrBrrr*rr+r,rGrAr:rFrH_slenrr3r idropout_probability attn_outputsattntensor_normalizeds r5rzFlaubertModel.forwards*2C1N-TXT_T_TqTq$8$D $++JjJj &1% $||D 6R +55a8??T K $$&2t*4 44  ::cm'tncmd^cm uo% &cm F cmrJrz The Flaubert Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). c!eZdZdgZfdZdZdZdZee jde e e ed dd eej"d eej"d eej"d eej"d eej"deej"deeeej"fdeej"deej"deej"deedeedeedeee ffdZxZS)FlaubertWithLMHeadModelzpred_layer.proj.weightct||t||_t ||_|j yr)rNrOrrr pred_layerrr]r^r_s r5rOz FlaubertWithLMHeadModel.__init__s5  (0+F3 rJc.|jjSrrKrrs r5get_output_embeddingsz-FlaubertWithLMHeadModel.get_output_embeddingss###rJc&||j_yrrNr s r5set_output_embeddingsz-FlaubertWithLMHeadModel.set_output_embeddingss-rJc P|jj}|jj}|jd}t j |df|tj |j}t j||gd}|t j||}nd}||dS)Nrr!r8ra)rr) r^ mask_token_idlang_idrqr*fullr<r:rx full_like)r]rrrSrTeffective_batch_size mask_tokenrs r5prepare_inputs_for_generationz5FlaubertWithLMHeadModel.prepare_inputs_for_generations 11 ++%%(q1ZZ!5q 9=PUPZPZclcscst IIy*51=  OOIw7EE&77rJbatch_size, sequence_lengthz )r'r(rrFrrrr)rrBrrr*labelsrr+r,r-c| | n|jj} |j||||||||| | | |  }|d}|j|| }| s||ddzSt | |dnd| |dn|d|j |j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N rrr)rrBrrr*rr+r,rr!rlogitsr3r )r^r4rrKrr3r )r]rrrr)rrBrrr*r[rr+r,transformer_outputsoutputrs r5rzFlaubertWithLMHeadModel.forwards:&1%2 'tn2 d^2  un$ %2 l2 rJrIz Flaubert Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c!eZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e eee jfd e e jd e e jd e e jde ede ede edeee ffdZxZS)!FlaubertForSequenceClassificationct|||j|_||_t ||_t ||_|jyr) rNrO num_labelsr^rrrsequence_summaryrrLs r5rOz*FlaubertForSequenceClassification.__init__sJ   ++ (0 / 7 rJrZr&rrrr)rrBrrr*r[rr+r,r-c$| | n|jj} |j||||||||| | | |  }|d}|j|}d}| |jj|j dk(rd|j_nl|j dkDrL| j tjk(s| j tjk(rd|j_nd|j_|jjdk(rIt}|j dk(r&||j| j}n||| }n|jjdk(r=t}||jd|j | jd}n,|jjdk(rt}||| }| s|f|ddz}||f|zS|St|||j |j" S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr]rr! regressionsingle_label_classificationmulti_label_classificationrkr^)r^r4rri problem_typerhr9r*r<rr squeezer rlr rr3r )r]rrrr)rrBrrr*r[rr+r,r`rar_rloss_fcts r5rz)FlaubertForSequenceClassification.forwards8&1%J 'tnJ d^J  u.. /J l J rJrfz Flaubert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. c!eZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e eee jfd e e jd e e jd e e jde ede ede edeee ffdZxZS)FlaubertForTokenClassificationc,t|||j|_t||_t j |j|_t j|j|j|_ |jyr) rNrOrhrrrDropoutrUrV hidden_size classifierrrLs r5rOz'FlaubertForTokenClassification.__init__Tsh   ++(0zz&..1 ))F$6$68I8IJ rJrZr&rrrr)rrBrrr*r[rr+r,r-c| | n|jj} |j||||||||| | | |  }|d}|j|}|j |}d}| 8 'tn8 d^8  u++ ,8 l 8 rJrrz Flaubert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). c#eZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e eee jfd e e jd e e jd e e jde e jde ede ede edeee ffdZxZS)"FlaubertForQuestionAnsweringSimplect||t||_t j |j |j|_|jyr) rNrOrrrrVrurh qa_outputsrrLs r5rOz+FlaubertForQuestionAnsweringSimple.__init__sG  (0))F$6$68I8IJ rJrZr&rrrr)rrBrrr*start_positions end_positionsrr+r,r-c.||n|jj}|j||||||||| | | | }|d}|j|}|j dd\}}|j dj }|j dj }d}| | t| jdkDr| j d} t| jdkDr| j d} |jd}| jd|} | jd|} t|}||| }||| }||zdz }|s||f|ddz}||f|zS|St||||j|j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nr]rr!rkra) ignore_indexr$)r start_logits end_logitsr3r )r^r4rr|splitrorsrbr?clampr rr3r )r]rrrr)rrBrrr*r}r~rr+r,r`rxr_rr total_loss ignored_indexrp start_lossend_lossras r5rz*FlaubertForQuestionAnsweringSimple.forwardsB&1%N 'tnN d^N u22 3!N l N rJrzz Flaubert Model with a beam-search span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). c^eZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZeejed<dZeeejed <dZeeejed <y) "FlaubertForQuestionAnsweringOutputa9 Base class for outputs of question answering models using a `SquadHead`. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned if both `start_positions` and `end_positions` are provided): Classification loss as the sum of start token, end token (and is_impossible if provided) classification losses. start_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top config.start_n_top start token possibilities (beam-search). start_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top config.start_n_top start token possibilities (beam-search). end_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). end_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). cls_logits (`torch.FloatTensor` of shape `(batch_size,)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the `is_impossible` label of the answers. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nrstart_top_log_probsstart_top_indexend_top_log_probs end_top_index cls_logitsr3r )rrrrrrr*r+__annotations__rrrDrrrr3rr rrJr5rr s>)-D(5$$ %,7;%"3"34;26OXe../659x 1 12904M8E,,-4.2J**+28s5  (0#F+ rJrZ)r(rrrrr)rrBrrr*r}r~ is_impossible cls_indexp_maskrr+r,r-c||n|jj}|j||||||||| ||| }|d}|j|| | | | ||}|s||ddzSt |j |j |j|j|j|j|j|jS)aL start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. is_impossible (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels whether a question has an answer or no answer (SQuAD 2.0) cls_index (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the classification token to use as input for computing plausibility of the answer. p_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Optional mask of tokens which can't be in answers (e.g. [CLS], [PAD], ...). 1.0 means token should be masked. 0.0 mean token is not masked. Returns: Example: ```python >>> from transformers import XLMTokenizer, XLMForQuestionAnswering >>> import torch >>> tokenizer = XLMTokenizer.from_pretrained("FacebookAI/xlm-mlm-en-2048") >>> model = XLMForQuestionAnswering.from_pretrained("FacebookAI/xlm-mlm-en-2048") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze( ... 0 ... ) # Batch size 1 >>> start_positions = torch.tensor([1]) >>> end_positions = torch.tensor([3]) >>> outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions) >>> loss = outputs.loss ```Nr]r)r}r~rrrr,r!)rrrrrrr3r ) r^r4rr|rrrrrrrr3r )r]rrrr)rrBrrr*r}r~rrrrr+r,r`rars r5rz$FlaubertForQuestionAnswering.forwardGsx&1%!c "'tn#c $d^%c & u88 9'c mlc rJrz Flaubert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. c!eZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e eee jfd e e jd e e jd e e jde ede ede edeee ffdZxZS)FlaubertForMultipleChoicect||g|i|t||_t ||_t j|jd|_ |jyr) rNrOrrrrirrVrh logits_projr)r]r^rrr_s r5rOz"FlaubertForMultipleChoice.__init__sY 3&3F3(0 / 799V%6%6: rJz(batch_size, num_choices, sequence_lengthr&rrrr)rrBrrr*r[rr+r,r-c| | n|jj} ||jdn| jd}|!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}| 1| jd| j d| j dnd} |t j dd}|j||||||||| | | |  }|d}|j|}|j|}|jd|}d}| t}||| }| s|f|ddz}||f|zS|St|||j|jS) aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr!rkzwThe `lengths` parameter cannot be used with the Flaubert multiple choice models. Please use the attention mask instead.) rrrr)rrBrrr*rr+r,rr^)r^r4rqrlr?loggerwarningrrirr rr3r )r]rrrr)rrBrrr*r[rr+r, num_choicesr`rar_reshaped_logitsrrps r5rz!FlaubertForMultipleChoice.forwards-<&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei 272C 2uzz"~.(   r=#5#5b#9=;M;Mb;Q R    NN* G"..))%'/!5#/  %Q'&&v.!!&) ++b+6  ')HOV4D%'*=ab*AAF)-)9TGf$ Ev E("-;;*55   rJrb)rrrrOrrArdrrBrrCrr*rErrFrGrrrrrs@r5rrs+!(()ST &-$-115(,15/3*.37,004)-,0/3&*L ELL)L !.L  % L !. L u||, L %,,'L S%,,./0L ELL)L   -L &L $D>L 'tnL d^L  u// 0L L rJr)rrrzrfrrrrIrr)Errry dataclassesrtypingrrrrnumpyr%r*rtorch.nnr r r activationsr generationrmodeling_outputsrrrrrrmodeling_utilsrrr pytorch_utilsrrrutilsrrrrrr configuration_flaubertr" get_loggerrrrBrCr6rIModulerLrFLAUBERT_START_DOCSTRINGrArrrrIrfrrrzrrr__all__rrJr5rs6, !// AA)JIll3   H %4"4]]BRYY* 8vh ' '  'V)o)XDm+DmN Y 5Y Y x \ (?\ \ ~ J %<J J Z ^ )@^ ^ B  ':': ':Vo #:o d _ 7_ _ D rJ