T^(hldZddlZddlmZddlmZmZmZmZddl Z ddl m Z ddl m Z m Z mZddlmZdd lmZdd lmZmZmZmZmZdd lmZdd lmZmZmZmZm Z m!Z!d dl"m#Z#e jHe%Z&dZ'dZ(d@dZ)dZ*Gdde jVZ,Gdde jVZ-Gdde jVZ.GddeZ/eGddeZ0eGddeZ1eGdd eZ2eGd!d"eZ3eGd#d$eZ4eGd%d&eZ5eGd'd(eZ6d)Z7d*Z8ed+e7Gd,d-e/Z9ed.e7Gd/d0e/eZ:ed1e7Gd2d3e/Z;ed4e7Gd5d6e/Z<ed7e7Gd8d9e/Z=ed:e7Gd;ded:e7Gd=d>e/Z?gd?Z@y)Az PyTorch XLNet model. N) dataclass)ListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GenerationMixin)PoolerAnswerClassPoolerEndLogitsPoolerStartLogitsPreTrainedModelSequenceSummary)apply_chunking_to_forward) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) XLNetConfigzxlnet/xlnet-base-casedrc"i}t|drt|dr|jj|d<t|drJd|vrF|jjj |d<|jjj|d<t|drl|j `d|j d |vrN|jj |d|j d <|jj|d|j d <|j}|j|jj |jd t|jD]\}}d |d }|j|dz|jjj |dz|jjj|dz|jj |dz|jj"|dz|jj$|dz|jj&|dz|jj(|dz|j*jj |dz|j*jj|dz|j*j,j |dz|j*j,j|dz|j*j.j |dz|j*j.ji |j0rg}g}g} g} |jD]}|j3|jj4|j3|jj6| j3|jj8| j3|jj:n4|j4g}|j6g}|j8g} |j:g} |j||| | d|S)z A map of modules from TF to PyTorch. I use a map to keep the PyTorch model as identical to the original PyTorch model as possible. transformerlm_losszmodel/lm_loss/biassequence_summaryz%model/sequnece_summary/summary/kernelz#model/sequnece_summary/summary/bias logits_projzmodel/regression_z /logit/kernelz /logit/bias)z-model/transformer/word_embedding/lookup_tablez#model/transformer/mask_emb/mask_embzmodel/transformer/layer_/zrel_attn/LayerNorm/gammazrel_attn/LayerNorm/betazrel_attn/o/kernelzrel_attn/q/kernelzrel_attn/k/kernelzrel_attn/r/kernelzrel_attn/v/kernelzff/LayerNorm/gammazff/LayerNorm/betazff/layer_1/kernelzff/layer_1/biaszff/layer_2/kernelzff/layer_2/bias)zmodel/transformer/r_r_biaszmodel/transformer/r_w_biaszmodel/transformer/r_s_biaszmodel/transformer/seg_embed)hasattrrbiasr summaryweightfinetuning_taskr!rupdateword_embeddingmask_emb enumeratelayerrel_attn layer_normoqkrvfflayer_1layer_2untie_rappendr_r_biasr_w_biasr_s_bias seg_embed) modelconfig tf_weights tf_to_pt_mapib layer_strr_r_listr_w_listr_s_listseg_embed_lists f/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/xlnet/modeling_xlnet.pybuild_tf_xlnet_to_pytorch_maprI1s Lum$ 5) $161C1CL- . 5, -2Y]g2gDIDZDZDbDbDiDiL@ ABGBXBXB`B`BeBeL> ? E= )&&2#F$:$:#;=IZWV[VgVgVnVnL,V-C-C,DMR STYTeTeTjTjL,V-C-C,DKP Q!!=B=Q=Q=X=X38>> %++& 1.qc3 66 8M8M8T8T55qzz7L7L7Q7Q//////////00!$$//2H2H//1E1E//1D1D--qtt||/@/@//1D1D--qtt||/@/@   *~~ 8A OOAJJ// 0 OOAJJ// 0 OOAJJ// 0  ! !!**"6"6 7  8 NN#NN#NN#//**2*2*2+9   c ddl}ddl}|j j |}i}|D]A\}}tjd|d||j j||} | ||<Ct|||} | jD]\}} tjd|||vrtj|d<||} d|vr2d |vsd |vsd |vr&tjd |j| } t| trt| | jdk(s$Jd t| d| jddt!| D]~\} } | | df} | j|jk(s"Jd| jd|jd tjd|d| t'j(|| _no | j| jk(s"Jd| jd| jd tjd|t'j(| | _|j-|d|j-|dzd|j-|dzdtjddj/|j1|S#t$rtj dwxYw#t"$r1}|xj$| j|jfz c_d}~wwxYw#t"$r1}|xj$| j| jfz c_d}~wwxYw)z&Load tf checkpoints in a pytorch modelrNzLoading a TensorFlow models in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.zLoading TF weight z with shape z Importing z( not in tf pre-trained weights, skippingkernelr4r%logit TransposingzPointer length z and array length z mismatched.zPointer shape z and array shape zInitialize PyTorch weight z for layer z/Adamz/Adam_1z%Weights not copied to PyTorch model: z, )numpy tensorflow ImportErrorloggererrortrainlist_variablesinfo load_variablerIitems transpose isinstancelistlenshaper+AssertionErrorargstorch from_numpydatapopjoinkeys)r=r>tf_pathnptf init_varsr?namer]arrayr@pointerrAp_iarr_ies rHload_tf_weights_in_xlnetrpsC''0IJ ! e (l5'BC&&w5 4! 1 KL%++-'/ g j'( z ! KK4& HI J 4  t d1BgQUo KK &LL'E gt $w<5;;q>1 !#g,/A%++a.AQQ\] 1$G, 33af 99 3( 3DU[[MQ\]3  8k!MN ++E2 3 }} 3$W]]O3DU[[MQ\]3 KK4TF; < ++E2GLtT"tg~t,ti'.O'/R KK7 *//BS8T7UVW Lw   Q   N&FFsyy%++66F" 7==%++66 s;K);K%;L" K"% L.,LL" M+,MMcpeZdZfdZdZeddZeddZ d dZd dZ d dZ xZ S) XLNetRelativeAttentionct||j|jzdk7r%t d|jd|j|j|_|j |_|j|_d|j dzz |_tjtj|j|j|j |_ tjtj|j|j|j |_ tjtj|j|j|j |_ tjtj|j|j|j |_tjtj|j|j|j |_tjtj|j|j |_tjtj|j|j |_tjtj|j|j |_tjtjd|j|j |_tj(|j|j*|_tj.|j0|_y)NrzThe hidden size (z6) is not a multiple of the number of attention heads (r?eps)super__init__d_modeln_head ValueErrord_headscaler Parameterr` FloatTensorr0r1r3r/r2r9r;r:r< LayerNormlayer_norm_epsr.Dropoutdropoutselfr> __class__s rHryzXLNetRelativeAttention.__init__s$  >>FMM )Q .#FNN#34 --*  mm mm ~~ &--,- e// T[[YZe// T[[YZe// T[[YZe// T[[YZe// T[[YZ U%6%6t{{DKK%PQ  U%6%6t{{DKK%PQ  U%6%6t{{DKK%PQ e&7&74;; &TU,,v~~6;P;PQzz&..1 rJctNNotImplementedError)rheadss rH prune_headsz"XLNetRelativeAttention.prune_heads!!rJc @|j}|j|d|d|d|d}|dddf}|j|d|ddz |d|d}tj|dtj||j tj }|S)zbnijr )rrzibnd,snd->ibnszijbs,ibns->bnijiz ijbn->bnijgꌠ9Y>)Fdimzbnij,jbnd->ibndz bnij->ijbn)r`einsumr:r9rr]r;r<r~rfloat16r functionalsoftmaxr)rq_headk_head_hv_head_hk_head_rseg_mat attn_mask head_maskoutput_attentionsacbdef attn_score attn_probattn_vecs rH rel_attn_corez$XLNetRelativeAttention.rel_attn_coresu\\+Vdmm-CX N\\+Vdmm-CX N  "((1+ 6 ?B.0FWB/"=B2gldjj0  %--/'%%,,|Y2W*WW '$lI1V*VV MM))*!)< LL+   !ELLy$III<< 19hG U\\, BB BrJctjd||j}|j|}|r||z}|j |}|S)zPost-attention processing.z ibnd,hnd->ibh)r`rr/rr.)rhrresidualattn_outoutputs rHpost_attentionz%XLNetRelativeAttention.post_attention7sG<<466B<<) !|H* rJc d||-|jdkDrtj||gd} n|} tjd| |j} tjd| |j } tjd||j }tjd||j}|j|| | |||| | }| r|\}}|j||}tjd||j}|Otjd||}|j|| | |||| | }| r|\}}tjd||}n |j|| | |||| | }| r|\}}|j||}| rf}n |-|jdkDrtj||gd} n|} tjd||j}tjd| |j} tjd| |j } tjd|j|j j|j }|j|| | |||| | }| r|\}}|j||}d}||f}| r|fz}|S)Nrrrz ibh,hnd->ibnd)rrrrzmbnd,mlb->lbndzlbnd,mlb->mbnd) rr`catrr1r3r2r0rrtyper)rrg attn_mask_h attn_mask_gr2rmemstarget_mappingrrrrrrq_head_h attn_vec_h attn_prob_houtput_hq_head_g attn_vec_g attn_prob_goutput_grroutputss rHforwardzXLNetRelativeAttention.forwardCs =DHHJNiiq q1||OS$&&AH||OS$&&AH||OQ?H||OQ?H++%#"3, J!*4' K**1j9H||OQ?H) <<(8(NS!//#)'&70  %.8+J "\\*:JW !//#)'&70  %.8+J **1j9H '4 DHHJNiiq q1||OQ?H||OS$&&AH||OS$&&AH||OQVVDFFLL5I466RH))%#"3* H!&.#)**1h7HHX&  ,GrJ)NNNF)T) __name__ __module__ __qualname__ryr staticmethodrrrrr __classcell__rs@rHrrrrsb28"    (2h (ErJrrc$eZdZfdZdZxZS)XLNetFeedForwardct|tj|j|j |_tj|j|j|_ tj|j|j|_ tj|j|_ t|jtrt |j|_y|j|_y)Nrv)rxryrrrzrr.Lineard_innerr5r6rrrZ ff_activationstrr activation_functionrs rHryzXLNetFeedForward.__init__s ,,v~~6;P;PQyy@ yy@ zz&..1 f**C 0'-f.B.B'CD $'-';';D $rJc|}|j|}|j|}|j|}|j|}|j|}|j ||z}|Sr)r5rrr6r.)rinprs rHrzXLNetFeedForward.forwardsff%))&1f%f%f%#. rJ)rrrryrrrs@rHrrs  <rJrc4eZdZfdZ ddZdZxZS) XLNetLayerct|t||_t ||_t j|j|_|j|_ d|_ yNr) rxryrrr-rr4rrrchunk_size_feed_forward seq_len_dimrs rHryzXLNetLayer.__init__sP .v6 "6*zz&..1 '-'E'E$rJc |j||||||||| |  } | dd\}}|,t|j|j|j|}t|j|j|j|}||f| ddz} | S)N)rrrrru)r-rff_chunkrr) rrrrrr2rrrrrrs rHrzXLNetLayer.forwards--     )/  %Ra[(  0 t;;T=M=MxH-T]]Dinitializer_ranger$zero_ Embedding padding_idxrfill_rrr0r1r3r/r2r9r;r:r< XLNetModelr*)rmoduleparams rH _init_weightsz"XLNetPreTrainedModel._init_weightss fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S )  6 7    P ""1N1N"O P + OO ( (ct{{7T7T ( U,rJN) rrr__doc__r config_classrpload_tf_weightsbase_model_prefixrrJrHrrs L.O%VrJrceZdZUdZej ed<dZee ej ed<dZ ee ej dfed<dZ ee ej dfed<y)XLNetModelOutputa@ Output type of [`XLNetModel`]. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, num_predict, hidden_size)`): Sequence of hidden-states at the last layer of the model. `num_predict` corresponds to `target_mapping.shape[1]`. If `target_mapping` is `None`, then `num_predict` corresponds to `sequence_length`. mems (`List[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. last_hidden_stateNr. hidden_states attentions) rrrrr`r__annotations__rrrrrrrrJrHrr>sm4(((.2D(4))* +2=AM8E%"3"3S"89:A:>Ju00#567>rJrceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejdfed<dZee ejdfed<y) XLNetLMHeadModelOutputa  Output type of [`XLNetLMHeadModel`]. Args: loss (`torch.FloatTensor` of shape *(1,)*, *optional*, returned when `labels` is provided) Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, num_predict, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). `num_predict` corresponds to `target_mapping.shape[1]`. If `target_mapping` is `None`, then `num_predict` corresponds to `sequence_length`. mems (`List[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlosslogitsr.rrrrrrrrr`rrrrrrrrrrJrHrr`s8)-D(5$$ %,*.FHU&& '..2D(4))* +2=AM8E%"3"3S"89:A:>Ju00#567>rJrceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejdfed<dZee ejdfed<y) $XLNetForSequenceClassificationOutputa_ Output type of [`XLNetForSequenceClassification`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `label` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). mems (`List[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nrrr.rrrrrJrHr r 2)-D(5$$ %,*.FHU&& '..2D(4))* +2=AM8E%"3"3S"89:A:>Ju00#567>rJr ceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejdfed<dZee ejdfed<y) !XLNetForTokenClassificationOutputa% Output type of [`XLNetForTokenClassificationOutput`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided) : Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_labels)`): Classification scores (before SoftMax). mems (`List[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nrrr.rrrrrJrHr r r rJr ceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejdfed<dZee ejdfed<y) XLNetForMultipleChoiceOutputad Output type of [`XLNetForMultipleChoice`]. Args: loss (`torch.FloatTensor` of shape *(1,)*, *optional*, returned when `labels` is provided): Classification loss. logits (`torch.FloatTensor` of shape `(batch_size, num_choices)`): *num_choices* is the second dimension of the input tensors. (see *input_ids* above). Classification scores (before SoftMax). mems (`List[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nrrr.rrrrrJrHrrs6)-D(5$$ %,*.FHU&& '..2D(4))* +2=AM8E%"3"3S"89:A:>Ju00#567>rJrceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ eeejdfed<dZeeejdfed <y) %XLNetForQuestionAnsweringSimpleOutputa Output type of [`XLNetForQuestionAnsweringSimple`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length,)`): Span-start scores (before SoftMax). end_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length,)`): Span-end scores (before SoftMax). mems (`List[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr start_logits end_logitsr.rr)rrrrrrr`rrrrrrrrrrrJrHrrs6)-D(5$$ %,04L(5,,-4.2J**+2.2D(4))* +2=AM8E%"3"3S"89:A:>Ju00#567>rJrceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZeejed<dZeeejed <dZeeejd fed <dZeeejd fed <y) XLNetForQuestionAnsweringOutputam Output type of [`XLNetForQuestionAnswering`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned if both `start_positions` and `end_positions` are provided): Classification loss as the sum of start token, end token (and is_impossible if provided) classification losses. start_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top config.start_n_top start token possibilities (beam-search). start_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top config.start_n_top start token possibilities (beam-search). end_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). end_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). cls_logits (`torch.FloatTensor` of shape `(batch_size,)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the `is_impossible` label of the answers. mems (`List[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states. Can be used (see `mems` input) to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nrstart_top_log_probsstart_top_indexend_top_log_probs end_top_index cls_logitsr.rr)rrrrrrr`rrrr LongTensorrrrrrrrrrrJrHrrs!F)-D(5$$ %,7;%"3"34;26OXe../659x 1 12904M8E,,-4.2J**+2.2D(4))* +2=AM8E%"3"3S"89:A:>Ju00#567>rJra? This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`XLNetConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aP Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) mems (`List[torch.FloatTensor]` of length `config.n_layers`): Contains pre-computed hidden-states (see `mems` output below) . Can be used to speed up sequential decoding. The token ids which have their past given to this model should not be passed as `input_ids` as they have already been computed. `use_mems` has to be set to `True` to make use of `mems`. perm_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length, sequence_length)`, *optional*): Mask to indicate the attention pattern for each input token with values selected in `[0, 1]`: - if `perm_mask[k, i, j] = 0`, i attend to j in batch k; - if `perm_mask[k, i, j] = 1`, i does not attend to j in batch k. If not set, each token attends to all the others (full bidirectional attention). Only used during pretraining (to define factorization order) or for sequential decoding (generation). target_mapping (`torch.FloatTensor` of shape `(batch_size, num_predict, sequence_length)`, *optional*): Mask to indicate the output tokens to use. If `target_mapping[k, i, j] = 1`, the i-th predict in batch k is on the j-th token. Only used during pretraining for partial prediction or for sequential decoding (generation). token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) input_mask (`torch.FloatTensor` of shape `{0}`, *optional*): Mask to avoid performing attention on padding token indices. Negative of `attention_mask`, i.e. with 0 for real tokens and 1 for padding which is kept for compatibility with the original code base. Mask values selected in `[0, 1]`: - 1 for tokens that are **masked**, - 0 for tokens that are **not masked**. You can only uses one of `input_mask` and `attention_mask`. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z_The bare XLNet Model transformer outputting raw hidden-states without any specific head on top.c!eZdZfdZdZdZdZdZdZe ddZ ddZ e e jd eeee  dd eej*d eej*d eej*deej*deej*deej*deej*deej*deej*deedeedeedeedeeeffdZxZS)rct|||j|_|j|_|j|_|j |_|j |_|j|_|j|_|j|_ tj|j|j|_ tjtj dd|j|_tj$t'|jDcgc] }t)|c}|_tj,|j.|_|j1ycc}wr)rxrymem_len reuse_lenrz same_length attn_typebi_data clamp_lenn_layerrr vocab_sizer)rr`rr* ModuleListrangerr,rr post_init)rr>_rs rHryzXLNetModel.__init__s  ~~ ))~~ !--))~~ ))~~  ll6+<+ < qlen > < qlen > ^ [0 0 0 0 0 1 1 1 1] [0 0 0 0 0 1 1 1 1] [0 0 0 0 0 0 1 1 1] [1 0 0 0 0 0 1 1 1] qlen [0 0 0 0 0 0 0 1 1] [1 1 0 0 0 0 0 1 1] [0 0 0 0 0 0 0 0 1] [1 1 1 0 0 0 0 0 1] v [0 0 0 0 0 0 0 0 0] [1 1 1 1 0 0 0 0 0] )rNrr)r`onesrr triltriu_)rqlenmlenmaskmask_los rH create_maskzXLNetModel.create_masks$zz4-dkkB   1ete8n))"-G JJtax ETENg %N  JJtax  rJc2|j|jdkDr|d|j}|j|jdk(rd}n |j }|||d}|j Stj||gd|d}|j S)Nrr)rrr`rdetach)rcurr_outprev_memcutoffnew_mems rH cache_memzXLNetModel.cache_mems >> %$..1*< 0$..1H << 4<<1#4Fll]F  vw'G~~ii8 4!idrr)r`rrsincosexpand)pos_seqinv_freqbsz sinusoid_inppos_embs rHpositional_embeddingzXLNetModel.positional_embeddingsh||IwA ))UYY|4eii 6MNTVW!T1*% ?nnRb1GrJctjd|jdtjj }dtj d||jz z }|j dk(r|| }}n-|j dk(r|d}}ntd |j d |jr7tj||d tjj }tj| | d tjj } |jdkDrN|j|j |j}| j|j |j} |-|j|||d z} |j| ||d z} n$|j||} |j| |} tj| | gd} | Stj||d tjj }|jdkDr'|j|j |j}|j|||} | S)Nrg@rri'biunirzUnknown `attn_type` .grrur) r`rrzint64floatpowr!r|r"r#clamprNr) rr9rrKfreq_seqrJbegend fwd_pos_seq bwd_pos_seq fwd_pos_emb bwd_pos_embrMs rHrelative_positional_encodingz'XLNetModel.relative_positional_encodings<<4<<EKKHNNPuyyDLL)@BB >>T !dUC ^^u $RC3DNN3C1EF F <<,,sCU[[IOOQK,,tcT3ekkJPPRK~~!)//P )//P "77 XsVWxX "77 XsVWxX "77 XN "77 XN iik :BG  ,,sCU[[IOOQK~~!)//P // XsKGrJbatch_size, sequence_length checkpoint output_typer input_idsattention_maskr perm_maskrtoken_type_ids input_maskr inputs_embedsuse_memsroutput_hidden_states return_dictreturnc F| | n|jj} | | n|jj} | | n|jj} d|vrt j dt |d} |jr| | n|jj} n| | n|jj} | | td|?|jddj}|jd|jd}}nL| ?| jddj} | jd| jd}}n td| |jddjnd}| |jddjnd}| |jddjnd}|!|jdddjnd}|!|jdddjnd}||d|djdnd}||z}|j}|j }|j"dk(r |j%||}|ddddddf}n*|j"d k(rd}ntd |j"| |Jd  ||d |z }| | |d|z}n|||d}n |||}nd}|v|dkDrMt'j(|jd||gj+|}t'j,||gd }||dddddddf}n||dddddddfz }||dkDj+|}|t'j.|j+| }|dkDr=t'j,t'j(||gj+||gd }||ddddddfzdkDj+|}nd}| | }n|j1|}|j3|}|<|j4j7|jd|d}|j3|}nd}||dkDrBt'j(||gt&j8|}t'j,||gd }n|}|dddf|dddfk7j9}t:j<j?|dj+|}nd}|jA|||} | j+|j } |j3| } ||jCdk(r^|jEdjEdjEdjEd}|j7|jFdddd}nB|jCdk(r/|jEdjEdjEd}|j+tI|jKj}ndg|jFz}d}!|dgtM|jNz}| rgnd}"| rgnd}#tQ|jND]p\}$}%| r|!|jS|||$fz}!| r|#jU|||fn||%||||| |||$|||$|  }&|&dd\}}| s]|"jU|&dr| r|#jU|||fn||j3||n|}'|'jdddj}'| sd}!| r'|tWd|#D}#ntWd|#D}#| r'|tWd|"D}"ntWd|"D}"| stWd|'|!|#|"fDStY|'|!|#|"S)N use_cachezgThe `use_cache` argument is deprecated and will be removed in a future version, use `use_mems` instead.zDYou cannot specify both input_ids and inputs_embeds at the same timerrz5You have to specify either input_ids or inputs_embedsrurRrQzUnsupported attention type: z8You can only use one of input_mask (uses 1 for padding) rrrrr) num_classes)rKrPr)rrr2rrrrrc3lK|],}|D]%}|jdddj'.ywrrruNpermute contiguous).0hsrs rH z%XLNetModel.forward..s5%j"gi%jbcaii1a&8&C&C&E%j&E%js24c3^K|]%}|jdddj'ywrsrt)rwrxs rHryz%XLNetModel.forward..s&%_2bjjAq&9&D&D&F%_s+-c3@K|]}td|Dyw)c3`K|]&}|jddddj(ywrur rrNrt)rw att_streams rHryz/XLNetModel.forward...s*Z**,,Q1a8CCEZ,.N)tuplerwts rHryz%XLNetModel.forward..s!#_`EZXYZZ#sc3`K|]&}|jddddj(ywr}rtrs rHryz%XLNetModel.forward..s("Z!199Q1a#8#C#C#E"Zrc3&K|] }|| ywrr)rwr3s rHryz%XLNetModel.forward..scqUVUbcs)rrrr)-r>rrkuse_return_dictwarningswarn FutureWarningtraininguse_mems_train use_mems_evalr|rYrvr]rurrr!r=r`zerostoreyer)rr*rHrrrone_hotr_r unsqueezer$next parametersr\r,r+rDr8rr)(rrdrerrfrrgrhrrirjrrkrlkwargsr9rKr:r dtype_floatrr data_mask mems_mask non_tgt_mask word_emb_kr word_emb_qrmem_padcat_idsrrMnew_memsrrrA layer_modulerrs( rHrzXLNetModel.forwards.2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "!++Aq1<<>I!*IOOA,>#D  &)33Aq9DDFM%++A. 0C0CA0F#DTU UHVHb11!Q7BBDhl@J@VZ))!Q/::<\` HVHb11!Q7BBDhl?H?TI%%aA.99;Z^ IWIc//1a8CCEim#'#3Q8KtAw}}QQRd{jj  >>U "((t4I!!Qd"23I ^^t #I;DNN;KLM M!^%;w=ww;g  ."<~-J  !i&;"4(94I  # (9"4(I  I$9!II  ax!KK);T3(GHKKIV !IIy)&hHM1a(335H # %%jP]%j j %%_Q^%_ _ )"#dn# #"Zz"ZZ cVX}j$Qcc c$8=]g  rJr) NNNNNNNNNNNNN)rrrryr-r1r4r=rDrrNr_rXLNET_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOCrr`Tensorboolrrrrrs@rHrrs (#-"8 *$L++A+H+HIf+gh&$$-115'+,01515-1,004#',0/3&*\ ELL)\ !.\ u||$ \ ELL) \ !. \ !.\ U\\*\ ELL)\   -\ 4.\ $D>\ 'tn\ d^\ u&& '!\ i \ rJrzt XLNet Model with a language modeling head on top (linear layer with weights tied to the input embeddings). c#teZdZdgZfdZdZdZddZee jde e e  ddeej d eej d eej d eej d eej d eej deej deej deej deej deedeedeedeedeee ffdZed eej dej deej fdZxZS)XLNetLMHeadModelzlm_loss.weightc t|||j|_|j|_t ||_t j|j|jd|_ |jy)NT)r$) rxryr!r rrrrrzr%rr(rs rHryzXLNetLMHeadModel.__init__ sb  ))!--%f-yy1B1BN  rJc|jSrrr,s rHget_output_embeddingsz&XLNetLMHeadModel.get_output_embeddingss ||rJc||_yrrr/s rHset_output_embeddingsz&XLNetLMHeadModel.set_output_embeddingss % rJc b |jd}tj|dftj|j}d |r$tj |dd df|gd}ntj ||gd}|jd}tj|||ftj |j}d|dddddf<tj|d|ftj |j} d| ddddf<||| |d} |rt fd |D| d <| S) Nrrrprurrr)rdrfrrjc3<K|]}|d ddddfywrr)rw layer_pastoffsets rHryzAXLNetLMHeadModel.prepare_inputs_for_generation..Fs""`*:hwh1n#="`sr)r]r`rrrrrUr) rrdpast_key_valuesrjreffective_batch_size dummy_tokensequence_lengthrfrinputsrs @rHprepare_inputs_for_generationz.XLNetLMHeadModel.prepare_inputs_for_generations0 )q1kk#7";5::V_VfVfg   9Q[#9;"GQOI 9k":BI$//!,KK !?O DEKK`i`p`p " !Q( !1o 6ekkR[RbRb $'q!Rx #",   ""`P_"``F6N rJr`rcrrdrerrfrrgrhrrilabelsrjrrkrlrmc ||n|jj}|j|f|||||||| | | | |d |}|j|d}d}| At }||j d|j d| j d}|s|f|ddz}||f|zS|St|||j|j|jS)aI labels (`torch.LongTensor` of shape `(batch_size, num_predict)`, *optional*): Labels for masked language modeling. `num_predict` corresponds to `target_mapping.shape[1]`. If `target_mapping` is `None`, then `num_predict` corresponds to `sequence_length`. The labels should correspond to the masked input words that should be predicted and depends on `target_mapping`. Note in order to perform standard auto-regressive language modeling a ** token has to be added to the `input_ids` (see the `prepare_inputs_for_generation` function and examples below) Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored, the loss is only computed for labels in `[0, ..., config.vocab_size]` Return: Examples: ```python >>> from transformers import AutoTokenizer, XLNetLMHeadModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("xlnet/xlnet-large-cased") >>> model = XLNetLMHeadModel.from_pretrained("xlnet/xlnet-large-cased") >>> # We show how to setup inputs to predict a next token using a bi-directional context. >>> input_ids = torch.tensor( ... tokenizer.encode("Hello, my dog is very ", add_special_tokens=False) ... ).unsqueeze( ... 0 ... ) # We will predict the masked token >>> perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float) >>> perm_mask[:, :, -1] = 1.0 # Previous tokens don't see last token >>> target_mapping = torch.zeros( ... (1, 1, input_ids.shape[1]), dtype=torch.float ... ) # Shape [1, 1, seq_length] => let's predict one token >>> target_mapping[ ... 0, 0, -1 ... ] = 1.0 # Our first (and only) prediction will be the last token of the sequence (the masked token) >>> outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping) >>> next_token_logits = outputs[ ... 0 ... ] # Output has shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size] >>> # The same way can the XLNetLMHeadModel be used to be trained by standard auto-regressive language modeling. >>> input_ids = torch.tensor( ... tokenizer.encode("Hello, my dog is very ", add_special_tokens=False) ... ).unsqueeze( ... 0 ... ) # We will predict the masked token >>> labels = torch.tensor(tokenizer.encode("cute", add_special_tokens=False)).unsqueeze(0) >>> assert labels.shape[0] == 1, "only one word will be predicted" >>> perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float) >>> perm_mask[ ... :, :, -1 ... ] = 1.0 # Previous tokens don't see last token as is done in standard auto-regressive lm training >>> target_mapping = torch.zeros( ... (1, 1, input_ids.shape[1]), dtype=torch.float ... ) # Shape [1, 1, seq_length] => let's predict one token >>> target_mapping[ ... 0, 0, -1 ... ] = 1.0 # Our first (and only) prediction will be the last token of the sequence (the masked token) >>> outputs = model(input_ids, perm_mask=perm_mask, target_mapping=target_mapping, labels=labels) >>> loss = outputs.loss >>> next_token_logits = ( ... outputs.logits ... ) # Logits have shape [target_mapping.size(0), target_mapping.size(1), config.vocab_size] ```N rerrfrrgrhrrirjrrkrlrrrrrrrr) r>rrrr viewsizerrrr)rrdrerrfrrgrhrrirrjrrkrlrtransformer_outputsrrloss_fctrs rHrzXLNetLMHeadModel.forwardJsp&1%{ 'tn{ d^{ " u,, -#{ ai{ zcT%,,/c5<<cDQVQ]Q]L^ccrJrz XLNet Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c#eZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e e jd e e jd e e jde ede ede ede edeee ffdZxZS)XLNetForSequenceClassificationct|||j|_||_t ||_t ||_tj|j|j|_ |jyr) rxry num_labelsr>rrrr rrrzr!r(rs rHryz'XLNetForSequenceClassification.__init__sg   ++ %f- / 799V^^V5F5FG rJr`rardrerrfrrgrhrrirrjrrkrlrmc b||n|jj}|j|f|||||||| | | | |d |}|d}|j|}|j |}d}| |jj |j dk(rd|j_nl|j dkDrL| jtjk(s| jtjk(rd|j_nd|j_|jj dk(rIt}|j dk(r&||j| j}n||| }n|jj dk(r=t}||jd|j | jd}n,|jj dk(rt}||| }|s|f|ddz}||f|zS|St!|||j"|j$|j& S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrr regressionsingle_label_classificationmulti_label_classificationrr)r>rrr r! problem_typerrr`rintr squeezer rr r rrr)rrdrerrfrrgrhrrirrjrrkrlrrrrrrs rHrz&XLNetForSequenceClassification.forwards<&1%P 'tnP d^P " u:: ;#P i P rJrz XLNet Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. c#eZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e e jd e e jd e e jde ede ede ede edeee ffdZxZS)XLNetForTokenClassificationct|||j|_t||_t j |j|j|_|jyr) rxryrrrrr hidden_size classifierr(rs rHryz$XLNetForTokenClassification.__init__HT   ++%f-))F$6$68I8IJ rJr`rardrerrfrrgrhrrirrjrrkrlrmc ||n|jj}|j||||||||| | | | | }|d}|j|}d}| rrrr rrr rrr)rrdrerrfrrgrhrrirrjrrkrlrrsequence_outputrrrrs rHrz#XLNetForTokenClassification.forwardRs:&1%< 'tn< d^< " u77 8#< i < rJrz XLNet Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RACE/SWAG tasks. c#eZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e e jd e e jd e e jde ede ede ede edeee ffdZxZS)XLNetForMultipleChoicect||t||_t ||_t j|jd|_ |jyr) rxryrrrr rrrzr!r(rs rHryzXLNetForMultipleChoice.__init__sM  %f- / 799V^^Q7 rJz(batch_size, num_choices, sequence_lengthrardrgrhrerrfrrrirrjrrkrlrmc ||n|jj}||jdn| jd}|!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}| 1| jd| j d| j dnd}|j |f||||||||| | | |d |}|d}|j |}|j|}|jd|}d}| "t}||| jd}|s|f|ddz}||f|zS|St|||j|j|jS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr) rgrhrerrfrrrirjrrkrlrr) r>rr]rrrr r!r rrrr)rrdrgrhrerrfrrrirrjrrkrlr num_choicesflat_input_idsflat_token_type_idsflat_attention_maskflat_input_maskflat_inputs_embedsrrrreshaped_logitsrrs rHrzXLNetForMultipleChoice.forwards<&1%?^bR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrvFPF\*//"joob.ABbf(   r=#5#5b#9=;M;Mb;Q R  /d..  .&.),/!5#  "%Q'&&v.!!&) ++b+6  ')HOV[[_=D%'*=ab*AAF)-)9TGf$ Ev E+"$))-;;*55   rJr)rrrryrrrrrrrrr`rrrrrrrs@rHrrs++A+H+HIs+tu&0$-115-115'+,015,004)-#',0/3&*L ELL)L !.L U\\* L !. L u||$ L ELL)L !.L ELL)L   -L &L 4.L $D>L 'tnL d^L " u22 3#L v L rJrz XLNet Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). c%eZdZfdZeej deee e  dde e jde e jde e jde e jde e jd e e jd e e jd e e jd e e jd e e jde e jde ede ede ede edeee ff dZxZS)XLNetForQuestionAnsweringSimplect|||j|_t||_t j |j|j|_|jyr) rxryrrrrrr qa_outputsr(rs rHryz(XLNetForQuestionAnsweringSimple.__init__rrJr`rardrerrfrrgrhrristart_positions end_positionsrjrrkrlrmc J||n|jj}|j|f|||||||| | | ||d |}|d}|j|}|j dd\}}|j dj }|j dj }d}| | t| jdkDr| j d} t| jdkDr| j d} |jd}| jd|} | jd|} t|}||| }||| }||zdz }|s||f|ddz}||f|zS|St||||j|j|j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrrrr) ignore_indexru)rrrrrr)r>rrrsplitrrvr\rrWr rrrr)rrdrerrfrrgrhrrirrrjrrkrlrrrrrr total_loss ignored_indexr start_lossend_lossrs rHrz'XLNetForQuestionAnsweringSimple.forwardsF&1%S 'tnS d^!S $ u;; <%S i S rJrc+teZdZfdZeej deee  dde e jde e jde e jde e jde e jd e e jd e e jd e e jd e e jd e e jde e jde e jde e jde e jde e de e de e de e deeeff&dZxZS)XLNetForQuestionAnsweringct|||j|_|j|_t ||_t ||_t||_ t||_ |jyr) rxry start_n_top end_n_toprrrrrrr answer_classr(rs rHryz"XLNetForQuestionAnswering.__init__tsj  !--))%f--f5)&1-f5 rJr`rrdrerrfrrgrhrrirr is_impossible cls_indexp_maskrjrrkrlrmc ||n|jj}|j|f|||||||| ||||d |}|d}|j||}|dd}| | | | | | fD]*}||j dkDs|j d,|j || |}t}||| }||| }||zdz }| ;| 9|j|| | }tj}||| } || d zz }|s |f|ddzSt||j|j|j S|j\}!}"}#tj j#|d }$t%j&|$|j(d \}%}&|&j+dj-dd|#}'t%j.|d |'}(|(j+dj-d|"dd}(|j+dj1|(})||j+dnd}|j |)|(|}tj j#|d }*t%j&|*|j2d \}+},|+j5d|j(|j2z}+|,j5d|j(|j2z},t%j6d||$}(|j||(| }|s|%|&|+|,|f}||ddzSt|%|&|+|,||j|j|jS)aJ start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. is_impossible (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels whether a question has an answer or no answer (SQuAD 2.0) cls_index (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the classification token to use as input for computing plausibility of the answer. p_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Optional mask of tokens which can't be in answers (e.g. [CLS], [PAD], ...). 1.0 means token should be masked. 0.0 mean token is not masked. Returns: Example: ```python >>> from transformers import AutoTokenizer, XLNetForQuestionAnswering >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("xlnet/xlnet-base-cased") >>> model = XLNetForQuestionAnswering.from_pretrained("xlnet/xlnet-base-cased") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze( ... 0 ... ) # Batch size 1 >>> start_positions = torch.tensor([1]) >>> end_positions = torch.tensor([3]) >>> outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions) >>> loss = outputs.loss ```Nrr)rrr)rrru)rrrt)rrrrrr) start_statesrz blh,bl->bh)rr)rrrrrrrr)r>rrrrsqueeze_rr rrr rrrrrrrr`topkrrrHgather expand_asrrr)-rrdrerrfrrgrhrrirrrrrrjrrkrlrrrrrrrrrrrr loss_fct_clscls_lossrKslenhszstart_log_probsrrstart_top_index_exprhidden_states_expanded end_log_probsrrs- rHrz!XLNetForQuestionAnswering.forwards|&1%>>6#,11"5"C"C2== +//1NCs mm33Lb3IO38::!1!1r4 0 #2";";B"?"F"Fr2s"S  << r;NOL'11!4;;Bb"ML%2%<%#a$'tn%a&d^'a* u55 6+ajiarJr) rrrrrrrrrpr)Arr dataclassesrtypingrrrrr`rtorch.nnr r r activationsr generationrmodeling_utilsrrrrr pytorch_utilsrutilsrrrrrrconfiguration_xlnetr get_loggerrrRrrrIrpModulerrrrrrrr r rrrXLNET_START_DOCSTRINGrrrrrrrr__all__rrJrHrs !// AA!)uu6-   H %.Pf@FARYYAHryy.//d'V?'VT ?{? ?B !?[!? !?H ?;? ?B ? ? ?B  ?; ?  ?F !?K!? !?H ,?k,? ,?^ FReb %b b J   Fc+_Fc  FcR c %9c c L M "6M M ` ] 1] ] @ d &:d d N q 4qqh rJ