T^(hdZddlZddlZddlZddlZddlmZmZmZm Z ddl Z ddl m Z ddl m Z mZmZddlmZddlmZmZmZmZdd lmZdd lmZdd lmZmZmZmZm Z m!Z!m"Z"dd l#m$Z$dd l%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4m5Z5ddl6m7Z7e.r ddl8m9Z9ddl:m;Z;e1jxe=Z>dZ?dZ@dZAdZBdZCGdde jZE ddlFmGZGeGZEe>jde&jeEGdde jZMGd d!e jZNGd"d#e jZOGd$d%e jZPGd&d'e jZQGd(d)e jZRGd*d+e jZSGd,d-e jZTGd.d/e$ZUGd0d1eUZVd2ZWd3ZXd4ZYd5ZZe,d6eWGd7d8eUZ[e,d9eWGd:d;eUeZ\e,deUZ]e,d?eWGd@dAeUZ^e,dBeWGdCdDeUZ_e,dEeWGdFdGeUZ`gdHZay#eI$rYpeJ$re>jdYwxYw)IzPyTorch T5 model.N)ListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache StaticCache)GenerationMixin)AttentionMaskConverter)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)ALL_LAYERNORM_LAYERS find_pruneable_heads_and_indicesprune_linear_layer) DUMMY_INPUTS DUMMY_MASKadd_start_docstrings%add_start_docstrings_to_model_forwardis_torch_flex_attn_availableis_torch_fx_proxyis_torchdynamo_compilingloggingreplace_return_docstrings)assert_device_mapget_device_map)T5Config) BlockMask)make_flex_block_causal_maskr*zgoogle-t5/t5-smallc  ddl}ddl}ddl}t jj|}tjd||jj|}g}i} |D]R\} } tjd| d| |jj|| } |j| | | | <T|D]E} | jd} td| Dr:tjd dj!| | j#| dad | d vr:tjd dj!| | j#| d|}| | } | D]}|j%d |r|jd |}n|g}|ddvrt'|d}nE|ddk(rt'|d}|d}n*|ddk(rt'|d}|d}n|ddk(rt'|d}|d}n|ddk(r2t)|dr t'|d}nt)|drt'|d}n|ddk(r t'|d}n|ddk(s|ddk(r t'|d}n|ddk(r t'|d}nt|ddk(r | dd k(r$|dd k(r t'|d!}nM|dd"k(r5t+|dkDr'|dj-rt'|d#|d}v t'||d}t+|dk\st1|d}||}ddvr t'|d}|dd$k7r6tjd%| j2d&| |j5| } |j2| j2k7r&t7d'|j2d(| j2d) tjd*| t=j>| jA|jB|_"| j#| dHtjd+d,j!| jGd-|S#t$rtj dwxYw#t.$r+tjd dj!| Y,wxYw#t8$r1}|xj:|j2| j2fz c_d}~wwxYw).z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape /c3$K|]}|dv yw))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0ns `/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py z(load_tf_weights_in_t5..ks  n n z Skipping _slot_z [A-Za-z]+_\d+z_(\d+))kernelscale embeddingweightself_attentionlayerenc_dec_attentionr)dense_relu_denserms_norm layer_normfinal_layer_normr> output_biasbetabiassquad classifierdecoderlogitslm_headwiwi_r?z"Transposing numpy weight of shape z for zPointer shape z and array shape mismatchedzInitialize PyTorch weight z%Weights not copied to PyTorch model: z, .)$renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendsplitanyjoinpop fullmatchgetattrhasattrlenisdigitAttributeErrorintshape transpose ValueErrorAssertionErrorargstorch from_numpyastypefloat32datakeys)modelconfigtf_checkpoint_pathrUnptftf_path init_varsnames tf_weightsnamernarraytxt_namepointerm_name scope_namesnumes r8load_tf_weights_in_t5rNs ggoo01G KK8 BC''0I EJ ! e (l5'BC&&w5 T 4 ! J'~~c"     KK)CHHTN#34 5 NN8T *  tBx  KK)CHHTN#34 5 NN8T * 8$* 'F||,f5 hhy&9 %h 1~!AA!'84Q#33!'73!!*Q#66!'73!!*Q#55!'73!!*Q:-7L1%g|G;1$+a.)!#,U* 'V q>!A Agx0G q>[ ( KK'--@QRWR]R]Q^^i!jkk,  078'' RZZ(@A x&UJ'X KK7 *//BS8T7UUVWX L   Q   R&KK)CHHTN+; <=  FFw}}ekk2 2F  s5 P>Q!&?R> Q!0RR S!,S  SaR This is an experimental feature and is a subject to change at a moment's notice. Uses a device map to distribute attention modules of the model across several devices. If no device map is given, it will evenly distribute blocks across all devices. Args: device_map (`Dict[int, list]`, *optional*): A dictionary that maps attention modules to devices. Note that the embedding module and LMHead are always automatically mapped to the first device (for esoteric reasons). That means that the first device should have fewer attention modules mapped to it than other devices. For reference, the t5 models have the following number of attention modules: - google-t5/t5-small: 6 - google-t5/t5-base: 12 - google-t5/t5-large: 24 - google-t5/t5-3b: 24 - google-t5/t5-11b: 24 Example: ```python # Here is an example of a device map on a machine with 4 GPUs using google-t5/t5-3b, which has a total of 24 attention modules: model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-3b") device_map = { 0: [0, 1, 2], 1: [3, 4, 5, 6, 7, 8, 9], 2: [10, 11, 12, 13, 14, 15, 16], 3: [17, 18, 19, 20, 21, 22, 23], } model.parallelize(device_map) ``` a4 Moves the model to cpu from a model parallel state. Example: ```python # On a 4 GPU machine with google-t5/t5-3b: model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-3b") device_map = { 0: [0, 1, 2], 1: [3, 4, 5, 6, 7, 8, 9], 2: [10, 11, 12, 13, 14, 15, 16], 3: [17, 18, 19, 20, 21, 22, 23], } model.parallelize(device_map) # Splits the model across several devices model.deparallelize() # Put the model back on cpu and cleans memory by calling torch.cuda.empty_cache() ``` c&eZdZdfd ZdZxZS) T5LayerNormct|tjt j ||_||_y)zc Construct a layernorm module in the T5 style. No bias and no subtraction of mean. N)super__init__r Parameterrsonesr@variance_epsilon)self hidden_sizeeps __class__s r8rzT5LayerNorm.__init__s1 ll5::k#:; #c|jtjjdj dd}|tj ||j zz}|jjtjtjfvr%|j|jj}|j|zS)NrEr<T)keepdim) torsrvpowmeanrsqrtrr@dtypefloat16bfloat16)r hidden_statesvariances r8forwardzT5LayerNorm.forwards !##EMM266q9>>r4>P% Ht?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r)gư>)__name__ __module__ __qualname__rr __classcell__rs@r8rrs $ +rr) FusedRMSNormzODiscovered apex.normalization.FusedRMSNorm - will use it instead of T5LayerNormzBdiscovered apex but it failed to load, falling back to T5LayerNormc*eZdZdeffd ZdZxZS)T5DenseActDenserzc^t|tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_ yNFrK)rrrLineard_modeld_ffrQwoDropout dropout_ratedropoutr dense_act_fnactrrzrs r8rzT5DenseActDense.__init__sn ))FNNFKKeD))FKKeDzz&"5"56 &--.rc|j|}|j|}|j|}t|jj t jr|j|jj jk7r`|jj jt jk7r/|j|jj j}|j |}|SN) rQrr isinstancerr@rsTensorrint8rrrs r8rzT5DenseActDense.forward"s . /  ]3 tww~~u|| 4##tww~~';';;$$ 2),,TWW^^-A-ABM . rrrrr*rrrrs@r8rrs/x/ rrc*eZdZdeffd ZdZxZS)T5DenseGatedActDenserzct|tj|j|j d|_tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_yr)rrrrrrwi_0wi_1rrrrr rrrs r8rzT5DenseGatedActDense.__init__1s IIfnnfkkF IIfnnfkkF ))FKKeDzz&"5"56 &--.rc,|j|j|}|j|}||z}|j|}t |j j tjr|j|j j jk7r`|j j jtjk7r/|j|j j j}|j |}|Sr) rrrrrrr@rsrrrr)rr hidden_gelu hidden_linears r8rzT5DenseGatedActDense.forward9shhtyy78  -0 #m3  ]3 tww~~u|| 4##tww~~';';;$$ 2),,TWW^^-A-ABM . rrrs@r8rr0s/x/rrc*eZdZdeffd ZdZxZS) T5LayerFFrzct||jrt||_nt ||_t |j|j|_ tj|j|_ y)Nr)rr is_gated_actrDenseReluDenserrrlayer_norm_epsilonrGrrrrrs r8rzT5LayerFF.__init__Ns_    "6v">D "1&"9D %fnn&:S:STzz&"5"56 rcr|j|}|j|}||j|z}|Sr)rGrr)rrforwarded_statess r8rzT5LayerFF.forwardXs=??=9../?@% 5E(FF rrrs@r8rrMs7x7rrcneZdZ ddedeeffd ZdZed dZ d dZ d dZ xZ S) T5Attentionrz layer_idxct||j|_||_|j|_|j |_|j |_|j|_|j|_ |j|_ |j|jz|_ ||_|9|jr-tj!d|j"j$dt'j(|j |jd|_t'j(|j |jd|_t'j(|j |jd|_t'j(|j|j d|_|jr/t'j2|j|j|_t7|_d|_y)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Fr)rr is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerd_kvkey_value_proj_dim num_headsn_headsrr inner_dimrrY warning_oncerrrrqkvo Embeddingrelative_attention_biasset pruned_headsgradient_checkpointingrrzrrrs r8rzT5Attention.__init__`ss  +++F(.4.S.S+/5/U/U,~~ "(++'' ** (?(??"     *4>>+B+B*CD,, 4<<eD4<<eD4<<eD4>>4<=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rrEr)) rrslongabsmin zeros_likelogfloatmath full_likewhere)relative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larges r8_relative_position_bucketz%T5Attention._relative_position_buckets(,  A K !2Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $y0&/ II'--/); <hh|i/0 1Y& ( "UZZ. &" &+YY &8RT_bcTc(d& " EKK2CE_``rc | |jjj}|.tj|tj |dddf}n|dddfj |}tj|tj |dddf}||z }|j||j |j|j}|j|} | jgdjd} | S)z%Compute binned relative position biasN)rdevice)rrr)rErr)r) rr@rrsarangerrrrrrpermute unsqueeze) r query_length key_lengthrcache_positioncontext_positionmemory_positionrrelative_position_bucketvaluess r8 compute_biaszT5Attention.compute_biass >1188??F  !$||L SYZ[\^b[bc -ag699&A ,,zFSTXZ[T[\+.>>#'#A#A #.;;== $B$  --.FG *44Q7 rc |jdd\} } |du} |j|}|j| d|j|jj dd}|@|j j|j}| r |j}n |j}| r|n|}| r7|5r3j|j}|j|j}n|j|}|j|}|j| d|j|jj dd}|j| d|j|jj dd}|D| s| nd} j|||jd| i\}}| rd|j |j<t!j"||j dd}||jd}||n| ddz}|j$sZt!j&d|j| |f|j(|j* }|j,rE|j.r9d|_n1|j3|||j(| }|dddd| dddf}|#|ddddddd|jdf}||z}|j4rRt!j6|jd}d |t9|j4<|dd|j;f}n|}||z }t<j>jA|jCd jE|}t<j>jG||jF|j. }|||z}t!j"||}|j ddjI}|j| d|jJ}|jM|}|||f}| r||fz}|S)z Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). NrEr<r)r Tr rr)rr rr)ptraining)'rnrviewrrro is_updatedgetrcross_attention_cacheself_attention_cache key_cache value_cacherrupdatersmatmulrzerosrrrr requires_gradrrrlistboolr functionalsoftmaxrtype_asr contiguousrr)rrmaskkey_value_states position_biaspast_key_valuelayer_head_maskr use_cacheoutput_attentionsr  batch_size seq_lengthis_cross_attention query_statesrcurr_past_key_valuecurrent_states key_states value_statesscoresr real_seq_length causal_maskposition_bias_masked attn_weights attn_outputoutputss r8rzT5Attention.forwards$"/!4!4Ra!8 J.T9vvm, #((RtG^G^_iijkmno  %'2266t~~FJ!&4&J&J#&4&I&I#-?)] ."<,66t~~FJ.::4>>JL/J66.1L#RtG^G^_iijkmnoJ',,ZT\\4KbKbcmmnoqrsL)7It+>+E+E dnn?OQ_>`,( L&@DN--dnn=lJ,@,@A,FG  #))"-J.:.FlN[]L^abLbO33 %  j*=fmm[a[g[g! ..4==26M/ $ 1 1#Z Vd!2! !.aZKL!.C D "1a,Bj.>.>r.B,B#BC - ;   ::m11!45D,-Dd''( )#0DIIK#@ #0 &&}},,V\\^,DLLVT }},,\T\\TXTaTa,b   &'/9Lll<> !++Aq1<<> !&&z2t~~F ff[)  > /GrFN)T )NN) NNNNNNFFN) rrrr*rrmrr staticmethodrrrrrs@r8rr_si%*#' !,!,C= !,F; - - ^.irrcBeZdZddeeffd Z ddZxZS)T5LayerSelfAttentionrct|t||||_t |j |j |_tj|j|_ y)Nrrr) rrr SelfAttentionrrrrGrrrrrs r8rzT5LayerSelfAttention.__init__DsT ( 0KW` &fnn&:S:STzz&"5"56 rc |j|} |j| |||||||} ||j| dz}|f| ddz} | S)N)r&r(r*r)r+r,r rr))rGrDr) rrattention_maskr(r*r)r+r,r normed_hidden_statesattention_outputr;s r8rzT5LayerSelfAttention.forwardLsv $}=-- '+)/).  & 5Ea5H(II  "%5ab%99rr<)NNNNFFNrrrrrmrrrrs@r8rArACs07XVY]7rrAcDeZdZddeeffd Z ddZxZS)T5LayerCrossAttentionrct|t|d||_t |j |j |_tj|j|_ y)NFrCr) rrrEncDecAttentionrrrrGrrrr)rrzrrs r8rzT5LayerCrossAttention.__init__hsN *6u`ij%fnn&:S:STzz&"5"56 rc |j|} |j| |||||||| |  } ||j| dz} | f| ddz}|S)N) r&r'r(r*r)r+rr,r rr))rGrMr)rrr'rFr(r*r)r+rr,r rGrH layer_outputr;s r8rzT5LayerCrossAttention.forwardns{ $}=// -'+)%/)0  %t||4DQ4G'HH /$4QR$88rr)NNNNFNFNrIrs@r8rKrKgs27(3-7rrKcLeZdZddeeffd Z ddZxZS)T5Blockrcpt||j|_tj|_|j j t||||jr&|j j t|||j j t|y)NrC)r) rrrr ModuleListrBrbrArKrrs r8rzT5Block.__init__s  ++]]_   E`lu v  ?? JJ  3FiP Q )F+,rc |jd||||| | | | }|dd\}} |dd}|jtjk(rtjtj |j tj|jjdz tj|jj}tj|| |}|jxr|du}|r|jd|||||| | ddz| |  }|dd\}} |jtjk(rtjtj |j tj|jjdz tj|jj}tj|| |}||ddz}|jd|}|jtjk(rtjtj |j tj|jjdz tj|jj}tj|| |}|f}| r || fz|z}|S||z}|S) Nr)rFr(r*r)r+r,r rEi)rmaxr)r<)r'rFr(r*r)rr+r,) rBrrsrrisinfrdfinforUclampr)rrrFr(encoder_hidden_statesencoder_attention_maskencoder_decoder_position_biasr*cross_attn_layer_head_maskr)r+r, return_dictr self_attention_outputsattention_outputs clamp_valuedo_cross_attentioncross_attention_outputsr;s r8rzT5Block.forwards "/A )'+)/) " )?r(B% ~2126   %-- /++ M*..0 M//044t; M//044K "KK K<[YM!__R1Fd1R &3djjm!65; :-+B/!3#"3 ' #-DBQ,G )M>""emm3#kkKK .224KK 3 34884?KK 3 3488 !& M |Q\ ] !24KAB4O O ' 2}5    %-- /++ M*..0 M//044t; M//044K "KK K<[YM "  114EEG 11Grr<) NNNNNNNNFFTNrIrs@r8rQrQs@ -XVY] -"#&*#'TrrQc`eZdZdZdeffd ZdejdejfdZxZ S)T5ClassificationHeadz-Head for sentence-level classification tasks.rzc(t|tj|j|j|_tj |j|_tj|j|j|_ y)N)r) rrrrrdenserclassifier_dropoutr num_labelsout_projrs r8rzT5ClassificationHead.__init__sZ YYv~~v~~> zzF$=$=>  &..&2C2CD rrreturnc|j|}|j|}tj|}|j|}|j |}|Sr)rrfrstanhrirs r8rzT5ClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r) rrr__doc__r*rrsrrrrs@r8rdrds/7ExE U\\ellrrdcXeZdZdZeZeZdZdZ dZ dZ dZ dZ dgZdgZedZdZd Zy ) T5PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. transformerTFrQrcvtjt}tjt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)rstensorrr)rrs input_mask dummy_inputss r8rwzT5PreTrainedModel.dummy_inputss6LL. \\*- !*"&0 rcx|jj}t|tr)|jj j |dzy t|ttttfr|jjj jd|dzt|drJ|jjs4|jjj jd|dzt|drz|j jj jd||jj"dzz|j j$j j'y y t|t(rpt|drc|j*jj jd|dz|j*j$j j'y y t|t,rM|j.jj jd||jj"dzzt|j.drD|j.j$.|j.j$j j'|j0jj jd||jj"dzzt|j0drF|j0j$/|j0j$j j'y y y t|t2rM|j4jj jd||jj"dzzt|j4drD|j4j$.|j4j$j j'|j6jj jd||jj8dzzt|j6drF|j6j$/|j6j$j j'y y y t|t:r|j<jj jd||jj"dzzt|j<drD|j<j$.|j<j$j j'|j>jj jd||jj"dzzt|j>drD|j>j$.|j>j$j j'|j6jj jd||jj8dzzt|j6drF|j6j$/|j6j$j j'y y y t|t@ri|jj"}|jjB}|jjD}|jFjj jd|||zdzz|jHjj jd||dzz|jJjj jd||dzz|jLjj jd|||zdzz|jNr8|jPjj jd||dzzy y y ) zInitialize the weights?)rstdrP qa_outputsrMrKN))rzinitializer_factorrrr@rwfill_T5ModelT5ForConditionalGenerationT5EncoderModelT5ForQuestionAnsweringsharednormal_ritie_word_embeddingsrPr|rrKzero_T5ForTokenClassificationrMrdrfrirrQrrrrrrrrrrrrrr)rmodulefactorrrrs r8 _init_weightszT5PreTrainedModel._init_weightss// fk * MM   $ $Vc\ 2   0.BX Y MM % % - -3FSL - Ivy)$++2Q2Q%%**22#2Nv|,!!((--553Ft{{ObObgkNkDl5m!!&&++113- 8 9v|,!!((--553FSL5Q!!&&++113- 4 5 LL   $ $ , ,#6dkkFYFY^bEb;c , dv||V,1B1B1N !!&&,,. OO " " ' ' / /SfI\I\aeHe>f / gv/FOO4H4H4T$$))//15U/  0 II   ! ! ) )s4;;CVCV[_B_8` ) avyy&)fiinn.H ##))+ II   ! ! ) )s4;;CSCSX\B\8] ) ^vyy&)fiinn.H ##))+/I)  4 5 KK   # # + +&T[[EXEX]aDa:b + cv{{F+ 0@0@0L   %%++- KK   # # + +&T[[EXEX]aDa:b + cv{{F+ 0@0@0L   %%++- II   ! ! ) )s4;;CSCSX\B\8] ) ^vyy&)fiinn.H ##))+/I)  ,kk))G!%!1!1 kk++G HHOO ( (cv'L^B^cgAg7h ( i HHOO ( (cv$7O ( P HHOO ( (cv$7O ( P HHOO ( (cv'L^B^cgAg7h ( i11..55::BBQW\chl[lQmBn2-rc|jj}|jj}| tdt |rGt j |jdddz|}t j||dddfgd}n>|j|j}|dddfj|dddf<||d<| td|j|d k(||S) Nzself.model.config.decoder_start_token_id has to be defined. In T5 it is usually set to the pad_token_id. See T5 docs for more information.r<)r).rr)).rz1self.model.config.pad_token_id has to be defined.) rzdecoder_start_token_id pad_token_idrpr#rsfullrncat new_zerosclone masked_fill_)rrsrrshifted_input_idss r8 _shift_rightzT5PreTrainedModel._shift_right\s!%!C!C{{// ! )4  Y ' % 9??3B+?$+FH^ _  % +Q*RXZ [  ) 3 3IOO D )238)<)B)B)D c12g &(> f %  PQ Q&&'8D'@,O  rN)rrrrmr* config_classrload_tf_weightsbase_model_prefixis_parallelizablesupports_gradient_checkpointing_supports_quantized_cache_supports_static_cache_supports_cache_class_no_split_modules_keep_in_fp32_modulespropertyrwrrr5rr8rorose L+O%&*# %! " !F ;oz!rrocbeZdZdfd ZeeddZeedZdZ dZ ddZ dde jde jd e jd ed ef d Zede jd edede j&de j(d e jdefdZxZS)T5Stackc t||||_|j|_t j t |jDcgc]}t|t|dk(|c}|_ t|j|j|_t j|j |_|j%d|_d|_d|_ycc}w)NrrCrF)rr embed_tokensrrrSrange num_layersrQr!blockrrrrHrrr post_initmodel_parallel device_mapr)rrzrirs r8rzT5Stack.__init__ys  ( ++]]]bcictct]u vXYWVa1fQR S v  !,FNN@Y@Y Zzz&"5"56  #&+# ws!C,ctjdt|Dtt |j t tjjn||_ t|jt |j d|_ d|jjvrdn.dtt|jjz|_dtt#|jjz|_|jj'D]E\}}|D];}dt|z}|j |j)||j |<=G|j*j)|j |_|j,j)|j$|_y)Na<`T5Stack.parallelize` is deprecated and will be removed in v5 of Transformers, you should load your model with `device_map='balanced'` in the call to `from_pretrained`. You can also provide your own `device_map` but it needs to be a dictionary module_name to device, so for instance {'block.0': 0, 'block.1': 1, ...}Tcpucuda:)warningswarn FutureWarningr(rjrrrscuda device_countrr'rrxstrr first_devicerU last_deviceitemsrrrH)rrrrrB cuda_devices r8 parallelizezT5Stack.parallelizesu  "   R\QcN3tzz?E%**2I2I2K,L Mis  $//3tzz?;"%*doo.B.B.D%DE'TWX[\`\k\k\p\p\rXsTtJt"ST__-A-A-C)D%EEOO))+ FDAq F%A. $(JJu$5$8$8$E 5! F F !--001B1BC $ 5 5 8 89I9I Jrctjdtd|_d|_d|_d|_tt|jD]-}|j|jd|j|</|jjd|_ |jjd|_ tjjy)N\Like `parallelize`, `deparallelize` is deprecated and will be removed in v5 of Transformers.Fr)rrrrrrrrrjrrrrHrsr empty_cache)rrs r8 deparallelizezT5Stack.deparallelizes j  $! s4::' 4A JJqM,,U3DJJqM 4 --007 $ 5 5 8 8 ?  rc|jSrrrs r8get_input_embeddingszT5Stack.get_input_embeddingss   rc||_yrrrnew_embeddingss r8set_input_embeddingszT5Stack.set_input_embeddingss *rcz|jrStjj|j|j j |j|_| | n|jj} | | n|jj} | | n|jj} | | n|jj} |$|"|jrdnd}td|d|d|&|j}|jd|d}n8||jdd}n"|jrdnd}td|d|d |j r%|j"r| rt$j'd d } |(|j td |j |}|\}}| d ur|jstd|dd }d }|jr| s|t)|t*r't)|t,sd }t-|t/}njt)|t,s-d }t$j'dt-j0|}n-|+t-t/t/}n|jsd}||j3nd}| %tj4|||z|j6} |1t9s'||z}tj:|||j6}|jjr$|j=||| | |j>nd| }n]|Y|ddddddf}|j |j@}d|z tjB|j@jDz}nd}|jr^|\|j\}}}||f}|0tj:||j6tjF}|jI|}nd}|jK||jjL}|jK||jjL}| rdnd}| rdnd}| r|jrdnd}d}d} |jO|}!tQ|jRD]9\}"}#||"}$||"}%|jrtjj|!j6||j |!j6}||j |!j6}||j |!j6}||j |!j6}| | j |!j6} |$|$j |!j6}$|%|%j |!j6}%| r||!fz}|j r5|j"r)|jU|#jV|!||||| |$|%d| | | | }&n|#|!||||| |$|%|| | | |  }&| d ur|&dddz|&ddz}&|&dd\}!}'|&d}|jr | |&| rdnd} | r||&dfz}|jr ||&dfz}|js|jXj[D]J\}(})|"|)dk(sdt]|(z|j^k7s+|!j dt]|(dzz}!L<|ja|!}!|jO|!}!| r||!fz}| r'nd}*|r |j>}*|r|jc}*| sted|!|*|||fDStg|!|*||| S)!Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer<zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fz.s = r:)last_hidden_statepast_key_valuesr attentionscross_attentions)4rrsr set_devicerrrrzr+r,output_hidden_statesuse_return_dictrrpsizerrrrYrrr rrfrom_legacy_cacheget_seq_lengthrrr$r_update_causal_maskrrrWrrinvert_attention_mask get_head_maskrr enumerater_gradient_checkpointing_funcrrrrrrHto_legacy_cachetupler)+rrsrFrYrZr head_maskcross_attn_head_maskrr+r,rr]r err_msg_prefix input_shaper-r.return_legacy_cachereturn_self_attention_cachepast_key_values_lengthmask_seq_lengthr7encoder_batch_sizeencoder_sequence_length_encoder_hidden_shapeencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsr(r[rr layer_moduler*r\ layer_outputsnext_decoder_cacherr next_caches+ r8rzT5Stack.forwards"    JJ ! !$"3"3 4 $ 1 1 4 4T5F5F GD !*!6IDKK+/??ZN*>*:.HXXvw  "#..*K!r;r?;I  &',,.s3K+/??ZN:>:J-XfWggtuv v  & &4==##p"    ( !_`` --i8M!, J  ?? #LTFRg!hii$&+# ?? _-H/51*_Vi:j.2+"5o|~"V1DE&*###` #6"G"G"X ("5lnln"U#OETE`!?!?!Afg  !"\\&(>(KTaThThN  !*B*D4zAO"ZZ OML`L`aN ;; ! !228G8S44Y]! K '(D$)9:K%..}/B/B.CK, M6U6X6XYfYmYm6n30<4Q4T4TUbUiUi4j1".&5&8&89M9M&NO-91K1N1N}OcOc1d.#$58H$H!**t}} $ A A ((!!)31#.%"! "!-!#."/*?+J2O$3/I#2'&7 +#1! $E! -bq 1G ;mAB>O O 0=bq0A -M- *!,M#8#D0=CTaZ[0\- !/=3C2E!E??+?=QRCSBU+U("" OO113ODAqAbEzgA&6$:J:J&J(5(8(83q1u:9M(N O_Q Of--m<  ]3  1]4D D +4'$ &(==J (88:J "%"(    9+&+%1   rrF input_tensorr rr,c |jjdk(r||dk(jr|Sy|jjdk(r7t|tj r t |}t|tr|S||jnd}t|t}|jjdk(r(|s&|s$tj||||jry|j|j} }|jd} |r|j!} n1t|tj r|jdn|| zdz} |j#|| | || ||jd } |jjdk(rQ|O|jj$d vr7|s5t j&|j(} tj*| | } | S) Nflash_attention_2rzflex_attentionrsdpa)rr is_trainingr)r<)sequence_length target_lengthrrr r-)rxpu)rz_attn_implementationrdrrsrr,r+rrr_ignore_causal_mask_sdparrrrnget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positiontyperWr_unmask_unattended)rrFrr rr,past_seen_tokensusing_static_cacherrrrr7 min_dtypes r8rzT5Stack._update_causal_masks ;; + +/B B)~/D.I.I.K%% ;; + +/? ?.%,,7!<^!L.)4%% @O?Z?99;`a'E ;; + +v 5>PYj%>>*'7 MM  $**L,?,?v&,,Q/ +??AMnell;$$R(%7!; PP +')#))!,Q  KK , , 6*%%**o=%  E*..I0CCKQZ[Krrrrrr-c ||jdk(r|}|Stj|j} tj||f| ||}|dk7rtj |d}|tj |||jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|jz} | dk(} |ddddddd| fj| | |ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. device (`torch.device`): The device to place the 4D attention mask on. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr) fill_valuerrr))diagonalrr<r)rrsrWrrtriurreshapeexpandrrnrr masked_fill) rFrrrrr r-kwargsr7r mask_length padding_masks r8rz=T5Stack._prepare_4d_causal_attention_mask_with_cache_positionsyD  %.*<*<*>!*C(K*' E*..I** -0Ye\bK!##jjqA 5<< fEH^H^_acdHee eK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12rr) NNNNNNNNNNNNN)F)rrrrr PARALLELIZE_DOCSTRINGrDEPARALLELIZE_DOCSTRINGrrrrrsrr r!rr?rmrrrrrs@r8rrxs+,&/0K1K612 !3 !!+ "#!!u |#( D DllD D  D  DL7 777{{ 7  7  777rra The T5 model was proposed in [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683) by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. It's an encoder decoder transformer pre-trained in a text-to-text denoising generative setting. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`T5Config`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. ay Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, `optional`: *hidden_states*, `optional`: *attentions*) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)` is a sequence of hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, target_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `decoder_inputs_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `decoder_input_ids` and `decoder_inputs_embeds` are both unset, `decoder_inputs_embeds` takes the value of `inputs_embeds`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a_ The input argument `head_mask` was split into two arguments `head_mask` and `decoder_head_mask`. Currently, `decoder_head_mask` is set to copy `head_mask`, but this feature is deprecated and will be removed in future versions. If you do not want to use any `decoder_head_mask` now, please set `decoder_head_mask = torch.ones(num_layers, num_heads)`. z\The bare T5 Model transformer outputting raw hidden-states without any specific head on top.c'eZdZdgZddgZdeffd Zeed!dZ ee dZ dZ d Z d Zd Zd Zd Zeeeee d"deej2deej4deej2deej6deej4deej4deej8deeeej4deeeej4deej8deej8deedeedeedeedeej2deeej4eff"d Z xZ!S)#rFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weightencoder.embed_tokens.weightdecoder.embed_tokens.weightrzct||tj|j|j |_tj|}d|_ d|_ d|_ t||j |_ tj|}d|_ d|_ |j|_t||j |_|j#d|_d|_yNFT)rrrr vocab_sizerrcopydeepcopyrr+is_encoder_decoderrencodernum_decoder_layersrrNrrrrrzencoder_configdecoder_configrs r8rzT5Model.__init__s  ll6#4#4fnnE v.$)!#( ,1)~t{{; v.$(!,1)$*$=$=!~t{{;  $rctjdt|Ntt |j j ttjjn||_ t|jt |j j |j j|j|jj|jd|_y)NaL`T5Model.parallelize` is deprecated and will be removed in v5 of Transformers, you should load your model with `device_map='balanced'` in the call to `from_pretrained`. You can also provide your own `device_map` but it needs to be a dictionary module_name to device, so for instance {'encoder.block.0': 0, 'encoder.block.1': 1, ...}T)rrrr(rjrrrrsrrrr'rrNrrrs r8rzT5Model.parallelizes  -   ! 3t||112E%**:Q:Q:S4T U  $//3t||/A/A+BC   1   1"rcxtjdt|jj |j j |jj d|_|j j d|_d|_d|_tjjyNrrF) rrrrrrNrrrrsrrrs r8rzT5Model.deparallelizes} j   ""$ ""$||u- ||u- #  rc|jSrrrs r8rzT5Model.get_input_embeddings {{rc~||_|jj||jj|yrrrrrNrs r8rzT5Model.set_input_embeddings -$  )).9 )).9rc|jjra|j|jj|j |j|j j|j yyrrzr_tie_or_clone_weightsrrrrNrs r8 _tie_weightszT5Model._tie_weights%P ;; * *  & &t||'@'@$++ N  & &t||'@'@$++ N +rc|jSrrrs r8 get_encoderzT5Model.get_encoder* ||rc|jSrrNrs r8 get_decoderzT5Model.get_decoder-r3rc|jD]7\}}|jj|jj |9y) Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)rrrB attentionrrheads_to_prunerBrs r8 _prune_headszT5Model._prune_heads0sE +002 CLE5 LL  u % / / ; ;E B Cr output_typerrsrFrrrtrdecoder_head_maskrencoder_outputsrrdecoder_inputs_embedsr+r,rr]r rjc| | n|jj} ||n|jj}|O|M|jj|jjk(r t j tt|}||j||| || ||}nI|rGt|ts7t|dt|dkDr|dndt|dkDr|dnd}|d}|jrtjj!|j"j$|j'|j"j$}|%|j'|j"j$}|%|j'|j"j$}|%|j'|j"j$}|j#||| | ||||| | ||| }|s||zSt)|j*|j,|j.|j0|j2|j*|j.|j0S) a Returns: Example: ```python >>> from transformers import AutoTokenizer, T5Model >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5Model.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for T5Model. >>> # This is not needed for torch's T5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```NrsrFrrr,rr]rr)rErrr rsrFrrrYrZrrr+r,rr]r )rrdecoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterYencoder_attentions)rzr+rrrrr_T5Model__HEAD_MASK_WARNING_MSGrrrrrjrrsrrrNrrrrrrrr)rrsrFrrrtrr?rr@rrrAr+r,rr]r rdecoder_outputss r8rzT5Model.forward8s=Z"+!6IDKK{{%%)G)GG 5}E$-!  ""ll#-+#"3%9'+OO_!M-"1!"4474H14Loa0RV14_1E1I?1-tO (*     JJ ! !$,,";"; <),,T\\-F-FGM ,$5$8$89R9R$S!)!/!2!24<<3L3L!M%1)?)B)B4<?+=O\156:8<=A159=7;EIEI048<$(,0/3&*59#p E,,-p !!2!23p $E$4$45 p !))9)9 : p E--. p $E$5$56p 'u||4p "%e.?.?(@"ABp "%e.?.?(@"ABp   -p  ( 5p D>p $D>p 'tnp d^!p "!!1!12#p $ uU&&');; <%p ]@p rrz0T5 Model with a `language modeling` head on top.c)eZdZdgZgdZdeffd Zeed$dZ ee dZ dZ dZ d Zd Zd Zd Zd Zeeeee d%deej4deej6deej4deej8deej6deej6deej:deeeej:deeeej:deej6deej6deej4deedeedeedeedeej4d e eej6eff$d!Z!dej:fd"Z"d#Z#xZ$S)&rr)rrzlm_head.weightrzcjt|||j|_t j |j |j|_tj|}d|_ d|_ d|_ t||j|_tj|}d|_ d|_ |j|_t||j|_t j$|j|j d|_|j)d|_d|_y)NFTr)rrr model_dimrrrrrrrr+rrrrrrNrrPrrrrs r8rz#T5ForConditionalGeneration.__init__s  ll6#4#4fnnE v.$)!#( ,1)~t{{; v.$(!,1)$*$=$=!~t{{; yy1B1BO  $rcVtjdt|Ntt |j j ttjjn||_ t|jt |j j |j j|j|jj|j|jj!|jj"|_d|_y)Na_`T5ForConditionalGeneration.parallelize` is deprecated and will be removed in v5 of Transformers, you should load your model with `device_map='balanced'` in the call to `from_pretrained`. You can also provide your own `device_map` but it needs to be a dictionary module_name to device, so for instance {'encoder.block.0': 0, 'encoder.block.1': 1, ...}T)rrrr(rjrrrrsrrrr'rrNrPrrrr"s r8rz&T5ForConditionalGeneration.parallelizes  A   ! 3t||112E%**:Q:Q:S4T U  $//3t||/A/A+BC   1   1||t||'@'@A "rctjdt|jj |j j |jj d|_|j j d|_|jj d|_d|_d|_ tjjyr$) rrrrrrNrrPrrrsrrrs r8rz(T5ForConditionalGeneration.deparallelizes j   ""$ ""$||u- ||u- ||u- #  rc|jSrr&rs r8rz/T5ForConditionalGeneration.get_input_embeddingsr'rc~||_|jj||jj|yrr)rs r8rz/T5ForConditionalGeneration.set_input_embeddingsr*rc|jjra|j|jj|j |j|j j|j yyrr,rs r8r.z'T5ForConditionalGeneration._tie_weightsr/rc||_yrrPrs r8set_output_embeddingsz0T5ForConditionalGeneration.set_output_embeddingss % rc|jSrr]rs r8get_output_embeddingsz0T5ForConditionalGeneration.get_output_embeddingsr3rc|jSrr1rs r8r2z&T5ForConditionalGeneration.get_encoderr3rc|jSrr5rs r8r6z&T5ForConditionalGeneration.get_decoderr3rr=rsrFrrrtrr?rr@rrrAlabelsr+r,rr]r rjc| | n|jj} ||n|jj}|O|M|jj|jjk(r t j tt|}||j||| ||||}nI|rGt|ts7t|dt|dkDr|dndt|dkDr|dnd}|d}|jr3tjj!|j"j$| || |j'| }|jrtjj!|j"j$|j)|j"j$}|%|j)|j"j$}|%|j)|j"j$}|%|j)|j"j$}|j#||| | ||||| |||| }|d}|jrtjj!|jj$|j*j)|jj$|_|j)|j*j,j.}|jj0r||j2dzz}|j+|}d}| ^t5d }| j)|j.} ||j7d |j9d | j7d }|s|f|ddz|z}||f|zS|St;|||j<|j>|j@|jB|jD|j>|j@ S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Returns: Examples: ```python >>> from transformers import AutoTokenizer, T5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer( ... "summarize: studies have shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model.generate(input_ids) >>> print(tokenizer.decode(outputs[0], skip_special_tokens=True)) >>> # studies have shown that owning a dog is good for you. ```NrCrr)rErDrEr}r ignore_indexr< lossrOrrFrGrrHrYrI)#rzr+rrrrr2_T5ForConditionalGeneration__HEAD_MASK_WARNING_MSGrrrrrjrrsrrrNrrrrPr@rrrVr rrrrrrrr)rrsrFrrrtrr?rr@rrrArcr+r,rr]r rrKsequence_output lm_logitsrhloss_fctoutputs r8rz"T5ForConditionalGeneration.forward sj"+!6IDKK{{%%)G)GG 5}E$-!  ""ll#-+#"3%9'+OO_!M-"1!"4474H14Loa0RV14_1E1I?1-tO (*    JJ ! !$,,";"; <  "3";@U@] $ 1 1& 9     JJ ! !$,,";"; <),,T\\-F-FGM ,$5$8$89R9R$S!)!/!2!24<<3L3L!M%1)?)B)B4<\%\ "' ](&%rr)NNNNNNNNNNNNNNNNN)%rrrrMrNr*rr rrrrrrr.r^r`r2r6r!rOr&rrPrrsrQrRrSrrr!rrrorxrrs@r8rrs| Q*&jx6/0#1#&12 !3 !: O &++>??Y156:8<=A159=7;@D@D59=A-1$(,0/3&*59%Y E,,-Y !!2!23Y $E$4$45 Y !))9)9 : Y E--. Y $E$5$56Y 'u||4Y "%ell(;"<=Y "%ell(;"<=Y   1 12Y  ((9(9:Y ))*Y D>Y $D>Y 'tn!Y "d^#Y $!!1!12%Y & uU&&'8 9'Y Z@Y v)ELL)&rrzfThe bare T5 Model transformer outputting encoder's raw hidden-states without any specific head on top.ceZdZdgZdgZdeffd ZeeddZ ee dZ dZ dZ d Zd Zd Zeeeee  dd eej0deej2deej2deej2deedeedeedeeej2effdZxZS)rrrNrzc:t||tj|j|j |_tj|}d|_ d|_ t||j |_ |jd|_d|_yNF)rrrrrrrrrr+rrrrrr)rrzrrs r8rzT5EncoderModel.__init__sy  ll6#4#4fnnE v.#( ,1)~t{{;  $rctjdt|Ntt |j j ttjjn||_ t|jt |j j |j j|jd|_y)NaC`T5EncoderModel.parallelize` is deprecated and will be removed in v5 of Transformers, you should load your model with `device_map='balanced'` in the call to `from_pretrained`. You can also provide your own `device_map` but it needs to be a dictionary module_name to device, so for instance {'block.0': 0, 'block.1': 1, ...}T)rrrr(rjrrrrsrrrr'rrr"s r8rzT5EncoderModel.parallelizes  "   ! 3t||112E%**:Q:Q:S4T U  $//3t||/A/A+BC   1"rctjdt|jj |jj d|_d|_d|_tjjyr$) rrrrrrrrrsrrrs r8rzT5EncoderModel.deparallelizesY j   ""$||u- #  rc|jSrr&rs r8rz#T5EncoderModel.get_input_embeddingsr'rcH||_|jj|yr)rrrrs r8rz#T5EncoderModel.set_input_embeddingss$  )).9rc|jjr1|j|jj|j yyr)rzrr-rrrrs r8r.zT5EncoderModel._tie_weightss2 ;; * *  & &t||'@'@$++ N +rc|jSrr1rs r8r2zT5EncoderModel.get_encoder r3rc|jD]D\}}|jj|jdjj |Fy)r8rN)rrrrBrDrr:s r8r<zT5EncoderModel._prune_headssP +002 PLE5 LL  u % + +A . < < H H O Prr=rsrFrrr,rr]rjc j||n|jj}|j|||||||}|S)a3 Returns: Example: ```python >>> from transformers import AutoTokenizer, T5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5EncoderModel.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```rC)rzrr) rrsrFrrr,rr]r@s r8rzT5EncoderModel.forwardsJ8&1% &'tn&d^& uU&&'8 9&ZH&rrz T5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c%eZdZdgZddgZdeffd Zeee e e  dde e jde e jd e e jd e e jd e e jd e e jd e e jde ee j"de e j"de e j"de e jde ede ede ede edeee ff dZxZS)T5ForSequenceClassificationrrrrzct||t||_t ||_|j d|_yr{)rrrrprdclassification_headrrrs r8rz$T5ForSequenceClassification.__init__Ls=  "6?#7#?  #rr=rsrFrrrtrr?rr@rrArcr+r,rr]rjc||n|jj}| d} |$| "td|jj| | | t d|j |}|j||||||||| | | | ||}|d}|j|jjj|j}ttj|jddkDr t d|j \}}}||ddfj#|d |ddd ddf}|j%|}d}| | j|j} |jj&|jj(dk(rd |j_nv|jj(dkDrL| j*tj,k(s| j*tj.k(rd |j_nd |j_|jj&d k(rSt1}|jj(dk(r&||j3| j3}n||| }n|jj&d k(rGt5}||j#d |jj(| j#d }n,|jj&d k(rt7}||| }|s|f|ddz}||f|zS|St9|||j:|j<|j>|j@|jB|jD|jF S)aD labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rFrrrtrr?rr@rrAr+r,rr]rr)z7All examples must have the same number of tokens.r< regressionsingle_label_classificationmulti_label_classificationrg)$rzrNotImplementedErrorrrrprrpeq eos_token_idrrrjrsunique_consecutivesumrnrr problem_typerhrrrmr squeezer rrrrFrGrrHrYrI)rrsrFrrrtrr?rr@rrArcr+r,rr]r;rjeos_maskr-rrsentence_representationrOrhrlrms r8rz#T5ForSequenceClassification.forwardVsQ4&1%>KbKbJcd   $)>)F  U !% 1 1) < "" )/#9/!5+'"7/!5##  "!*<< 8 89<<_=S=ST u'' Q8 9A =VW W%4%:%:" A{"1(A+">"C"CJPRT_"`abdfhiai"j))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./Y,F)-)9TGf$ Ev E.#33")"?"?&99$55&-&G&G")"?"?&99  r)NNNNNNNNNNNNNNN)rrrrMrNr*rr!rOr&rrPrrsrQrrrRr!rrrrrs@r8rrAs+s)s&79VW$x$++>?+JYhi15158<=A,0487;=A59=A-1$(,0/3&*!k E,,-k !.k $E$4$45 k !))9)9 : k ELL) k $ELL1k 'u||4k "$u'8'8"9:k   1 12k  ((9(9:k ))*k D>k $D>k 'tnk d^!k " u55 6#k j@k rrz T5 Encoder Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cPeZdZdgZdeffd Zeeee e  dde e jde e jde e jde e jd e e jd e ed e ed e ed eee je ffdZxZS)rz'transformer.encoder.embed_tokens.weightrzc,t|||j|_t||_t j |j|_t j|j|j|_ |jyr) rrrhrrprrrgrrrrMrrs r8rz!T5ForTokenClassification.__init__sj   ++)&1zz&";";< ))F$6$68I8IJ rr=rsrFrrrcr,rr]rjc ||n|jj}|j|||||||} | d} |j| } |j | } d} |?+@_-115,004)-,0/3&*. ELL). !.. ELL) .   - . & . $D>. 'tn. d^.  uU\\"$99 :. `@. rrz T5 Model with a span classification head on top for extractive question-answering tasks like SQuAD (linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). c'neZdZdgZddgZdeffd ZdZdZdZ d Z d Z e e eee  dd eej&d eej(deej&deej*deej(deej(deej,deeeej,deej&deej&deej(deej(deedeedeedeedeeej(eff"dZxZS)rrrrrzczt|||j|_t j |j |j|_tj|}d|_ d|_ d|_ t||j|_tj|}d|_ d|_ |j|_t||j|_|j$|_t j&|j(|j$|_|j-d|_yr)rrrrVrrrrrrrr+rrrrrrNrhrrr|rrrs r8rzT5ForQuestionAnswering.__init__ s  ll6#4#4fnnE v.$)!#( ,1)~t{{; v.$(!,1)$*$=$=!~t{{;  ++))F$6$68I8IJ #rc|jSrr&rs r8rz+T5ForQuestionAnswering.get_input_embeddings3 r'rc~||_|jj||jj|yrr)rs r8rz+T5ForQuestionAnswering.set_input_embeddings6 r*rc|jjra|j|jj|j |j|j j|j yyrr,rs r8r.z#T5ForQuestionAnswering._tie_weights; r/rc|jSrr1rs r8r2z"T5ForQuestionAnswering.get_encoder@ r3rc|jSrr5rs r8r6z"T5ForQuestionAnswering.get_decoderC r3rr=rsrFrrrtrr?rr@start_positions end_positionsrrAr+r,rr]rjcf||n|jj}| | n|jj} | | d} | | | td|j |}| | n|jj} ||n|jj}|O|M|jj |jj k(r tjtt|}||j||| ||||}nI|rGt|ts7t|dt|dkDr|dndt|dkDr|dnd}|d}|j||| d||||| ||| }|d}|j!|}|j#dd \}}|j%d j'}|j%d j'}d}| | t| j)dkDr*| j%d j+|j,} t| j)dkDr*| j%d j+|j,} |j)d}| j/d|} | j/d|} t1| }||| }||| }||zdz }|s||f|ddz|z}||f|zS|St3||||j4|j6|j8|j:|j<|j6|j8 S)a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (*sequence_length*). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (*sequence_length*). Position outside of the sequence are not taken into account for computing the loss. Returns: NFrrCrr)rErD) rsrFrrrYrZrrr+r,rr]r<rre) rh start_logits end_logitsrrFrGrrHrYrI)rzrr+rprrrrr._T5ForQuestionAnswering__HEAD_MASK_WARNING_MSGrrrrrjrNr|rcrr%rrrrXr rrrrrr)rrsrFrrrtrr?rr@rrrrAr+r,rr]rrKrjrOrr total_loss ignored_indexrl start_lossend_lossrms r8rzT5ForQuestionAnswering.forwardF sx@&1%)F  U !% 1 1) < !*!6IDKK{{%%)G)GG 5}E$-!  ""ll#-+#"3%9'+OO_!M-"1!"4474H14Loa0RV14_1E1I?1-tO (* ,,'1/ "/#1'!5/!5#'  *!,1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""="@"@ATAT"U=%%'(1, - 5 5b 9 < ?+N]lm156:8<=A159=7;@D6:4859=A$(,0/3&*#C E,,-C !!2!23C $E$4$45 C !))9)9 : C E--. C $E$5$56C 'u||4C "%ell(;"<=C "%"2"23C   0 01C   1 12C  ((9(9:C D>C $D>C 'tn!C "d^#C $ uU&&')LL M%C n@C rr)rrrrorrrr)brmrrr[rtypingrrrrrsrtorch.nnrr r activationsr cache_utilsr rrr generationrmodeling_attn_mask_utilsrmodeling_outputsrrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrr r!r"r#r$r%r&utils.model_parallel_utilsr'r(configuration_t5r*!torch.nn.attention.flex_attentionr+integrations.flex_attentionr, get_loggerrrYrP_CHECKPOINT_FOR_DOCrrrModulerapex.normalizationrr^rX ExceptionrqrbrrrrrArKrQrdrorT5_START_DOCSTRINGrOr__HEAD_MASK_WARNING_MSGrrrrrr__all__r5rr8rs // AA!PP)>.gg   L& !;J   H %*f\ B(+"))+2  /K KKabK(bii,299: $a"))aH!299!H#BII#LabiiaH299$r!r!j}}@*^@#LbK K K \LN`aY&!2OY&bY&xlp&p pf { "3{ { | > 0> > B v .v v r }E   NNWX sII?&I?>I?