T^(h* dZddlZddlmZmZmZmZmZddlm Z ddl Z ddl m Zddl ZddlmZmZmZddlmZmZddlmZddlmZddlmZmZdd lmZd d lm Z m!Z!m"Z"m#Z#m$Z$d d l%m&Z&m'Z'm(Z(m)Z)m*Z*d d l+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1e.jde3Z4dZ5dZ6ejnZ7dejpde9de9dejpfdZ:d^dejpde9de9de9dejpf dZ;dejpde9de9dejpfdZd!ejpde9dejpfd"Z?d#ejpde9dejpfd$Z@d#ejpd%e9deejpejpffd&ZAd#ejpd%e9dejpfd'ZBd(ejpd)ejpd*e9dejpfd+ZCGd,d-e jZEGd.d/e jZFGd0d1e jZGGd2d3e jZHGd4d5e jZIGd6d7e jZJGd8d9e jZKGd:d;e jZLGd<d=e jZMGd>d?e jZNGd@dAe jZOGdBdCe jZPGdDdEe jZQGdFdGe jZRGdHdIe jZSdJZTdKZUdLZVGdMdNe'ZWdOZXe,dPeXGdQdRe jZYGdSdTeWZZe(eZe5e$e6dUZ[e*eZeVe[ze)eZe#e6Ve,dWeXGdXdYe jZ\GdZd[eWZ]d\Z^e*e]eVe^ze)e]e#e6Vgd]Z_y)_zFlax LongT5 model.N)AnyCallableListOptionalTuple) FrozenDictfreezeunfreeze) combine_masksmake_causal_mask) partitioning)dot_product_attention_weights) flatten_dictunflatten_dict)PRNGKey)FlaxBaseModelOutput-FlaxBaseModelOutputWithPastAndCrossAttentions%FlaxCausalLMOutputWithCrossAttentionsFlaxSeq2SeqLMOutputFlaxSeq2SeqModelOutput)ACT2FNFlaxPreTrainedModelappend_call_sample_docstring append_replace_return_docstringsoverwrite_call_docstring)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) LongT5Configzgoogle/long-t5-local-baser" input_ids pad_token_iddecoder_start_token_idreturnctj|}|jddddfj|ddddf}|jdddfj|}tj|dk(||}|S)z1 Shift input ids one token to the right. Nr!ri)jnp zeros_likeatsetwhere)r#r$r%shifted_input_idss m/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/longt5/modeling_flax_longt5.pyshift_tokens_rightr0:sy1),,QU377 !SbS&8IJ),,QT2667MN "3t";\K\] x block_lenaxis pad_valuec|j| |z}dg|jz}d|f||<tj||d|}|S)zHPad an array so that a sequence length will be a multiple of `block_len`rrrconstant pad_widthmodeconstant_values)shapendimr)pad)r2r3r4r5pad_lenr?s r/_pad_to_multiplerAFsLwwt}ny(G (QVV CG CI Sz9MA Hr1c|j||zdk7rt|||d}|j||z}|jd|||fz|j|dzdz}|j|S)zSplit an input array into blocks of a given `block_len` along the given `axis`. If the dimension length is not a multiple of `block_len`, it will be padded first with selected `pad_value`. r)r5Nr!)r=rAreshape)r2r3r4 num_blocks output_shapes r/_split_into_blocksrFOsw  wwt}y A% Q 41 =)+J775D>Z$;;aggtaxl>SSL 99\ ""r1 block_axis sequence_axiscd|j|}dg|jz}d||<tj||d|}g}t dD]M}t ddg|jz}t |||z||<t |}|j||Otj||S) zConcatenate three consecutive blocks for each input block for local attentiont. For more information, see: https://arxiv.org/pdf/2112.07916.pdf. r7r!r!r8r9rrNr4) r=r>r)r?rangeslicetupleappend concatenate) r2rGrHr5rDr? blocks_listiindicess r/_concatenate_3_blocksrT[s$J (QVV CC O Sz9MA"$K 1X'D>"QVV+#Aq:~6 .1W:& ' ??;] ;;r1ctjd|ztj}||| }|dddf|dddfz }|S)z:Makes 3-blocked relative position ids for local attention.rdtypeN)r)arangeint32)r3 position_idscenter_position_idsrelative_position_idss r/"_make_3block_relative_position_idsr]qsM::a)m399=L&y)<(q14G44PP  r1local_attention_maskct|}tj||k}|ddddddf}tj||S)znMask local attention mask to enforce that tokens are not allowed to attend tokens farther than ``local_radius.N)r]r)abs logical_and)r^r3r\ locality_masks r/_mask_local_attention_maskrcysF>yIGG12Y>M!$a"23M ??/ ??r1attention_maskct||d}t|dd}|d}|ddddf}tj||}t ||}|ddddfS)z;Prepare attention mask to be applied for a local attention.r!rKrGrH.N.N)rFrTr)rarc)rdr3_blocked_attention_mask_3blocked_attention_maskr^s r/_get_local_attention_maskrkst1QRS45LYZjkl5i@7T1 E??+BD\]56JIV 4 --r1global_block_sizec& |jdd\} dtjdtjf fd }tj|z }tj |d|z }tj |dk7d d }tjtj||zd z tjd |j }||z|dz z}||} z}|d kDr0tj|jddddf|d}n#tj|d f|j }tj tj||fddz } tj | |kdd } || fS)aObtain the "fixed block" global id corresponding to each input token. This implementation is a simlified version of the original Flaxformr implementation adopted from: https://github.com/google/flaxformer/blob/main/flaxformer/architectures/longt5/long_attention.py. In our scenario, as we use this strategy only for a decoder, orphan tokens, i.e. those tokens which do not make for the whole fixed block, are assigned to the preceding block. Padding tokens from the original sequence are represented by -1. Nrf block_idsr&ctjzdz k(}tj||dk\}|jdd}tj||dz }|S)Nr!rr(rh)r)rXrasumminimum)rn block_endstrue_block_ends full_blocksrlseq_lens r/handle_orphan_tokensz:_make_global_fixed_block_ids..handle_orphan_tokenssfjj),==BSVWBWW //*i1nE%))"-i8 KK ;?; r1r!rK?g@grVrr()repeatsr4)r=npndarrayr) ones_likecumsumr-maximumfloorarrayrWrepeatmaxzerosones) rdrl batch_sizervfixed_block_maskmaskglobal_block_ids num_globals_sequence_block_ids_maxglobal_segment_idsrus ` @r/_make_global_fixed_block_idsrs)..r2J s{{}}^47HHzz"2;>NN 99^s*C 9D{{ $))C/0#))DH\H\2])>9nq>PQ+,<=..KQ"%**-=-A-Ar-A-J1d7-S]hop"q"%))ZOCSCYCY"ZCHHj+-F$GbQTUU#59P#PRSUVW / //r1c|t||\}}|jd}tj|}||dz }|S)zBCreate the relative position tensor for local -> global attention.r(rh)rr=r)rX)rdrlrnrglobal_seq_lenglobal_positionsside_relative_positions r/ _make_side_relative_position_idsrsJ$@Qb$c!I!'--b1Nzz.1- )0DD !!r1 hidden_statesrnrcptjj||}tjd||S)zFCompute individual block aggregates by summing over individual blocks.z...nd,...ng->...gd)jaxnnone_hotr)einsum)rrnrone_hot_block_idss r/_create_global_aggregatesrs.y.A ::*M;L MMr1ceZdZUeed<ej Zejed<dZe ed<e jjjZedej"fed<dZdZy ) FlaxLongT5LayerNorm hidden_sizerWgư>eps. weight_initc^|jd|j|jf|_y)Nweight)paramrrrselfs r/setupzFlaxLongT5LayerNorm.setups%jj4+;+;d>N>N=PQ r1ctj|jddjdd}|tj||j zz }|j |zS)zg Construct a layernorm module in the LongT5 style; No bias and no subtraction of mean. f4rfr(T)r4keepdims)r)powerastypemeansqrtrr)rrvariances r/__call__zFlaxLongT5LayerNorm.__call__s[ 99]11$7;@@bSW@X%DHH1D(EE {{]**r1N)__name__ __module__ __qualname__int__annotations__r)float32rWrfloatrr initializersrrrrzr{rrr1r/rrsV{{E399"C-0VV-@-@-E-EK#rzz/*ER+r1rc\eZdZUeed<ej Zejed<dZddZ y)FlaxLongT5DenseActDenseconfigrWc|jj|jjdzz}|jj|jjdzz}t j |jjdt jjj||j|_ t j |jjdt jjj||j|_ t j|jj|_t|jj |_yNFuse_bias kernel_initrW)rinitializer_factord_modeld_ffrDenserrnormalrWwiwoDropout dropout_ratedropoutr dense_act_fnactr wi_init_std wo_init_stds r/rzFlaxLongT5DenseActDense.setupskk44 8K8KT8QR kk44 8H8H$8NO (( KK  ++22;?**   (( KK  ++22;?**   zz$++":":; $++223r1c|j|}|j|}|j||}|j|}|SN deterministic)rrrr)rrrs r/rz FlaxLongT5DenseActDense.__call__sD . /  ]- P  . r1NT rrrr"rr)rrWrrrr1r/rrs$ {{E399"4&r1rcZeZdZUeed<ej Zejed<dZdZ y)FlaxLongT5DenseGatedActDenserrWc|jj|jjdzz}|jj|jjdzz}t j |jjdt jjj||j|_ t j |jjdt jjj||j|_ t j |jjdt jjj||j|_ t j|jj|_t |jj"|_yr)rrrrrrrrrrWwi_0wi_1rrrrrrrrs r/rz"FlaxLongT5DenseGatedActDense.setups;kk44 8K8KT8QR kk44 8H8H$8NO HH KK  ++22;?**   HH KK  ++22;?**   (( KK  ++22;?**   zz$++":":; $++223r1c|j|j|}|j|}||z}|j||}|j |}|Sr)rrrrr)rrr hidden_gelu hidden_linears r/rz%FlaxLongT5DenseGatedActDense.__call__sWhhtyy78  -0 #m3  ]- P  . r1Nrrr1r/rrs$ {{E399"42r1rc\eZdZUeed<ej Zejed<dZddZ y)FlaxLongT5LayerFFrrWc|jjr't|j|j|_n&t |j|j|_t |jj|jj|j|_ tj|jj|_ y)NrVrrW)r is_gated_actrrWDenseReluDenserrrlayer_norm_epsilon layer_normrrrrrs r/rzFlaxLongT5LayerFF.setup,s ;; # #">t{{RVR\R\"]D "9$++TZZ"XD - KK  T[[%C%C4:: zz$++":":; r1cz|j|}|j||}||j||z}|Sr)rrr)rrrforwarded_statess r/rzFlaxLongT5LayerFF.__call__7sG??=9../?}.]% 5EUb (cc r1Nrrrr1r/rr(s$ {{E399" <r1rceZdZUeed<dZeed<dZeed<ejZ ejed<dZ e ddZ dZd Zd Zej$d Zd Z ddZy )FlaxLongT5AttentionrFhas_relative_attention_biascausalrWc*|jj|_|jj|_|jj|_|jj|_|jj |_|jj|_ |j|j z|_ |jj|j|j zdzz}|jj|jdzz}|jj|jdzz}tj|jdtjjj!||j"|_tj|jdtjjj!||j"|_tj|jdtjjj!||j"|_tj|jdtjjj!||j"|_|j,rdtj.|j|jtjjj!||j"|_yy)NrFrembedding_initrW)rrelative_attention_num_bucketsrelative_attention_max_distancerd_kvkey_value_proj_dim num_headsn_headsrr inner_dimrrrrrrrWqkvorEmbedrelative_attention_biasr q_init_std kv_init_std o_init_stds r/rzFlaxLongT5Attention.setupEs.2kk.X.X+/3{{/Z/Z,{{** "&++"2"2{{,, {{// (?(??[[33I`I`8`ei7ij kk448LM [[33t~~t7KL  NN++22:>**    NN++22;?**    NN++22;?**    LL++22:>**    + ++-8833 "vv2299+Fjj ,D ( ,r1cd}|r&|dz}||dkD|zz }tj|}ntj|d }|dz}||k}|tj||z tj||z z ||z zz}tj||dz }|tj|||z }|j dSav Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on rrf)a_maxr!i4r)r`cliplogr-rrelative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larges r/_relative_position_bucketz-FlaxLongT5Attention._relative_position_bucketss  A K !2Q!6+ E E  #(9 : !$*;1!E E  1$ $y0&/ GG% 1 2SWW\I=U5V VZehqZq r& "&)XX.HP[^_P_%`"CIIh0AC]^^&&t,,r1cNtj|ddddf}tj|ddddf}||z }|j||j |j|j }|j |}|jddddddddf}|S)%Compute binned relative position biasrrVNrr r rfrr!)r)rXrrrrr transpose)r query_length key_lengthcontext_positionmemory_positionrrelative_position_bucketvaluess r/ compute_biasz FlaxLongT5Attention.compute_biass::l$?4H**Zt>#'#A#A #{{?;;== $B$  --.FG!!),T1a]; r1cp|j|jdd|j|jfzSNrfrCr=rrrrs r/ _split_headsz FlaxLongT5Attention._split_heads4$$]%8%8!%< dNeNe?f%fggr1cZ|j|jdd|jfzSrrCr=rrs r/ _merge_headsz FlaxLongT5Attention._merge_headss,$$]%8%8!%<?P%PQQr1cR|jdd}|jddtj|j|j }|jddtj|j|j }|jddd}|r |j j^} } } } |j } dt| z| ddfz}tjj|j ||}tjj|j ||}||_||_|jd}|j |z|_tjtj| | |zkt| d|| fz}t||}|||fS) a\ This function takes projected key, value states from a single input token and concatenates the states to cached states from previous steps. This function is slightly adapted from the official Flax repository: https://github.com/google/flax/blob/491ce18759622506588784b4fca0e4bf05f8c8cd/flax/linen/attention.py#L252 cache cached_key cached_value cache_indexcLtjdtjS)NrrV)r)rrYrr1r/z;FlaxLongT5Attention._concatenate_to_cache..sCIIaWZW`W`Dar1rrr!) has_variablevariabler)rr=rWvaluelenrlaxdynamic_update_slice broadcast_torXrNr )rkeyr/queryrdis_initializedr'r(r) batch_dims max_lengthrdepth_per_head cur_indexrSnum_updated_cache_vectorspad_masks r/_concatenate_to_cachez)FlaxLongT5Attention._concatenate_to_caches|**7LA]]7L#))SYYPSPYPYZ }}WnciiV[VaVab mmG]ZY#))IS_, 1a/@@G''..z/?/?gNCGG001C1CUGTE"J !&L (- A % + 1 14M MK '' :&5N)NNj!Q(A:$NNH+8^DNE>))r1c|jxr|jddxr| }|jd}|r|n|jd} |jr|j | |} nG|t j |} n/t jd|j| |f|j} |rR|jddjd} tjj| dd|dfd|j|| f} | S)Nr&r'r!rVr)rr-r=rrr)r*rrrW variablesrr1 dynamic_slice) r key_states query_statesrd init_cache seq_lengthcausal_attention_mask_shiftcache_is_filledrr position_biasmax_decoder_lengths r/_create_position_biasz)FlaxLongT5Attention._create_position_biass++g$*;*;G\*Rg\fXf%%a( %4z,:L:LQ:O  + + --lJGM  'NN>:MIIq$,, j&QY]YcYcdM !%!8!F!L!LQ!O GG11A2A6DLL*.@AM r1Nc "|jdd\} } |j|} ||j|n|j|} ||j|n|j|} |j | } |j | } |j | } | t j | jdz} |jddr|jr|jddnd}|jrt|d }|jddrH|jddjd }tjj|dd|dfd d | |f}t j|| f|jd dz}t jt j|d |j}t!||}n|t j|d }|jr,|jdds|r|j#| | | |\} } }|t j$|j&j(}tjj+|dkDt j,|jd j/|j&t j,|j|j/|j&}||j1| | ||| |}|||z}d}|s |j2d kDr|j5d}t7| | |||j2d||j&}t j8d|| }|j;|}|j=|}||f}|r||fz}|S) Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). Nrfr(r&r'r)rboolrVr!)rKrwrTbias dropout_rngrbroadcast_dropoutrrW...hqk,...khd->...qhd)r=rrrr r)rr-rr?r rr1r@r3 expand_dimsr r=finforWminselectfullrrIrmake_rngrrr$r)rrrdkey_value_statesrG use_cacheoutput_attentionsrrCrrDrBrA value_statesrEcausal_attention_maskrH mask_valuerQ attn_weights attn_outputoutputss r/rzFlaxLongT5Attention.__call__s|"/!4!4Ra!8 Jvvm, .>.FTVVM*DFFScLd 0@0Htvvm,dffUeNf ((6 &&z2 ((6  !3!3B!788 8<7H7HR^7_dhdodoDNN7 #M 2vw $ ;;$4^6$R !  ,7%)^^G%<\%J%P%PQR%S"(+(=(=)6::'9:)%%($4$4% }7L7R7RSTSU7V'V% !!--X>@U@[@[N+>;PQN  ' __^(KN ;;D--g|D 7;7Q7QL,8 4J n  %4::.22J WW^^"--s3::4::F--z:AA$**MN   66L.*jRmM) - >  !3-- 2K5  #"'**  jj!8, U '' 4 ff[)  . /Gr1T )NNNFFTF)rrrr"rrrLrr)rrWr staticmethodrrr r$rcompactr=rIrrr1r/rr?s (--FD{{E399",\!-!-F"hRZZ**@6qr1rceZdZUeed<dZeed<ejZ ejed<dZ e ddZ de fdZd Zd Zd e d eej&d ej&fdZ ddZy)FlaxLongT5LocalAttentionrFrrWcr|jj|_|jj|_|jj|_|jj|_|jj |_|jj|_|jdz|_ |jj|_ |j|j z|_ |jj|j|j zdzz}|jj|jdzz}|jj|jdzz}tj|jdt jj"j%||j&|_tj|jdt jj"j%||j&|_tj|jdt jj"j%||j&|_tj|jdt jj"j%||j&|_|j0rYtj2|j|jt jj"j%||_yy)Nr!rFrr)rrrrrrrr local_radiusr3rrrrrrrrrrWrrrrrrrrs r/rzFlaxLongT5LocalAttention.setup`s;.2kk.X.X+/3{{/Z/Z,{{** "&++"2"2{{,,  KK44**Q.{{// (?(??[[33I`I`8`ei7ij kk448LM [[33t~~t7KL  NN++22:>**    NN++22;?**    NN++22;?**    LL++22:>**    + ++-8833 "vv2299+F,D ( ,r1cd}|r&|dz}||dkD|zz }tj|}ntj|d }|dz}||k}|tj||z tj||z z ||z zz}tj||dz }|tj|||z }|j dSrrrs r/rz2FlaxLongT5LocalAttention._relative_position_bucket  A K !2Q!6+ E E  #(9 : !$*;1!E E  1$ $y0&/ GG% 1 2SWW\I=U5V VZehqZq r& "&)XX.HP[^_P_%`"CIIh0AC]^^&&t,,r1 block_lengthctjd|zd}||| }|dddf|dddfz }|j|d|j|j}|j |}|j dddddddddf}|SrrrrVNTrrr)rXrrrrrrrorrrrrs r/rz%FlaxLongT5LocalAttention.compute_bias**Q%5TB*< F+D!G47G47PP#'#A#A ;;== $B$  --.FG!!),T4Aq-@A r1cp|j|jdd|j|jfzSrrrs r/r z%FlaxLongT5LocalAttention._split_headsr!r1cV|j|jdd|jSNrr(r#rs r/r$z%FlaxLongT5LocalAttention._merge_heads&$$]%8%8%;RPPr1r3rdr&c|jr|j|}|S|tj|}|Stjdd|j |d|zf|j }|SNr!rrVrrr)r*rrrWrr3rdrGs r/rIz.FlaxLongT5LocalAttention._create_position_biass  + + --i8M   'NN>:M IIq!T\\9a)m&T\`\f\fgMr1Nc N|jdd\}}|j|} ||j|n|j|} ||j|n|j|} |j | } |j | } |j | } t | |j d} t | |j d} t | |j d} t| dd} t| dd} | tj| jdz} |t||j }tjj|dkDtj|jdj|j tj|jd j|j }|3|j#|j |}|||j%ddz}d} |s |j&dkDr|j)d } t+| | || |j&d ||j } tj,d | | }|j/|}|ddd|ddf}|j1|}||f}|r|| fz}|S)rKNrfr!rKrgr(rrw _rTrOrS)r=rrrr rFr3rTr)rrkrr1rWrXrrWrIswapaxesrrYrrr$r)rrrdrZrGr\rrrDrBrAr]rQr`rarbs r/rz!FlaxLongT5LocalAttention.__call__s"/!4!4Ra!8 Jvvm, .>.FTVVM*DFFScLd 0@0Htvvm,dffUeNf ((6 &&z2 ((6 *,QO ' DNNK ),QO +:!STU ,\aWXY  !3!3B!788  %6~t~~VN!WW^^"--s3::4::F--u5<!>{{// (?(??[[33I`I`8`ei7ij kk448LM [[33t~~t7KL  NN++22:>**    NN++22;?**    NN++22;?**    LL++22:>**    + ++-8833 "vv2299+F,D (  + +24((33 "vv2299+F3D / (; KK  T[[%C%C4::( $r1cd}|r&|dz}||dkD|zz }tj|}ntj|d }|dz}||k}|tj||z tj||z z ||z zz}tj||dz }|tj|||z }|j dSrrrs r/rzd||}|jA|}|ddd|ddf}|jC|}||f}|r||fz}|S)rKNrfr(r!rKrg.rrwrrNrTrOrS)"r=rr)rrlrrrrrr rFr3rTr>tilerPrrkrr1rWrXrrWrIrrrrYrrr$r)rrrdrZrGr\rrrDrnr_global_seq_len global_inputsrBrAr]side_key_statesside_value_statesrepsr^side_position_biasrQr`rarbs r/rz+FlaxLongT5TransientGlobalAttention.__call__s2"/!4!4Ra!8 J )E,8Nchh T^G_>`  " ") % % -22261-O\ 44]C vvm, .>.FTVVM*DFFScLd 0@0Htvvm,dffUeNf ((6 &&z2 ((6 &&/ FF=1++O< --.?@*,QO ' DNNK ),QO +:!STU ,\aWXY so**Q./""1%Q((?1dC<#@$GHH%6q$|%DdK__j/%BK  6G'HqQ  !3!3B!788  %#<^T^^#\ #&77>>$q(-33S9@@L-33U;BB4::N$ $(   66t~~~VM#/ -0D0M0MaQR0S S %!$:z*B!C!%!7!7HZ![ !34F]_!` !$.@!Q!G OO]aA   ,0,L,LVjj     ;; JJ83q6Y]YcYcdf fJ ! #  (3?Q;RZ^ZdZdegg r1Nc |jd||||| | } | d}| dd} |jxr|du} | r(|jd|||||| }|d}| |ddz} |jd|| }|f}|| z}|S)Nrrr!)rZrdrGr\rr(r)rr)rrrdrGencoder_hidden_statesencoder_attention_maskencoder_decoder_position_biasr\ return_dictrrCself_attention_outputsattention_outputsdo_cross_attentioncross_attention_outputsrbs r/rzFlaxLongT5Block.__call__ s"/A )'/'! " /q1 2126![[N-B$-N &3djjm!65;"3+ ' #4A6M!24KAB4O O ' 2}MR  "--r1) NNNNNFTTFrrr1r/rrsN (--{{E399"h@"#&*0r1rcveZdZUeed<eed<ejZejed<dZ ddZ y)FlaxLongT5LayerCollectionrrrWcft|j|j|j|_y)Nr)rrrrWrrs r/rzFlaxLongT5LayerCollection.setupBs&$ KKT5U5U]a]g]g  r1Nc 6|j|||||||||  S)N)rdrGrrrr\rrC)r) rrrdrGrrrr\rrCs r/rz"FlaxLongT5LayerCollection.__call__Gs5zz )'"7#9*G/'!  r1)NNNNNFTF) rrrr"rrLr)rrWrrrr1r/rr=sD !%%{{E399" "#&* r1rc eZdZUeed<ej Zejed<dZe ed<dZ d de de d e d e fd Z y) FlaxLongT5BlockCollectionrrWFgradient_checkpointingc |jj|_|jrnttd}t |jj Dcgc].}||j|dk(|jt|0c}|_ yt |jj Dcgc]1}t |j|dk(|jt|3c}|_ ycc}wcc}w)N))static_argnumsr)rrWr) rrrrematrrL num_layersrWrblocks)rFlaxLongT5CheckpointLayerrRs r/rzFlaxLongT5BlockCollection.setupfskk((  & &(-.GXa(b %t{{556 *KK12a**Q DK"t{{556 *KK12a**Q DKs 3C676C;Nr\output_hidden_statesrrCc T|rdnd} |rdnd} |r|jrdnd} d} d} t|jD]`\}}|r| |fz} |||| ||| ||| }|d}|d} |jr | ||rdnd} |sB| |dfz} |jsX| |dfz} bt|| | | S)Nrrr!rrflast_hidden_stater attentionscross_attentions)r enumeraterr)rrrdrrr\rrrCall_hidden_statesall_attentionsall_cross_attentionsrGrrR layer_module layer_outputss r/rz"FlaxLongT5BlockCollection.__call__~s#7BD0d&7DKKrd (,%(5 VOA|#$58H$H!(%&-! M*!,M *!,M{{4@0=CTaZ[0\- !/=3C2E!E;;+?=QRCSBU+U(; V>=++%1   r1)NNNNFFTF) rrrr"rr)rrWrrLrrrr1r/rrasq {{E399"#(D(4"#"'%*" 6  6 #6 6 6 r1rc eZdZUeed<ej ed<ejZ ejed<dZ e ed<dZ dde d e d e d e d e f d Z y)FlaxLongT5Stackr embed_tokensrWFrc|jj|_t|j|j|j|_t |jj|jj|j|_ tj|jj|_ y)NrWrr)rrrrWrblockrrrfinal_layer_normrrrrrs r/rzFlaxLongT5Stack.setups~kk(( . KKtzz$B]B]  !4 KK  T[[%C%C4::! zz$++":":; r1Nr\rrrrCc n|j|} |j| |} |j| ||||||| } | d} |j| } |j| |} d} |r| j} | | fz} |s|r | | f| ddzS| f| ddzSt | | | j | jS)Nr)rdrrr\rrrCrrfr!r)rrrrrrrr) rr#rdrrr\rrrrCrrbrs r/rzFlaxLongT5Stack.__call__s)))4  ]- P ** )"7#9/!5'!    --m<  ]- P !  ' 5 5  1]4D D #!%ABK  "#gabk1 1<++))$55   r1) NNNNFFTTF)rrrr"rrrr)rrWrrLrrrr1r/rrs (({{E399"#(D( <"#"'%* " 3  3 #3 3 3 3 r1ra Args: input_ids (`jnp.ndarray` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. LongT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [LONGT5 Training](./longt5#training). attention_mask (`jnp.ndarray` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a1 Args: decoder_input_ids (`jnp.ndarray` of shape `(batch_size, target_sequence_length)`): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) For training, `decoder_input_ids` should be provided. encoder_outputs (`tuple(tuple(jnp.ndarray)`): Tuple consists of (`last_hidden_state`, *optional*: `hidden_states`, *optional*: `attentions`) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`jnp.ndarray` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`jnp.ndarray` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. past_key_values (`Dict[str, np.ndarray]`, *optional*, returned by `init_cache` or when passing previous `past_key_values`): Dictionary of pre-computed hidden-states (key and values in the attention blocks) that can be used for fast auto-regressive decoding. Pre-computed key and value hidden-states are of shape *[batch_size, max_length]*. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`jnp.ndarray` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. LongT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [LONGT5 Training](./longt5#training). attention_mask (`jnp.ndarray` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`jnp.ndarray` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) LONGT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [LONGT5 Training](./longt5#training). decoder_attention_mask (`jnp.ndarray` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. encoder_outputs (`tuple(tuple(jnp.ndarray)`, *optional*): Tuple consists of (`last_hidden_state`, `optional`: *hidden_states*, `optional`: *attentions*) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)` is a sequence of hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`tuple(tuple(jnp.ndarray))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. ceZdZUdZeZdZdZeje d<dde jdfded e ed ed e jd ef fd ZdZd#dej(j*d e dedefdZee d$de j4dee j4de j4dee j4deedeedeedededefdZdZee e!e"e d%de j4dee j4deedeedeedededefdZ#ee$e!e%e d$d ee j4dee j4d!edeedeedeedededefd"Z&xZ'S)&FlaxLongT5PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. transformerN module_classrJrTr input_shapeseedrW_do_initc Z|jd||d|}t| ||||||y)N)rrW)rrrWrr)rsuper__init__) rrrrrWrrmodule __class__s r/rz"FlaxLongT5PreTrainedModel.__init__s=#""H&HH [tSXcklr1c^|j|j|jd|_y)NT)rrWr)rrrW_modulers r/enable_gradient_checkpointingz7FlaxLongT5PreTrainedModel.enable_gradient_checkpointings*((;;**#')  r1rngparamsr&ctj|d}tj|}tj|}tj|}tjj |\}} || d} |j j| ||||d} |dtt| } tt|}|jD] } | | || < t|_ tt|S| S)NrrV)rrr)r)rr|rrandomsplitrinitrr _missing_keysr,r r) rrrrr#rddecoder_input_idsdecoder_attention_mask params_rngrQrngs random_params missing_keys r/ init_weightsz&FlaxLongT5PreTrainedModel.init_weightssIIk6 y1MM)4!$y!9"%**"2"23"7 K$= ((     "      (-)@AM!(6"23F#11 A &3K&@{# A!$D .01 1 r1r#rdr r r\rrtrainrQc 8||n|jj}||n|jj}||n|jj}| t d|t j |}|t j |}| d| ini} |jjd| xs |jit j|dt j|dt j|dt j|d|||| |  S)NzfMake sure to provide both `input_ids` and `decoder_input_ids`. `decoder_input_ids` is not passed here.rrrrV) r#rdr r r\rrrr) rr\rrrr)r|rapplyrr) rr#rdr r r\rrrrrQrs r/rz"FlaxLongT5PreTrainedModel.__call__s#2C1N-TXT_T_TqTq$8$D $++JjJj &1%FlaxLongT5PreTrainedModel.init_cache.._decoder_forward-#779N!!& r1rT)r r rrCmethodr&) r)rr|rr rrrr )rrr8encoder_outputsr r rinit_variabless r/rCz$FlaxLongT5PreTrainedModel.init_caches HHj*%=TJ!$/@!A )) JJ  q !/#9"1!"4# * w/00r1 output_type config_classc ||n|jj}||n|jj}||n|jj}|t j |}i} ||| d<d} |j jd|xs |jit j|dt j|d|||| | |  S)a Returns: Example: ```python >>> from transformers import AutoTokenizer, FlaxLongT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-base") >>> model = FlaxLongT5ForConditionalGeneration.from_pretrained("google/long-t5-local-base") >>> text = "My friends are cool but they eat too many carbs." >>> inputs = tokenizer(text, return_tensors="np") >>> encoder_outputs = model.encode(**inputs) ```rc6|j}|||fi|Sr)_get_encoder_module)rr#rdr encode_modules r/_encoder_forwardz:FlaxLongT5PreTrainedModel.encode.._encoder_forward;s""668M NEfE Er1rrrV)r#rdr\rrrrr) rr\rrr)r|rrrr) rr#rdr\rrrrrQrr)s r/encodez FlaxLongT5PreTrainedModel.encodes82C1N-TXT_T_TqTq$8$D $++JjJj &1%>> from transformers import AutoTokenizer, FlaxLongT5ForConditionalGeneration >>> import jax.numpy as jnp >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-base") >>> model = FlaxLongT5ForConditionalGeneration.from_pretrained("google/long-t5-local-base") >>> text = "My friends are cool but they eat too many carbs." >>> inputs = tokenizer(text, return_tensors="np") >>> encoder_outputs = model.encode(**inputs) >>> decoder_start_token_id = model.config.decoder_start_token_id >>> decoder_input_ids = jnp.ones((inputs.input_ids.shape[0], 1), dtype="i4") * decoder_start_token_id >>> outputs = model.decode(decoder_input_ids, encoder_outputs) >>> logits = outputs.logits ```Nrrfrrr&Fc6|j}|||fi|Srrrs r/rz:FlaxLongT5PreTrainedModel.decode.._decoder_forwardrr1rrV r r rrr\rrrrmutablerr+r!) rr\rrr=r)rrrrrr )rr r rr r+r\rrrrrQrrsequence_lengthrinputsr/rrbpasts r/decodez FlaxLongT5PreTrainedModel.decodeKsL2C1N-TXT_T_TqTq$8$D $++JjJj &1%>LGr1r NNNNNNFNN)NNNNFNN)(rrrrr"r$base_model_prefixrrModulerr)rrrrWrLrrrrrrrrLONGT5_INPUTS_DOCSTRINGr{rdictrrCrLONGT5_ENCODE_INPUTS_DOCSTRINGr rr*LONGT5_DECODE_INPUTS_DOCSTRINGrr3 __classcell__)rs@r/rrs  L%"L"))" #);; m m3Z m m yy m  m ! 2 2!!PZ!fp!:++BC15)-8<,0/3&*#/ ;;/ !-/ ;; / !) 5 / $D> / 'tn/ d^/ / / / D/ b"1H89+>\Z15,0/3&*#6 ;;6 !-6 $D> 6 'tn 6 d^ 6 6 6 6 [:6 p89+Xgst 9=8< $,0/3&*#c!) 5 c !) 5 c  c$D>c'tncd^ccccu:cr1ra The LongT5 model was proposed in [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://arxiv.org/abs/2112.07916) by Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung and Yinfei Yang. It's an encoder-decoder transformer pre-trained in a text-to-text denoising generative setting. LongT5 model is an extension of T5 model, and it enables using one of the two different efficient attention mechanisms - (1) Local attention, or (2) Transient-Global attention. This model inherits from [`FlaxPreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a Flax Linen [flax.nn.Module](https://flax.readthedocs.io/en/latest/_autosummary/flax.nn.module.html) subclass. Use it as a regular Flax Module and refer to the Flax documentation for all matter related to general usage and behavior. Finally, this model supports inherent JAX features such as: - [Just-In-Time (JIT) compilation](https://jax.readthedocs.io/en/latest/jax.html#just-in-time-compilation-jit) - [Automatic Differentiation](https://jax.readthedocs.io/en/latest/jax.html#automatic-differentiation) - [Vectorization](https://jax.readthedocs.io/en/latest/jax.html#vectorization-vmap) - [Parallelization](https://jax.readthedocs.io/en/latest/jax.html#parallelization-pmap) Parameters: config ([`LongT5Config`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~FlaxPreTrainedModel.from_pretrained`] method to load the model weights. dtype (`jax.numpy.dtype`, *optional*, defaults to `jax.numpy.float32`): The data type of the computation. Can be one of `jax.numpy.float32`, `jax.numpy.float16` (on GPUs) and `jax.numpy.bfloat16` (on TPUs). This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. If specified all the computation will be performed with the given `dtype`. **Note that this only specifies the dtype of the computation and does not influence the dtype of model parameters.** If you wish to change the dtype of the model parameters, see [`~FlaxPreTrainedModel.to_fp16`] and [`~FlaxPreTrainedModel.to_bf16`]. z_The bare LONGT5 Model transformer outputting raw hidden-stateswithout any specific head on top.ceZdZUeed<ej Zejed<dZe ed<dZ dZ dZ d d e fd Z y) FlaxLongT5ModulerrWFrc|jSrencoderrs r/r'z$FlaxLongT5Module._get_encoder_module ||r1c|jSrdecoderrs r/rz$FlaxLongT5Module._get_decoder_modulerAr1ctj|jj|jjt jj j|jjdz|j|_ tj|j}d|_ t||j|j|j|_tj|j}d|_ |jj"|_t||j|j|j|_y)NrxrF)rrWrT)rrr vocab_sizerrrrrrWsharedcopydeepcopyrrrr@num_decoder_layersrrDrencoder_configdecoder_configs r/rzFlaxLongT5Module.setupshh KK " " KK  66..55dkk6T6TWZ6Z[**   t{{3 %& **#'#>#>   t{{3 $$(KK$B$B!& **#'#>#>   r1Nrc p||n|jj}|j|||||| }|j|||d||||| } |s| |zSt | j | j | j| j| j|j |j|jS)Nr#rdr\rrrrr#rdrrr\rrr)rr+decoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentions) ruse_return_dictr@rDrrr+rrr) rr#rdr r r r\rrrdecoder_outputss r/rzFlaxLongT5Module.__call__s&1%>> from transformers import AutoTokenizer, FlaxLongT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-base") >>> model = FlaxLongT5Model.from_pretrained("google/long-t5-local-base") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="np" ... ).input_ids >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="np").input_ids >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ``` r"z4LONGT5 Model with a `language modeling` head on top.ceZdZUeed<ej Zejed<dZe ed<dZ dZ dZ d d e fd Z y) (FlaxLongT5ForConditionalGenerationModulerrWFrc|jSrr?rs r/r'z{K  **HxAQASAS6T+UWfgI _5I>> from transformers import AutoTokenizer, FlaxLongT5ForConditionalGeneration >>> import jax.numpy as jnp >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-base") >>> model = FlaxLongT5ForConditionalGeneration.from_pretrained("google/long-t5-local-base") >>> text = "summarize: My friends are cool but they eat too many carbs." >>> inputs = tokenizer(text, return_tensors="np") >>> encoder_outputs = model.encode(**inputs) >>> decoder_start_token_id = model.config.decoder_start_token_id >>> decoder_input_ids = jnp.ones((inputs.input_ids.shape[0], 1), dtype="i4") * decoder_start_token_id >>> outputs = model.decode(decoder_input_ids, encoder_outputs) >>> logits = outputs.logits ```Nrrfrrr&Fc|j}|||fi|}|d} jjr| jjdzz} jjrJ|jj dd}|j jdd|jii|}||fS|j |}||fS)Nrrrrdre) rrrgrrGr?rbrrh) rr r rrrVrirjrkrs r/rzCFlaxLongT5ForConditionalGeneration.decode.._decoder_forward s#779N,!&O .a0O{{..#2T[[5H5H$5N"O{{..#)==#:#:8#D[#Q "NN00(XGWGYGY>LGr1rdc |j\}}|j|||} tj||fd} |!tj j | |d} | ||| dS)NrrVr7)r+r rr )r=rCr)rrr1r2) rr r8rdr r rrrDr+extended_attention_masks r/prepare_inputs_for_generationz@FlaxLongT5ForConditionalGeneration.prepare_inputs_for_generationR sy"3!8!8 J//*j/R#&((J +C4"P ! -&)gg&B&B')?' # /.&4&=   r1c$|j|d<|S)Nr+)r+)r model_outputs model_kwargss r/update_inputs_for_generationz?FlaxLongT5ForConditionalGeneration.update_inputs_for_generationo s*7*G*G &'r1r4)NNN)rrrr\rrr:r rr"rr)r{r8rLrr3rArrayrrrvrr1r/rmrms ;L89+P_kl 9=8< $,0/3&*#!) 5  !) 5   $D>'tnd^m:J/36:  !+  !) 3  :r1rma Returns: Example: ```python >>> from transformers import AutoTokenizer, FlaxLongT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-base") >>> model = FlaxLongT5ForConditionalGeneration.from_pretrained("google/long-t5-local-base") >>> ARTICLE_TO_SUMMARIZE = "summarize: My friends are cool but they eat too many carbs." >>> inputs = tokenizer([ARTICLE_TO_SUMMARIZE], return_tensors="np") >>> # Generate Summary >>> summary_ids = model.generate(inputs["input_ids"]).sequences >>> print(tokenizer.decode(summary_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)) ``` )rmrZrr,)`rrHtypingrrrrr flax.linenlinenrr jax.numpynumpyr)rzflax.core.frozen_dictrr r r r r nn_partitioningflax.linen.attentionrflax.traverse_utilrr jax.randomrmodeling_flax_outputsrrrrrmodeling_flax_utilsrrrrrutilsrrrr configuration_longt5r" get_loggerrlogger_CHECKPOINT_FOR_DOC_CONFIG_FOR_DOCrr{rr0rArFrTr]rcrkrrrr6rrrrrrirrrrrrrrrr9r:r7rLONGT5_START_DOCSTRINGr=rZFLAX_LONGT5_MODEL_DOCSTRINGr\rm,FLAX_LONGT5_CONDITIONAL_GENERATION_DOCSTRING__all__rr1r/rs[ 77 >>66>;ut.   H %1  #++ S Z] bebmbm     3 3 WZWbWb  ##++ ## #S #S[[ #Q`opPRhik ryyk jk \e)BeP0,*&(?Bn(n!&4GVe ar1