T^(h\dZddlZddlmZmZmZmZmZddlZddl Zddlm Z ddl m Z ddl mZmZmZmZddlmZdd lmZdd lmZmZmZmZmZdd lmZdd lmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+m,Z,m-Z-e%r ddl.m/Z/ddl0m1Z1e(jde3Z4dZ5Gdde jlZ7 ddl8m9Z9e9Z7e4judej|e7Gdde jlZ?Gdde jlZ@Gdde jlZAGdde jlZBGd d!e jlZCGd"d#eZDd$ZEd%ZFe#d&eEGd'd(eDZGGd)d*e jlZHGd+d,e jlZIGd-d.e jlZJGd/d0e jlZKGd1d2e jlZLGd3d4e jlZMd5ZNd6ZOd7ZPe#d8eNGd9d:eDZQe#d;eNGd<d=eDeZRgd>ZSy#e;$rYae<$re4j{dYywxYw)?zPix2Struct modeling fileN)DictListOptionalTupleUnion)nn)ACT2FN)Cache DynamicCacheEncoderDecoderCache StaticCache)GenerationMixin)AttentionMaskConverter)BaseModelOutputBaseModelOutputWithPooling!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)ALL_LAYERNORM_LAYERS) DUMMY_INPUTS DUMMY_MASKadd_start_docstrings%add_start_docstrings_to_model_forwardis_torch_flex_attn_availableis_torch_fx_proxyis_torchdynamo_compilingloggingreplace_return_docstrings)Pix2StructConfigPix2StructTextConfigPix2StructVisionConfig) BlockMask)make_flex_block_causal_maskr"c&eZdZdfd ZdZxZS)Pix2StructLayerNormct|tjt j ||_||_y)zc Construct a layernorm module in the T5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s p/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/pix2struct/modeling_pix2struct.pyr+zPix2StructLayerNorm.__init__As1 ll5::k#:; #c|jtjjdj dd}|tj ||j zz}|jjtjtjfvr%|j|jj}|j|zS)NT)keepdim) tor-float32powmeanrsqrtr0r/dtypefloat16bfloat16)r1 hidden_statesvariances r5forwardzPix2StructLayerNorm.forwardIs !##EMM266q9>>r4>P% Ht?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r6)gư>__name__ __module__ __qualname__r+rE __classcell__r4s@r5r(r(@s $ +r6r() FusedRMSNormzWDiscovered apex.normalization.FusedRMSNorm - will use it instead of Pix2StructLayerNormzJDiscovered apex but it failed to load, falling back to Pix2StructLayerNormcdeZdZdZdeddffd ZdejdejfdZxZ S)Pix2StructVisionEmbeddingsa- Construct the embeddings from patch. In `Pix2Struct` the input is different from classic Vision-transformer models. Here the input is a sequence of `seq_len` flattened patches that also combines padding patches (tokens). Each patch is represented by a vector of `hidden_size` values. configreturnNct|tj|j|j |_tj|j|j |_ tj|j|j |_ tj|j|_ yN)r*r+rLinearpatch_embed_hidden_sizer2patch_projection Embeddingseq_len row_embeddercolumn_embedderDropout dropout_ratedropoutr1rOr4s r5r+z#Pix2StructVisionEmbeddings.__init__ps}  " &*H*H&J\J\ ]LL9K9KL!||FNNFA \\*- r6) rGrHrI__doc__r"r+r-TensorrErJrKs@r5rNrNis7 7/7D7%,,r6rNc.eZdZfdZ ddZxZS)Pix2StructVisionAttentionc|t||j|_|j|_|j |_|j|_|j |jz|_ tj|j|jd|_ tj|j|jd|_ tj|j|jd|_tj|j|jd|_d|_yNFbias)r*r+r2d_kvkey_value_proj_dimnum_attention_headsn_headsattention_dropoutr\ inner_dimrrSquerykeyvalueoutputgradient_checkpointingr]s r5r+z"Pix2StructVisionAttention.__init__s !--"(++11 // (?(??YYt//eL 99T--t~~EJYYt//eL ii0@0@uM &+#r6c6|jdd\}fd}|j|}|j|} |j|} t j || j dd} |t jdj||f| j| j}jrjrd|_ |jdk(r*||ddddddfj|jz}nw|||j|jz}nVt!sLt j"|f|j|j}||j|jz}d|z }|j%|dk(t j&| jj(} | | z } t j*| t j,t j&| jj(} t.j0j3| dtj4 j7| } t.j0j9| j8j } || |z} t j | | }|j ddj;j=dj>}jA|}|f|fz}|r|| fz}|S) z& Self-attention block Nr8c|jjdjjj ddS) projectionr9r!r8) contiguousviewrqro transpose)states batch_sizer1s r5to_projection_shapez>Pix2StructVisionAttention.forward..to_projection_shapes<$$&++JDLL$JaJabllmnpqr rr6r r!devicer@Tr9)dimr@ptraining)!shapertrurvr-matmulr~zerosrqrr@rxr requires_gradrr;rr. masked_fillfinfominmaxtensorr functionalsoftmaxr<type_asr\r|r}rsrw)r1rCattention_mask position_biaslayer_head_maskoutput_attentions seq_lengthr query_states key_states value_statesscoresposition_bias_masked attn_weights attn_outputoutputsrs` @r5rEz!Pix2StructVisionAttention.forwards"/!4!4Ra!8 J s +4::m+DE )-)@A *4::m+DE lJ,@,@A,FG  !KKDLL*j9&--W]WcWcM**t}}.2 +!!#q( -q$a?O0P0S0STaThTh0i i + -0A0A-BVBV0W W -/!&,]5I5IQ^QdQd"!.0A0A-BVBV0W W  -M,88!9KU[[Y_YeYeMfMjMjk&&65<< FLL0I0M0M#NO}},,V5==,QYYZ`a }},,\T\\TXTaTa,b   &'/9Lll<> "++Aq1<<>CCJPRTXTbTbc kk+. .M#33 /Gr6)NNNFrFrKs@r5riris,& Mr6ric*eZdZdeffd ZdZxZS)Pix2StructVisionMlprOct|tj|j|j d|_tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_yrkr*r+rrSr2d_ffwi_0wi_1worZr[r\r dense_act_fnactr]s r5r+zPix2StructVisionMlp.__init__ IIf00&++EJ IIf00&++EJ ))FKK););%Hzz&"5"56 &--.r6c,|j|j|}|j|}||z}|j|}t |j j tjr|j|j j jk7r`|j j jtjk7r/|j|j j j}|j |}|SrR rrrr\ isinstancerr/r-rgr@int8r;r1rC hidden_gelu hidden_linears r5rEzPix2StructVisionMlp.forwardhhtyy78  -0 #m3  ]3 tww~~u|| 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r6)rGrHrIr$r+rErJrKs@r5rrs/5/r6rceZdZdeddffd Z d dej deej deej dede e ej ej fe ej ff d Z xZ S) Pix2StructVisionLayerrOrPNc*t||j|_d|_t ||_t ||_t|j|j|_ t|j|j|_ y)Nr!r3) r*r+chunk_size_feed_forward seq_len_dimri attentionrmlpr(r2layer_norm_epspre_mlp_layer_normpre_attention_layer_normr]s r5r+zPix2StructVisionLayer.__init__ ss '-'E'E$26:&v."5f6H6HfNcNc"d(;Fz2Pix2StructVisionEncoder.forward..dsmq_`_lmslast_hidden_staterC attentions) enumeraterrxr_gradient_checkpointing_func__call__tupler) r1rCrrrrrall_hidden_statesall_self_attentionsi layer_moduler layer_outputss r5rEzPix2StructVisionEncoder.forward>s#7BD$5b4(4 POA|#$58H$H!.7.CilO**t}} $ A A ))!"#% ! !-]NO]n o )!,M &9]1=Mrt|jtr |jjjn|jj}|j@jj jd||dzzyt|t8jBt8jDfrt8jFjI|jj jKtLjNd|jjPjK|jjR|j_|j%|jj j!yyt|tr3|j&|jj j dyyt|t8j:rz|jj jd|jjP|j<2|jj |j<j!yyy)zInitialize the weights?g)r>stdrmN)*rOinitializer_factorrr(r/datafill_ Pix2StructTextDenseGatedActDenser" text_configr2rrnormal_hasattrrmzero_rrPix2StructTextAttentionrn num_headsrtrurvrwhas_relative_attention_biasrelative_attention_biasrrV padding_idxPix2StructTextModellm_headrSConv2dinit trunc_normal_r;r-r<initializer_ranger@)r1modulefactorr2rrorqs r5 _init_weightsz'Pix2StructPreTrainedModel._init_weightss// f1 2 MM   $ $Vc\ 2  @ Adkk+;< ''33[[,,  4>dkkK[3\4;;**//bfbmbmbrbrD KK   # # + +&[UYDY:Z + [v{{F+ 0@0@0L   %%++- KK   # # + +&[UYDY:Z + [v{{F+ 0@0@0L   %%++- II   ! ! ) )s4D.8Q ) Rvyy&)fiinn.H ##))+/I)  7 8 dkk+;< ''33[[,,  1;4;;HX0Y '',,_c_j_j_v_v  dkk+;< ''11[[**  LL   $ $ , ,#6kTfFfkoEo;p , q JJ   " " * *;PTCT9U * V LL   $ $ , ,#6[RVEV;W , X MM % % - -3FwQcGchlFl.  3 4dkk+;< ''33[[,,  NN ! ! & & . .CV X\G\=] . ^ BII 6 7"$!6!6 ""%%emm43DKKDaDa"7"b$$% MM {{&   &&('  3 4}}( ""((-)  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>..r6c|jj}|jj}| tdt |rGt j |jdddz|}t j||dddfgd}n>|j|j}|dddfj|dddf<||d<| td|j|d k(||S) Nzself.model.config.decoder_start_token_id has to be defined. In Pix2Struct it is usually set to the pad_token_id. See Pix2Struct docs for more information.r9)r!.rr!).rz1self.model.config.pad_token_id has to be defined.) rOdecoder_start_token_id pad_token_id ValueErrorrr-fullrcat new_zerosclone masked_fill_)r1rrr shifted_input_idss r5 _shift_rightz&Pix2StructPreTrainedModel._shift_rights!%!C!C{{// ! )<  Y ' % 9??3B+?$+FH^ _  % +Q*RXZ [  ) 3 3IOO D )238)<)B)B)D c12g &(> f %  PQ Q&&'8D'@,O  r6N) rGrHrIrfr" config_class_supports_cache_class_supports_static_cachepropertyrrrrr6r5rrls8 $L " M?`!r6raL This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`Pix2StructConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: flattened_patches (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_channels x patch_height x patch_width)`): Flattened and padded pixel values. These values can be obtained using [`AutoImageProcessor`]. See [`Pix2StructVisionImageProcessor.__call__`] for details. Check the [original paper](https://arxiv.org/abs/2210.03347) (figure 5) for more details. attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zjThe bare Pix2StructVision Model transformer outputting raw hidden-states without any specific head on top.c(eZdZeZdZdZdgZdeffd Z dZ de e e e fdd fd Zeeeee  ddeej,d eej,d eej,deedeedeedeeeffdZxZS)Pix2StructVisionModelr^TrrOct||||_t||_t ||_t|j|j|_ |jyNr) r*r+rOrNrcrencoderr(r2r layernorm post_initr]s r5r+zPix2StructVisionModel.__init__sU   4V<.v6 ,V-?-?VEZEZ[ r6c.|jjSrR)rcrUr1s r5get_input_embeddingsz*Pix2StructVisionModel.get_input_embeddings)s///r6heads_to_prunerPNc|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrr prune_heads)r1r rheadss r5 _prune_headsz"Pix2StructVisionModel._prune_heads,sE +002 CLE5 LL  u % / / ; ;E B Cr6 output_typerrrrrrc(||n|jj}||n|jj}||n|jj}| t d|#|j ddk7j }|j||jj}|j|}|j||||||}|d} |j| } |s | f} | |ddzSt| |j|jS) a. Returns: Example: ```python >>> import requests >>> from PIL import Image >>> from transformers import AutoProcessor, Pix2StructVisionModel >>> image_processor = AutoProcessor.from_pretrained("google/pix2struct-textcaps-base") >>> model = Pix2StructVisionModel.from_pretrained("google/pix2struct-textcaps-base") >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = image_processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 2048, 768] ``` Nz%You have to specify flattened_patchesr9rr)rrrrrr!r)rOrruse_return_dictr sumfloat get_head_maskrrcrrrrCr) r1r^rrrrrembedding_outputencoder_outputssequence_output head_outputss r5rEzPix2StructVisionModel.forward4s8H2C1N-TXT_T_TqTq$8$D $++JjJj &1% K 'tn K d^K  u00 1K eOK r6rc*eZdZdeffd ZdZxZS)rrOct|tj|j|j d|_tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_yrkrr]s r5r+z)Pix2StructTextDenseGatedActDense.__init__rr6c,|j|j|}|j|}||z}|j|}t |j j tjr|j|j j jk7r`|j j jtjk7r/|j|j j j}|j |}|SrRrrs r5rEz(Pix2StructTextDenseGatedActDense.forwardrr6rGrHrIr#r+rErJrKs@r5rrs/3/r6rc*eZdZdeffd ZdZxZS)Pix2StructTextLayerFFrOct|t||_t |j |j |_tj|j|_ yr) r*r+rDenseReluDenser(r2layer_norm_epsilon layer_normrrZr[r\r]s r5r+zPix2StructTextLayerFF.__init__sK >vF-f.@.@fF_F_`zz&"5"56 r6cr|j|}|j|}||j|z}|SrR)r@r>r\)r1rCforwarded_statess r5rEzPix2StructTextLayerFF.forwards=??=9../?@% 5E(FF r6r:rKs@r5r<r<s737r6r<cfeZdZ ddedeeffd ZeddZd dZ d dZ xZ S) rrO layer_idxct|||_|j|_|j|_|j |_|j |_|j|_ |j|_ |j|jz|_ ||_ |-tjd|j j"dt%j&|j |j d|_t%j&|j |j d|_t%j&|j |j d|_t%j&|j |j d|_|jr/t%j0|j|j|_t5|_d|_y)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Frl)r*r+rrelative_attention_num_bucketsrelative_attention_max_distancer2rnrorrqr[r\rsrDlogger warning_oncer4rGrrSrtrurvrwrVrset pruned_headsrxr1rOrrDr4s r5r+z Pix2StructTextAttention.__init__ss +F(.4.S.S+/5/U/U,!--"(++'' ** (?(??"     *4>>+B+B*CD,, YYt//1A1AN 99T--t/?/?eLYYt//1A1AN ii 0 0$2B2BO  + ++-<<8[8[]a]i]i+jD (E&+#r6cTd}|rC|dz}||dkDjtj|zz }tj|}n*tj|tj | }|dz}||k}|tj |j|z tj ||z z ||z zjtjz}tj|tj||dz }|tj|||z }|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr8r!) r;r-r`absr zeros_likelogr+math full_likewhere)relative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larges r5_relative_position_bucketz1Pix2StructTextAttention._relative_position_buckets(.  A K !2Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $y0&/ II'--/); <hh|i/0 1Y& ( "UZZ. &" &+YY &8RT_bcTc(d& " EKK2CE_``r6c| |jjj}|.tj|tj |dddf}n|dddfj |}tj|tj |dddf}||z }|j|d|j|j}|j|} | jgdjd} | S)z%Compute binned relative position biasN)r@rF)rUrVrW)r8rr!r) rr/rr-aranger`r;r\rFrGpermute unsqueeze) r1 query_length key_lengthrcache_positioncontext_positionmemory_positionrTrelative_position_bucketvaluess r5 compute_biasz$Pix2StructTextAttention.compute_biass >1188??F  !$||L SYZ[\^b[bc -ag699&A ,,zFSTXZ[T[\+.>>#'#A#A ;;== $B$  --.FG *44Q7 r6c |jdd\} } |du} |j|}|j| d|j|jj dd}|@|j j|j}| r |j}n |j}| r|n|}| r7|r5r3j|j}|j|j}n|j|}|j|}|j| d|j|jj dd}|j| d|j|jj dd}|D| s| nd} j|||jd| i\}}| rd|j |j<t!j"||j dd}||jd}||n| ddz}|j$sZt!j&d|j| |f|j(|j* }|j,rE|j.r9d|_n1|j3|||j(| }|dddd| dddf}|#|ddddddd|jdf}||z}|j4rRt!j6|jd}d |t9|j4<|dd|j;f}n|}||z }t<j>jA|jCd jE|}t<j>jG||jF|j. }|||z}t!j"||}|j ddjI}|j| d|jJ}|jM|}|||f}| r||fz}|S)z Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). Nr8r9r!rcTr r)rrcrrr)'rrtr}rqror~ is_updatedgetrDcross_attention_cacheself_attention_cache key_cache value_cacherurvupdater-rrrrr@rxrrrhrKr.listrrrrr+rr\r|rsrw)r1rCmaskkey_value_statesrpast_key_valuerra use_cacherrcrris_cross_attentionrrkcurr_past_key_valuecurrent_statesrrrrbreal_seq_length causal_maskrrrrs r5rEzPix2StructTextAttention.forwards$"/!4!4Ra!8 J.T9zz-0 #((RtG^G^_iijkmno  %'2266t~~FJ!&4&J&J#&4&I&I#-?)] .Z,66t~~FJ.::4>>JL.1J::n5L#RtG^G^_iijkmnoJ',,ZT\\4KbKbcmmnoqrsL)7It+>+E+E dnn?OQ_>`,( L&@DN--dnn=lJ,@,@A,FG  #))"-J.:.FlN[]L^abLbO33 %  j*=fmm[a[g[g! ..4==26M/ $ 1 1#Z Vd!2! !.aZKL!.C D "1a,Bj.>.>r.B,B#BC - ;   ::m11!45D,-Dd''( )#0DIIK#@ #0 &&}},,V\\^,DLLVT }},,\T\\TXTaTa,b   &'/9Lll<> !++Aq1<<> !&&z2t~~F kk+.  > /Gr6FN)T )NN) NNNNNNFFN) rGrHrIr#rr4r+ staticmethodr\rhrErJrKs@r5rrs^jn,*,ZbcfZg,>- - `0ir6rcBeZdZddeeffd Z ddZxZS) Pix2StructTextLayerSelfAttentionrDct|t||||_t |j |j |_tj|j|_ y)NrrDr r*r+rrr(r2r?r@rrZr[r\rLs r5r+z)Pix2StructTextLayerSelfAttention.__init__sU 0 0KW` .f.@.@fF_F_`zz&"5"56 r6c |j|} |j| |||||||} ||j| dz}|f| ddz} | S)N)rsrrrurvrrcrr!r@rr\) r1rCrrrrurvrrcnormed_hidden_statesrrs r5rEz(Pix2StructTextLayerSelfAttention.forwardst $}=>> '+)/)*  & 5Ea5H(II  "%5ab%99r6r|)NNNNFFNrGrHrIrr4r+rErJrKs@r5rrs07XVY]7r6rcDeZdZddeeffd Z ddZxZS)!Pix2StructTextLayerCrossAttentionrDct|t|d||_t |j |j |_tj|j|_ y)NFrrr)r1rOrDr4s r5r+z*Pix2StructTextLayerCrossAttention.__init__sP 0UZfop-f.@.@fF_F_`zz&"5"56 r6c |j|} |j| |||||||| |  } ||j| dz} | f| ddz}|S)N) rsrtrrrurvrarrcrr!r)r1rCrtrrrrurvrarrcrrrrs r5rEz)Pix2StructTextLayerCrossAttention.forwardsy $}=>> -'+)%/)*  %t||4DQ4G'HH /$4QR$88r6rR)NNNNFNFNrrKs@r5rrs27(3-7r6rcLeZdZddeeffd Z ddZxZS)Pix2StructTextBlockrDct|t||||_t |||_t ||_y)Nr)rD)r*r+rself_attentionrencoder_decoder_attentionr<rrLs r5r+zPix2StructTextBlock.__init__sH > (C  *K * & )0r6c (|j||||| | | | }|dd\}} |dd}|jtjk(rhtj|j rEtj |jjdz }tj|| |}|du}|r|j|||||| | ddz| |  }|dd\}} |jtjk(rhtj|j rEtj |jjdz }tj|| |}||ddz}|j|}|jtjk(rhtj|j rEtj |jjdz }tj|| |}|f}| r || fz|z}|S||z}|S)N)rrrrurvrrcr8i)rrr9r!)rtrrrrurarvr) rr@r-rAisinfanyrrclamprr)r1rCrrencoder_hidden_statesencoder_attention_maskencoder_decoder_position_biasrcross_attn_layer_head_maskrurvrrrcrattention_outputs clamp_valuedo_cross_attentioncross_attention_outputsrs r5rEzPix2StructTextBlock.forwards "&!4!4 )'+)/)"5 " )?r(B% ~2126   %-- /EKK 4N4R4R4T++m&9&9:>>EK!KK K<[YM2$> &*&D&D!65; :-+B/!3#"3'E ' #-DBQ,G )M>""emm3 M8R8V8V8X#kk-*=*=>BBTI % M |Q\ ] !24KAB4O O /    %-- /EKK 4N4R4R4T++m&9&9:>>EK!KK K<[YM "  114EEG 11Gr6r|) NNNNNNNNFFTNrrKs@r5rrs@1XVY]1&"#&*#'Hr6ra The Pix2Struct model was proposed in [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://arxiv.org/abs/2210.03347) by Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova. It's an encoder decoder transformer pre-trained in a image-to-text setting. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (Union[`Pix2StructConfig`, `Pix2StructTextConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Pix2StructText is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [Pix2StructText Training](./t5#training). attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Pix2StructText uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [Pix2StructText Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, `optional`: *hidden_states*, `optional`: *attentions*) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)` is a sequence of hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention layers. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, target_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `decoder_inputs_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `decoder_input_ids` and `decoder_inputs_embeds` are both unset, `decoder_inputs_embeds` takes the value of `inputs_embeds`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. a Args: flattened_patches (`torch.FloatTensor` of shape `(batch_size, seq_length, hidden_size)`): Flattened pixel patches. the `hidden_size` is obtained by the following formula: `hidden_size` = `num_channels` * `patch_size` * `patch_size` The process of flattening the pixel patches is done by `Pix2StructProcessor`. attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Pix2StructText uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [Pix2StructText Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, `optional`: *hidden_states*, `optional`: *attentions*) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)` is a sequence of hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention layers. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, target_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `decoder_inputs_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `decoder_input_ids` and `decoder_inputs_embeds` are both unset, `decoder_inputs_embeds` takes the value of `inputs_embeds`. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss for the decoder. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z)The standalone text decoder of Pix2Structc$ eZdZeZdgZdgZdZfdZdZ dZ dZ dZ d Z eeeee  d$d eej*d eej,d eej,deej,deej*deej,deej.deeeej,deedeedeedeej*deedeej*deeej,dfeffdZ d%d ej.dej.dej.dedef dZed ej.deded ej@d!ejBdej.d"efd#Z"xZ#S)&rrzlm_head.weightTc Vt||tj|j|j |_tjt|jDcgc]}t|t|dk(|c}|_ t|j |j|_tj |j"|_tj&|j |jd|_|j+d|_ycc}w)NrrrFrl)r*r+rrV vocab_sizer2 embed_tokensrr num_layersrrrr(r?final_layer_normrZr[r\rSrrrx)r1rOrr4s r5r+zPix2StructTextModel.__init__s  LL):):F\%\ "' ](&%r6c|jSrRrrs r5rz(Pix2StructTextModel.get_input_embeddings:s   r6c||_yrRrr1new_embeddingss r5set_input_embeddingsz(Pix2StructTextModel.set_input_embeddings=s *r6c|jSrRrrs r5get_output_embeddingsz)Pix2StructTextModel.get_output_embeddings@ ||r6c||_yrRrrs r5set_output_embeddingsz)Pix2StructTextModel.set_output_embeddingsCs % r6r&rrrr inputs_embedsrcross_attn_head_maskrrvrrlabelsrrcrP.c | | n|jj} | | n|jj} | | n|jj} | | n|jj} | | t d|&|j }|jd|d}n!||j dd}n t d|$|jJd|j|}|\}}d}d}| s|t|tr't|tsd}t|t}n[t|ts-d}tjdtj|}n|ttt}d }||d }n||j!}|%t#j$|||z|j& }|9||j!|zn|}t#j(|||j& }|jj*r$|j-|||| |j.nd| }nX|ddddddf}|j1|j2 }d |z t#j4|j2j6z}|M|j \}}}||f}|!t#j(||j& }|j9|}nd}|j;||jj<}|j;||jj<}| rd nd}| rd nd}| rd nd}d} d}!|j?|}"tA|jBD]\}#}$||#}%||#}&| r||"fz}|jDrM|jFrA| rtjIdd} |jK|$jL|"|| |||!|%|&d| | | }'n|$|"|| |||!|%|&|| | | }'| dur|'dddz|'ddz}'|'dd\}"}(|'d} | |'| rdnd}!| s||'dfz}|||'dfz}|jO|"}"|j?|"}"|jQ|"})| r||"fz}d}*| | j1|)j&} tSjTdd}+|+|)jWjd|)j d| jWjd}*| r(nd},|r |j.},|r|jY},| st[d|*|)|,|||fDSt]|*|)|,|||S)a Returns: Example: ```python >>> from transformers import AutoProcessor, Pix2StructTextModel >>> processor = AutoProcessor.from_pretrained("google/pix2struct-textcaps-base") >>> model = Pix2StructTextModel.from_pretrained("google/pix2struct-textcaps-base") >>> inputs = processor(text="Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ``` NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer9zEYou have to specify either decoder_input_ids or decoder_inputs_embedsz) ignore_index reductionc3$K|]}|| ywrRrrs r5rz.Pix2StructTextModel.forward..s = s)losslogitsrrCrcross_attentions)/rOrvrrr)r sizer}rrr r r rHrIfrom_legacy_cacheget_seq_lengthr-r^rr. is_decoder_update_causal_maskrnr;r@rrinvert_attention_maskr,rr\rrrxrrrrErrrCrossEntropyLossr|to_legacy_cacherr)-r1rrrrrrrrrvrrrrrckwargs input_shaperrreturn_legacy_cachereturn_self_attention_cachepast_key_values_lengthmask_seq_lengthr{encoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskrall_attentionsall_cross_attentionsrrrCrrrrrnext_decoder_cacherrloss_fct next_caches- r5rEzPix2StructTextModel.forwardFsH"+!6IDKKst t  "#..*K!r;r?;I  &',,.s3Kde e  $$0 p2p p0 --i8M!, J$&+# 3/51*_Vi:j.2+"5o|~"V1DE&*###` #6"G"G"X ("5lnln"U!"  %%3A%6 "  (%4%C%C%E "  !"\\&(>(KTaThThN  !BQA\..0:=bl #ZZ OML`L`aN ;; ! !228G8S44Y]! K)D$)9:K%..}/B/B.CK, MO O 0=bq0A -M- *!,M$00=CTaZ[0\- !/=3C2E!E(4+?=QRCSBU+U(y< V|--m<  ]3 m, 1]4D D   YYv}}-F**OHF--/44RRI6K\K\K^KcKcdfKghD+4'$ &(==J (88:J %"(    1&+%1   r6 input_tensorc |jjdk(r||dk(jr|Sy|jjdk(r7t|tj r t |}t|tr|S||jnd}t|t}|jjdk(r(|s&|s$tj||||jry|j|j} }|jd} |r|j!} n1t|tj r|jdn|| zdz} |j#|| | || ||jd } |jjdk(rQ|O|jj$d vr7|s5t j&|j(} tj*| | } | S) Nflash_attention_2rflex_attentionrsdpa)rr is_trainingr!r9)sequence_length target_lengthr@rrcr)cudaxpu)rO_attn_implementationrrr-rgr&r%rrr_ignore_causal_mask_sdparr@rrget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positiontyperr_unmask_unattended)r1rrrcrrpast_seen_tokensusing_static_cacher@rrrr{ min_dtypes r5rz'Pix2StructTextModel._update_causal_mask4s ;; + +/B B)~/D.I.I.K%% ;; + +/? ?.%,,7!<^!L.)4%% @O?Z?99;`a'E ;; + +v 5>PYj%>>*'7 MM  $**L,?,?v&,,Q/ +??AMnell;$$R(%7!; PP +')#))!,Q  KK , , 6*%%**o=%  E*..I0CCKQZ[Kr6rrr@rrc ||jdk(r|}|Stj|j} tj||f| ||}|dk7rtj |d}|tj |||jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|jz} | dk(} |ddddddd| fj| | |ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. device (`torch.device`): The device to place the 4D attention mask on. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr) fill_valuer@rr!)diagonalrr9r)rr-rrr triur^reshapeexpandrrr;rr) rrrr@rrcrrr{r mask_length padding_masks r5rzIPix2StructTextModel._prepare_4d_causal_attention_mask_with_cache_positionzsyD  %.*<*<*>!*C(K*' E*..I** -0Ye\bK!##jjqA 5<< fEH^H^_acdHee eK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r6)NNNNNNNNNNNNNN)F)$rGrHrIr#rr3_tied_weights_keysr2r+rrrrrr PIX2STRUCT_TEXT_INPUTS_DOCSTRINGr rr6rr- LongTensor FloatTensorrgrrrrEr rrr4r@rrrJrKs@r5rrs (L./*+&*#,(&<!+&++KL+L[jk156:=A>B48157;EI$(,0/3-1&*59i E,,-i !!2!23i  ((9(9: i !)):): ; i   0 01 i E--.i 'u||4i "%e.?.?(@"ABi D>i $D>i 'tni ))*i d^i !!1!12i " uU&&+,.OO P#i lMi d#( D DllD D  D  DL7 777{{ 7  7  777r6rzhA conditional generation model with a language modeling head. Can be used for sequence generation tasks.c'eZdZeZdZdgZdeffd ZdZdZ de jfdZ d Z d Zd Zeeeee  ddeej.d eej.deej0deej2deej.deej.deej4deeeej.deeeej.deej0deej4deedeedeedeedeej0deeej.eff"dZxZS)"Pix2StructForConditionalGenerationr^zdecoder.lm_head.weightrOct||t|j|_t |j |_|j|_|jyrR) r*r+r vision_configrrrdecoderis_vqarr]s r5r+z+Pix2StructForConditionalGeneration.__init__sK  ,V-A-AB *6+=+=> mm  r6c6|jjSrR)rrrs r5rz7Pix2StructForConditionalGeneration.get_input_embeddingss||0022r6c:|jj|yrR)rrrs r5rz7Pix2StructForConditionalGeneration.set_input_embeddingss )).9r6rPc6|jjSrR)rrrs r5rz8Pix2StructForConditionalGeneration.get_output_embeddingss||1133r6c:|jj|yrR)rrrs r5rz8Pix2StructForConditionalGeneration.set_output_embeddingss **>:r6c|jSrR)rrs r5 get_decoderz.Pix2StructForConditionalGeneration.get_decoderrr6c|jSrR)rrs r5 get_encoderz.Pix2StructForConditionalGeneration.get_encoderrr6r&rrrrdecoder_head_maskrr.rrdecoder_inputs_embedsrvrrrrcc$| | n|jjj} ||n|jj}||j |||| ||}nI|rGt |t s7t |dt|dkDr|dndt|dkDr|dnd}|d}| U|S| Q|j| }||n2|j|jjj}d|dddf<|j||| | ||||| | || ||}|s||zSt|j|j|j |j"|j$|j&|j(|j"|j$ S) a Returns: Example: Inference: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Pix2StructForConditionalGeneration >>> processor = AutoProcessor.from_pretrained("google/pix2struct-textcaps-base") >>> model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-textcaps-base") >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> # autoregressive generation >>> generated_ids = model.generate(**inputs, max_new_tokens=50) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_text) A stop sign is on a street corner. >>> # conditional generation >>> text = "A picture of" >>> inputs = processor(text=text, images=image, return_tensors="pt", add_special_tokens=False) >>> generated_ids = model.generate(**inputs, max_new_tokens=50) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_text) A picture of a stop sign with a red stop sign ``` Training: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Pix2StructForConditionalGeneration >>> processor = AutoProcessor.from_pretrained("google/pix2struct-base") >>> model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-base") >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "A stop sign is on the street corner." >>> inputs = processor(images=image, return_tensors="pt") >>> labels = processor(text=text, return_tensors="pt").input_ids >>> # forward pass >>> outputs = model(**inputs, labels=labels) >>> loss = outputs.loss >>> print(f"{loss.item():.5f}") 5.94282 ```N)r^rrrrrrr!r8r)rrrrrrrrrvrrrrrc) rrrdecoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentions)rOrrvr)rrrrrner r+rrrrrrCrrr)r1r^rrrrr rr.rrr rvrrrrcrCdecoder_outputss r5rEz*Pix2StructForConditionalGeneration.forwards`"+!6IDKK6:8<=A159=7;EIEI-18<$(,0/3&*59#N #E$5$56N !!2!23N $E$4$45 N !))9)9 : N E--. N $E$5$56N 'u||4N "%e.?.?(@"ABN "%e.?.?(@"ABN ))*N  ( 5N D>N $D>N 'tnN d^!N "!!1!12#N $ uU&&');; <%N ]HN r6r)rrrr)TrfrQtypingrrrrrr-torch.utils.checkpointr activationsr cache_utilsr r r r generationrmodeling_attn_mask_utilsrmodeling_outputsrrrrrmodeling_utilsr pytorch_utilsrutilsrrrrrrrrr configuration_pix2structr"r#r$!torch.nn.attention.flex_attentionr%integrations.flex_attentionr& get_loggerrGrHr6rr(apex.normalizationrLinfo ImportError ExceptionrappendrNrirrrr!PIX2STRUCT_VISION_START_DOCSTRINGr5rrr<rrrrPIX2STRUCT_START_DOCSTRINGrrrr__all__rr6r5r.sm 55 !PP)>.1   ed !;J   H %%+"))+2  /& KKij/0!!H^ ^D")):(BII(V2 bii2 j~!~!B %!&"6p%j 5j j \ryy: BII  PbiiPh!ryy!J# #LY"))Yx,`$ DVr/q3q qh nr )BOr r j a8   NN_` s8H**I 2I  I