T^(hDdZddlZddlZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZdd lmZmZdd lmZmZdd lmZdd lmZmZmZd dlmZej:eZdZ dZ!GddejDZ#GddejDZ$GddejJZ&GddejJZ'GddejJZ(GddeZ)dZ*Gdde)Z+ed e*Gd!d"e)Z,ed#e*Gd$d%e)eZ-d%dgZ.y)&z/PyTorch TrOCR decoder model (based on RoBERTa).N)OptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)GenerationMixin)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)add_start_docstringsloggingreplace_return_docstrings) TrOCRConfigrz microsoft/trocr-base-handwrittencVeZdZdZdedeffd Zddejdeffd ZxZ S) TrOCRLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcNd|_t| ||jz|y)N)offsetsuper__init__)selfrr __class__s f/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/trocr/modeling_trocr.pyrz(TrOCRLearnedPositionalEmbedding.__init__.s$  $++5}E input_idspast_key_values_lengthc|jdd\}}tj|||ztj|jj j |d}t|!||jzS)z3`input_ids' shape is expected to be [bsz x seqlen].Nr)dtypedevice) shapetorcharangelongweightr&expandrforwardr)rr"r#bszseq_len positionsrs r r.z'TrOCRLearnedPositionalEmbedding.forward4so!r* WLL "$:W$DEJJ_c_j_j_q_q &b/ wy4;;677r!r) __name__ __module__ __qualname____doc__intrr)Tensorr. __classcell__rs@r rr)s:FsF3F 88s88r!rc `eZdZdZd dedededeeffd Zdejffd Z xZ S) TrOCRScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. rr padding_idx embed_scalec6t||||||_yN)rrr>)rrrr=r>rs r rz!TrOCRScaledWordEmbedding.__init__Es  D&r!r"c<t|||jzSr@)rr.r>)rr"rs r r.z TrOCRScaledWordEmbedding.forwardIswy)D,<,<<=CCEOOPQRUXUbUbcdUeeii338a@EEnVXY 1  !))S%++na"@AqIC  ""#C Q vve--/00r!r"r#cP|j\}}|j||j|j|j}|jdz|z}|j ||j jdkDr,|j ||j|j|_|j j|j|_|j jd|jdj||dj}|S)Nrrr') size"create_position_ids_from_input_idsr=rZr&rJrIrrH index_selectrXdetach)rr"r#r/r0 position_idsmax_posxs r r.z*TrOCRSinusoidalPositionalEmbedding.forwardks ~~' W>>y$JZJZ\rsvv    ""Q&0 << 7T\\->->q-A#A--gt7I7I4K[K[\DL||t'9'9: LL % %a):):2)> ? D DS'SU V ] ] _r!c|j|j}tj|dj ||z|z}|j |zS)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. rrN)ner7r)cumsumtype_asr+)rr"r=r#maskincremental_indicess r r`zETrOCRSinusoidalPositionalEmbedding.create_position_ids_from_input_ids~sW||K(,,.$||Da8@@FI__cgg"'')K77r!r@r2)r3r4r5r6r7rr staticmethodrIr)no_gradr8r.r`r9r:s@r rErEMsNDcD#DHUXMD1c1#1HUXM11$U]]_s&bc 8 847 8QYZ]Q^ 8r!rEceZdZdZ ddededeedeedededed effd Zd e jd ed efdZ dde jdee jdee e jdee jdee jdede e jee jee e jffdZ xZS)TrOCRAttentionz>Multi-headed attention from 'Attention Is All You Need' paper. embed_dim num_headskdimvdimdropout is_decoderbiasis_cross_attentionc Bt |||_||n||_||n||_||_||_||z|_|j|z|jk(std|jd|d|jdz|_ ||_ tj|j|||_ tj|j|||_tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rv)rrrprrrsrqrthead_dim ValueErrorscalingrurLineark_projv_projq_projout_proj) rconfigrprqrrrsrtrurvrwrs r rzTrOCRAttention.__init__s " ,D)  ,D) " !Y.  )T^^;MdnnM]^;b" }}d* $ii 94@ ii 94@ ii 94@  )YTB r!tensorr0r/c|j|||j|jjddj S)Nrr)rXrqrz transpose contiguous)rrr0r/s r _shapezTrOCRAttention._shapes7{{3GQQRSUVWbbddr! hidden_stateskey_value_statespast_key_valueattention_masklayer_head_maskoutput_attentionsreturnc |du}|j\}} } |j||jz} |r| |d} |d} n |rE|j|j |d|} |j|j |d|} n|}|j|j |d|} |j|j |d|} t j|d| gd} t j|d| gd} nD|j|j |d|} |j|j |d|} |jr| | f}||jzd|jf}|j| | |j|} | j|} | j|} | jd}t j| | jdd}|j||jz| |fk7r/td||jz| |fd|j|{|j|d| |fk7r#td |d| |fd|j|j||j| ||z}|j||jz| |}tj j#|d}||j|jfk7r*td |jfd|j|jdddd|j||j| |z}|j||jz| |}|r?|j||j| |}|j||jz| |}nd}tj j%||j$|j& }t j|| }|j||jz| |jfk7r7td ||j| |jfd|j|j||j| |j}|jdd}|j)|| | }|j+|}|||fS) z#Input shape: Batch x Time x ChannelNrrr'rrNz$Attention weights should be of size z , but is z!Attention mask should be of size z/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )r_rr|rr~rr)rUrurqrzrXbmmrr{r functionalsoftmaxrtrreshaper)rrrrrrrrwr/tgt_lenrp query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r r.zTrOCRAttention.forwards.T9"/"4"4"6Wi{{=1DLL@ ."<'*J)!,L T[[1A%BBLJ;;t{{3C'Db#NL  'T[[%?SIJ;;t{{='A2sKLN1$5z#BJJ 99nQ&7%FANLT[[%?SIJ;;t{{='A2sKL ??),7NDNN*B > Ct{{<#>CCZP $Z__j1 (|((*5 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL',,S4>>-A7GTL}},,\r,B  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVZVdVdfmov?wwL',,S4>>-A7GTL  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<AAr!)NNFTF)NNNNF)r3r4r5r6r7rrCboolrr)r8rrr.r9r:s@r rorosaH#" #(CC C sm C sm CCCC!CBeU\\eCece 488<1526"'kB||kB#5<<0kB!u||!45 kB !. kB "%,,/ kB kB u||Xell3XeELL>Q5RR SkBr!roc$eZdZdeffd Z d dej deej deej deej deej deej d eeej d ee d ee fd Z xZ S)TrOCRDecoderLayerrc ^t||j|_t ||j|j |j d|_|j|_t|j|_ |j|_ tj|j|_|j rnt ||j|j |j"|j"|j dd|_tj|j|_tj(|j|j*|_tj(|j*|j|_tj|j|_y)NT)rprqrtru)rprqrrrsrtrurw)rr hidden_sizerprodecoder_attention_headsattention_dropout self_attnrtr activation_function activation_fnactivation_dropoutr LayerNormself_attn_layer_normrucross_attention_hidden_size encoder_attnencoder_attn_layer_normr}decoder_ffn_dimfc1fc2final_layer_normrrrs r rzTrOCRDecoderLayer.__init__!s7 ++' nn44,,  ~~ #F$>$>?"(";";$&LL$@!    ... 88777700#' !D ,.<<+GD (99T^^V-C-CD99V33T^^D " T^^ <r!rrencoder_hidden_statesencoder_attention_maskrcross_attn_layer_head_maskrr use_cachec t|} ||ddnd} |j|| |||\}} } tjj||j|j}| |z}|j |}d}d}|w|} ||ddnd}|j ||||||\}}}tjj||j|j}| |z}|j|}| |z} |} |j|j|}tjj||j|j}|j|}tjj||j|j}| |z}|j|}|f}|r|| |fz }| r|| fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size *(decoder_attention_heads,)*. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. Nr)rrrrrr)rrrrrr) rrrrtrrrrrrrrr)rrrrrrrrrrresidualself_attn_past_key_valueself_attn_weightspresent_key_valuecross_attn_present_key_valuecross_attn_weightscross_attn_past_key_valueoutputss r r.zTrOCRDecoderLayer.forwardCs <!:H9S>"1#5Y] >Bnn'3)+/ ?M? ; (*; --mt||VZVcVc-d  =0 11-@ (,$! ,$H@N?Yrs(;_c %NRN_N_+!65 :8"3 O`O KM-/KMM11-4< >G  )+ +Gr!)NNNNNNFT) r3r4r5rrr)r8rrrr.r9r:s@r rr s ={ =J268<9=26=A8<,1$([||[!.[ ( 5 [ !) 6 [ "%,,/ [%-U\\$:[!u||!45[$D>[D>[r!rc$eZdZeZdZdZdgZdZy)TrOCRPreTrainedModelmodelTrc6|jj}t|tjtj frY|j jjd||j%|jjjyyt|tjrf|j jjd||j2|j j|jjyyy)Nr)meanstd) rinit_std isinstancerr}Conv1dr,datanormal_rvzero_ Embeddingr=)rmodulers r _init_weightsz"TrOCRPreTrainedModel._init_weightsskk"" fryy"))4 5 MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>..r!N) r3r4r5r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modulesrr!r rrs!L&*#,- ?r!raJ This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`TrOCRConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. cTeZdZdZdeffd ZdZdZ ddZxZ S) TrOCRDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`TrOCRDecoderLayer`] Args: config: TrOCRConfig rc\t|||j|_|j|_|j |_|jrtj|jnd}t|j|j|j ||_ |jr&t|j |j|_n@t%|j |j zdz|j|j |_|j&r%t)j*|j|_nd|_t)j,t/|j0Dcgc] }t3|c}|_d|_|j9ycc}w)NrB)r>rF)rrrtdecoder_layerdrop layerdrop pad_token_idr=scale_embeddingrPsqrtrr< vocab_size embed_tokensuse_learned_position_embeddingsrmax_position_embeddingsembed_positionsrElayernorm_embeddingrr ModuleListrangedecoder_layersrlayersgradient_checkpointing post_init)rrr>_rs r rzTrOCRDecoder.__init__sK  ~~ 11!..7=7M7Mdii 2 23SV 4   v1143C3CQ\   1 1#B6CaCacicucu#vD #E..1A1AAAE""  $D  % %')||F4F4F'GD $'+D $mmfNcNcHd$e1%6v%>$ef &+#  %fs3F)c|jSr@rrs r get_input_embeddingsz!TrOCRDecoder.get_input_embeddingss   r!c||_yr@rrvalues r set_input_embeddingsz!TrOCRDecoder.set_input_embeddingss !r!c n| | n|jj} | | n|jj} | | n|jj} | | n|jj} | | t d|"|} |j d| jd}n-| |jdd}|dddddf} n t d||ddjdnd}||j|}|jjr|j| |}n|j||}||z}|j|j|}tjj||j|j }| j}t#||||}||t%||j&|d }|j(r%|j r| rt*j-d d } | rd nd}| rd nd}| r|d nd}| rd nd}t/||gd dgD]j\}}| |jdt1|j2k7s3t d|dt1|j2d|jddt5|j2D]\}}| r||fz }|j r%t7j8g}||j:kr?|||nd}|j(r?|j r3|j=|j>|||||||nd|||ndd| | }n||||||||nd|||nd|| |  }|d}| r ||| rdndfz }| s||dfz }|||dfz }| r||fz }| r|nd}| stAd|||||fDStC|||||S)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in encoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer'zEYou have to specify either decoder_input_ids or decoder_inputs_embedsrr)r#r)rz^`use_cache = True` is incompatible with gradient checkpointing. Setting `use_cache = False`...Fr head_maskcross_attn_head_maskzThe `z` should be specified for z layers, but it is for .)rrrrrrrrrrc3$K|]}|| ywr@r).0vs r z'TrOCRDecoder.forward..s=s)last_hidden_statepast_key_valuesr attentionscross_attentions)"rroutput_hidden_statesruse_return_dictr{rXr(r_rrrrrrrtrr r r%rlogger warning_onceziplenr enumerater)randr_gradient_checkpointing_func__call__tupler )rr"rrrrrr inputs_embedsrrr return_dictinput input_shaper# embed_posrall_hidden_statesall_self_attnsall_cross_attentionsnext_decoder_cache attn_mask mask_nameidx decoder_layerdropout_probabilityr layer_outputs next_caches r r.zTrOCRDecoder.forwards`2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKst t  "E!r5;;r?;I  &',,.s3K!!Q(+Ede eDSC^!3A!6!>#A&3t{{+;<$ {*DSEUDVW%NN,Q/03 #,DKK"8/ @ C#!m%55!}}&+jjn#&75D5P_S1VZN**t}} $ A A!**!")*&/&;IcN1E1Q(-W[% ! !.!#1*?+A7@7LYs^RV5I5U,S1[_#1&7' ! *!,M"}:KQQR'S&UU" =#3"55(4(]1-=,??(_/ @d -!1 1 +4'$ '5FXlm  9+&+%1   r!) NNNNNNNNNNNN) r3r4r5r6rrrrr.r9r:s@r rrsJ{>!" "#!!Y r!rzMThe TrOCR Model with a language modeling head. Can be used for summarization.c(eZdZdZfdZdZxZS)TrOCRDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cDt||t||_yr@)rrrdecoderrs r rzTrOCRDecoderWrapper.__init__s  #F+ r!c&|j|i|Sr@)r )rargskwargss r r.zTrOCRDecoderWrapper.forwardst||T,V,,r!)r3r4r5r6rr.r9r:s@r rrs  ,-r!rzThe TrOCR Decoder with a language modeling head. Can be used as the decoder part of [`EncoderDecoderModel`] and [`VisionEncoderDecoder`].c eZdZdgZfdZdZdZdZdZdZ dZ e e e  dd eej d eej"d eej$d eej deej"deej"deeeej$deej$deej deedeedeedeedeee ffdZedZxZS)TrOCRForCausalLMzoutput_projection.weightc tj|}d|_d|_t||t ||_tj|j|jd|_ |jy)NTFry)copydeepcopyruis_encoder_decoderrrrrrr}rroutput_projectionrrs r rzTrOCRForCausalLM.__init__shv& $)!  (0 !#6+=+=v?P?PW\!] r!cB|jjjSr@rr rrs r rz%TrOCRForCausalLM.get_input_embeddingsszz!!...r!c:||jj_yr@r,rs r rz%TrOCRForCausalLM.set_input_embeddingss*/ 'r!c|jSr@r*rs r get_output_embeddingsz&TrOCRForCausalLM.get_output_embeddingss%%%r!c||_yr@r/)rnew_embeddingss r set_output_embeddingsz&TrOCRForCausalLM.set_output_embeddingss !/r!c&||j_yr@rr )rr s r set_decoderzTrOCRForCausalLM.set_decoders$ r!c.|jjSr@r5rs r get_decoderzTrOCRForCausalLM.get_decoderszz!!!r!) output_typerr"rrrrrrr labelsrrrrrcD| | n|jj} | | n|jj} | | n|jj} |jj ||||||||| | | |  }|j |d}d}| Ft}||jd|jj| jd}| s|f|ddz}||f|zS|St|||j|j|j|jS)aO Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. The two additional tensors are only required when the model is used as a decoder in a Sequence to Sequence model. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Returns: Example: ```python >>> from transformers import ( ... TrOCRConfig, ... TrOCRProcessor, ... TrOCRForCausalLM, ... ViTConfig, ... ViTModel, ... VisionEncoderDecoderModel, ... ) >>> import requests >>> from PIL import Image >>> # TrOCR is a decoder model and should be used within a VisionEncoderDecoderModel >>> # init vision2text model with random weights >>> encoder = ViTModel(ViTConfig()) >>> decoder = TrOCRForCausalLM(TrOCRConfig()) >>> model = VisionEncoderDecoderModel(encoder=encoder, decoder=decoder) >>> # If you want to start from the pretrained model, load the checkpoint with `VisionEncoderDecoderModel` >>> processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten") >>> model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten") >>> # load image from the IAM dataset >>> url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> pixel_values = processor(image, return_tensors="pt").pixel_values >>> text = "industry, ' Mr. Brown commented icily. ' Let us have a" >>> # training >>> model.config.decoder_start_token_id = processor.tokenizer.eos_token_id >>> model.config.pad_token_id = processor.tokenizer.pad_token_id >>> model.config.vocab_size = model.config.decoder.vocab_size >>> labels = processor.tokenizer(text, return_tensors="pt").input_ids >>> outputs = model(pixel_values, labels=labels) >>> loss = outputs.loss >>> round(loss.item(), 2) 5.30 >>> # inference >>> generated_ids = model.generate(pixel_values) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> generated_text 'industry, " Mr. Brown commented icily. " Let us have a' ```N) r"rrrrrrr rrrrrr'r)losslogitsrrrr)rrrrrr r*rrXrrrrrr)rr"rrrrrrr r:rrrrrr=r<loss_fctoutputs r r.zTrOCRForCausalLM.forwardsHJ2C1N-TXT_T_TqTq$8$D $++JjJj &1%.s.nU_j--aZ=N=N1OPns58)r )rrCreordered_past layer_pasts ` r _reorder_cachezTrOCRForCausalLM._reorder_caches=) J ncmnn N r!) NNNNNNNNNNNNN)r3r4r5_tied_weights_keysrrrr0r3r6r8rr_CONFIG_FOR_DOCrr) LongTensorr8rLrrrr.rlrFr9r:s@r r%r%s 55 /0&0%"+L[jk1515=A=A,07;EI59-1$(,0/3&*l E,,-l !.l  ((9(9: l !))9)9 : l ELL) l 'u||4l "%e.?.?(@"ABl   1 12l ))*l D>l $D>l 'tnl d^l  u77 8l ll \r!r%)/r6r'rPtypingrrrr)rtorch.nnr activationsr generationr modeling_attn_mask_utilsr r modeling_outputsr rmodeling_utilsrutilsrrrconfiguration_trocrr get_loggerr3rrH_CHECKPOINT_FOR_DOCrrr<ModulerErorrTROCR_START_DOCSTRINGrrr%__all__rr!r rXs66 )) %!)el-MM,   H %88bll8. =r|| =;8;8|RBRYYRBj~ ~B???$"F 'F RS -. -  -! W+_W  Wt 5 6r!