S^(hv dZddlZddlZddlZddlmZddlmZm Z m Z m Z ddl Z ddl Z ddl mZmZddlmZmZmZddlmZdd lmZmZmZmZmZmZdd lmZdd lm Z m!Z!m"Z"dd l#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)dd l*m+Z+ddl,m-Z-e'j\e/Z0dZ1dZ2gdZ3dZ4dZ5eGddeZ6dUde jde7de8de jfdZ9GddejtZ;GddejtZ<Gd d!ejtZ=Gd"d#ejtZ>Gd$d%e>Z?Gd&d'ejtZ@e>e?d(ZAGd)d*ejtZBGd+d,ejtZCGd-d.ejtZDGd/d0ejtZEGd1d2ejtZFGd3d4ejtZGGd5d6eZHd7ZId8ZJe%d9eIGd:d;eHZKGd<d=ejtZLe%d>eIGd?d@eHZMe%dAeIGdBdCeHZNGdDdEejtZOGdFdGejtZPGdHdIejtZQGdJdKejtZRGdLdMejtZSe%dNeIGdOdPeHZTe%dQeIGdRdSeHe+ZUgdTZVy)VzPyTorch BEiT model.N) dataclass)ListOptionalTupleUnion)Tensornn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BackboneOutputBaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedLMOutputSemanticSegmenterOutput)PreTrainedModel)#compile_compatible_method_lru_cache find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings torch_int) BackboneMixin) BeitConfigr!z%microsoft/beit-base-patch16-224-pt22k)r izmicrosoft/beit-base-patch16-224ztabby, tabby catceZdZdZy)BeitModelOutputWithPoolinga Class for outputs of [`BeitModel`]. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N)__name__ __module__ __qualname____doc__d/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/beit/modeling_beit.pyr$r$Asr*r$input drop_probtrainingreturnc|dk(s|s|Sd|z }|jdfd|jdz zz}|tj||j|j z}|j |j||z}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. r r)r )dtypedevice)shapendimtorchrandr2r3floor_div)r,r-r. keep_probr4 random_tensoroutputs r+ drop_pathr=[sCx I I [[^  Q 7 7E 5 ELL YYM YYy !M 1F Mr*cxeZdZdZd deeddffd ZdejdejfdZ de fdZ xZ S) BeitDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nr-r/c0t|||_yN)super__init__r-)selfr- __class__s r+rCzBeitDropPath.__init__rs "r* hidden_statescDt||j|jSrA)r=r-r.rDrFs r+forwardzBeitDropPath.forwardvs FFr*c8dj|jS)Nzp={})formatr-rDs r+ extra_reprzBeitDropPath.extra_reprys}}T^^,,r*rA) r%r&r'r(rfloatrCr6rrIstrrM __classcell__rEs@r+r?r?osGb#(5/#T#GU\\GellG-C-r*r?c eZdZdZdeddffd Zdejdededejfd Z dd ejd e ejd e e dejfd Z xZS)BeitEmbeddingszc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. configr/Nc2t|tjt j dd|j |_|jr:tjt j dd|j |_ nd|_ t||_ |j|_ t|jtj j"r |jn|j|jf|_|jj$}|j&r=tjt j d|dz|j |_nd|_tj*|j,|_y)Nr )rBrCr Parameterr6zeros hidden_size cls_tokenuse_mask_token mask_tokenBeitPatchEmbeddingspatch_embeddings patch_size isinstance image_size collectionsabcIterable num_patches use_absolute_position_embeddingsposition_embeddingsDropouthidden_dropout_probdropout)rDrTrdrEs r+rCzBeitEmbeddings.__init__s$ ekk!Q8J8J&KL   ll5;;q!V=O=O+PQDO"DO 3F ; ++&++[__-E-EF   ##V%6%67  ++77  2 2')||EKK;QR?TZTfTf4g'hD $'+D $zz&"<"<= r* embeddingsheightwidthc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r Ng?rr bicubicFsizemode align_cornersdim)r4rfr6jit is_tracingr^rreshapepermuter functional interpolateviewcat) rDrjrkrlrd num_positionsclass_pos_embedpatch_pos_embedrv new_height new_widthsqrt_num_positionss r+interpolate_pos_encodingz'BeitEmbeddings.interpolate_pos_encodings`!&&q)A- 0066q9A= yy##%+*F6UZ?++ +221bqb59221ab59r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr* pixel_valuesbool_masked_posrc8|j|tjd|j\}}}}|j |\}\}} |j \} } }|K|j j| | d} |jdj| } |d| z z| | zz}|jj| dd}tj||fd}|j||j|||z}|j|}||| ffS)Nz`interpolate_pos_encoding` argument has no effect for BEiTEmbeddings, embeddings are always interpolated to the input image size. The argument will be removed in transformers v4.51.0.rnr ru)rfwarningswarnr4r]rrr[expand unsqueezetype_asrYr6r~rri)rDrrr_rkrlrj patch_height patch_width batch_sizeseq_len mask_tokensw cls_tokenss r+rIzBeitEmbeddings.forwards-  # # /4L4X MMn  +001fe262G2G 2U/ /\;!+!2 GQ  &//00WbIK))"-55kBA#q1u- a?J^^**:r2> YY J7Q?  # # /#d&C&CJPVX]&^^J\\*- L+666r*NN)r%r&r'r(r!rCr6rintrr BoolTensorboolrIrPrQs@r+rSrSs >z>d>.&D5<<&D&DUX&D]b]i]i&DV7;37 7ll7"%"2"237#+4. 7  7r*rScZeZdZdZfdZdej dej fdZxZS)r\z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}|d|dz|d|dzf}||_||_||_||_ ||_ tj|||||_y)Nr r kernel_sizestride)rBrCr`r^ num_channelsrXr_rarbrcrd patch_shaper Conv2d projection) rDrTr`r^rrXrdrrEs r+rCzBeitPatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY !!} 1 5z!} ST 7UV $$(&&))L+:^hir*rr/c|j\}}}}||jk7r td|j|}|jd|jd}}|j dj dd}|||ffS)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.ror r )r4r ValueErrorrflatten transpose) rDrrrrkrlrjrrs r+rIzBeitPatchEmbeddings.forwards2>2D2D/ L&% 4,, ,w __\2 $.$4$4Q$79I9I!9Lk ''*44Q: L+666r*) r%r&r'r(rCr6rrIrPrQs@r+r\r\s) j" 7ELL 7U\\ 7r*r\ceZdZddedeeddffd ZdZ ddejdeejd e d eejd e d ee e de e eje ejejfffd ZxZS)BeitSelfAttentionNrT window_sizer/c<t|||_|j|jzdk7r2t |ds&t d|jd|jd|j|_t|j|jz |_|j|jz|_ tj|j|j|_ tj|j|jd|_ tj|j|j|_tj|j |_t%||_|j&rt)|||_yy) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .F)biasr)rBrCrTrXnum_attention_headshasattrrrattention_head_size all_head_sizer Linearquerykeyvaluergattention_probs_dropout_probrirhas_relative_position_biasBeitRelativePositionBiasrelative_position_biasrDrTrrEs r+rCzBeitSelfAttention.__init__ sP     : : :a ?PVXhHi"6#5#5"67334A7  $*#=#= #&v'9'9FS"S* **,CD 6G=/2O\M]r*rANFNFN)r%r&r'r!rtuplerCrr6rrrrrrIrPrQs@r+rr sdzddSWd.%-1"'9=).+/3||3ELL)3 3 !) 6 3 #' 3U3Z(3 uU\\"E%,, *D$EE F3r*rceZdZ d dejdeejdedeejdedeeede eejeejejffffd Z xZ S) BeitSdpaSelfAttentionrFrrrrrr/c |s|*tjdt| ||||||S|j |}|j |j |}|j |j|} |j |} d} |jrX|\} } | |jjz| |jjzf}|j|||jd} | | |} n| |z } dtj|jz }t j"j$j'| || | |j(r|jj*ndd|}|j-dd dd j/}|j1dd |j2fz}|j4|}|dfS) Na`BeitSdpaSelfAttention` is used but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rFrrrrrr rr1F) attn_mask dropout_p is_causalscalerror r)logger warning_oncerBrIrrrrrrTr^rr4rrrr6r r{scaled_dot_product_attentionr.rrzrrrrr})rDrFrrrrrrrrr attn_biasrkrlrscalingrrrEs r+rIzBeitSdpaSelfAttention.forward_s  5   w  7?+#"3'=)A% # !JJ}5--dhh}.EF // =0IJ //0AB   * *&MFE!T[[%;%;;UdkkF\F\=\]K335 @S@STU@V4I " - 2 33 dii 8 899++HH   BF--dkk>>UXI &--aAq9DDF "/"4"4"6s";t?Q?Q>S"S* **,CD d""r*r) r%r&r'r6rrrrrrrIrPrQs@r+rr^s-1"'9=).+/:#||:#ELL):# :# !) 6 :# #' :#U3Z(:# uU\\"E%,, *D$EE F:#:#r*rc~eZdZdZdeddffd Zd dejdejdejfdZxZ S) BeitSelfOutputz The residual connection is defined in BeitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rTr/Nct|tj|j|j|_tj |j|_yrA) rBrCr rrXdensergrhrirDrTrEs r+rCzBeitSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r*rF input_tensorcJ|j|}|j|}|SrArri)rDrFrgammas r+rIzBeitSelfOutput.forward$ =1  ]3 r*rA) r%r&r'r(r!rCr6rrIrPrQs@r+rrsD >z>d> U\\^c^j^jr*r)eagersdpaceZdZddedeeddffd ZdZ ddejdeejd e d eejd e d ee e de e eje ejejfffd ZxZS) BeitAttentionNrTrr/ct|t|j|||_t ||_t|_y)Nr) rBrCBEIT_SELF_ATTENTION_CLASSES_attn_implementation attentionrr<set pruned_headsrs r+rCzBeitAttention.__init__s? 4V5P5PQRXfqr$V, Er*c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrr ru)lenrrrrrrrrrr<rrunion)rDheadsindexs r+ prune_headszBeitAttention.prune_headss u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r*rFrrrrrcn|j||||||}|j|d|}|f|ddz} | S)Nrr )rr<) rDrFrrrrr self_outputsattention_outputrs r+rIzBeitAttention.forwardsS~~ 9&79OQiku  ;;|A F#% QR(88r*rAr)r%r&r'r!rrrCrr6rrrrrrIrPrQs@r+rrs"z""SW" ;*-1"'9=).+/||ELL)  !) 6  #' U3Z( uU\\"E%,, *D$EE Fr*rc`eZdZdeddffd Zdej dej fdZxZS)BeitIntermediaterTr/Nct|tj|j|j |_t|jtrt|j|_ y|j|_ yrA) rBrCr rrXintermediate_sizerr_ hidden_actrOrintermediate_act_fnrs r+rCzBeitIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r*rFcJ|j|}|j|}|SrA)rrrHs r+rIzBeitIntermediate.forwards& =1 00? r* r%r&r'r!rCr6rrIrPrQs@r+rrs19z9d9U\\ellr*rc`eZdZdeddffd Zdej dej fdZxZS) BeitOutputrTr/Nct|tj|j|j |_tj|j|_ yrA) rBrCr rrrXrrgrhrirs r+rCzBeitOutput.__init__sB YYv779K9KL zz&"<"<= r*rFcJ|j|}|j|}|SrArrHs r+rIzBeitOutput.forwardrr*rrQs@r+r r s1>z>d> U\\ellr*r ceZdZdZddedeededdffd Z dde jd ee jd e d ee jd e d ee e dee e je e je jfffdZxZS) BeitLayerz?This corresponds to the Block class in the timm implementation.NrTrdrop_path_rater/ct||j|_d|_t |||_t ||_t||_ tj|j|j|_|dkDr t|ntj |_tj|j|j|_|j&}|dkDrytj(|t+j,|jzd|_tj(|t+j,|jzd|_yd\|_|_y) Nr repsr1rT) requires_gradr)rBrCchunk_size_feed_forward seq_len_dimrrr intermediater r<r LayerNormrXlayer_norm_epslayernorm_beforer?Identityr=layernorm_afterlayer_scale_init_valuerVr6oneslambda_1lambda_2)rDrTrr init_valuesrEs r+rCzBeitLayer.__init__s '-'E'E$&v;G,V4 ( " V-?-?VEZEZ [9G#9Mn5SUS^S^S`!||F,>,>FDYDYZ33 ?LLuzz6CUCU7W)WgklDMLLuzz6CUCU7W)WgklDM+5 (DM4=r*rFrrrrrc|j|j||||||}|d}|dd} |j|j|z}|j||z}|j |} |j | } |j | } |j|j| z} |j| |z} | f| z} | S)N)rrrrrr )rrrr=rrr<r) rDrFrrrrrself_attention_outputsrr layer_outputs r+rIzBeitLayer.forwards"&  ! !- 0 /#9%=! "0" 2!4(, == $#}}/?? '78=H ++M: ((6 {{<0 == $==<7L~~l3mC /G+r*)Nr1r)r%r&r'r(r!rrrNrCr6rrrrrrIrPrQs@r+r r sI6z66`e6pt6*-1"'9=).+/)||)ELL)) ) !) 6 ) #' )U3Z() uU\\"E%,, *D$EE F)r*r ceZdZdededdffd Zeddeeefde jfdZ d d e de jfd Z xZS) rrTrr/Nct|||_d|dzdz d|dzdz zdz|_t j t j|j|j|_ y)Nrorr r ) rBrCrnum_relative_distancer rVr6rWrrelative_position_bias_tablers r+rCz!BeitRelativePositionBias.__init__?sr &&'+a.&81&<[QR^ASVWAW%X[\%\",.LL KK22F4N4N O- )r* )maxsizecd|dzdz d|dzdz zdz}|d|dz}tjtj|dtj|dd}tj|}tj|d}|dddddf|dddddfz }|j dddj }|dddddfxx|ddz z cc<|dddddfxx|ddz z cc<|dddddfxxd|dzdz zcc<tj|dzfdz|j}|jd |ddddf<|dz |dddf<|dz |dddf<|dz |d <|S) z This method creates the relative position index, modified to support arbitrary window sizes, as introduced in [MiDaS v3.1](https://arxiv.org/abs/2307.14460). rorr r ij)indexingN)rrr2rn)rr) r6meshgridarangestackrrzrrWr2sum) rDrr% window_areagridcoordscoords_flattenrelative_coordsrelative_position_indexs r+ generate_relative_position_indexz9BeitRelativePositionBias.generate_relative_position_indexHs "#[^!3a!7A A:ELLUV)B12&)>)BA&(=(A%&&r*rcd|jdzdz }d|jdzdz }d|dzdz }d|dzdz }|j}|j} ||zdz} |d| dz } | jd||dj dddd} t j j| t|t|fd} | j ddddj| dz d} tj| || dz dg} |j|}| |jd}|j|d|dzdz|d|dzdzd}|j dddj}|rCt j j|jd||fdd jd}|jdS) zu Modification of timm.models.beit.py: Attention._get_rel_pos_bias to support arbitrary window sizes. rorr r Nrnbilinear)rrrsFrq)rr&r%ryrzr r{r|rr6r~r6r}rrsqueeze)rDrrr old_height old_widthrr old_relative_position_bias_tableold_num_relative_distancenew_num_relative_distance old_sub_table new_sub_table new_relative_position_bias_tabler5rs r+rIz BeitRelativePositionBias.forwardas-))!,,q0 ((++a/ Q'!+  A&* +/+L+L($($>$>!$.$:Q$>!89X;TWX;XY %--aJKSSTUWXZ[]^_  11 :!6 )8L MT^2 &--aAq9AAB[^_B_acd +099 <=VYZ=Z=\] ^, (#'"G"G "T!ABYB^B^_aBb!c"8!%>&003)# &?& gaj #&//22r*)FN)r%r&r'r!rrCrrrr6rr6rrIrPrQs@r+rr>sm z  $ )4'E#s(O'PUP\P\'5'0-3T-3]b]i]i-3r*rceZdZddedeeddffd Z ddejdeejde d e d e d ee e e fd e de ee ffd ZxZS) BeitEncoderNrTrr/c t|||_|j|_|jrt |||_tjd|j|jDcgc]}|j}}tjt|jDcgc]!}t||j r|nd||#c}|_d|_ycc}wcc}w)Nrr)rrF)rBrCrT!use_shared_relative_position_biasrrrr6linspacernum_hidden_layersitemr ModuleListranger use_relative_position_biaslayergradient_checkpointing)rDrTrrdprirEs r+rCzBeitEncoder.__init__s  *0*R*R'  * **B6Wb*cD '"'63H3H&JbJb!cdAqvvxdd]]v778    /5/P/P VZ#&q6    ',#e s 3C,2&C1rFrroutput_hidden_statesrr return_dictc N|rdnd}|rdnd} t|jD]\} } |r||fz}|jrY|\} } | |jjz| |jjzf}|j |||j d}nd}||| nd}|jr.|jr"|j| j||||||}n | ||||||}|d}|s| |dfz} |r||fz}|std||| fDSt||| S)Nr)r )rrrc3&K|] }|| ywrAr)).0vs r+ z&BeitEncoder.forward..smq_`_lms)last_hidden_staterF attentions) enumeraterLrrTr^rr4rMr._gradient_checkpointing_func__call__rr)rDrFrrrPrrrQall_hidden_statesall_self_attentionsrO layer_modulerkrlrrlayer_head_mask layer_outputss r+rIzBeitEncoder.forwards#7BD$5b4(4& POA|#$58H$H!.. * %)?)??$++J`J`A`a )-)D)D:R]j]p]pqr]s*E*&*.&.7.CilO**t}} $ A A ))!#%*,! !-!#%*, ! *!,M &9]1=M ||> ELL)>  > # > #' > U38_-> >  uo% &> r*rCc6eZdZdZeZdZdZdZdgZ dgZ dZ dZ y) BeitPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. beitrTr z.*relative_position_index.*ct|tjtjtjfrm|j j jd|jj|j%|jj jyyt|tjrz|j j jd|jj|j2|j j |jjyyt|tjrJ|jj j|j j jdyt|t r|j"j j|j$$|j$j j|j&%|j&j jyyt|t(r%|j*j jyt|t,r|j.s|j.j j|jj0|j2j j|jj0yyy)zInitialize the weightsr1)meanstdNg?)r_r rrConvTranspose2dweightdatanormal_rTinitializer_rangerzero_ Embedding padding_idxrfill_rSrYr[rfrr&r rrr)rDmodules r+ _init_weightsz!BeitPreTrainedModel._init_weightss fryy"))R5G5GH I MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S )  /    ! ! ' ' )  ,!!&&,,.))5**//5576  8 9  / / 4 4 : : <  **$$**4;;+M+MN$$**4;;+M+MN++r*N) r%r&r'r(r! config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules"_keys_to_ignore_on_load_unexpected_supports_sdparqr)r*r+rbrbs; L$O&*#$ *H)I&NOr*rbaF This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`BeitConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`BeitImageProcessor.__call__`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the pre-trained position encodings. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare Beit Model transformer outputting raw hidden-states without any specific head on top.c eZdZddededdffd ZdZdZee e e e e de  dd ej d eej$d eej d eedeededeedeee ffdZxZS) BeitModelrTadd_pooling_layerr/Nct||||_t||_t ||jj j|_|jrtjn*tj|j|j|_|r t!|nd|_|j%y)Nrr)rBrCrTrSrjrCr]rencoderuse_mean_poolingr rrrXr layernorm BeitPoolerpooler post_init)rDrTr{rEs r+rCzBeitModel.__init__<s   (0"6t7W7W7c7cd $44BKKM",,vGYGY_e_t_t:u ->j(4  r*c.|jjSrArjr]rLs r+get_input_embeddingszBeitModel.get_input_embeddingsK///r*c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr}rLrr)rDheads_to_prunerLrs r+ _prune_headszBeitModel._prune_headsNsE +002 CLE5 LL  u % / / ; ;E B Cr*vision) checkpoint output_typerrmodalityexpected_outputrrrrrPrrQc :||n|jj}||n|jj}||n|jj}|j ||jj }|j ||\}} |jdd} |j||||| ||} | d} |j| } |j|j| nd} |s| | | fn| f}|| ddzSt| | | j| jS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). N)rro)rrrPrrQrrr )rW pooler_outputrFrX)rTrrPuse_return_dict get_head_maskrGrjr4r}rrr$rFrX)rDrrrrrPrrQembedding_outputrrencoder_outputssequence_output pooled_output head_outputss r+rIzBeitModel.forwardVsE,2C1N-TXT_T_TqTq$8$D $++JjJj &1% 4 'tn 4 #'4 d^4  u00 14 B4 r*rzc`eZdZdeddffd Zdej dej fdZxZS)rrTr/Nct||jr1tj|j |j |_yd|_y)Nr)rBrCr~r rrXrrrs r+rCzBeitPooler.__init__sA KQKbKbBLL++1F1F G hl r*rFc|j0|ddddddf}|j|jd}|S|dddf}|S)Nr r)rre)rDrF patch_tokensrs r+rIzBeitPooler.forwardsU >> %(AB2L NN<+<+ f&8&8f>S>STyy!3!3V5F5FG  r*rrrrrrlabelsrrPrrQc j||n|jj}|j|||||||} | d} |j| } |j | ddddf} d} |t } | | ||} |s| f| ddz}| | f|zS|St | | | j| jS)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, BeitForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> model = BeitForMaskedImageModeling.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, logits = outputs.loss, outputs.logits >>> list(logits.shape) [1, 196, 8192] ```N)rrrrPrrQrr losslogitsrFrX) rTrrcrrr rrFrX)rDrrrrrrPrrQrrprediction_scoresmasked_lm_lossloss_fctr<s r+rIz"BeitForMaskedImageModeling.forwards`&1%X046:,0)-,0/3).&*L u||,L "%"2"23L ELL) L & L $D> L 'tnL #'L d^L  un$ %L YBL r*rz Beit Model transformer with an image classification head on top (a linear layer on top of the average of the final hidden states of the patch tokens) e.g. for ImageNet. ceZdZdeddffd Zeeeee e e  dde e jde e jde e jd e ed e ed ed e edeee ffd ZxZS)BeitForImageClassificationrTr/Nc.t|||j|_t|d|_|jdkDr*t j |j|jnt j|_ |jy)NTrr) rBrCrrzrcr rrXr classifierrrs r+rCz#BeitForImageClassification.__init__ss   ++f= OUN_N_bcNc"))F$6$68I8IJikititiv r*)rrrrrrrrrrPrrQc4||n|jj}|j||||||}|r |jn|d} |j | } d} ||jj |j dk(rd|j_nl|j dkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt} |j dk(r&| | j|j} n| | |} n|jj dk(r=t} | | jd|j |jd} n,|jj dk(rt} | | |} |s| f|ddz} | | f| zS| St!| | |j"|j$ S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrrPrrQr regressionsingle_label_classificationmulti_label_classificationrnror)rTrrcrr problem_typerr2r6longrr r9r r}r rrFrX)rDrrrrrPrrQrrrrrr<s r+rIz"BeitForImageClassification.forward"s.&1% = 'tn = #'= d^=  u++ ,= B= r*rceZdZdZ d dededeeeeeffdeeeeefefdedeeeeeffdd ffd Z d e jde jfd Z xZ S)BeitConvModuleaD A convolutional block that bundles conv/norm/activation layers. This block simplifies the usage of convolution layers, which are commonly used with a norm layer (e.g., BatchNorm) and activation layer (e.g., ReLU). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. in_channels out_channelsrpaddingrdilationr/Nct|tj|||||||_tj ||_tj|_y)N)rrrrrr) rBrCr rconv BatchNorm2dbnReLU activation)rDrrrrrrrEs r+rCzBeitConvModule.__init__qsQ II#%#   ...'')r*r,cl|j|}|j|}|j|}|SrA)rrr)rDr,r<s r+rIzBeitConvModule.forwards05!( r*)rFr )r%r&r'r(rrrrOrrCr6rrIrPrQs@r+rris5601$$$3c3h/0 $ sE#s(OS01 $  $U38_,-$ $*U\\ellr*rcheZdZdedededdffd Zdej dej fdZxZS) BeitPyramidPoolingBlock pool_scalerchannelsr/Nct|tj|t ||dg|_t |j D] \}}|jt||"y)Nr r) rBrCr AdaptiveAvgPool2drlayersrY add_modulerO)rDrrrrOrLrEs r+rCz BeitPyramidPoolingBlock.__init__sa    , ;a @  "$++. +HAu OOCFE * +r*r,c<|}|jD] }||} |SrA)r)rDr, hidden_staterLs r+rIzBeitPyramidPoolingBlock.forwards* [[ /E .L /r*) r%r&r'rrCr6rrIrPrQs@r+rrs?+3+S+C+D+U\\ellr*rc eZdZdZdeedfdedededdf fd Zd ejde ejfd Z xZ S) BeitPyramidPoolingModulea Pyramid Pooling Module (PPM) used in PSPNet. Args: pool_scales (tuple[int]): Pooling scales used in Pooling Pyramid Module. in_channels (int): Input channels. channels (int): Channels after modules, before conv_seg. align_corners (bool): align_corners argument of F.interpolate. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. pool_scales.rrrtr/Nct|||_||_||_||_g|_t|D]I\}}t|||}|j j||jt||Ky)N)rrr) rBrCrrtrrblocksrYrappendrrO) rDrrrrtrOrblockrEs r+rCz!BeitPyramidPoolingModule.__init__s &*&   &{3 +MAz+z{emnE KK  u % OOCFE * +r*rcg}|jD]Y}||}tjj||j ddd|j }|j |[|S)Nror8rq)rr r{r|rrrtr)rDrppm_outsppmppm_outupsampled_ppm_outs r+rIz BeitPyramidPoolingModule.forwardsn;; /C!fG " 9 9affhqrl4K]K]!:!  OO- .  / r*) r%r&r'r(rrrrCr6rrrIrPrQs@r+rrs[  +E#s(O +# +QT +ei +nr +$u||*<r*rcjeZdZdZdeddffd ZdZdejdejfdZ xZ S) BeitUperHeadz Unified Perceptual Parsing for Scene Understanding. This head is the implementation of [UPerNet](https://arxiv.org/abs/1807.10221). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rTr/Nct||j|_|jgdz|_|j|_d|_tj|j |jd|_ t|j|jd|j |j |_ t|jdt|j|j zz|j dd|_tj |_tj |_|jddD]s}t||j d}t|j |j dd}|j"j'||j$j'|utt|j|j z|j dd|_y) NFr rrn)rtr rr)rBrCrrXrrrtr rrrr psp_modulesrr bottleneckrI lateral_convs fpn_convsrfpn_bottleneck)rDrTrl_convfpn_convrEs r+rCzBeitUperHead.__init__s !--"../!3** "))DMM63D3DRST4      R MM,,   )   R 3t'7'7#84==#H H MM   ]]_++CR0 ,K#KANF%dmmT]]PQ[\]H    % %f - NN ! !( +  , -   !DMM 1 MM  r*c|d}|g}|j|j|tj|d}|j |}|S)Nrnr ru)extendrr6r~r)rDinputsrpsp_outsr<s r+ psp_forwardzBeitUperHead.psp_forwardsL 2J3((+,99X1-* r*encoder_hidden_statescPt|jDcgc]\}}|||}}}|j|j|t |}t |dz ddD]V}||dz j dd}||dz tjj|||d|jz||dz <Xt |dz Dcgc]}|j|||}}|j|dt |dz ddD]E}tjj|||dj ddd|j||<Gtj|d}|j|}|j|}|Scc}}wcc}w)Nr rrnror8rqru)rYrrrrrJr4r r{r|rtrr6r~rr) rDrrO lateral_convlateralsused_backbone_levels prev_shapefpn_outsr<s r+rIzBeitUperHead.forwardsR[\`\n\nRopq,L!6q!9:pp(()>?@ #8}+a/B7 A!!a%..qr2J&q1uo 0I0I *:TM_M_1J1HQUO =BBVYZBZ<[\q%DNN1%hqk2\\ %+a/B7 A--33 (1+"3"3AB"7jX\XjXj4HQK 99X1-$$X.( 3q]s FF#) r%r&r'r(r!rCrr6rrIrPrQs@r+rrs<$ z$ d$ LU\\ellr*rc eZdZdZ d dedededeeeeeffddf fd Zd e jde jfd Z xZ S) BeitFCNHeada Fully Convolution Networks for Semantic Segmentation. This head is implemented of [FCNNet](https://arxiv.org/abs/1411.4038>). Args: config (BeitConfig): Configuration. in_channels kernel_size (int): The kernel size for convs in the head. Default: 3. dilation (int): The dilation rate for convs in the head. Default: 1. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rTin_indexrrr/Nc <t||j|_|j|_|j |_|j|_ ||_ |dz|z}g}|jt|j|j |||t|jdz D]5}|jt|j |j |||7|jdk(rtj|_ntj"||_|jr8t|j|j z|j ||dz|_tj&|j |j(d|_y)Nro)rrrr rrr)rBrCrXrauxiliary_channelsrauxiliary_num_convs num_convsauxiliary_concat_input concat_inputrrrrJr rconvs Sequentialconv_catrrr) rDrTrrr conv_paddingrrOrEs r+rCzBeitFCNHead.__init__(sX !--11 33"99  #q(H4    $--[R^iq  t~~)* A LLMM4==kS_jr   >>Q DJ.DJ   *  4==0$--[bmqrbrDM))DMM63D3DRSTr*rc||j}|j|}|jr(|jt j ||gd}|j |}|S)Nr ru)rrrrr6r~r)rDrrFr<s r+rIzBeitFCNHead.forwardJsX-dmm< M*   ]]599mV-D!#LMF( r*)ror r ) r%r&r'r(r!rrrrCr6rrIrPrQs@r+rrsv tu U  U,/ UBE UUZ[^`efiknfn`o[oUp U  UDU\\ellr*rzf Beit Model transformer with a semantic segmentation head on top e.g. for ADE20k, CityScapes. ceZdZdeddffd ZdZeeee e  dde e jde e jd e e jd e ed e ed ed e edeee ffdZxZS)BeitForSemanticSegmentationrTr/Nc xt|||j|_t|d|_t |j jdk7r tdtjtj|j|jddtj|jtjtj|j|jdd|_tjtj|j|jdd|_tj"|_tj&dd|_t+||_|j.r t1|nd|_|j5y)NFrrzBeitForSemanticSegmentation requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.ror)rBrCrrzrcrrT out_indicesrr rrgrXrGELUfpn1fpn2rfpn3 MaxPool2dfpn4r decode_headuse_auxiliary_headrauxiliary_headrrs r+rCz$BeitForSemanticSegmentation.__init__[sO   ++f>  t{{&& '1 ,-  MM   v1163E3EST]^ _ NN6-- . GGI   v1163E3EST]^ _   MM   v1163E3EST]^ _  KKM LLQq9 (/5;5N5Nk&1TX r*cntjj||jdddd}|0tjj||jdddd}t |j j }|||}|}|%||} ||j j| zz }|S)Nrr8Frq) ignore_index)r r{r|r4r rTsemantic_loss_ignore_indexauxiliary_loss_weight) rDrauxiliary_logitsrupsampled_logitsupsampled_auxiliary_logitsr main_lossrauxiliary_losss r+ compute_lossz(BeitForSemanticSegmentation.compute_loss{s==44 bc*55   ')+)B)B v||BC'8zY^*C* &$1W1WX-v6   '%&@&IN DKK55F FD r*rrrrrrPrrQc T||n|jj}||n|jj}|$|jjdk(r t d|j |||d||}|r |j n|d} t| D cgc]#\} } | dz|jjvs"| %} } } |jd} |jj|jjz}| Dcgc]3}|ddddddfjdddj| d||5} }|j|j|j |j"g}t%t'| D]}||| || |<|j)| }d}|j*|j+| }d}||j-|||}|s|r |f|ddz}n |f|ddz}||f|zS|St/|||r |j nd|j0 Scc} } wcc}w) aV labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth semantic segmentation maps for computing the loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1`, a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, BeitForSemanticSegmentation >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> model = BeitForSemanticSegmentation.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # logits are of shape (batch_size, num_labels, height, width) >>> logits = outputs.logits ```Nr z/The number of labels should be greater than oneTrrrornr)rTrrPrrrcrFrYr r4r`r^rzryrrrrrJrrrrrrX)rDrrrrrPrrQrridxfeaturefeaturesrpatch_resolutionropsrOrrrr<s r+rIz#BeitForSemanticSegmentation.forwardsbJ&1%8H%r)r%r&r'r!rCrrrrrrrr6rrrrrIrPrQs@r+r r Tszd@&++@A+BQ`a04,0)-,0/3).&*Z u||,Z ELL)Z & Z $D> Z 'tn Z #'Z d^Z  u-- .Z bBZ r*r zM BEiT backbone, to be used with frameworks like DETR and MaskFormer. c eZdZfdZdZeeeee  d de de e de e de e def d Z xZS) BeitBackbonec t||t| |t|jdzDcgc]}|j c}|_t||_t||jjj|_ |jrt|jj dk7r t#d|j }t%j&t%j(||ddt%j*||j,t%j.t%j(||dd|_t%j&t%j(||dd|_t%j4|_t%j8dd|_|j=ycc}w)Nr rrzBeitBackbone requires config.out_indices to be a list of 4 integers, specifying which features to use from the backbone. One can use [3, 5, 7, 11] in case of a base-sized architecture.rorr)rBrC_init_backbonerJrGrX num_featuresrSrjrCr]rr}add_fpnrrTr rr rrgrbatch_norm_epsr rrrrrrr)rDrTrrXrEs r+rCzBeitBackbone.__init__s[   v&9>v?W?WZ[?[9\]AV//](0"6t7W7W7c7cd >>4;;**+q0 1 !,,K ""; STU{0E0EF ""; STU DI b&8&8k_`ij&klDI DI 1=DI 1^sGc.|jjSrArrLs r+rz!BeitBackbone.get_input_embeddingsrr*rrrPrrQr/c||n|jj}||n|jj}||n|jj}|jd}|j |\}\}}|jdd} |j |d|| |} |r | jn| d} d} t|j| D]e\} }| |jvs|jjr5|ddddddf}|jddd}|j|d||}| |fz } g|jjrY|j| d|j!| d|j#| d|j%| d g} t'| } |s|r | f| ddz}|S| f| ddz}|St)| |r | jnd| j* S) aL Returns: Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224") >>> model = AutoBackbone.from_pretrained( ... "microsoft/beit-base-patch16-224", out_features=["stage1", "stage2", "stage3", "stage4"] ... ) >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```NrroT)rPrrrQr r)rnr ) feature_mapsrFrX)rTrrPrr4rjr}rFzip stage_names out_featuresreshape_hidden_statesrzryr+rrrrrrrX)rDrrPrrQrrrrrrrFr/stagerr<s r+rIzBeitBackbone.forwardsF&1%X04,0&* S S 'tnS $D> S d^ S  S YBS r*r')rrr rzrbr')r1F)Wr(collections.abcrarr dataclassesrtypingrrrrr6torch.utils.checkpointrr torch.nnr r r activationsrmodeling_outputsrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrrrrrutils.backbone_utilsrconfiguration_beitr! get_loggerr%rrrrrrr$rNrr=Moduler?rSr\rrrrrrr r rrCrbBEIT_START_DOCSTRINGrrzrrrrrrrrr r'__all__r)r*r+rEs' !// AA!.vv2*   H %>&<1 !; 2U\\eTV[VbVb( -299 - c7RYYc7L#7"))#7LP Pf;#-;#|RYY& ! )BII)X ryy   > >BP3ryyP3fT "))T n(O/(OV 2dW #W W t&s \ !4\ \ ~ Q !4Q Q h"RYY"Jbii""ryy"JR299Rj8"))8v  P "5P  P f  w & w  w t r*