S^(htdZddlZddlmZmZmZmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZddlmZmZmZmZdd lmZmZdd lmZmZdd lm Z m!Z!m"Z"m#Z#m$Z$m%Z%dd l&m'Z'd dl(m)Z)e#jTe+Z,dZ-dZ.gdZ/dZ0dZ1Gdde jdZ3Gdde jdZ4 dFde jdde jjde jjde jjdee jjde6de6fdZ7Gd d!e jdZ8Gd"d#e jdZ9Gd$d%e jdZ:Gd&d'e jdZ;dGd(e jjd)e6d*eGd/d0e jdZ?Gd1d2e jdZ@Gd3d4e jdZAGd5d6e jdZBGd7d8eZCd9ZDd:ZEd;ZFe!deCZGe!d?eDGd@dAeCZHe!dBeDGdCdDeCe'ZIgdEZJy)HzPyTorch DINOv2 model.N)CallableDictListOptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BackboneOutputBaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings torch_int) BackboneMixin) Dinov2Configr zfacebook/dinov2-base)riiz(facebook/dinov2-small-imagenet1k-1-layerztabby, tabby catceZdZdZdeddffd Zdejdededejfd Z d d ejd e ejdejfd Z xZ S)Dinov2EmbeddingszM Construct the CLS token, mask token, position and patch embeddings. configreturnNczt|tjt j dd|j |_|jr8tjt jd|j |_ t||_ |jj}tjt j d|dz|j |_tj|j |_|j$|_|j|_||_y)Nr)super__init__r Parametertorchrandn hidden_size cls_tokenuse_mask_tokenzeros mask_tokenDinov2PatchEmbeddingspatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropout patch_sizer#)selfr#r2 __class__s h/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/dinov2/modeling_dinov2.pyr'zDinov2Embeddings.__init__As ekk!Q8J8J&KL   ll5;;q&:L:L+MNDO 5f =++77 #%<< A{QPVPbPb0c#d zz&"<"<=  ++$33 embeddingsheightwidthc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}|j} tjj|jtj| | fdd j| }|jddddjdd|}tj ||fd S) a- This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing and interpolation at torch.float32 precision. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rrbicubicF)sizemode align_cornersdtypedim)shaper3r)jit is_tracingr7rreshapepermuterGr functional interpolatetofloat32viewcat) r8r<r=r>r2 num_positionsclass_pos_embedpatch_pos_embedrI new_height new_widthsqrt_num_positions target_dtypes r:interpolate_pos_encodingz)Dinov2Embeddings.interpolate_pos_encodingOs!&&q)A- 0066q9A= yy##%+*F6UZ?++ +221bqb59221ab59r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=&,, --33   u}} -i( 4 "<" *11!Q1=BB1b#Nyy/?;CCr; pixel_valuesbool_masked_poscD|j\}}}}|jjjj}|j|j |}|d|j rXtj|jd|jj |jjd|}|jj|dd} tj| |fd}||j|||z}|j|}|S)NrFr@rrrH)rJr1 projectionweightrGrQr-r)where unsqueezer/r,expandrTr\r6) r8r]r^ batch_size_r=r>r[r< cls_tokenss r:forwardzDinov2Embeddings.forwardws'3'9'9$ Avu,,77>>DD **+>))"-t/A/A*BRBR/S/]/]^_/`blJ ^^**:r2> YY J7Q?  $"?"? FTY"ZZ \\*- r;N) __name__ __module__ __qualname____doc__r r'r)Tensorintr\rrh __classcell__r9s@r:r"r"<s| |  &D5<<&D&DUX&D]b]i]i&DPELL8ELLCYejeqeqr;r"cZeZdZdZfdZdej dej fdZxZS)r0z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct||j|j}}|j|j }}t |tjjr|n||f}t |tjjr|n||f}|d|dz|d|dzz}||_||_||_||_ tj|||||_ y)Nrr) kernel_sizestride)r&r' image_sizer7 num_channelsr+ isinstance collectionsabcIterabler2r Conv2dr`)r8r#rvr7rwr+r2r9s r:r'zDinov2PatchEmbeddings.__init__s !'!2!2F4E4EJ $*$7$79K9Kk #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&))L+:^hir;r]r$c|jd}||jk7rtd|jd|d|j|j dj dd}|S)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rA)rJrw ValueErrorr`flatten transpose)r8r]rwr<s r:rhzDinov2PatchEmbeddings.forwardsz#))!, 4,, ,!../yaI __\2::1=GG1M r;) rjrkrlrmr'r)rnrhrprqs@r:r0r0s) jELLU\\r;r0modulequerykeyvalueattention_maskscalingr6c tj||jdd|z}tjj |dtj j|j}tjj|||j}|||z}tj||} | jddj} | |fS)Nr@)rIrG)ptrainingrrA) r)matmulrr rOsoftmaxrRrQrGr6r contiguous) rrrrrrr6kwargs attn_weights attn_outputs r:eager_attention_forwardrs<<s}}R'<=GL==((2U]](SVVW\WbWbcL==((6??([L!#n4 ,,|U3K''1-88:K  $$r;c eZdZdeddffd Zdej dej fdZ d deej de de e ej ej fe ej ffd Z xZ S) Dinov2SelfAttentionr#r$Nc2t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |j|_ |jdz|_ d|_ tj|j|j|j |_tj|j|j|j |_tj|j|j|j |_y) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads r~gFbias)r&r'r+num_attention_headshasattrrr#roattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalr Linearqkv_biasrrrr8r#r9s r:r'zDinov2SelfAttention.__init__sF     : : :a ?PVXhHi"6#5#5"67334A7   #)#=#= #&v'9'9FO##L '>dkk>^>^&_#)<     nnLL#}}C$2C2C * & #0"4"4"6s";t?Q?Q>S"S%--.EF 6G=/2O\M]r;NF)rjrkrlr r'r)rnrrboolr rrhrprqs@r:rrs]|]](%ell%u||% bg!(0(>!Z^! uU\\5<</0% 2EE F!r;rc|eZdZdZdeddffd ZdejdejdejfdZxZ S) Dinov2SelfOutputz The residual connection is defined in Dinov2Layer instead of here (as is the case with other models), due to the layernorm applied before each block. r#r$Nct|tj|j|j|_tj |j|_yri) r&r'r rr+denser4r5r6rs r:r'zDinov2SelfOutput.__init__sB YYv1163E3EF zz&"<"<= r;r input_tensorcJ|j|}|j|}|Sri)rr6)r8rrs r:rhzDinov2SelfOutput.forwards$ =1  ]3 r;) rjrkrlrmr r'r)rnrhrprqs@r:rr sD >|>> U\\RWR^R^r;rc eZdZdeddffd ZdeeddfdZ d dejde ejd e de e ejejfe ejffd ZxZS) Dinov2Attentionr#r$Nct|t||_t ||_t |_yri)r&r'r attentionroutputset pruned_headsrs r:r'zDinov2Attention.__init__!s0 ,V4&v. Er;headsc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)NrrrH)lenrrrrrrrrrrrrunion)r8rindexs r: prune_headszDinov2Attention.prune_heads's u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r;rrrch|j|||}|j|d|}|f|ddz}|S)Nrr)rr)r8rrr self_outputsattention_outputrs r:rhzDinov2Attention.forward9sE ~~mY@QR ;;|A F#% QR(88r;r)rjrkrlr r'rrorr)rnrrr rrhrprqs@r:rr s"|"" ;S;d;*-1"' || ELL)   uU\\5<</0% 2EE F r;rcXeZdZdfd Zdej dej fdZxZS)Dinov2LayerScaler$ct|tj|jt j |jz|_yri) r&r'r r(layerscale_valuer)onesr+lambda1rs r:r'zDinov2LayerScale.__init__Hs8 ||F$;$;ejjI[I[>\$\] r; hidden_statec ||jzSri)rr8rs r:rhzDinov2LayerScale.forwardLsdll**r;r$Nrjrkrlr'r)rnrhrprqs@r:rrGs$^+ELL+U\\+r;rinput drop_probrr$c|dk(s|s|Sd|z }|jdfd|jdz zz}|tj||j|j z}|j |j||z}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rrr)r)rGdevice)rJndimr)randrGrfloor_div)rrr keep_probrJ random_tensorrs r: drop_pathrQsCx I I [[^  Q 7 7E 5 ELL YYM YYy !M 1F Mr;cxeZdZdZd deeddffd ZdejdejfdZ de fdZ xZ S) Dinov2DropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nrr$c0t|||_yri)r&r'r)r8rr9s r:r'zDinov2DropPath.__init__is "r;rcDt||j|jSri)rrr)r8rs r:rhzDinov2DropPath.forwardms FFr;c8dj|jS)Nzp={})formatrr8s r: extra_reprzDinov2DropPath.extra_reprps}}T^^,,r;ri) rjrkrlrmrfloatr'r)rnrhstrrrprqs@r:rrfsGb#(5/#T#GU\\GellG-C-r;rcXeZdZdfd Zdej dej fdZxZS) Dinov2MLPr$c~t||jx}}t|j|jz}t j ||d|_t|jtrt|j|_ n|j|_ t j ||d|_ y)NTr)r&r'r+ro mlp_ratior rfc1rx hidden_actrr activationfc2r8r# in_features out_featureshidden_featuresr9s r:r'zDinov2MLP.__init__us %+%7%77 lf0063C3CCD99[/E f'' -$V%6%67DO$//DO99_lFr;rcl|j|}|j|}|j|}|Sri)rrrrs r:rhzDinov2MLP.forwards2xx - |4 xx - r;rrrqs@r:rrts$ GELLU\\r;rcXeZdZdfd Zdej dej fdZxZS)Dinov2SwiGLUFFNr$c0t||jx}}t|j|jz}t|dzdz dzdzdz}t j |d|zd|_t j ||d|_y)NrArTr) r&r'r+rorr r weights_in weights_outrs r:r'zDinov2SwiGLUFFN.__init__s %+%7%77 lf0063C3CCD2Q67!;AAE))K_1D4P99_lNr;rc|j|}|jdd\}}tjj ||z}|j |S)NrAr@rH)rchunkr rOsilur)r8rx1x2hiddens r:rhzDinov2SwiGLUFFN.forwardsS|4 ##A2#.B##B'",''r;rrrqs@r:rrs$O(ELL(U\\(r;rc eZdZdZdeddffd Z d dejdeejde de e ejejfe ejffd Z xZ S) Dinov2LayerzCThis corresponds to the Block class in the original implementation.r#r$Nct|tj|j|j |_t||_t||_ |jdkDrt|jntj|_tj|j|j |_|j rt#||_nt'||_t||_y)Nepsr)r&r'r LayerNormr+layer_norm_epsnorm1rrr layer_scale1drop_path_raterIdentityrnorm2use_swiglu_ffnrmlpr layer_scale2rs r:r'zDinov2Layer.__init__s \\&"4"4&:O:OP (0,V4BHBWBWZ]B](=(=>cecncncp\\&"4"4&:O:OP  &v.DH (DH,V4r;rrrcD|j|j|||}|d}|j|}|dd}|j||z}|j |}|j |}|j |}|j||z}|f|z}|S)N)rrr)rrrrrrr)r8rrrself_attention_outputsrr layer_outputs r:rhzDinov2Layer.forwards "& JJ} % /"0"  2!4,,-=>(,'78=H zz-0 xx - ((6 ~~l3mC /G+r;r)rjrkrlrmr r'r)rnrrr rrhrprqs@r:r r s~M5|55&-1"' ||ELL)  uU\\5<</0% 2EE F r;r ceZdZdeddffd Z d dej deej deded ede e e ff d Z xZ S) Dinov2Encoderr#r$Nct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) r&r'r#r ModuleListrangenum_hidden_layersr layergradient_checkpointingr8r#rfr9s r:r'zDinov2Encoder.__init__sN  ]]vG_G_A`#aAK$7#ab &+#$bsA#rrroutput_hidden_states return_dictct|rdnd}|rdnd}t|jD]h\}} |r||fz}|||nd} |jr+|jr|j | j || |} n | || |} | d}|s`|| dfz}j|r||fz}|st d|||fDSt|||S)Nrrc3&K|] }|| ywrir)).0vs r: z(Dinov2Encoder.forward..smq_`_lms)last_hidden_stater attentions) enumerater#r$r_gradient_checkpointing_func__call__tupler) r8rrrr&r'all_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r:rhzDinov2Encoder.forwards#7BD$5b4(4 POA|#$58H$H!.7.CilO**t}} $ A A ))!#% ! !-]OM^ _ )!,M &9]1=M