T^(hpddlZddlZddlmZddlmZmZmZmZm Z ddl Z ddl Z ddl mZddlmcmZddl mZmZmZddlmZddlmZddlmZdd lmZmZmZdd lm Z m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(d d l)m*Z*m+Z+m,Z,e'jZe.Z/dZ0eGdde#Z1eGdde#Z2eGdde#Z3GddejhZ5 dLdejhde jlde jlde jldee jlde7de7fdZ8Gdd ejhZ9Gd!d"ejhZ:Gd#d$ejhZ;Gd%d&ejhZGd*d+ejhZ?d,Z@ dMd-e jld.e7d/e7d0e7d1e7d2e jlf d3ZAdNd4ZBd5ZCd6ZDd7ZEGd8d9ejhZFd:ZGd;ZHGd<d=e!ZIe$d>eGGd?d@eIZJGdAdBejhZKe$dCeGGdDdEeIZLe$eGGdFdGeIZMe$dHeGGdIdJeIZNgdKZOy)ON) dataclass)AnyCallableOptionalTupleUnion)BCEWithLogitsLossCrossEntropyLossMSELoss)_calculate_fan_in_and_fan_out)ACT2FN)_prepare_4d_attention_mask)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardcan_return_tupleloggingreplace_return_docstrings) Siglip2ConfigSiglip2TextConfigSiglip2VisionConfigrceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<y)Siglip2VisionOutputa Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r!rtorch FloatTensor__annotations__r"r#rr$j/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/siglip2/modeling_siglip2.pyr r 6sr*15L(5,,-459x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r-r ceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<y)Siglip2TextOutputa Base class for text model's outputs that also contains a pooling of the last hidden states. Args: text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N text_embedsr".r#r$) r%r&r'r(r1rr)r*r+r"r#rr$r,r-r.r0r0Ssr*04K%++,359x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r-r0ceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd Zy) Siglip2Outputa Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`Siglip2TextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`Siglip2VisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`Siglip2TextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`Siglip2VisionModel`]. Nlosslogits_per_imagelogits_per_textr1r!text_model_outputvision_model_outputreturncHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw))r7r8N)getattrto_tuple).0kselfs r. z)Siglip2Output.to_tuple..s=  LLDGRYZ^`aRbRkRkRm m s-0)tuplekeysr@s`r.r=zSiglip2Output.to_tuples# YY[   r-)r%r&r'r(r4rr)r*r+r5r6r1r!r7rr8rrr=r,r-r.r3r3ps()-D(5$$ %,48hu001837OXe//07/3K%++,304L(5,,-448186:3: %* r-r3c eZdZdeffd Zedejdejde dejfdZ dejdejdejfd Z xZ S) Siglip2VisionEmbeddingsconfigct|||_|j|_|j |_t j|j|j z|j z|j|_ |j|_ t|jdz|_ t j|j|j|_y)N) in_features out_featuresg?)super__init__rG hidden_size embed_dim patch_sizennLinear num_channelspatch_embedding num_patchesintposition_embedding_size Embeddingposition_embeddingr@rG __class__s r.rLz Siglip2VisionEmbeddings.__init__s  ++ ++!yy++doo=O  "--'*4+;+;S+@'A$"$,,t/?/?"Pr-positional_embeddingsspatial_shapes max_lengthr9cb|jd}|jd}|j}tj|||f|j|}|j dddj d}|jjdk(r|jtj}t|D]w}||\}} tj||| fddd } | j||| zjdd} | j|} | ||d || zf<| d|||| zd f<y|S) ac Resize positional embeddings to image-specific size and pad to a fixed size. Args: positional_embeddings (`torch.Tensor`): Position embeddings of shape (height, width, embed_dim) spatial_shapes (`torch.LongTensor`): Spatial shapes of shape (batch_size, 2) to resize the positional embeddings to max_length (`int`): Maximum length of the positional embeddings to pad resized positional embeddings to Returns: `torch.Tensor`: Embeddings of shape (batch_size, max_length, embed_dim) r)devicedtypercpubilinearFT)sizemode align_corners antialiasN)shaperar)emptyr`permute unsqueezetypetofloat32rangeF interpolatereshape transpose) r[r\r] batch_sizerN source_dtyperesulted_positional_embeddingsiheightwidthresized_embeddingss r.resize_positional_embeddingsz4Siglip2VisionEmbeddings.resize_positional_embeddingssc($))!, )//3 ,22 ). Y /(//* &!6 = =aA F P PQR S ! ' ' , , 5$9$<$.>+> ?BTUVBW *1fun.>+> ?% X(.-r- pixel_valuescJ|jjj}|j|j|}|jjj |j |j d}|j|||jd}||z}|S)aH Args: pixel_values (`torch.FloatTensor`): Pixel values of shape (batch_size, max_num_patches, num_channels * patch_size * patch_size) spatial_shapes (`List[Tuple[int, int]]`): Spatial shapes of shape (batch_size, 2) to resize the positional embeddings to )rar_r)r]) rSweightrarnrXrsrVr|ri)r@r}r\ target_dtype patch_embedsr[resized_positional_embeddings embeddingss r.forwardzSiglip2VisionEmbeddings.forwards++2288 ++LOO,O,OP !% 7 7 > > F F  ( ($*F*F! )-(I(I !>l>P>PQR>S)J) % "$AA r-)r%r&r'rrL staticmethodr)Tensor LongTensorrUr|r*r __classcell__rZs@r.rFrFs Q2 Q8.$||8.((8.8.  8.8.tE$5$5uGWGW\a\h\hr-rFmodulequerykeyvalueattention_maskscalingdropoutc tj||jdd|z}|||z}tjj |dtj j|j}tjj|||j}tj||} | jddj} | |fS)Nr_)dimra)ptrainingrrb) r)matmulrtrP functionalsoftmaxrornrarr contiguous) rrrrrrrkwargs attn_weights attn_outputs r.eager_attention_forwardrs<<s}}R'<=GL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r-c eZdZdZdeeefffd Z d dejde ejde e de eje ejffdZ xZS) Siglip2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperrGct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ d|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).F)rKrLrGrMrNnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutr is_causalrPrQk_projv_projq_projout_projrYs r.rLzSiglip2Attention.__init__s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r-r#routput_attentionsr9c |j\}}}|j|}|j|}|j|} |j |||j |j jdd}|j |||j |j jdd}| j |||j |j jdd} t} |jjdk7rN|jjdk(r|rtjdnt|jj} | |||| ||j|j|j sdn |j"\} } | j%|||j'} |j)| } |sd} | | fS) z#Input shape: Batch x Time x Channelrrbeagersdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rrrN)rirrrviewrrrtrrG_attn_implementationlogger warning_oncerrrrrrsrr) r@r#rrru seq_lengthrNqueriesrCvaluesattention_interfacerrs r.rzSiglip2Attention.forward(s-:,?,?) J ++m,{{=)]+,,z:t~~t}}U__`acdeyyZOYYZ[]^_ZT^^T]]S]]^_abc(? ;; + +w 6{{//69>O##L '>dkk>^>^&_#$7     nnJJ#}}C$,, % ! \"))*j)LWWY mmK0  LL((r-NF)r%r&r'r(rrrrLr)rrboolrrrrs@r.rrs{GBu%8:K%KLB.26,1 -)||-)!.-)$D> -) u||Xell33 4 -)r-rcVeZdZfdZdej dej fdZxZS) Siglip2MLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yN) rKrLrGr hidden_act activation_fnrPrQrMintermediate_sizefc1fc2rYs r.rLzSiglip2MLP.__init__Ysd  #F$5$5699V//1I1IJ99V55v7I7IJr-r#r9cl|j|}|j|}|j|}|Sr)rrr)r@r#s r.rzSiglip2MLP.forward`s4/ **=9 / r-)r%r&r'rLr)rrrrs@r.rrXs$KU\\ellr-rc eZdZdeeefffd Z ddejdejde e de ejfdZ xZS) Siglip2EncoderLayerrGcDt||j|_t j |j|j |_t||_ t j |j|j |_ t||_ y)Neps) rKrLrMrNrP LayerNormlayer_norm_eps layer_norm1r self_attn layer_norm2rmlprYs r.rLzSiglip2EncoderLayer.__init__hsm ++<<F $ u  ! $r-rc xeZdZdZdeffd Ze d deejdee dee de fdZ xZ S) Siglip2Encoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`Siglip2EncoderLayer`]. Args: config: Siglip2Config rGct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) rKrLrGrP ModuleListrpnum_hidden_layersrlayersgradient_checkpointing)r@rG_rZs r.rLzSiglip2Encoder.__init__sP  mm%PVPhPhJi$jQ%8%@$jk &+#%ksA#rroutput_hidden_statesr9c||n|jj}||n|jj}|rdnd}|rdnd}|}|jD]]}|r||fz}|jr+|j r|j |j|||} n ||||} | d}|sU|| dfz}_|r||fz}t|||S)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr,)rrr)r"r#r$) rGrrrrr_gradient_checkpointing_func__call__r) r@ inputs_embedsrrrencoder_statesall_attentionsr# encoder_layer layer_outputss r.rzSiglip2Encoder.forwards<2C1N-TXT_T_TqTq$8$D $++JjJj  40d% ![[ FM#!/=2B!B**t}} $ A A!**!"% ! !.!"&7! *!,M !/=3C2E!E) F, +}.>>N+(%  r-NNN)r%r&r'r(rrLrrr)rrrrrrs@r.rrsr,},26,0/3 C !.C $D> C 'tn C  C C r-ra Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the pre-trained position encodings. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. ceZdZdeffd Zeeeee e d de jde jde jdeedeed e f d ZxZS) Siglip2VisionTransformerrGcrt|||_|j}t ||_t ||_tj||j|_ t|dsdn |j|_|jrt||_|j"dk(|_y)Nrvision_use_headTflash_attention_2)rKrLrGrMrFrrencoderrPrrpost_layernormhasattrruse_head$Siglip2MultiheadAttentionPoolingHeadheadr_use_flash_attention_2r@rGrNrZs r.rLz!Siglip2VisionTransformer.__init__s  && 1&9%f-  ll9&:O:OP$+F4E$FFLbLb == * 'tn *  $* iL* r-rc eZdZdeffd Z ddeejdeejdeejdejfdZ xZ S) Siglip2TextEmbeddingsrGcNt||j}tj|j ||_tj|j||_|jdtj|jjddy)N position_ids)rr_F) persistent) rKrLrMrPrW vocab_sizetoken_embeddingmax_position_embeddingsrXregister_bufferr)arangeexpandrs r.rLzSiglip2TextEmbeddings.__init__?s && !||F,=,=yI"$,,v/M/My"Y  ELL)G)GHOOPWXej  r- input_idsrrr9c8||jdn|jd}|jjjd}||kDrtd|d|||jddd|f}||j |}|j|}||z}|S)Nr_rrzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rirXrrrr)r@r rrrmax_position_embeddingposition_embeddingsrs r.rzSiglip2TextEmbeddings.forwardKs -6,AY__R(}GZGZ[]G^ !%!8!8!?!?!E!Ea!H . .d,<=Ssk  0  153759 E,,-u//0  1 12   r-rcd}||d|zz ks ||d|zzkDrtjdd|||z |z }|||z |z }|jd|zdz d|zdz |j|j |t j dz|j||j||y)Ncddtj|tjdz zdz S)N?@)matherfsqrt)xs r.norm_cdfz _trunc_normal_..norm_cdfis(dhhq499S>122c99r-rbzjmean is more than 2 std from [a, b] in nn.init.trunc_normal_. The distribution of values may be incorrect.) stacklevelrr)minmax) warningswarnuniform_erfinv_mul_rradd_clamp_)tensormeanstdabrlus r._trunc_normal_r(fs: q1s7{q1s7{ 2  ;  !d(c!"A!d(c!"A OOAEAIq1uqy) NN KKdiin$% KK MMaQMr-r!r"r#r$r%r9ctj5t|dd|||j|j |dddy#1swYyxYw)anFills the input Tensor with values drawn from a truncated normal distribution. The values are effectively drawn from the normal distribution :math:`\mathcal{N}( ext{mean}, ext{std}^2)` with values outside :math:`[a, b]` redrawn until they are within the bounds. The method used for generating the random values works best when :math:`a \leq ext{mean} \leq b`. NOTE: this 'tf' variant behaves closer to Tensorflow / JAX impl where the bounds [a, b] are applied when sampling the normal distribution with mean=0, std=1.0 and the result is subsequently scaled and shifted by the mean and std args. Args: tensor: an n-dimensional `torch.Tensor` mean: the mean of the normal distribution std: the standard deviation of the normal distribution a: the minimum cutoff value b: the maximum cutoff value rrN)r)no_gradr(rr)r!r"r#r$r%s r.trunc_normal_tf_r+sI* $vq#q!, Cd#$$$s 0AAc,t|\}}|dk(r|}n|dk(r|}n |dk(r||zdz }|z }|dk(r$t|tj|dz y|dk(rCt j 5|j tj|dddy|d k(rItjd |z}t j 5|j| |dddytd |#1swYyxYw#1swYyxYw) Nfan_infan_outfan_avgrbtruncated_normalg۶%?r#normaluniformr zinvalid distribution ) r r+rrr)r*normal_rr) r!rrf distributionr-r.denomvariancebounds r.variance_scaling_r9s3F;OFG x    '!Q&u}H))TYYx%8;N%NO  ! ]]_ 4 NNtyy2N 3 4 4  " !h,' ]]_ + OOUFE * + +0?@@ 4 4 + +s3&C>D >D Dc t|ddy)Nr-r0rfr5r9r!s r. lecun_normal_r>sf8:LMr-c t|ddy)Nr-r2r;r<r=s r.default_flax_embed_initr@sf8(Cr-a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. ceZdZdeffd Zeeeee e d de e jde e jde e jde e de e d e f d ZxZS) Siglip2TextTransformerrGcFt|||_|j}t ||_t ||_tj||j|_ tj||j|_|jdk(|_y)Nrr)rKrLrGrMrrrrrPrrfinal_layer_normrQprojection_sizerrrrs r.rLzSiglip2TextTransformer.__init__s}  && /7%f- " YF 2 'tn 2  $2 gJ2 r-rBa@ This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`Siglip2Config`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the pre-trained position encodings. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. c2eZdZdZeZdZdZgdZdZ dZ dZ y)Siglip2PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. siglip2T)rrrFrrc t|trt|jtr |jjj n|jj }t jj|jjdtj|z yt|t jrt|jyt|trt jj!|j"jt jj!|j$jt jj!|j&jt jj!|j(jt jj+|j"j,t jj+|j$j,t jj+|j&j,t jj+|j(j,yt|t.rt jj!|j0jt jj!|j2jt jj|j0j,dt jj|j2j,dyt|t4rt jj!|j6j8t jj!|j:j<j8t jj+|j:j>j8yt|t@rrtCjDtCjFd}|jHj8jK||jLj8jOyt|tPrnt jj|jRj|jjj dz|jjTzyt|t jVt jXfrLt[|j|j,*t jj+|j,yyt|t j\rJ|j,j8jO|jj8jKdyy)zInitialize the weightsrr1gư>rrN)/ isinstancerFrGr vision_configrMrPinitr4rXrnprrWr@rxavier_uniform_rrrrzeros_biasrrrrprobedata attentionin_proj_weight in_proj_bias Siglip2Modelr)logr! logit_scalefill_ logit_biaszero_Siglip2ForImageClassification classifierinitializer_factorrQConv2dr>r)r@rrzlogit_scale_inits r. _init_weightsz$Siglip2PreTrainedModel._init_weightsus f5 6dkk=9 ))55[[,,  GGOOF55<  -$yyc):;     # # ) )*: ;    " " ( ( *  = > GGOO!!((KK--994?$++B`B``  BII 6 7 &-- ({{&v{{+'  - KK   " " $ MM   $ $S ).r-N) r%r&r'r(rrbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_2_supports_sdparfr,r-r.rLrLas4 !L!&*#"N,*r-rLzBThe text model from Siglip2 without any head or projection on top.ceZdZeZdeffd ZdejfdZdZ e e e e ee d deej"deej"d eej"d eed eedef d ZxZS)Siglip2TextModelrGcdt||t||_|j yr)rKrLrB text_model post_initrYs r.rLzSiglip2TextModel.__init__s&  08 r-r9cB|jjjSrrorrrDs r.get_input_embeddingsz%Siglip2TextModel.get_input_embeddingss))999r-c:||jj_yrrr)r@rs r.set_input_embeddingsz%Siglip2TextModel.set_input_embeddingss5:""2r-rr rrrrc.|j|||||S)a Returns: Examples: ```python >>> from transformers import AutoTokenizer, Siglip2TextModel >>> model = Siglip2TextModel.from_pretrained("google/siglip2-base-patch16-224") >>> tokenizer = AutoTokenizer.from_pretrained("google/siglip2-base-patch16-224") >>> # important: make sure to set padding="max_length" as that's how the model was trained >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding="max_length", return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```r rrrr)ro)r@r rrrrs r.rzSiglip2TextModel.forwards)<)%/!5   r-rI)r%r&r'rrrLrPModulersrurrrJrrrr)rrrrrs@r.rmrms %L0 :bii:;*+HI+ETef-115/3,0/3 ! ELL)! !.! u||, ! $D> ! 'tn !  $! gJ! r-rmceZdZdZdeffd ZddejdeejdejfdZ xZ S) rzMultihead Attention Pooling.rGct|tjt j dd|j |_tjj|j |jd|_ tj|j |j|_ t||_|j|_y)NrT) batch_firstr)rKrLrP Parameterr)randnrMrVMultiheadAttentionrrXrr layernormrrrrYs r.rLz-Siglip2MultiheadAttentionPoolingHead.__init__s \\%++aF4F4F"GH 44V5G5GIcIcqu4vf&8&8f>S>STf%33r- hidden_staterr9c|jd}|jj|dd}|f|jd|jd}}t||j|}|jd|j |d}|j d||}|j||||d}|}|j|}||j|z}|dddfS)Nrrr_) attn_mask) rirVrepeatrrarrsrXrr)r@rrrurV target_len source_lenrs r.rz,Siglip2MultiheadAttentionPoolingHead.forwards!''*  !!*a3  %%*[[^\5G5G5J J7 HZHZ\fgN+221dnnjRSTN+33B JON~~e\>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Siglip2VisionModel >>> model = Siglip2VisionModel.from_pretrained("google/siglip2-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled features ```r}rr\rr)r)r@r}rr\rrs r.rzSiglip2VisionModel.forwards,B  %/)/!5 !  r-r)r%r&r'rrmain_input_namerLrPrxrsrrrrrr)r*rrrrrrrs@r.rrs 'L$O2 $ 'tn $  $$ iL$ r-rceZdZeZdeffd Zee ddee jdee jdee jdee dee de jf d Z ee dd ee jd ee jd ee jdee dee de jf d Zeeeeee ddee jd ee jd ee jd ee jdee jdee jdee dee dee defdZxZS)r[rGct||t|jts"t dt |jdt|jts"t dt |jd|j}|j}tj|}tj|}|j|_ |j|_ tjt!j"d|_tjt!j"d|_|j)y)NzNconfig.text_config is expected to be of type Siglip2TextConfig but is of type .zRconfig.vision_config is expected to be of type Siglip2VisionConfig but is of type r)rKrLrO text_configr TypeErrorrmrPrrm _from_configrrorrPr|r)r}r]r_rp)r@rGrrProrrZs r.rLzSiglip2Model.__init__>s  &,,.?@++,-Q0  &..0CD--./q2  (( ,, &22;? )66}E %//(55<< A7,,u{{1~6 r-r rrrrr9c||n|jj}||n|jj}|j|||||}|j}|S)aM Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`Siglip2TextModel`]. Examples: ```python >>> from transformers import AutoTokenizer, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip2-base-patch16-224") >>> tokenizer = AutoTokenizer.from_pretrained("google/siglip2-base-patch16-224") >>> # important: make sure to set padding="max_length" as that's how the model was trained >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding="max_length", return_tensors="pt") >>> with torch.no_grad(): ... text_features = model.get_text_features(**inputs) ```rw)rGrrror)r@r rrrr text_outputsrHs r.get_text_featureszSiglip2Model.get_text_features^sr:2C1N-TXT_T_TqTq$8$D $++JjJj 48??)%/!5 4C4 %22 r-r}rr\c||n|jj}||n|jj}|j|||||}|j}|S)a Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`Siglip2VisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip2-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... image_features = model.get_image_features(**inputs) ```r)rGrrrr)r@r}rr\rrvision_outputsrHs r.get_image_featureszSiglip2Model.get_image_featuressuD2C1N-TXT_T_TqTq$8$D $++JjJj 6:5F5F%/)/!5 6G6 '44 r-r return_lossc ||n|jj}| | n|jj} |j||||| } |j ||||| } | j } | j } | | j dddz } | | j dddz } tj| | jj| j}|jj| j|jj| j}}||jz|z}|j}d}|rtj|j!d|j }tj"| d|zz}tj$j&j)||z}tj*|d  }|j-}t/|||| | | | S) aa Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AutoModel >>> import torch >>> model = AutoModel.from_pretrained("google/siglip2-base-patch16-224") >>> processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-224") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["a photo of 2 cats", "a photo of 2 dogs"] >>> # important: we pass `padding=max_length` since the model was trained with this >>> inputs = processor(text=texts, images=image, padding="max_length", return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image >>> probs = torch.sigmoid(logits_per_image) # these are the probabilities >>> print(f"{probs[0][0]:.1%} that image 0 is '{texts[0]}'") 31.9% that image 0 is 'a photo of 2 cats' ```Nrrwrbr_T)rrkeepdimr)r`r)r4r5r6r1r!r7r8)rGrrrrornormr)rtrnr`r]r_expeyere ones_likerPr logsigmoidsumr"r3)r@r r}rr\rrrrrrrr!r1r6r]r_r5r4rm1_diag1logliknlls r.rzSiglip2Model.forwardsZ2C1N-TXT_T_TqTq$8$D $++JjJj 6:5F5F%/)/!5 6G6 48??)%/!5 4C4 &33 "00 $l&7&7!T&7&RR !K$4$4qb$$4$OO  ,,{LNN4D4G4G HZHZ4[\"&"2"2"5"5k6H6H"I4??K]K]^i^p^pKqZ )KOO,== J*,,. ))O003O + 'tn +   +J+Z++JK597;59,0/3 0u0010'u||40!!1!12 0 $D> 0 'tn 0   0L0d*+CD=}U15487;591537&*,0/3_ E,,-_ u001_ 'u||4 _ !!1!12 _ !. _ u//0_ d^_ $D>_ 'tn_  _ VE_ r-r[z Siglip2 vision encoder with an image classification head on top (a linear layer on top of the pooled final hidden states of the patch tokens) e.g. for ImageNet. c eZdZdZdeddffd Zeeee e e  d de e jde e jde e jd e e jd e ed e ede fd ZxZS)rar}rGr9Nc~t|||j|_tj |j }|j |_|jdkDr4tj|j j|jntj|_ |jy)Nr) rKrL num_labelsrrrPrrPrQrMIdentityrbrp)r@rGrrZs r.rLz&Siglip2ForImageClassification.__init__.s   ++*66v7K7KL (55OUN_N_bcNcBIIf**668I8I Jikititiv  r-rrr\labelsrrc8||n|jj}||n|jj}|j|||||}|j}|Q|dj |j } tj|| zdtj| dz }ntj|d}|j|} d} ||j | j }|jj|jdk(rd|j_ nl|jdkDrL|jtjk(s|jtjk(rd|j_ nd|j_ |jjdk(rIt!} |jdk(r&| | j#|j#} n| | |} n|jjdk(r=t%} | | j'd |j|j'd } n,|jjdk(rt)} | | |} t+| | |j,|j. S) a; labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, Siglip2ForImageClassification >>> import torch >>> from PIL import Image >>> import requests >>> torch.manual_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # note: we are loading a `Siglip2Model` from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random if seed is not set above. >>> image_processor = AutoImageProcessor.from_pretrained("google/siglip2-base-patch16-224") >>> model = Siglip2ForImageClassification.from_pretrained("google/siglip2-base-patch16-224") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the two classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: LABEL_1 ```N)rr\rr).Nrr regressionsingle_label_classificationmulti_label_classificationr_)r4logitsr#r$)rGrrrr"rnr`r)rr"rb problem_typerralongrUr squeezer rr rr#r$) r@r}rr\rrrrsequence_output pool_maskrr4loss_fcts r.rz%Siglip2ForImageClassification.forward@s8Z2C1N-TXT_T_TqTq$8$D $++JjJj /3.?.? /)/!5 /@/ "33 +,Y7::?;Q;QRI#ii)(CKeiiXaghNiiO#jja@O1  YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./$!//))   r-)NNNNNN)r%r&r'rrrLrrrrr_CONFIG_FOR_DOCrr)rrrrrrs@r.rara$s%O}$*+CD+@_047;59)-,0/3a u||,a 'u||4a !!1!12 a & a $D> a 'tna  a `Ea r-ra)r[rLrmrra)r)rrgr)rr-r2)Prr dataclassesrtypingrrrrrnumpyrRr)torch.nnrPtorch.nn.functionalrrqr r r torch.nn.initr activationsrmodeling_attn_mask_utilsrmodeling_outputsrrrmodeling_utilsrrutilsrrrrrrconfiguration_siglip2rrr get_loggerr%rrr r0r3rxrFrfloatrrrrrrrrr(r+r9r>r@rJrBSIGLIP2_START_DOCSTRINGrrLrmrrr[ra__all__r,r-r.rs* !88 AA7!BbbFYX   H %" ?+? ?8 ? ? ?8 ! K!  ! HbbiibX% II% <<% % << % U\\* %  %%.D)ryyD)N  -"))-`T RYYT n#$; ryy; |%BII%P! J\_$ LL$ %$27$BG$SX$ \\$4A2ND!BA RYYA H %P@*_@*FL3 -3 3 l"299">N6 /6 6 r-.f )f /f R y $:y y x r-