T^(hddlmZddlmZmZmZmZmZddlZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZmZdd lmZmZdd lmZmZmZmZddlm Z ddl!m"Z"m#Z#ejHe%Z&dZ'dZ(ede(GddeZ)Gdde jTZ+ d=de jTdejXdejXdejXdeejXde-de-fd Z.Gd!d"e jTZ/Gd#d$e jTZ0Gd%d&e jTZ1Gd'd(e jTZ2d)Z3ed*e3Gd+d,e)Z4eGd-d.eZ5Gd/d0e jTZ6Gd1d2e jTZ7d3Z8ed4e(Gd5d6e)Z9eGd7d8eZ:ed9e(Gd:d;e)eZ;gd<Z) dataclass)CallableListOptionalTupleUnionN)nn)CrossEntropyLoss)ACT2FN) DynamicCache)GenerationMixin)_prepare_4d_attention_mask)BaseModelOutput ModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) AutoModel) SmolVLMConfigSmolVLMVisionConfigrag This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`SmolVLMConfig`] or [`SmolVLMVisionConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zUThe bare SmolVLM Model outputting raw hidden-states without any specific head on top.c:eZdZeZdZdZddgZdZdZ dZ dZ dZ dZ y)SmolVLMPreTrainedModelmodelTSmolVLMVisionAttentionSmolVLMDecoderLayerpast_key_valuesct|jdr|jjn#|jjj}t|dr'|jj j d|t|tjtjfrY|jj j d||j%|jj jyyt|tjrf|jj j d||j2|jj |jjyyy)Ninitializer_rangeclass_embedding)meanstd)hasattrconfigr$get_text_configr%datanormal_ isinstancer LinearConv2dweightbiaszero_ Embedding padding_idx)selfmoduler(s j/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/smolvlm/modeling_smolvlm.py _init_weightsz$SmolVLMPreTrainedModel._init_weightsRs$t{{$78 KK ) ),,.@@ 6, -  " " ' ' / /Sc / B fryy"))4 5 MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>..N)__name__ __module__ __qualname__r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_flex_attn_supports_cache_classr9r:r8rrCsC !L&*#13HI"3!N ?r:rcxeZdZdZdeffd ZdejdejdejfdZ xZ S)SmolVLMVisionEmbeddingsaP This is a modified version of `siglip.modelign_siglip.SiglipVisionEmbeddings` to enable images of variable resolution. The modifications are adapted from [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://arxiv.org/abs/2307.06304) which allows treating images in their native aspect ratio and without the need to resize them to the same fixed size. In particular, we start from the original pre-trained SigLIP model (which uses images of fixed-size square images) and adapt it by training on images of variable resolutions. r*ct||j|_|j|_|j |_t j|j|j|j |j d|_ |j|j z|_ |jdz|_ |j|_ t j|j|j|_y)Nvalid) in_channels out_channels kernel_sizestridepaddingr)super__init__ hidden_size embed_dim image_size patch_sizer r0 num_channelspatch_embeddingnum_patches_per_side num_patches num_positionsr4position_embeddingr6r* __class__s r8rRz SmolVLMVisionEmbeddings.__init__qs ++ ++ ++!yy++??  %)OOt$F!44a7!--"$,,t/A/A4>>"Rr: pixel_valuespatch_attention_maskreturnc|j\}}}}|j|}|jdjdd}||jz||jz} } t j d|jz dd|jz } t j|| | zfd} t|D]\} }|dddfj}|dj}t j ddd|z }t j ddd|z }t j|| d}t j|| d}|dddf|jz|zj}|| | |jd j<| j|jj j"} ||j| z}|S) Nrrg?r)size fill_valueg !?T)right)shaperXflatten transposerVtorcharangerYfull enumeratesum bucketizeviewcputor\r1device)r6r_r` batch_size_max_im_hmax_im_w patch_embeds embeddingsmax_nb_patches_hmax_nb_patches_w boundaries position_ids batch_idx p_attn_mask nb_patches_h nb_patches_wfractional_coords_hfractional_coords_wbucket_coords_hbucket_coords_wpos_idss r8forwardzSmolVLMVisionEmbeddings.forwards,8,>,>) Ax++L9 !))!,66q!< -5-H(VZVeVeJe*\\!d&?&?"?a$JcJcFcd zz 4DGW4W'Xefg &/0D&E J "I{&q!t,002L&q>--/L"',,q(A ? J$t'>'>'E'E'L'LM $"9"9,"GG r:) r;r<r=__doc__rrRrj FloatTensor BoolTensorTensorr __classcell__r^s@r8rIrIfsDS2S&E$5$5UM]M]bgbnbnr:rIr7querykeyvalueattention_maskscalingdropoutc tj||jdd|z}|||z}tjj |dtj j|j}tjj|||j}tj||} | jddj} | |fS)Nrf)dimdtype)ptrainingrr) rjmatmulrir functionalsoftmaxfloat32rrrrr contiguous) r7rrrrrrkwargs attn_weights attn_outputs r8eager_attention_forwardrs<<s}}R'<=GL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r:c eZdZdZfdZ ddej deej deede ej eej ffdZ xZ S) r z=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_d|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gF)rQrRr*rSrTnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutrr r/k_projv_projq_projout_proj is_causalr]s r8rRzSmolVLMVisionAttention.__init__s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r: hidden_statesroutput_attentionsrac |j\}}}|j|}|j|}|j|} |j |||j |j jdd}|j |||j |j jdd}| j |||j |j jdd} t} |jjdk7rN|jjdk(r|rtjdnt|jj} | |||| ||j|j|j sdn |j"\} } | j%|||j'} |j)| } |sd} | | fS) z#Input shape: Batch x Time x Channelrreagersdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.r&)rrrN)rgrrrrprrrirr*_attn_implementationlogger warning_oncerrrrrreshaperr) r6rrrrt seq_lengthrTquerieskeysvaluesattention_interfacerrs r8rzSmolVLMVisionAttention.forwards-:,?,?) J ++m,{{=)]+,,z:t~~t}}U__`acdeyyZOYYZ[]^_ZT^^T]]S]]^_abc(? ;; + +w 6{{//69>O##L '>dkk>^>^&_#$7     nnJJ#}}C$,, % ! \"))*j)LWWY mmK0  LL((r:NF) r;r<r=rrRrjrrboolrrrrs@r8r r sfG226,1 -)||-)!.-)$D> -) u||Xell33 4 -)r:r cVeZdZfdZdej dej fdZxZS)SmolVLMVisionMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yN) rQrRr*r hidden_act activation_fnr r/rSintermediate_sizefc1fc2r]s r8rRzSmolVLMVisionMLP.__init__sd  #F$5$5699V//1I1IJ99V55v7I7IJr:rracl|j|}|j|}|j|}|Sr)rrr)r6rs r8rzSmolVLMVisionMLP.forwards4/ **=9 / r:)r;r<r=rRrjrrrrs@r8rrs$KU\\ellr:rc eZdZdeffd Z ddej dej deede ejfdZ xZ S) SmolVLMEncoderLayerr*cDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ y)Neps) rQrRrSrTr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2r]s r8rRzSmolVLMEncoderLayer.__init__sm ++/7<<F $ u  ! $r:rc eZdZdZdeffd Z d deejdee dee dee de e e ff d Z xZS) SmolVLMEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`SmolVLMEncoderLayer`]. Args: config: SmolVLMConfig r*ct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) rQrRr*r ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)r6r*rur^s r8rRzSmolVLMEncoder.__init__HsP  mm%PVPhPhJi$jQ%8%@$jk &+#%ksA#rroutput_hidden_states return_dictrac||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|}|jD]]} |r||fz}|j r+|j r|j| j|||} n | |||} | d}|sU|| dfz}_|r||fz}|std|||fDSt|||S)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NrG)rrrc3&K|] }|| ywrrG.0vs r8 z)SmolVLMEncoder.forward..seqWXWdelast_hidden_stater attentions) r*rruse_return_dictrrr_gradient_checkpointing_func__call__tupler) r6 inputs_embedsrrrrencoder_statesall_attentionsr encoder_layer layer_outputss r8rzSmolVLMEncoder.forwardOs7<2C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]NN$See e+>Vd  r:NNNN)r;r<r=rrrRrrjrrrrrrrrs@r8rr?s~,},26,0/3&* E !.E $D> E 'tn E d^ E  uo% &E r:raR This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`SmolVLMVisionConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zDThe SmolVLM Vision Transformer Model outputting raw image embedding.c eZdZeZdZdZdZdeffd ZdZ dZ d de e jde ede ed e ed eeeff d ZxZS) SmolVLMVisionTransformerTr*ct|||j}t||_t ||_|j|_tj||j|_ |jdk(|_ y)Nrflash_attention_2)rQrRrSrIryrencoderrVr rrpost_layernormr_use_flash_attention_2)r6r*rTr^s r8rRz!SmolVLMVisionTransformer.__init__sm  && 1&9%f-  ++ ll9&:O:OP&,&A&AEX&X#r:c|jSrryr6s r8get_input_embeddingsz-SmolVLMVisionTransformer.get_input_embeddingss r:c||_yrrr6rs r8set_input_embeddingsz-SmolVLMVisionTransformer.set_input_embeddingss r:r`rrrrac||n|jj}||n|jj}||n|jj}|j d}|s|j }t j||j d|z|j d|zf}|jt j|j}|j||}|j|d}t j|sd}n"|jst||j }|j#|||||} | d} |j%| } |s | f| ddzSt'| | j(| j* S) Nrrr rrsr_r`rf)rrrrrrr)r*rrrrcrVrjonesrrrrsryrpanyrrrrrrrr) r6r_r`rrrrtrVrencoder_outputsrs r8rz SmolVLMVisionTransformer.forwards2C1N-TXT_T_TqTq$8$D $++JjJj &1%RTaTgTg#h ,,'//!5# ' ,A. //0AB%'/!"*== =/)77&11  r:r)r;r<r=rr>rD_supports_flash_attention_2rErRrrrrjrrrrrrrrs@r8rrs 'LN"&Y2Y <@,0/3&* 7 'u'7'787 $D> 7 'tn 7 d^ 7  uo% &7 r:rceZdZUdZdZeejed<dZ ee e ejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<y)SmolVLMBaseModelOutputWithPasta Base class for SmolVLM model's outputs that may also contain a past key/values (to speed up sequential decoding). Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and optionally if `config.is_encoder_decoder=True` 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder Nrr"rrimage_hidden_states)r;r<r=rrrrjr__annotations__r"rrrr rGr:r8r r s<6:x 1 129AEOXeE%*;*;$<=>E8B%(9(9":;Br:r c$eZdZfdZdZxZS)SmolVLMSimpleMLPct||jj|jdzz}|j j}t j||d|_y)NrFr2) rQrR vision_configrS scale_factor text_configr r/proj)r6r* input_size output_sizer^s r8rRzSmolVLMSimpleMLP.__init__$sT ))559L9La9OP ((44 IIj+EB r:c$|j|Sr)r)r6xs r8rzSmolVLMSimpleMLP.forward*syy|r:)r;r<r=rRrrrs@r8rr#sC r:rc,eZdZfdZddZdZxZS)SmolVLMConnectorcdt||j|_t||_yr)rQrRrrmodality_projectionr]s r8rRzSmolVLMConnector.__init__/s) "//#3F#; r:c|j\}}}t|dzx}}|j||||}|j||t||z ||z}|jdddd}|j |t||z t||z ||dzz}|jdddd}|j |t||dzz ||dzz}|S)Ng?rrrr )rcintrppermuter)r6rrbszseqrTheightwidths r8 pixel_shufflezSmolVLMConnector.pixel_shuffle4sffhS)S#X& FF3y 1 FF3EL$8 99|;S T IIaAq ! IIc3u|34c&<:O6PR[_kmn_nRo p IIaAq ! IIc3slAo67lTUo9V Wr:c`|j||j}|j|}|Sr)r&rr)r6r s r8rzSmolVLMConnector.forward?s4"001DdFWFWX"667JK""r:)r)r;r<r=rRr&rrrs@r8rr.s< #r:ra Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)): The tensors corresponding to the input images. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details ([]`LlavaProcessor`] uses [`CLIPImageProcessor`] for processing images). pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. image_hidden_states (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The hidden states of the image encoder after modality projection. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. zOSmolVLM model consisting of a SIGLIP vision encoder and Llama3 language decoderc eZdZdZdeffd ZdZdZdZdZ de jd e jd e jfd Z ed e ddee jd ee jdee jdeee j$d ee j$dee j$dee j&d ee j$deedeedeedeedee jdeeeffdZxZS) SmolVLMModelz A subclass of Idefics3Model. We do *not* remove or block the call to inputs_merger in forward. Instead, we override inputs_merger here with custom logic. r*ct|||jjj|_|jjj |_tj|j|_ t||_ tj|j|_t!|jj"|jj$zdz|j&dzz |_|jj*|_|jj,dk(|_|j1y)Nrr)rQrRr*r pad_token_idr5 vocab_sizer _from_configr vision_modelr connectorr from_config text_modelr rUrVr image_seq_lenimage_token_idrr post_initr]s r8rRzSmolVLMModel.__init__s  ;;22??++11<<4AA&BVBVW)&1#//0B0BC ""--1E1E1P1PPUV V[a[n[npq[q r #kk88&,&8&8&M&MQd&d# r:cfdd}|jj||_|jj||_y)aE Enables the gradients for the input embeddings. This is useful for lora when using gradient checkpointing. c.f. https://github.com/huggingface/peft/issues/1402#issuecomment-1913675032 Override to set output.requires_grad = True for both the decoder's and vision model's embeddings. ctt|jdk(r|St|jdS)Nr)lenlistchildren)r7get_lowest_modules r8r:zBSmolVLMModel.enable_input_require_grads..get_lowest_modules?4)*+q0 )foo.?)@)CDDr:c&|jdyNTrequires_grad_r7inputoutputs r8make_inputs_require_gradszJSmolVLMModel.enable_input_require_grads..make_inputs_require_grads  ! !$ 'r:N)rregister_forward_hook_text_require_grads_hookr._vision_require_grads_hook)r6rBr:s @r8enable_input_require_gradsz'SmolVLMModel.enable_input_require_gradssN E ()-(A(A(C(Y(YZs(t%*;D