S^(hi dZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZd dlmZm Z m!Z!d dl"m#Z#m$Z$ejJe&Z'dZ(de jRde jRfdZ*de jRde jRfdZ+eGddeZ,eGddeZ-eGddeZ.eGddeZ/Gdde j`Z1Gd d!e j`Z2Gd"d#e j`Z3Gd$d%e j`Z4Gd&d'e j`Z5Gd(d)eZ6d*Z7d+Z8d,Z9d-Z:Gd.d/e j`Z;Gd0d1e6Z<ed2e7Gd3d4e6Z=ed5e7Gd6d7e6eZ>ed8e7Gd9d:e6eZ?ed;e7Gd<d=e6Z@gd>ZAy)?zPyTorch BLIP model.N) dataclass)AnyOptionalTupleUnion)nn) normalize)ACT2FN)GenerationMixin)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings torch_int) BlipConfigBlipTextConfigBlipVisionConfig)BlipTextLMHeadModel BlipTextModelzSalesforce/blip-vqa-baselogitsreturnctjj|tjt ||j S)Ndevice)r functional cross_entropytorcharangelenr )rs d/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/blip/modeling_blip.pycontrastive_lossr'0s/ == & &vu||CKPVP]P]/^ __ similaritycZt|}t|j}||zdz S)Ng@)r't)r) caption_loss image_losss r& blip_lossr.5s,#J/L!*,,.1J : % ,,r(c2eZdZUdZdZeeeje d<dZ eeeje d<dZ eeje d<dZ eeje d<dZ eeejdfe d<dZeeejdfe d <ed Zy) 'BlipForConditionalGenerationModelOutputa Adapted from the base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. This class also adds the loss term from the text decoder. Args: loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Languge modeling loss from the text decoder. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`, *optional*): Prediction scores of the language modeling head of the text decoder model. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*): The image embeddings obtained after applying the Vision Transformer model to the input image. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossr image_embedslast_hidden_state. hidden_states attentionscNtjdt|jS)Nz`decoder_logits` attribute is deprecated and will be removed in version 5 of Transformers. Please use the `logits` attribute to retrieve the final output instead.)warningswarn FutureWarningrselfs r&decoder_logitsz6BlipForConditionalGenerationModelOutput.decoder_logits^s#  W  {{r()__name__ __module__ __qualname____doc__r1rrr# FloatTensor__annotations__rr2r3r4r5propertyr<r(r&r0r0;s404D(5**+ ,315FHU5,,- .504L(5,,-459x 1 129=AM8E%"3"3S"89:A:>Ju00#567> r(r0ceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<y) BlipTextVisionModelOutputa Adapted from the base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. This class also adds the loss term from the text decoder. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Languge modeling loss from the text decoder. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr1r2r3.r4r5)r=r>r?r@r1rr#rArBr2r3r4rr5rDr(r&rFrFhs0)-D(5$$ %,04L(5,,-459x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r(rFcleZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ ee ejdfed<dZeejed <dZee ejdfed <dZee ejed <y) BlipImageTextMatchingModelOutputa Adapted from the base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. This class also adds the loss term from the text decoder as well as the image-text similarity scores. Args: itm_score (`torch.FloatTensor`): The image-text similarity scores. loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Languge modeling loss from the text decoder. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. vision_pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`, *optional*): Last layer hidden-state of the vision of the vision-only branch of the model. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. question_embeds (`torch.FloatTensor`): The question embeddings obtained by the text projection layer. N itm_scorer1r2r3.r4vision_pooler_outputr5question_embeds)r=r>r?r@rIrr#rArBr1r2r3r4rrJr5rKrDr(r&rHrHs>.2Ix))*1(,D(5$$ %,04L(5,,-459x 1 129=AM8E%"3"3S"89:A8<(5#4#45<:>Ju00#567>:>OXeE$5$567>r(rHceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd Zy) BlipOutputa Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image:(`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text:(`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`BlipTextModel`]. image_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`BlipVisionModel`]. text_model_output(`BaseModelOutputWithPooling`): The output of the [`BlipTextModel`]. vision_model_output(`BaseModelOutputWithPooling`): The output of the [`BlipVisionModel`]. Nr1logits_per_imagelogits_per_text text_embedsr2text_model_outputvision_model_outputrcHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw))rQrRN)getattrto_tuple).0kr;s r& z&BlipOutput.to_tuple..s=  LLDGRYZ^`aRbRkRkRm m s-0)tuplekeysr:s`r&rVzBlipOutput.to_tuples# YY[   r()r=r>r?r@r1rr#rArBrNrOrPr2rQrrRrrrVrDr(r&rMrMs()-D(5$$ %,48hu001837OXe//07/3K%++,304L(5,,-448186:3: %* r(rMceZdZdeffd Zdej dededej fdZd dejd e dej fd Z xZ S) BlipVisionEmbeddingsconfigcpt|||_|j|_|j |_|j |_tjtjdd|j|_ tjd|j|j |j |_ |j |j zdz|_|jdz|_tjtjd|j|j|_y)Nrr ) in_channels out_channels kernel_sizestride)super__init__r^ hidden_size embed_dim image_size patch_sizer Parameterr#randnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingr;r^ __class__s r&rfzBlipVisionEmbeddings.__init__s  ++ ++ ++!||EKK1dnn,MN!yyDOO\`\k\k !OOt>1D!--1"$,,u{{1d>P>PRVR`R`/a"br( embeddingsheightwidthrc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rr rdbicubicF)sizemode align_cornersdim)shaperrr#jit is_tracingrjrreshapepermuterr! interpolateviewcat) r;rurvrwrprqclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r&interpolate_pos_encodingz-BlipVisionEmbeddings.interpolate_pos_encodings`!&&q)A- //55a81< yy##%+*F6UZ?** *11!RaR%811!QR%8r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr( pixel_valuesrc|j\}}}}|jjj}|j|j |}|j dj dd}|jj|ddj |} tj| |gd} |r|j| ||} n |j} | | ddd| jdddfj |z} | S)Ndtyperdrryr~)rroweightrtoflatten transposermexpandr#rrrrr{) r;rr batch_size_rvrw target_dtype patch_embeds class_embedsrurrs r&forwardzBlipVisionEmbeddings.forwards'3'9'9$ Avu++2288 ++LOO,O,OP #++A.88A> ++22:q"EHHV YY l;C #!%!>!>z6SX!Y !%!8!8 "4Q8L*//!:L8La5O"P"S"ST`"aa r(F) r=r>r?rrfr#TensorintrrAboolr __classcell__rts@r&r]r]smc/c$&D5<<&D&DUX&D]b]i]i&DP E$5$5 QU bgbnbn r(r]c eZdZdeffd Z ddeejdeejdeejdejfdZ xZ S) BlipTextEmbeddingsr^cNt||j}tj|j ||_tj|j||_|jdtj|jjddy)N position_ids)rryF) persistent) rerfrgr Embedding vocab_sizetoken_embeddingmax_position_embeddingsrrregister_bufferr#r$rr;r^rhrts r&rfzBlipTextEmbeddings.__init__%s && !||F,=,=yI"$,,v/M/My"Y  ELL)G)GHOOPWXej  r( input_idsr inputs_embedsrc8||jdn|jd}|jjjd}||kDrtd|d|||jddd|f}||j |}|j|}||z}|S)NryrzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rrrr ValueErrorrr)r;rrr seq_lengthmax_position_embeddingposition_embeddingsrus r&rzBlipTextEmbeddings.forward1s -6,AY__R(}GZGZ[]G^ !%!8!8!?!?!E!Ea!H . .d,<=Sr?rrfrr# LongTensorrArrrrs@r&rr$sj  ~  153759 E,,-u//0  1 12   r(rceZdZdZfdZdej dedefdZ d dej de ej d e e d e ej e ej e e ej ffd Z xZ S) BlipAttentionz=Multi-headed attention from 'Attention Is All You Need' paperczt|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ tj|j|_ tj|jd|jz|_tj|j|j|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gr )rerfr^rgrhnum_attention_heads num_headshead_dimrscalerDropoutattention_dropoutdropoutLinearqkv projectionrss r&rfzBlipAttention.__init__Os  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( zz&":":; 99T^^Q-?@))DNNDNNCr(tensorseq_lenbszc|j|||j|jjddj S)Nrrd)rrrr contiguous)r;rrrs r&_shapezBlipAttention._shapeas7{{3GQQRSUVWbbddr(r4 head_maskoutput_attentionsrc|j\}}}|j|j||d|j||jzj ddddd}|d|d|d} } }t j || jdd} | |jz} tjj| d} |j| } || |z} t j | | j dddd} | jd d|jfz}| j|} |j| }|r|| f}|S|d f}|S) z#Input shape: Batch x Time x Channelr rdrrryrr~N)r{rrrrr#matmulrrrr!softmaxrrhr)r;r4rrrtgt_lenrh mixed_qkv query_states key_states value_statesattention_scoresattention_probs context_layernew_context_layer_shapeoutputoutputss r&rzBlipAttention.forwarddsm#0"4"4"6Wi HH] # WS'1dnni4>>6Q R WQ1a #  2;1y|YWX\,j !<< j6J6J2r6RS+djj8--//0@b/I,,7  - 9O _lCKKAqRSUVW "/"4"4"6s";t~~>O"O%--.EF //@6?+HNtnr(NF)r=r>r?r@rfr#rrrrrrrrrs@r&rrLsGD$eU\\eCece -1,1 *||*ELL)*$D> * u||Xell3XeELL>Q5RR S *r(rcVeZdZfdZdej dej fdZxZS)BlipMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yN) rerfr^r hidden_act activation_fnrrrgintermediate_sizefc1fc2rss r&rfzBlipMLP.__init__sd  #F$5$5699V//1I1IJ99V55v7I7IJr(r4rcl|j|}|j|}|j|}|Sr)rrr)r;r4s r&rzBlipMLP.forwards4/ **=9 / r()r=r>r?rfr#rrrrs@r&rrs$KU\\ellr(rc eZdZdeffd Z ddej dej deede ejfdZ xZ S) BlipEncoderLayerr^cDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yN)eps) rerfrgrhr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rss r&rfzBlipEncoderLayer.__init__sl ++&v.<<Fr?rrfr#rrrrrArrrs@r&rrsXSzS-2 $||$ $$D> $ u  ! $r(rc0eZdZdZeZdZdZddgZdgZ dZ y) BlipPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. blipTrrpast_key_valuec0|jj}t|tjs4t|tj st|tj rc|jjjd|t|dr0|j$|jjjt|trt|jdr |jjj}tjj!|j"d|tjj!|j$d|yt|tj&rJ|jjj|jjj)dyt|tj r2|j%|jjjyyy)zInitialize the weightsg)meanstdbiasN vision_configg?)r^initializer_range isinstancerrnrrrdatanormal_hasattrrzero_r]rinit trunc_normal_rrrmrfill_)r;modulefactors r& _init_weightsz!BlipPreTrainedModel._init_weightsst.. fbii (Jvr||,LPZ[aceclclPm MM   & &CV & <vv&6;;+B   &&( f2 3t{{O422DD GG ! !)) "  GG ! !&& "   - KK   " " $ MM   $ $S )  *v{{/F KK   " " $0G *r(N) r=r>r?r@r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placementr rDr(r&rrs4 L&*#+-AB#3"4%r(ra= This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`BlipConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aN Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoProcessor`]. See [`BlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`BlipImageProcessor`]. See [`BlipImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the pre-trained position encodings. as Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoProcessor`]. See [`BlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`BlipImageProcessor`]. See [`BlipImageProcessor.__call__`] for details. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the pre-trained position encodings. c eZdZdZdeffd Z d deejdee dee dee de e e ff d Z xZS) BlipEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`BlipEncoderLayer`]. Args: config (`BlipConfig`): The corresponding vision configuration for the `BlipEncoder`. r^ct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) rerfr^r ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)r;r^rrts r&rfzBlipEncoder.__init__ksO  mmuVMeMeGf$g!%5f%=$gh &+#%hsA#rroutput_hidden_states return_dictrc||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|}t |j D]`\} } |r||fz}|j r+|jr|j| j|||} n | |||} | d}|sX|| dfz}b|r||fz}|std|||fDSt|||S)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NrD)rrrc3&K|] }|| ywrrD)rWvs r&rYz&BlipEncoder.forward..seqWXWde)r3r4r5) r^rruse_return_dict enumeraterrtraining_gradient_checkpointing_func__call__rZr ) r;rrrrrencoder_statesall_attentionsr4idx encoder_layer layer_outputss r&rzBlipEncoder.forwardqsA82C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]NN$See e+>Vd  r(NNNN)r=r>r?r@rrfrr#rrrrr rrrs@r&rras~,z,26,0/3&* C !.C $D> C 'tn C d^ C  uo% &C r(rceZdZdZeZdeffd Zeee e e d de e jde ede ede eded eee ff d Zd ZxZS) BlipVisionModelrr^ct||||_|j}t ||_t ||_tj||j|_ |jyr) rerfr^rgr]rurencoderrrrpost_layernorm post_initrs r&rfzBlipVisionModel.__init__s]   && .v6"6*  ll9&:O:OP r( output_typer rrrrrc||n|jj}||n|jj}||n|jj}| t d|j ||}|j ||||}|d}|j|}|dddddf} |j| } |s || f|ddzSt|| |j|jS)z Returns: Nz You have to specify pixel_values)r)rrrrrr)r3 pooler_outputr4r5) r^rrrrrur+r,rr4r5) r;rrrrrr4encoder_outputsr3 pooled_outputs r&rzBlipVisionModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj &1%r?main_input_namerr rfrBLIP_VISION_INPUTS_DOCSTRINGrrrr#rArrrrr5rrs@r&r)r)s$O#L / ++GH+ETde59,0/3&*). , u001, $D>, 'tn , d^ , #' ,  u00 1, fI, \r(r)z This model is going to be deprecated in future versions. Please use `BlipForConditionalGeneration`, `BlipForQuestionAnswering` or `BlipForImageTextRetrieval` depending on your usecase. ceZdZeZdeffd ZdZdZee  dde e jde e jde e jde e d e jf d Zee dd e e jde e d e d e jfd Zee dde e j&d e e jde e jde e d e d e jf dZeeeee dde e j&d e e jde e jde e j&de e de e de e de e d e d eeeffdZxZS) BlipModelr^czt||t|jts"t dt |jdt|jts"t dt |jd|j}|j}|j|_ |j|_ |j|_ t||_t||_t#j$|j|jd|_t#j$|j|jd|_t#j*t-j.|j0j2|_t6j9d|j;y)NzKconfig.text_config is expected to be of type BlipTextConfig but is of type .zOconfig.vision_config is expected to be of type BlipVisionConfig but is of type F)rz`BlipModel` is going to be deprecated in future release, please use `BlipForConditionalGeneration`, `BlipForQuestionAnswering` or `BlipForImageTextRetrieval` depending on your usecase.)rerfr text_configr TypeErrortyperrprojection_dimrgtext_embed_dimvision_embed_dimr text_modelr) vision_modelrrvisual_projectiontext_projectionrkr#rr^logit_scale_init_value logit_scaleloggerwarningr-)r;r^r=rrts r&rfzBlipModel.__init__sh  &,,n=++,-Q0  &..0@A--./q2  (( ,, $33)55 - 9 9' 4+M:!#4+@+@$BUBU\a!b!yy)<)Q>QX]^<< T[[5W5W(XY G r(c6|jjSr)rCr5r:s r&r5zBlipModel.get_input_embeddings's3355r(c:|jj|yr)rCset_input_embeddingsr;values r&rMzBlipModel.set_input_embeddings*s ,,U3r(rrrrrc||n|jj}|j||||}|d}|j|}|S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`BlipTextModel`]. Examples: ```python >>> from transformers import AutoProcessor, BlipModel >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```)rrrrr)r^rrCrF)r;rrrr text_outputsr3 text_featuress r&get_text_featureszBlipModel.get_text_features-sZ0&1%>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipModel >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```)rrrr)r^rrDrE)r;rrrvision_outputsr3image_featuress r&get_image_featureszBlipModel.get_image_featuresSsZ:&1%r(c4||n|jj}|j|dd||}|d}tj|j ddtj }|j|||||} | d} |j| } | S) a Returns: multimodal_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The multimodal embeddings obtained by applying the image embeddings to the text encoder using the cross-attention mechanism. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipModel >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["a photo of a cat", "a photo of a dog"] >>> inputs = processor(images=image, text=texts, padding=True, return_tensors="pt") >>> multimodal_features = model.get_multimodal_features(**inputs) ```NTrrrrrrryrrrencoder_hidden_statesencoder_attention_maskrr) r^rrDr#onesr{longrCrF) r;rrrrrrUr2 image_attsrQr3multimodal_featuress r&get_multimodal_featuresz!BlipModel.get_multimodal_features}s<&1%>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipModel >>> model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```NrY)rrrrrrrrdryT)prkeepdimr)r r)r1rNrOrPr2rQrR)r^rrrrDrCrErFnormrHexprr rr#rr+r.rM)r;rrrrrbrrrrrUrQr2rPrHrOrNr1rs r&rzBlipModel.forwardsN2C1N-TXT_T_TqTq$8$D $++JjJj &1%r?rr rfr5rMrBLIP_TEXT_INPUTS_DOCSTRINGrr#rrrArSr8rWBLIP_INPUTS_DOCSTRINGrrarrMrrrrrs@r&r:r:sL"z"H64++EF-115/3&* #ELL)#!.#u||, # d^ #    #G#J++GH59&*). 'u001'd^'#' '    'I'R++@A154815&*). 4#E,,-4#u0014#!. 4# d^ 4# #' 4#   4#B4#l++@A:JO15481537&*,0/3&*).\ E,,-\ u001\ !. \ u//0 \ d^ \ $D>\ 'tn\ d^\ #'\  uj !\ PB\ r(r:a BLIP Model for image captioning. The model consists of a vision encoder and a text decoder. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the text decoder continue the prompt. Otherwise, the decoder starts generating text from the [BOS] (beginning-of-sequence) token. will start generating the caption from the text input. If no text input is provided, the decoder will start with the [BOS] token only. ceZdZeZdgZdZdeffd ZdZdZ e e e e e ddej deej$d eej$d eed eed eej$d eededeee ffdZej. ddej deej$d eej$dedej$f dZxZS)BlipForConditionalGeneration)text_decoder.cls.predictions.decoder.biasrr^ct||t|j|_t |j |_|j j|_ |j j|_ |jyr) rerfr)rrDrr= text_decoder bos_token_iddecoder_input_ids pad_token_iddecoder_pad_token_idr-rss r&rfz%BlipForConditionalGeneration.__init__#sj  +F,@,@A/0B0BC!'!3!3!@!@$*$6$6$C$C! r(c6|jjSr)ror5r:s r&r5z1BlipForConditionalGeneration.get_input_embeddings0  5577r(c:|jj|yr)rorMrNs r&rMz1BlipForConditionalGeneration.set_input_embeddings3 ..u5r(r.rrrrlabelsrrrc ||n|jj}||n|jj}||n|jj}|j |||||} | d} |j ||| ||d} |s4| | d| dfn| df} | | | df| ddzz } t d| DSt| j| j| | j| j| j S) a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipForConditionalGeneration >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "A picture of" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) ```NrYrr)rrr[rxr reductionrrdc3&K|] }|| ywrrDrWrs r&rYz7BlipForConditionalGeneration.forward..uLF9KLr)r1rr2r3r4r5) r^rrrrDrorZr0r1rr3r4r5) r;rrrrrrxrrrUr2rs r&rz$BlipForConditionalGeneration.forward6s8F&1%!";MM MGLgLL L6>>%,>>(66%00   r(c 4|jd}|j||}|d}tj|j ddtj |j } t|trtj|}nk|itj|j|jjjggj|dj|j }|jjj |dddf<| |ddddfnd}|j"j$d|ddddf|jjj&|jjj(||| d|} | S) a Overrides *generate* function to be able to use the model as a conditional generator Parameters: pixel_values (*torch.FloatTensor* of shape *(batch_size, num_channels, image_height, image_width)*: Input image to be processed input_ids (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): The sequence used as a prompt for the generation. attention_mask (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipForConditionalGeneration >>> model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model.generate(**inputs) >>> print(processor.decode(outputs[0], skip_special_tokens=True)) two cats sleeping on a couch ``` rrrNryrr r)r eos_token_idrrrr[r\rD)rrDr#r]r{r^r rlistrrqr^r=rrepeatrrprogenerate sep_token_idrr) r;rrrrgenerate_kwargsrrUr2image_attention_maskrs r&rz%BlipForConditionalGeneration.generates}R"''* **%%=+  &a( $zz,*;*;*=cr*B%**]i]p]pq i &((3I    4#9#94;;;R;R;_;_"`!ab A&L''(  ++11>> !Q$3A3M3B3/SW,$##,, 3B3'00==00==)".#7   r()NNNNNNFrh)r=r>r?rr _tied_weights_keysr7rfr5rMrr8rr0rr#rArrrrrrno_gradrrrs@r&rlrlsLEF$O z 86++GH+Raqr1559,0/3-1&*).F ''F E,,-F !!1!12 F $D> F 'tn F ))*F d^F #'F  u== >F sIF PU]]_1559). H''HE,,-H!!1!12 H #' H   HHr(rlaS BLIP Model for visual question answering. The model consists of a vision encoder, a text encoder as well as a text decoder. The vision encoder will encode the input image, the text encoder will encode the input question together with the encoding of the image, and the text decoder will output the answer to the question. ceZdZeZdgZdeffd ZdZdZe e e e e  ddejdej d eejd eejd eejd eed eedeejdeededeee ffdZej, ddejdej d eejdedejf dZxZS)BlipForQuestionAnsweringrmr^cPt||t|j|_t |j d|_t|j |_ |j j|_ |j j|_ |jy)NFadd_pooling_layer)rerfr)rrDrr= text_encoderrrorrrsrpdecoder_start_token_idr-rss r&rfz!BlipForQuestionAnswering.__init__s  +F,@,@A)&*<*>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipForQuestionAnswering >>> model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # training >>> text = "How many cats are in the picture?" >>> label = "2" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> labels = processor(text=label, return_tensors="pt").input_ids >>> inputs["labels"] = labels >>> outputs = model(**inputs) >>> loss = outputs.loss >>> loss.backward() >>> # inference >>> text = "How many cats are in the picture?" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model.generate(**inputs) >>> print(processor.decode(outputs[0], skip_special_tokens=True)) 2 ```NaEither `decoder_input_ids` or `labels` should be passed when calling `forward` with `BlipForQuestionAnswering`. if you are training the model make sure that `labels` is passed, if you are using the model for inference make sure that `decoder_input_ids` is passed or call `generate`rYrryrrZr)rrr[r\rxrrzrdc3&K|] }|| ywrrDr|s r&rYz3BlipForQuestionAnswering.forward..Vr}r)r1r2r3r4r5)rr^rrrrDr#r]r{r^rr3ror1rrZrFr4r5)r;rrrqrrrrrxrrrUr2rrK answer_output decoder_lossrs r&rz BlipForQuestionAnswering.forwardsb >/7u  &1%!3DEWXWYHZZGLgLL L(%,>>(66%00   r(c |j||}|d}tj|jddtj|j }t |trtj|}|j||||d} | d} tj| jddtj| j } tj| jddf|j| j } |jjd | |jjj |jjj"| | d |} | S) aD Overrides *generate* function to be able to use the model as a conditional generator Parameters: input_ids (*torch.LongTensor* of shape *(batch_size, sequence_length)*): The sequence used as a prompt for the generation. pixel_values (*torch.FloatTensor* of shape *(batch_size, num_channels, image_height, image_width)*: Input image to be processed attention_mask (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`. `1` for tokens that are NOT MASKED, `0` for MASKED tokens. **generate_kwargs: Additional arguments passed to the *generate* function of the decoder Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipForQuestionAnswering >>> model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are in the picture?" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model.generate(**inputs) >>> print(processor.decode(outputs[0], skip_special_tokens=True)) 2 ``` rrNryrFrZr) fill_valuer )rrrrr[r\rD)rDr#r]r{r^r rrrrfullrrorr^r=rrr)r;rrrrrrUr2rquestion_outputsrKquestion_attention_maskbos_idsrs r&rz!BlipForQuestionAnswering.generate`shX**%%=+  &a( $zz,*;*;*=cr*B%**]i]p]pq i &((3I,,)".#7 - +1-"'**  "3B 'uzz/BXBX# **  ! !! $a (T5P5PYhYoYo -$##,, 00==00=="1#:    r()NNNNNNNFr)r=r>r?rr rrfrMr5rr8rrFrr#rrArrrrrrrrrs@r&rrsLEF z 68++GH+DScd 9==A59,0/3-1&*).n ##n ''n $E$4$45 n !))9)9 : n !!1!12 n $D>n 'tnn ))*n d^n #'n  u// 0n eIn `U]]_ 6:). R##R''R!!1!12 R #' R   RRr(ra BLIP Model with a vision and text projector, and a classification head on top. The model is used in the context of image-text retrieval. Given an image and a text, the model returns the probability of the text being relevant to the image. ceZdZeZdeffd ZdZdZee e e e  dde jde jdeed ee jd eed eed eed edeee ffdZxZS)BlipForImageTextRetrievalr^ct||t|j|_t |j d|_tj|jj|j|_ tj|j j|j|_ tj|j jd|_t|ds|j j n |j"|_t|ds|j j$n |j&|_|j)y)NFrrdrsr)rerfr)rrDrr=rrrrgimage_text_hidden_size vision_proj text_projitm_headrrrrsrprr-rss r&rfz"BlipForImageTextRetrieval.__init__s  +F,@,@A)&*<*>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, BlipForImageTextRetrieval >>> model = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-itm-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "an image of a cat" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) ``` NrYrryrrZ)rrrr~rdc3&K|] }|| ywrrDr|s r&rYz4BlipForImageTextRetrieval.forward..2r}r)rIr3r4r5rK)r^rrrrDr#r]r{r^rr3rr rrr+rZrHr4r5)r;rrrrrrrrrUr2r_rKr image_feat text_featrs r&rz!BlipForImageTextRetrieval.forwardsD&1%>(66%00+   r()TNNNNF)r=r>r?rr rfr5rMrr8rrFrr#rrArrrrrrrs@r&rrsLz<86++GH+DScd (,59,0/3&*).S ##S ''S tn S !!1!12 S $D> S 'tnS d^S #'S  u// 0S eIS r(r)r:rrlrr)rr)Br@r7 dataclassesrtypingrrrrr#torch.utils.checkpointrtorch.nn.functionalr activationsr generationr modeling_outputsr rmodeling_utilsrutilsrrrrrrconfiguration_bliprrrmodeling_blip_textrr get_loggerr=rI_CHECKPOINT_FOR_DOCrr'r.r0rFrHrMModuler]rrrrrBLIP_START_DOCSTRINGrir8rjrr)r:rlrr__all__rDr(r&rsw!.. )!)K-MLB   H %0`U\\`ell` -%,,-5<<-  )k) )X ? ? ?@ '?{'? '?T ! !  ! HG299GV%%PBBIIBL bii -ryy-`'%/'%T > "$NS "))S l@)@F  R #R  R j k#6kk\  _2O__D  | 3| | ~ r(