S^(h$dZddlmZddlZddlmZddlmZmZm Z m Z ddl Z ddl mZmZddlmZmZmZmZmZmZmZdd lmZmZdd lmZmZmZmZm Z d d l!m"Z"m#Z#m$Z$d d l%m&Z&m'Z'm(Z(ejRe*Z+dZ,d:dZ-d;dZ.eGddeZ/eGddeZ0eGddeZ1eGddeZ2GddejfjhZ5GddejfjhZ6GddejfjhZ7Gdd ejfjhZ8Gd!d"ejfjhZ9Gd#d$eZ:d%Z;d&ZGd*d+e:Z?Gd,d-ejfjhZ@Gd.d/e:ZAed0e;Gd1d2e:ZBed3e;Gd4d5e:ZCed6e;Gd7d8e:ZDgd9ZEy)s/#J/L!",,z":;J : % ,,r.cveZdZUdZdZded<dZded<dZded<dZded <dZ d ed <dZ d ed <e d Z y))TFBlipForConditionalGenerationModelOutputa Adapted from the base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. This class also adds the loss term from the text decoder. Args: loss (`tf.Tensor`, *optional*, returned when `labels` is provided, `tf.Tensor` of shape `(1,)`): Languge modeling loss from the text decoder. logits (`tf.Tensor` of shape `(batch_size, sequence_length, config.vocab_size)`, *optional*): Prediction scores of the language modeling head of the text decoder model. image_embeds (`tf.Tensor` of shape `(batch_size, output_dim)`, *optional*): The image embeddings obtained after applying the Vision Transformer model to the input image. last_hidden_state (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.` NTuple[tf.Tensor] | Nonelossr+tf.Tensor | None image_embedsOptional[tf.Tensor]last_hidden_stateTuple[tf.Tensor, ...] | None hidden_states attentionscNtjdt|jS)Nz`decoder_logits` attribute is deprecated and will be removed in version 5 of Transformers. Please use the `logits` attribute to retrieve the final output instead.)warningswarn FutureWarningr+selfs r,decoder_logitsz8TFBlipForConditionalGenerationModelOutput.decoder_logitsgs#  W  {{r.) __name__ __module__ __qualname____doc__r8__annotations__r+r:r<r>r?propertyrFr.r,r6r6Ds^4%)D !(&*F #*%)L")-1*126M/6/3J,3 r.r6cXeZdZUdZdZded<dZded<dZded<dZded <dZ ded <y) TFBlipTextVisionModelOutputa Adapted from the base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. This class also adds the loss term from the text decoder. Args: loss (`tf.Tensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Languge modeling loss from the text decoder. image_embeds (`tf.Tensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr9r8r:r;r<r=r>r?) rGrHrIrJr8rKr:r<r>r?rMr.r,rOrOqsA0"D !%)L")-1*126M/6/3J,3r.rOceZdZUdZdZded<dZded<dZded<dZded<dZ d ed <dZ ded <dZ d ed <dZ d ed<y)"TFBlipImageTextMatchingModelOutputa9 Adapted from the base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. This class also adds the loss term from the text decoder as well as the image-text similarity scores. Args: itm_score (`tf.Tensor`): The image-text similarity scores. loss (`tf.Tensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Languge modeling loss from the text decoder. image_embeds (`tf.Tensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(tf.Tensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `tf.Tensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. vision_pooler_output (`tf.Tensor` of shape `(batch_size, hidden_size)`, *optional*): Last layer hidden-state of the vision of the vision-only branch of the model. attentions (`tuple(tf.Tensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `tf.Tensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. question_embeds (`tf.Tensor`): The question embeddings obtained by the text projection layer. Nr9 itm_scorer8r:r;r<r=r>vision_pooler_outputr?r7question_embeds) rGrHrIrJrRrKr8r:r<r>rSr?rTrMr.r,rQrQsc>#'I&!D !%)L")-1*126M/6-1*1/3J,3/3O,3r.rQc|eZdZUdZdZded<dZded<dZded<dZded<dZ ded <dZ d ed <dZ d ed <dd Z y) TFBlipOutputa Args: loss (`tf.Tensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image:(`tf.Tensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text:(`tf.Tensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds(`tf.Tensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`BlipTextModel`]. image_embeds(`tf.Tensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`BlipVisionModel`]. text_model_output(`BaseModelOutputWithPooling`): The output of the [`BlipTextModel`]. vision_model_output(`BaseModelOutputWithPooling`): The output of the [`BlipVisionModel`]. Nr9r8r;logits_per_imagelogits_per_text text_embedsr:r text_model_outputvision_model_outputcHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw))rZr[N)getattrto_tuple).0krEs r, z(TFBlipOutput.to_tuple..s=  LLDGRYZ^`aRbRkRkRm m s-0)tuplekeysrDs`r,r_zTFBlipOutput.to_tuples# YY[   r.)returnz Tuple[Any]) rGrHrIrJr8rKrWrXrYr:rZr[r_rMr.r,rVrVs^("D !,0)0+/O(/'+K$+(,L%,6:3:8<5< r.rVc0eZdZdfd ZddZddZxZS)TFBlipVisionEmbeddingsc t|di|||_|j|_|j |_|j |_tjj|j|j |j t|jjdd|_ |j |j zdz|_ |jdz|_y)N channels_lastpatch_embedding)filters kernel_sizestrideskernel_initializer data_formatnamerrM)super__init__config hidden_size embed_dim image_size patch_sizerlayersConv2Dr initializer_rangerj num_patches num_positionsrErtkwargs __class__s r,rszTFBlipVisionEmbeddings.__init__s "6" ++ ++ ++$||22NNOO.t{{/L/LM'" 3 !OOt>1D!--1r.c"|jdd|jft|jjdd|_|jd|j |jft|jjdd|_|jryd|_t|ddPtj|jj5|jjgddddyy#1swYyxYw)NrTclass_embeddingshape initializer trainablerpposition_embeddingrj)NNNr ) add_weightrvr rtr{rr}rbuiltr^r% name_scoperjrpbuildrE input_shapes r,rzTFBlipVisionEmbeddings.builds#a(' (E(EF" / #'//d(($..9' (E(EF% #2#  ::  4*D 1 =t33889 B$$**+@A B B > B Bs DDctj|d}tj|d}|j|}tj|||j df}tj |j|d|jf}tj||gd}||jdddtj|dddfz}|S)Nr)rrqr rpermraxis) r%rr0rjreshaper| broadcast_torrvconcatr)rE pixel_values batch_size patch_embeds class_embeds embeddingss r,callzTFBlipVisionEmbeddings.call sXXl+A. ||L|D ++L9 zz,T=M=Mr0RS t';';j!T^^=\] YY l;!D $"9"9!=Vrxx ?STU?V=VXY:Y"ZZ r.rtrN)r tf.TensorrerrGrHrIrsrr __classcell__rs@r,rgrgs2&B, r.rgcLeZdZdfd Zddfd Z d ddZxZS) TFBlipTextEmbeddingsc Tt|di||j|_||_y)NrM)rrrsrurvrtr~s r,rszTFBlipTextEmbeddings.__init__s' "6"++ r.ctjd5|j|jj|j ft |jj|jjzdd|_ dddtjd5|j|jj|j ft |jj|jjzdd|_ dddt|5|y#1swYxYw#1swY%xYw)Ntoken_embeddingTweightrrr)r%rrrt vocab_sizervr initializer_factorr{rmax_position_embeddingsrrrr)rErrs r,rzTFBlipTextEmbeddings.build#s ]], - //{{--t~~>+DKK,J,JT[[MjMj,jk *DK ]]/ 0 &*oo{{::DNNK+DKK,J,JT[[MjMj,jk! '6'D #   k"!    sA/D*#A/D6*D36D?c| | td|At||jjt j |j |}t|dd}|/t jt jd|dd}t j |j|}t j||dddf }||z}|S) z Applies embedding based on inputs tensor. Returns: final_embeddings (`tf.Tensor`): output embedding tensor. Nz5You have to specify either input_ids or inputs_embeds)paramsindicesrr)startlimitrr)input multiples) ValueErrorrrtrr%gatherrr expand_dimsr*rtile)rE input_ids position_ids inputs_embedsrposition_embedsfinal_embeddingss r,rzTFBlipTextEmbeddings.call6s  !6TU U  *9dkk6L6L MIIT[[)LM /4  >>"((+b/*RYZ))4+B+BLY''KPQNTUWXCYZ(?:r.)rtrr)rztf.TensorShape)NNN)rr;rr;rr;rerrrs@r,rrsC#**.,0-1  & * +    r.rcLeZdZdZfdZ d ddZddZxZS)TFBlipAttentionz=Multi-headed attention from 'Attention Is All You Need' paperc t|d i|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ tjj|jd|_tjjd|jzt!|j"d |_tjj|jt!|j"d |_y) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).dropoutrpr qkvrnrp projectionrM)rrrsrtrurvnum_attention_heads num_headshead_dimrscalerryDropoutattention_dropoutrDenser r{rrr~s r,rszTFBlipAttention.__init__Xs/ "6" ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( ||++F,D,D9+U <<%%  ?6C[C[3\ch&  ,,,, NNv?W?W/X_k- r.c`t|\}}}|j|}tj|||d|j|j f}tj |d}|d|d|d} } } | tj | dz} | |jz} t| d } |j| | } || |z} tj | | zd }t|d d |jgz}tj||}|j|}|r|| f}|S|d f}|S)z#Input shape: Batch x Time x Channelr )rqrr rrrrrq)rrr rqrr)trainingN)rrqrr ) rrr%rrrr0rrrrvr)rEr> head_maskoutput_attentionsrbsztgt_lenrv mixed_qkv query_states key_states value_statesattention_scoresattention_probs context_layernew_context_layer_shapeoutputoutputss r,rzTFBlipAttention.callnsA#-]";WiHH]+ JJy3DNNDMM*Z[ LLA 1:1y|YWX\,j (",,z<*PP+djj8))9C,,,J  - 9O _|%C,W ",]";CR"@DNNCS"S =2IJ //@6?+HNtnr.c|jryd|_t|ddMtj|jj 5|jj ddddt|ddZtj|jj 5|jj dd|jgdddt|dd[tj|jj 5|jj dd|jgdddyy#1swYxYw#1swY~xYw#1swYyxYw)NTrrr) rr^r%rrrprrrvrrs r,rzTFBlipAttention.builds ::  4D ) 5t||001 ) ""4( ) 4 % 1txx}}- =dDNN;< = 4t , 8t334 D%%tT4>>&BC D D 9  ) ) = = D Ds$ D?%)E  )E?E EE )NFN) r>rrr9rOptional[bool]rrrez;Tuple[tf.Tensor, tf.Tensor | None, Tuple[tf.Tensor] | None]r)rGrHrIrJrsrrrrs@r,rrUsSG 2'+,1#' * *$** * ! * E *X Dr.rc0eZdZdfd ZddZddZxZS) TFBlipMLPc t|di|t|j|_|j dzd|j zdzz}d|j zdz}tjj|jt|d|_ tjj|j t|d|_ ||_y)Nrrqfc1)unitsrnrpfc2rM)rrrsr hidden_act activation_fnrunum_hidden_layersrryrintermediate_sizer rrrt)rErtr in_proj_stdfc_stdrs r,rszTFBlipMLP.__init__s "6".v/@/@A))4/Q9Q9Q5QVZ4Z[ f(((T1<<%%**v?V]b& <<%%$$9U\a&  r.cp|j|}|j|}|j|}|S)N)inputs)rrr)rEr>s r,rzTFBlipMLP.calls8 6 **=9  6 r.c"|jryd|_t|dddtj|jj 5|jj dd|jjgdddt|ddetj|jj 5|jj dd|jjgdddyy#1swY|xYw#1swYyxYw)NTrr) rr^r%rrrprrtrurrrs r,rzTFBlipMLP.builds ::  4 % 1txx}}- FdDKK,C,CDE F 4 % 1txx}}- LdDKK,I,IJK L L 2 F F L Ls 3C9<3D9DDrtr)r>rrerrrGrHrIrsrrrrs@r,rrs  Lr.rcHeZdZdfd Z d ddZddZxZS)TFBlipEncoderLayerc Nt|di||j|_t |d|_t jj|jd|_ t|d|_ t jj|jd|_ y)N self_attnr layer_norm1epsilonrpmlp layer_norm2rM)rrrsrurvrrrryLayerNormalizationlayer_norm_epsrrrrr~s r,rszTFBlipEncoderLayer.__init__s "6"++(kB <<::6CXCX_l:mV%0 <<::6CXCX_l:mr.c|}|j|}|j||||\}}||z}|}|j|}|j|}||z}|f}|r||fz }|S)a9 Args: hidden_states (`tf.Tensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`tf.Tensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. `(config.encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r>rrr)rrrr)rEr>attention_maskrrresidual attn_weightsrs r,rzTFBlipEncoderLayer.calls"!((7 &*nn'$/ '5' # | &0  ((7 / %0  "    &Gr.c|jryd|_t|ddMtj|jj 5|jj ddddt|ddZtj|jj 5|jj dd|jgdddt|ddMtj|jj 5|jj ddddt|dd[tj|jj 5|jj dd|jgdddyy#1swY4xYw#1swYxYw#1swYxYw#1swYyxYw)NTrrrr) rr^r%rrrprrrvrrrs r,rzTFBlipEncoderLayer.buildsp ::  4d + 7t~~223 +$$T* + 4 - 9t//445 E  &&dDNN'CD E 4 % 1txx}}- %t$ % 4 - 9t//445 E  &&dDNN'CD E E : + + E E % % E Es0 F%)F& F2&)F>F#&F/2F;>Gr)FN) r>rrrrrrrrezTuple[tf.Tensor]rrrs@r,rrsMn-2#' & &"&* & ! &  &PEr.rceZdZdZeZdZdgZy)TFBlipPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. bliprN)rGrHrIrJr config_classbase_model_prefix_keys_to_ignore_on_load_missingrMr.r,rrs L'6&7#r.ra: This model inherits from [`TFPreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a [keras.Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) subclass. Use it as a regular TF 2.0 Keras Model and refer to the TF 2.0 documentation for all matter related to general usage and behavior. Parameters: config ([`BlipConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~TFPreTrainedModel.from_pretrained`] method to load the model weights. a= Args: pixel_values (`tf.Tensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`BlipImageProcessor`]. See [`BlipImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`tf.Tensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoProcessor`]. See [`BlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) pixel_values (`tf.Tensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`BlipImageProcessor`]. See [`BlipImageProcessor.__call__`] for details. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cbeZdZeZ dfd Ze d ddZddZxZ S) TFBlipEncoderc t|di|||_t|jDcgc]}t |d|c}|_ycc}w)Nz layers_._rrM)rrrsrtr*rrry)rErtrirs r,rszTFBlipEncoder.__init__isI "6" QVW]WoWoQpqA)&1#Gq qsAc||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|} t |j D]*\} } |r|| fz}| | |||} | d} |s"|| dfz},|r|| fz}|st d| ||fDSt| ||S)a Args: inputs_embeds (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NrM)rrrrc3&K|] }|| ywrrM)r`vs r,rbz%TFBlipEncoder.call..seqWXWde)r<r>r?)rtroutput_hidden_statesuse_return_dict enumerateryrcr ) rErrrr return_dictrencoder_statesall_attentionsr>idx encoder_layer layer_outputss r,rzTFBlipEncoder.callns<2C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]NN$See e +>Vd  r.c|jryd|_t|ddK|jD];}tj|j 5|j dddd=yy#1swYIxYw)NTry)rr^ryr%rrpr)rErlayers r,rzTFBlipEncoder.buildsp ::  44 ( 4 &]]5::.&KK%&& & 5&&s A..A7 rNNNNN) rr9rrrrrrrrrezUnion[Tuple, TFBaseModelOutput]r) rGrHrIrrrsrrrrrs@r,rr]szLr ,0,0/3&*#'= )= * = - = $ = !=  )= = ~&r.rceZdZdZeZdfd Zd dZee e e e e d d dZ dZd dZxZS) TFBlipVisionModelrct||g|i|||_t|d|_t |d|_tjj|jd|_ |j|_ y)Nrrencoderpost_layernormr)rrrsrtrgrrrrryrrrrurvrErtargsrrs r,rszTFBlipVisionModel.__init__sm 1$1&1 0lK$V)< #ll==fF[F[br=s++r.c$|jjrtj|jnd}|jj rtj|j nd}t|j|j||S)Nr< pooler_outputr>r?) rtrr%convert_to_tensorr>rr?r r<r$)rErhsattnss r,serving_outputz TFBlipVisionModel.serving_outputsq;?;;;[;[R ! !&"6"6 7ae;?;;;X;X$$V%6%67^b+$66 ..   r. output_typerc||n|jj}||n|jj}||n|jj}| t d|j |}|j |||||}|d}|j|}|dddddf} |jtj| d} tj| d} |s || f|ddzSt|| |j|jS)z Returns: Nz You have to specify pixel_values)rrrrrrrr#)rtrrrrrrrr%rsqueezer r>r?) rErrrrrr>encoder_outputsr< pooled_outputs r,rzTFBlipVisionModel.calls-2C1N-TXT_T_TqTq$8$D $++JjJj &1%?)E 2D;>E Er)rr rer r) rr9rrrrrrrrrez*Union[Tuple, TFBaseModelOutputWithPooling]r)rGrHrImain_input_namerrrsr(rrBLIP_VISION_INPUTS_DOCSTRINGrr rr0rrrs@r,rrs$O#L,  *+GH+GVfg*.,0/3&*#' / &/ */ - / $ / ! /  4/ hI/ b Hr.rcxeZdZeZdfd ZddZe d ddZxZ S)TFBlipMainLayerct||i|t|jts"t dt |jdt|jts"t dt |jd|j}|j}|j|_ |j|_ |j|_ t|d|_t|d|_t"j$j'|jdt)|j*d |_t"j$j'|jdt)|j*d |_||_y) NzKconfig.text_config is expected to be of type BlipTextConfig but is of type .zOconfig.vision_config is expected to be of type BlipVisionConfig but is of type text_modelr vision_modelFvisual_projection)use_biasrnrptext_projection)rrrs isinstance text_configr TypeErrortype vision_configrprojection_dimrutext_embed_dimvision_embed_dimr r8rr9rryrr r{r:r<rt)rErtr!rr>rArs r,rszTFBlipMainLayer.__init__sh $)&)&,,n=++,-Q0  &..0@A--./q2  (( ,, $33)55 - 9 9)+LI-m.Q!&!3!3   .v/G/GH$ "4"  %||11   .v/G/GH" 2  r.c*|jdgtjj|jj d|_|jryd|_t|ddMtj|jj5|jjddddt|ddMtj|jj5|jjddddt|ddZtj|jj5|jjdd|j gdddt|dd[tj|j"j5|j"jdd|j$gdddyy#1swY4xYw#1swYxYw#1swYxYw#1swYyxYw)N logit_scaleT)rprrrr8r9r:r<)rr initializersConstantrtlogit_scale_init_valuerFrr^r%rr8rprr9r:rDr<rCrs r,rzTFBlipMainLayer.buildAs??**33DKK4V4VW +  ::  4t , 8t334 ,%%d+ , 4 . :t00556 .!!''- . 4,d 3 ?t55::; R&&,,dD$:O:O-PQ R 4*D 1 =t33889 N$$**D$8K8K+LM N N > , , . . R R N Ns0G$0G1 )G=1)H $G.1G:=H Hc ||n|jj}||n|jj}||n|jj}|j ||||| } |j ||||||| } | d} |j | } | d} |j| } | tj| dddz } | tj| dddz } tj|j}tj| | d|z}tj|}d}|r!t|}tj|d }|s||| | | | f}||f|zS|St!|||| | | | S) Nrrrrr)rrrrrrrrrqrT)ordrkeepdims transpose_br)r8rWrXrYr:rZr[)rtrrrr9r8r:r<r%normexprFmatmulr0r4rrV)rErrrr return_lossrrrrvision_outputs text_outputsr:rYrFrXrWr8rs r,rzTFBlipMainLayer.callYs2C1N-TXT_T_TqTq$8$D $++JjJj &1%>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, TFBlipModel >>> model = TFBlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="tf", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = tf.nn.softmax(logits_per_image, axis=1) # we can take the softmax to get the label probabilities ```) rrrrrTrrrr)r) rErrrrrTrrrrrs r,rzTFBlipModel.calls:N))%)%#/!5#  r.c||n|jj}|jj||||}|d}|jj |}|S)a Returns: text_features (`tf.Tensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`TFBlipTextModel`]. Examples: ```python >>> from transformers import AutoProcessor, TFBlipModel >>> model = TFBlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="tf") >>> text_features = model.get_text_features(**inputs) ```)rrrrr)rtrrr8r<)rErrrrrVr. text_featuress r,get_text_featureszTFBlipModel.get_text_featuressd0&1%>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, TFBlipModel >>> model = TFBlipModel.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="tf") >>> image_features = model.get_image_features(**inputs) ```)rrr)rtrrr9r:)rErrrUr.image_featuress r,get_image_featureszTFBlipModel.get_image_featuressX8&1%rk bos_token_iddecoder_input_ids pad_token_iddecoder_pad_token_idr s r,rsz'TFBlipForConditionalGeneration.__init__Jsl 1$1&1-f.B.BX1&2D2D>Z!'!3!3!@!@$*$6$6$C$C!r.cB|jjjSrr9rrjrDs r,r0z3TFBlipForConditionalGeneration.get_input_embeddingsT  ++;;;r.r)c ||n|jj}|j|||||} | d} |j||| |d|} |s'| d| d| | df| ddz} t d| DS| | d} | d} nd} | d} | /| j j dk(rtj| d } t| | | | j| j| j S) a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, TFBlipForConditionalGeneration >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> model = TFBlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "A picture of" >>> inputs = processor(images=image, text=text, return_tensors="tf") >>> outputs = model(**inputs) ```NrKrF)rrencoder_hidden_stateslabelsrrrrqc3&K|] }|| ywrrMr`rs r,rbz6TFBlipForConditionalGeneration.call..LF9KLrrP)r8r+r:r<r>r?) rtrr9rkrcrrankr%rr6r<r>r?)rErrrrrrurrrUr:rr8r+s r,rz#TFBlipForConditionalGeneration.callWs.H&1%>(66%00   r.c |jd}|j|}|d}tjt |ddtj }t |tr&tj|tj }nj|htj|j|jjjggtj }tj||df}tjtj|dftj |jjjz|ddddfgd}| |ddddfnd}|j j"d |ddddf|jjj$|jjj&|||d|} | S) ay Overrides *generate* function to be able to use the model as a conditional generator Parameters: pixel_values (`tf.Tensor` of shape `(batch_size, num_channels, image_height, image_width)`: Input image to be processed input_ids (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): The sequence used as a prompt for the generation. attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, TFBlipForConditionalGeneration >>> model = TFBlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="tf") >>> outputs = model.generate(**inputs) >>> print(processor.decode(outputs[0], skip_special_tokens=True)) two cats sleeping on a couch ``` rrNrdtyperr)r eos_token_idrnrrtencoder_attention_maskrM)rr9r%onesrint32r=listr%rmrtr>r~rrrlrkgenerate sep_token_idrn) rErrrgenerate_kwargsrrUr:image_attention_maskrs r,rz'TFBlipForConditionalGeneration.generatesN"''* ** *E%a( !wwz,'?'DBHHU i &,,YbhhGI  ,,(($++*A*A*N*NOPXZX`X`I J?;III WWj!_BHH 5 8O8O8\8\ \^ghiklkmhm^n ovw 4B3M3B3/SW,$##,, 3B3'00==00==)".#7   r.c|jryd|_t|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swYexYw#1swYyxYw)NTr9rk)rr^r%rr9rprrkrs r,rz$TFBlipForConditionalGeneration.builds ::  4 . :t00556 .!!''- . 4 . :t00556 .!!''- . . ; . . . .s C %C CC rrezkeras.layers.Layer)NNNNNNN)rrrr9rr9rrrrrur9rrrrrez7Union[Tuple, TFBlipForConditionalGenerationModelOutput]rf)rrrr9rr9rerr)rGrHrIrrrr2rsr0rrr3rr6rrrrrs@r,riri<sL'S&T#$OD<*+GH+Tcmn'++/,0/3#'&*#'J J $J ) J * J - J !J $J !J  AJ oIJ ^'++/ GG$G) G  GR .r.riaS BLIP Model for visual question answering. The model consists of a vision encoder, a text encoder as well as a text decoder. The vision encoder will encode the input image, the text encoder will encode the input question together with the encoding of the image, and the text decoder will output the answer to the question. ceZdZeZdgZd fd Zd dZdZe e e e e e d d dZ d ddZd dZxZS)TFBlipForQuestionAnsweringr[cBt||g|i|t|jd|_t |j dd|_t|j d|_ |j j|_ |j j|_ y)Nr9r text_encoderFrpadd_pooling_layerrk)rrrsrrAr9r r>rrrkrnrorldecoder_start_token_idr s r,rsz#TFBlipForQuestionAnswering.__init__s 1$1&1-f.B.BX+F,>,>^glm1&2D2D>Z$*$6$6$C$C!&,&8&8&E&E#r.cB|jjjSrrqrDs r,r0z/TFBlipForQuestionAnswering.get_input_embeddingsrrr.c F|j}|j}|| tdtjt |ddf|}tj ||j}tj||ddddfgd}tj|dk(tj tjt |||j|}tjj|tjd|j|S)Nz8decoder_start_token_id and pad_token_id must be defined!rrrir|) rrorr%fillrcastr}rwhere debuggingassert_greater_equalconstant)rErrrn start_tokensshifted_input_idss r, _shift_rightz'TFBlipForQuestionAnswering._shift_rights!%!TU ww|Y__= II|Yq#2#v5F&GLHH  % GGBGGJ'89<HJ[JaJa b   ))*;R[[RcRiRi=jk  r.r)c | | td| | n|jj} |j|||| | } | d} t j t | ddtj} |j||| | | | }| s|dn |j}|||}|j|||||| | }|:| rt j|jnt j|d}nd}| s || | df| d dz}td |DSt|| | j| j| j S) a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, TFBlipForQuestionAnswering >>> model = TFBlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # training >>> text = "How many cats are in the picture?" >>> label = "2" >>> inputs = processor(images=image, text=text, return_tensors="tf") >>> labels = processor(text=label, return_tensors="tf").input_ids >>> inputs["labels"] = labels >>> outputs = model(**inputs) >>> loss = outputs.loss >>> # inference >>> text = "How many cats are in the picture?" >>> inputs = processor(images=image, text=text, return_tensors="tf") >>> outputs = model.generate(**inputs) >>> print(processor.decode(outputs[0], skip_special_tokens=True)) 2 ```Na Either `decoder_input_ids` or `labels` should be passed when calling `TFBlipForQuestionAnswering`. if you are training the model make sure that `labels` is passed, if you are using the model for inference make sure that `decoder_input_ids` is passed or call `generate`rKrrr|rrrtrrr)rrrtrrurrrqc3&K|] }|| ywrrMrws r,rbz2TFBlipForQuestionAnswering.call..rxr)r8r:r<r>r?)rrtrr9r%rrint64rr<rkr'r8rcrOr>r?)rErrrmdecoder_attention_maskrrrrurrrUr:rrT answer_output decoder_lossrs r,rzTFBlipForQuestionAnswering.call.sb >/7u  &1%>-*<*<=RTR`R`anopaqRrLL#\>!3DEWXWYHZZGLgLL L*%,>>(66%00   r.c |j|}|d}tjt|ddtj}t |t rtj|}|j||||d}|d} tjt| ddtj} tjtj| ddftj|j|j } |jjd | |j j"j$|j j"j&| | d |} | S) aC Overrides *generate* function to be able to use the model as a conditional generator Parameters: input_ids (`tf.Tensor` of shape `(batch_size, sequence_length)`): The sequence used as a prompt for the generation. pixel_values (`tf.Tensor` of shape `(batch_size, num_channels, image_height, image_width)`: Input image to be processed attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`. `1` for tokens that are NOT MASKED, `0` for MASKED tokens. generate_kwargs (dict, *optional*): Additional arguments passed to the `generate` function of the decoder Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, TFBlipForQuestionAnswering >>> model = TFBlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are in the picture?" >>> inputs = processor(images=image, text=text, return_tensors="tf") >>> outputs = model.generate(**inputs) >>> print(processor.decode(outputs[0], skip_special_tokens=True)) 2 ``` r{rNrr|F)rrrtrrr)value)rr~rnrtrrM)r9r%rrrr=rTensorrrrrrr}rkrrtr>rrn) rErrrrrUr:rquestion_outputsrTquestion_attention_maskbos_idsrs r,rz#TFBlipForQuestionAnswering.generatesMT** *E%a( !wwz,'?'DBHHU i & ),I,,)".#7 - +1-"$''*_*Ecr*JRTRZRZ"['' XXo &q )1 -RWWT=X=XZcZiZi5j -$##,, 00==00=="1#:    r.c|jryd|_t|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddNtj|jj 5|jj ddddyy#1swYxYw#1swYqxYw#1swYyxYw)NTr9rrk) rr^r%rr9rprrrkrs r,rz TFBlipForQuestionAnswering.builds  ::  4 . :t00556 .!!''- . 4 . :t00556 .!!''- . 4 . :t00556 .!!''- . . ;  . . . . . .s$ D%%D1?D=%D.1D:=ErrrW)rrrr9rmr9rr9rr9rrrrrur9rrrrrez)Union[Tuple, TFBlipTextVisionModelOutput]r)rrrrrr9rer)rGrHrIrrrrsr0rrrr3rrOrrrrrrs@r,rrs*L'S&T# F<!.*+GH+FUef*..237+/,0/3#'&*#'j j 'j , j !1 j ) j *j -j !j $j !j  3j gIj `,0 LL L) L  L\ .r.ra BLIP Model with a vision and text projector, and a classification head on top. The model is used in the context of image-text retrieval. Given an image and a text, the model returns the probability of the text being relevant to the image. ceZdZeZdfd ZddZeee e e e  d d dZ d dZxZS) TFBlipForImageTextRetrievalct||g|i|t|jd|_t |j dd|_tjj|jt|jd|_tjj|jt|jd|_tjjd t|jd |_t#|d s|j j$n |j&|_t#|d s|j j(n |j*|_||_y) Nr9rrFr vision_projr text_projrqitm_headror)rrrsrrAr9r r>rrryrimage_text_hidden_sizer r{rrrhasattrrnrorlrrtr s r,rsz$TFBlipForImageTextRetrieval.__init__sM 1$1&1-f.B.BX+F,>,>^glm!<<--  ) ).v/G/GH. ++  ) ).v/G/GH,  ** /&2J2J"KR\+  6#9:    + +,, !6#;<    + +.. #  r.cB|jjjSrrqrDs r,r0z0TFBlipForImageTextRetrieval.get_input_embeddings+rrr.r)c ||n|jj}|j|||||} | d} tjt | ddtj } |j||| | ||} |s| dn | j} |j| dddddf} |j||||}|s|dn |j}tjj|j| dddddfdd \}}tjj|j|dddddfdd \}}tj||d }|r| }| }n|}|}|s#|| df| ddz|fz}td |DSt!|| j| j"| j$| S)a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, TFBlipForImageTextRetrieval >>> model = TFBlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco") >>> processor = AutoProcessor.from_pretrained("Salesforce/blip-itm-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "an image of a cat" >>> inputs = processor(images=image, text=text, return_tensors="tf") >>> outputs = model(**inputs) ``` NrKrrr|r)rrrrrq)rLrTrNc3&K|] }|| ywrrMrws r,rbz3TFBlipForImageTextRetrieval.call..rxr)rRr<r>r?rT)rtrr9r%rrrrr<rlinalg normalizerrrSrcrQr>r?)rErr use_itm_headrrrrrrUr: image_attsitm_question_embeds itm_outputno_itm_question_embeds image_feat_ text_feat no_itm_outputrrTrs r,rz TFBlipForImageTextRetrieval.call.sF&1%T>f>f  ++D,<,<\!QPQ'=R,SYZac+d Ayy**4>>:PQRTUWXQX:Y+Z`ahj*k 1 *iTJ F1O"F4O~a01N124FF/I[[GLgLL L1,>>(66%00+   r.c|jryd|_t|ddMtj|jj 5|jj ddddt|ddMtj|jj 5|jj ddddt|ddntj|jj 5|jj dd|jjjgdddt|ddntj|jj 5|jj dd|jjjgdddt|ddotj|jj 5|jj dd|jjjgdddyy#1swYxYw#1swYxYw#1swYxYw#1swYxYw#1swYyxYw)NTr9rrrr)rr^r%rr9rprrrrtrArurr>rrs r,rz!TFBlipForImageTextRetrieval.builds ::  4 . :t00556 .!!''- . 4 . :t00556 .!!''- . 4 - 9t//445 \  &&dDKK4M4M4Y4Y'Z[ \ 4d + 7t~~223 X$$dD$++2I2I2U2U%VW X 4T * 6t}}112 W ##T41H1H1T1T$UV W W 7 . . . . \ \ X X W Ws< H<%I ?=I:=I#5=I/<I II #I,/I8rr)NTNNNNN)rrrr9rrrr9rrrrrrrrrez0Union[Tuple, TFBlipImageTextMatchingModelOutput]r)rGrHrIrrrsr0rrr3rrQrrrrrs@r,rrsL$L<*+GH+M\lm*.'++/,0/3&*#'] ] '] % ] ) ] * ] -] $] !]  :] nI] ~Wr.r)rZrrirrr r)r+rrer)r1rrer)FrJ __future__rrA dataclassesrtypingrrrr tensorflowr%modeling_tf_outputsr r modeling_tf_utilsr r rrrrrtf_utilsrrutilsrrrrrconfiguration_bliprrrmodeling_tf_blip_textrrr get_loggerrGlogger_CHECKPOINT_FOR_DOCr-r4r6rOrQrVryLayerrgrrrrrBLIP_START_DOCSTRINGr3rgrrr5rZrirr__all__rMr.r,rs"!..RGMLee   H %0-  ) ) )X 4+4 4@ '4'4 '4T ! ;!  ! H5U\\//5r7 5<<--7 tQDell((QDh L "" LF@E++@EF8-8  "JW&ELL&&W&W&t[H-[H|I ell((I XW&'W&t s.%:s.s.l  t.!6t.t.n  `W"7`W`WF r.