T^(hT/ddlZddlmZddlmZmZmZmZddlZddlm Z ddl m Z ddl m Z ddlmZdd lmZmZmZmZdd lmZdd lmZmZmZdd lmZmZmZmZm Z m!Z!d dl"m#Z#m$Z$ddl%m&Z&m'Z'm(Z(ejRe*Z+eGddeZ,Gdde jZZ.Gdde jZZ/Gdde jZZ0Gdde jZZ1GddeZ2Gdde jZZ3dZ4Gd d!e2Z5Gd"d#e jZZ6Gd$d%e jZZ7Gd&d'e jZZ8Gd(d)e jZZ9Gd*d+e jZZ:Gd,d-e jZZ;Gd.d/e jZZ<Gd0d1e jZZ=Gd2d3e2Z>d4Z?d5Z@ed6e?Gd7d8e2eZAy)9N) dataclass)AnyOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)GenerationMixin)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings torch_int)AutoModelForCausalLMAutoModelForSeq2SeqLM)InstructBlipVideoConfigInstructBlipVideoQFormerConfigInstructBlipVideoVisionConfigceZdZUdZdZeeeje d<dZ eeeje d<dZ eeje d<dZ eeeje d<dZ eeeje d<deefd Zy) 4InstructBlipVideoForConditionalGenerationModelOutputa. Class defining the outputs of [`InstructBlipVideoForConditionalGeneration`]. Args: loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Language modeling loss from the language model. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head of the language model. vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). language_model_outputs (`CausalLMOutputWithPast` or `Seq2SeqLMOutput`): Outputs of the language model. Nlosslogitsvision_outputsqformer_outputslanguage_model_outputsreturncHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw))r&r'r(N)getattrto_tuple).0kselfs ~/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/instructblipvideo/modeling_instructblipvideo.py zPInstructBlipVideoForConditionalGenerationModelOutput.to_tuple..TsB WW Gq!**, - s-0)tuplekeysr0s`r1r-z=InstructBlipVideoForConditionalGenerationModelOutput.to_tupleSs% YY[    )__name__ __module__ __qualname____doc__r$rrtorch FloatTensor__annotations__r%r&r'r(rr-r6r1r#r#;s 04D(5**+ ,315FHU5,,- .526NHU../6:>OXeE$5$567>AEHU5+<+<%=>E %* r6r#ceZdZdeffd Zdej dededej fdZd dejd e dej fd Z xZ S) !InstructBlipVideoVisionEmbeddingsconfigcpt|||_|j|_|j |_|j |_tjtjdd|j|_ tjd|j|j |j |_ |j |j zdz|_|jdz|_tjtjd|j|j|_y)Nrr ) in_channels out_channels kernel_sizestrider)super__init__rA hidden_size embed_dim image_size patch_sizer Parameterr;randnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingr0rA __class__s r1rHz*InstructBlipVideoVisionEmbeddings.__init__]s  ++ ++ ++!||EKK1dnn,MN!yyDOO\`\k\k !OOt>1D!--1"$,,u{{1d>P>PRVR`R`/a"br6 embeddingsheightwidthr)c|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rr rbicubicF)sizemode align_cornersdim)shaperTr;jit is_tracingrLrreshapepermuter functional interpolateviewcat) r0rWrXrYrRrSclass_pos_embedpatch_pos_embedra new_height new_widthsqrt_num_positionss r1interpolate_pos_encodingz:InstructBlipVideoVisionEmbeddings.interpolate_pos_encodingos`!&&q)A- //55a81< yy##%+*F6UZ?** *11!RaR%811!QR%8r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr6 pixel_valuesrpc|j\}}}}|jjj}|j|j |}|j dj dd}|jj|ddj |} tj| |gd} |r|j| ||} n |j} | | ddd| jdddfj |z} | S)Ndtyperrr[r`)rbrQweightrttoflatten transposerOexpandr;rjrprTr]) r0rqrp batch_size_rXrY target_dtype patch_embeds class_embedsrWrTs r1forwardz)InstructBlipVideoVisionEmbeddings.forwards'3'9'9$ Avu++2288 ++LOO,O,OP #++A.88A> ++22:q"EHHV YY l;C #!%!>!>z6SX!Y !%!8!8 "4Q8L*//!:L8La5O"P"S"ST`"aa r6F) r7r8r9r!rHr;Tensorintrpr<boolr __classcell__rVs@r1r@r@\smc<c$&D5<<&D&DUX&D]b]i]i&DP E$5$5 QU bgbnbn r6r@ceZdZdZfdZdej dedefdZ d dej de ej d e e d e ej e ej e e ej ffd Z xZ S) InstructBlipVideoAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ tj|j|_ tj|jd|jzd|_|j retj"t%j&|j}tj"t%j&|j}nd}d}|Qt%j(|t%j*|d|f}tj"||j_tj|j|j|_y) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gr F)bias) requires_grad)rGrHrArIrJnum_attention_heads num_headshead_dim ValueErrorscalerDropoutattention_dropoutdropoutLinearqkvqkv_biasrMr;zerosrj zeros_liker projection)r0rAq_biasv_biasrrVs r1rHz#InstructBlipVideoAttention.__init__sv  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( zz&":":; 99T^^Q-?eL ??\\%++dnn"=>F\\%++dnn"=>FFF  yy&%*:*:6QV*WY_!`aHLL2DHHM))DNNDNNCr6tensorseq_lenbszc|j|||j|jjddj S)Nrr)rirrrx contiguous)r0rrrs r1_shapez!InstructBlipVideoAttention._shapes7{{3GQQRSUVWbbddr6 hidden_states head_maskoutput_attentionsr)c|j\}}}|j|}|j||d|j||jzj ddddd}|d|d|d} } }t j || jdd} | |jz} tjj| d} |j| } || |z} t j | | j dddd} | jd d|jfz}| j|} |j| }|r|| f}|S|d f}|S) z#Input shape: Batch x Time x Channelr rrrr[r`N)r]rrerrfr;matmulrxrrrgsoftmaxrrJr)r0rrrrtgt_lenrJ mixed_qkv query_states key_states value_statesattention_scoresattention_probs context_layernew_context_layer_shapeoutputoutputss r1rz"InstructBlipVideoAttention.forwardsu#0"4"4"6WiHH]+ %%c7At~~yTXTbTbGbckk q!Q 2;1y|YWX\,j !<< j6J6J2r6RS+djj8--//0@b/I,,7  - 9O _lCKKAqRSUVW "/"4"4"6s";t~~>O"O%--.EF //@6?+HNtnr6NF)r7r8r9r:rHr;rrrrrrrrrs@r1rrsGD<eU\\eCece -1,1 *||*ELL)*$D> * u||Xell3XeELL>Q5RR S *r6rcVeZdZfdZdej dej fdZxZS)InstructBlipVideoMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yN) rGrHrAr hidden_act activation_fnrrrIintermediate_sizefc1fc2rUs r1rHzInstructBlipVideoMLP.__init__sd  #F$5$5699V//1I1IJ99V55v7I7IJr6rr)cl|j|}|j|}|j|}|Sr)rrrr0rs r1rzInstructBlipVideoMLP.forwards4/ **=9 / r6r7r8r9rHr;rrrrs@r1rrs$KU\\ellr6rc eZdZdeffd Z ddej dej deede ejfdZ xZ S) InstructBlipVideoEncoderLayerrAcDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yNeps) rGrHrIrJr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rUs r1rHz&InstructBlipVideoEncoderLayer.__init__sm ++3F;<<F $ u  ! $r6rc*eZdZdZeZdZdZgdZdZ y) InstructBlipVideoPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. blipT)"InstructBlipVideoQFormerEmbeddingsr*InstructBlipVideoQFormerMultiHeadAttention"InstructBlipVideoQFormerSelfOutputcd|jj}t|tjs4t|tj st|tj rc|jjjd|t|dr0|j$|jjjt|trt|jdr:t|jts |jjj}tj j#|j$d|tj j#|j&d|yt|tj(rJ|jjj|jjj+dyt|tj r2|j%|jjjyyy)zInitialize the weightsg)meanstdrN vision_config?)rAinitializer_range isinstancerrP Embeddingrrudatanormal_hasattrrzero_r@r!rinit trunc_normal_rTrOrfill_)r0modulefactors r1 _init_weightsz.InstructBlipVideoPreTrainedModel._init_weightsGsp.. fbii (Jvr||,LPZ[aceclclPm MM   & &CV & <vv&6;;+B   &&( f? @t{{O4Z Ur=s22DD GG ! !&";";#6 ! R GG ! !&"8"8s ! O  - KK   " " $ MM   $ $S )  *v{{/F KK   " " $0G *r6N) r7r8r9r:r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modulesrr>r6r1rr6s' +L&*#%r6rc eZdZdZdeffd Z d deejdee dee dee de e e ff d Z xZS) InstructBlipVideoEncodera" Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipVideoEncoderLayer`]. Args: config (`InstructBlipVideoConfig`): The corresponding vision configuration for the `InstructBlipVideoEncoder`. rAct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) rGrHrAr ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)r0rAr{rVs r1rHz!InstructBlipVideoEncoder.__init__fsQ  mmTYZ`ZrZrTs$tq%B6%J$tu &+#%usA#rroutput_hidden_states return_dictr)c||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|}t |j D]`\} } |r||fz}|j r+|jr|j| j|||} n | |||} | d}|sX|| dfz}b|r||fz}|std|||fDSt|||S)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr>rrrc3&K|] }|| ywrr>r.vs r1r2z3InstructBlipVideoEncoder.forward..seqWXWdes)last_hidden_stater attentions) rArruse_return_dict enumeraterrtraining_gradient_checkpointing_func__call__r3r ) r0 inputs_embedsrrrrencoder_statesall_attentionsridx encoder_layer layer_outputss r1rz InstructBlipVideoEncoder.forwardlsA82C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]NN$See e+>Vd  r6)NNNN)r7r8r9r:rrHrr;rrrrr rrrs@r1rr\s,6,26,0/3&* C !.C $D> C 'tn C d^ C  uo% &C r6ra Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`InstructBlipVideoProcessor`]. See [`InstructBlipVideoProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the pre-trained position encodings. ceZdZdZeZdeffd Zeee e e d de e jde ede ede eded eee ff d Zd ZxZS) InstructBlipVideoVisionModelrqrAct||||_|j}t ||_t ||_tj||j|_ |jyr) rGrHrArIr@rWrencoderrrrpost_layernorm post_init)r0rArJrVs r1rHz%InstructBlipVideoVisionModel.__init__s]   && ;FC/7  ll9&:O:OP r6 output_typerrrrrpr)c||n|jj}||n|jj}||n|jj}| t d|j ||}|j ||||}|d}|j|}|dddddf} |j| } |s || f|ddzSt|| |j|jS)z Returns: Nz You have to specify pixel_values)rp)rrrrrr)r pooler_outputrr) rArrrrrWr r rrr) r0rqrrrrprencoder_outputsr pooled_outputs r1rz$InstructBlipVideoVisionModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj &1%, 'tn , d^ , #' ,  u00 1, sV, \r6r cReZdZdfd ZdZdZdZdZdZ d dZ xZ S) rct|||_|j|jzdk7r0t |ds$t d|j|jfz|j|_t|j|jz |_|j|jz|_ tj|j|j|_ |r_tj|j|j|_tj|j|j|_n^tj|j|j|_tj|j|j|_tj |j"|_t'|dd|_|j(dk(s|j(dk(rF|j*|_tj,d|j*zd z |j|_d |_y) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)position_embedding_typeabsolute relative_keyrelative_key_queryrrF)rGrHrArIrrrrattention_head_size all_head_sizerrqueryencoder_hidden_sizekeyvaluerattention_probs_dropout_probrr,rmax_position_embeddingsrdistance_embeddingsave_attentionr0rAis_cross_attentionrVs r1rHz3InstructBlipVideoQFormerMultiHeadAttention.__init__s     : : :a ?PVXhHi^%%v'A'ABC  $*#=#= #&v'9'9F 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD ##r6c||_yrattn_gradients)r0r/s r1save_attn_gradientsz>InstructBlipVideoQFormerMultiHeadAttention.save_attn_gradients$s ,r6c|jSrr.r5s r1get_attn_gradientsz=InstructBlipVideoQFormerMultiHeadAttention.get_attn_gradients's"""r6c||_yr attention_map)r0r5s r1save_attention_mapz=InstructBlipVideoQFormerMultiHeadAttention.save_attention_map*s *r6c|jSrr4r5s r1get_attention_mapzfz}|j|}|r||fn|f}||fz}|S)Nrrr`rr[rrr rtdevicerszbhld,lrd->bhlrzbhrd,lrd->bhlrr ) r<r%r&r;rjr#rrxrr]arangelongr?rir)r(rvrteinsummathsqrtr!rSoftmaxr*r6 register_hookr0rrfrr")r0rrrencoder_hidden_statesencoder_attention_maskpast_key_valuerr, key_layer value_layermixed_query_layer query_layerr seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_scores_dtyperattention_probs_droppedrrrs r1rz2InstructBlipVideoQFormerMultiHeadAttention.forward5s3$> 11$((;P2QRI33DJJ?T4UVK3N  '11$((=2IJI33DJJ}4MNK >!#4i"@aHI))^A%6 $D!LK11$((=2IJI33DJJ}4MNK JJ}5//0AB #[1!<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q&++-a0J"\\*EJJ}OcOcdiijlnopN"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||AABXY $"5"5  # #O 4  ) )$*B*B C#',,"?  &= &I # %S"S* **,CD 6G=/2mM]^--r6rNNNNNF) r7r8r9rHr0r2r6r8r<rrrs@r1rrs;$8-#+"%"#Tr6rcneZdZfdZdej dej dej fdZxZS)rc(t|tj|j|j|_tj |j|j|_tj|j|_ yr) rGrHrrrIdenserrrhidden_dropout_probrrUs r1rHz+InstructBlipVideoQFormerSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r6r input_tensorr)cr|j|}|j|}|j||z}|Srr[rrr0rr]s r1rz*InstructBlipVideoQFormerSelfOutput.forward7 =1  ]3 }|'CD r6rrs@r1rr1> U\\RWR^R^r6rceZdZd fd ZdZ d dej deejdeejdeejdeejdee e ejd ee d e ej fd Z xZ S)!InstructBlipVideoQFormerAttentionct|t|||_t ||_t |_yr)rGrHr attentionrrset pruned_headsr+s r1rHz*InstructBlipVideoQFormerAttention.__init__s3 CFL^_8@ Er6c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr`)lenrrfrr!rhrr#r%r&rr[r"union)r0headsindexs r1 prune_headsz-InstructBlipVideoQFormerAttention.prune_headss u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r6rrrrGrHrIrr)c p|j|||||||}|j|d|} | f|ddz} | S)Nrr)rfr) r0rrrrGrHrIr self_outputsattention_outputrs r1rz)InstructBlipVideoQFormerAttention.forwardsW~~    ! "    ;;|A F#% QR(88r6rrX) r7r8r9rHrnr;rrr<rrrrrs@r1rdrds" ;*7;15=A>BDH,1||!!2!23E--.  ((9(9:  !)):): ; !uU->->'?!@A$D> u|| r6rdcVeZdZfdZdej dej fdZxZS)$InstructBlipVideoQFormerIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) rGrHrrrIrr[rrstrr intermediate_act_fnrUs r1rHz-InstructBlipVideoQFormerIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r6rr)cJ|j|}|j|}|Sr)r[rvrs r1rz,InstructBlipVideoQFormerIntermediate.forwards& =1 00? r6rrs@r1rsrss#9U\\ellr6rscneZdZfdZdej dej dej fdZxZS)InstructBlipVideoQFormerOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) rGrHrrrrIr[rrrr\rrUs r1rHz'InstructBlipVideoQFormerOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= r6rr]r)cr|j|}|j|}|j||z}|Srr_r`s r1rz&InstructBlipVideoQFormerOutput.forwardrar6rrs@r1ryryrbr6ryc@eZdZfdZ ddZdZdZxZS)InstructBlipVideoQFormerLayercft||j|_d|_t ||_||_||jzdk(rt |d|_d|_ nd|_ t||_ t||_ t||_t||_y)NrrT)r,F)rGrHchunk_size_feed_forward seq_len_dimrdrf layer_idxcross_attention_frequencycrossattentionhas_cross_attentionrs intermediateryrintermediate_query output_queryr0rArrVs r1rHz&InstructBlipVideoQFormerLayer.__init__s '-'E'E$:6B" v77 71 <"CF_c"dD '+D $',D $@H4V< "Fv"N:6Br6c ||ddnd} |j||||| } | d} | dd} | d} |dkDr| ddd|ddf}|jr1| td|j||||||}|d}| |ddz} t |j |j |j|}| jd|kDr~t |j|j |j| dd|dddf}tj||gd}n,t |j|j |j| }|f| z} | | fz} | S) Nr)rrIrrr[z>encoder_hidden_states must be given for cross-attention layersrr`) rfrrrrfeed_forward_chunk_queryrrrbfeed_forward_chunkr;rj)r0rrrrGrHrIr query_lengthself_attn_past_key_valueself_attention_outputsrqrpresent_key_valuequery_attention_outputcross_attention_outputs layer_outputlayer_output_texts r1rz%InstructBlipVideoQFormerLayer.forwards:H9S>"1#5Y] !%   /3 "0" 2!4(2.226 ! %5a,6I%J "''(0$%eff*.*=*=*")*&7 +>+'*A)C&!$;Ab$AA4--,,  & L %%a(<7$=++00$$$Q q%89 %! %yy,8I)JPQR 4'',,   L  /G+.00r6cL|j|}|j||}|Sr)rrr0rqintermediate_outputrs r1rz0InstructBlipVideoQFormerLayer.feed_forward_chunkDs,"//0@A{{#68HI r6cL|j|}|j||}|Sr)rrrs r1rz6InstructBlipVideoQFormerLayer.feed_forward_chunk_queryIs."556FG(()<>NO r6)NNNNNFr)r7r8r9rHrrrrrs@r1r}r}s1C."#EN r6r}c:eZdZfdZ ddZxZS)InstructBlipVideoQFormerEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}wr) rGrHrArrrrr}layerrrs r1rHz(InstructBlipVideoQFormerEncoder.__init__PsW  ]]OTU[UmUmOn o) *69 = o  ',# psA$c j| rdnd} |rdnd} |rdnd}|rdnd}t|jjD]}|j|}| r| |fz} |||nd}|||nd}t |jddrF|j r:|rt jdd}|j|j|||||}n||||||||| }|d}|r ||dfz }|s| |dfz} |js||dfz}| r| |fz} | std ||| | |fDSt||| | | S) Nr>rFzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...rr[rrc3$K|]}|| ywrr>rs r1r2z:InstructBlipVideoQFormerEncoder.forward..s = s)rpast_key_valuesrrcross_attentions) rrArrr,rloggerwarningrrrr3r)r0rrrrGrHr use_cacherrrrall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_maskrIrs r1rz'InstructBlipVideoQFormerEncoder.forwardXs#7BD$5b4%6rD#,R$t{{445( VA::a=L#$58H$H!.7.CilO3B3N_Q/TXNt{{$S>STzz&"<"<=   ELL)G)GHOOPWXej  (/v7PR\']$ r6c||jd}nd}|&|jdd|||zfj}|k|j|}|jdk(r/|j |j |j}||z}|tj||fd}n|}|j |jjj}|j|}|j|}|S)Nrrrr`)r]rclonerrrrvr?r;rjrrurtr)r0 input_idsr query_embedspast_key_values_lengthrNrWrs r1rz*InstructBlipVideoQFormerEmbeddings.forwards  ")!,JJ  ,,Q0FVlIl0l-lmssuL  --i8J++z9&*&>&>|zO`O`?a&b#'*== '"YY j'AqI %J]]4>>#8#8#>#>? ^^J/ \\*- r6)NNNr)r7r8r9r:rHrrrs@r1rrsE$ r6rceZdZdZdeffd ZdZdZdZ dde jde e d e jd ed e jf d Z dd e j dee j$dee j dee jdee j$dee j$dee j$dee e e j$deedeedeedeed ee e j$effdZxZS)InstructBlipVideoQFormerModelz Querying Transformer (Q-Former), used in InstructBlipVideo. Slightly modified from BLIP-2 as it also takes the instruction as input. rAct||||_t||_t ||_|jyr)rGrHrArrWrr rrUs r1rHz&InstructBlipVideoQFormerModel.__init__s9     r6c.|jjSrrWrr5s r1rz2InstructBlipVideoQFormerModel.get_input_embeddingss...r6c&||j_yrrr0r&s r1set_input_embeddingsz2InstructBlipVideoQFormerModel.set_input_embeddingss*/'r6c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr rrfrn)r0heads_to_prunerrls r1 _prune_headsz*InstructBlipVideoQFormerModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B Cr6r input_shaper? has_queryr)c|jdk(r|dddddddf}n=|jdk(r|ddddddf}ntd|d|jd|j|j}d|z d z}|S) a> Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`Tuple[int]`): The shape of the input to the model. device: (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. r Nrz!Wrong shape for input_ids (shape z) or attention_mask (shape )rsrg)rarrbrvrt)r0rrr?rextended_attention_masks r1get_extended_attention_maskz9InstructBlipVideoQFormerModel.get_extended_attention_masks.    1 $&4Qa]&C #    !Q &'5QdA5E&F #3K=@[\j\p\p[qqrs #:"<"<4::"<"N#&)@#@H"L&&r6rrrrrGrHrrrrrc | | n|jj} | | n|jj} | | n|jj} | | t d|,|ddj d|jj z nd} ||j dnd}|j|||| }|jdd}|\}}|j}|tj||| zf|}|j|||}|t|tr|dj\}}}n|j\}}}||f}t|tr|Dcgc]}|j|}}n?|)tj||}|j|}n|j|}nd}|j!||jj"}|j%||||||| | | | | }|d}|dddddf}| s ||f|ddzSt'|||j(|j*|j,|j. Scc}w) a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of: shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Nz7You have to specify query_embeds when input_ids is Nonerrr)rrrrr[)r?) rrrGrHrrrrrr)rrrrrr)rArrrrrbrrWr]r?r;onesrrlistinvert_attention_mask get_head_maskrr rrrrr)r0rrrrrrGrHrrrrrrrembedding_outputrrzrNr?rencoder_batch_sizeencoder_sequence_lengthr{encoder_hidden_shapemaskencoder_extended_attention_maskrsequence_outputrs r1rz%InstructBlipVideoQFormerModel.forward*sD2C1N-TXT_T_TqTq$8$D $++JjJj &1%S^`f"g ! ,/6AVWXAYA^A^A`>"$;QAVA[A[A]>"$;Q$68O#P 0$7`v2wX\43M3Md3S2w/2w'/).4HQW)X&262L2LMc2d/262L2LMc2d/.2 +&&y$++2O2OP ,, 2"7#B+/!5#%'  *!,'1a0 #]3oab6II I;-'+;;)77&11,==   G3xs!Ir) NNNNNNNNNNN)r7r8r9r:r rHrrrr;rrrr?rr LongTensorrr<rrrrrs@r1rrs =/0C )' )'3Z)' )'  )'  )'\7;37/315=A>BEI$(,0/3&*x ##x !!2!23x u//0 x u||, x E--. x  ((9(9:x !)):): ;x "%e.?.?(@"ABx D>x $D>x 'tnx d^x  uU&&')UU Vx r6raJ This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`InstructBlipVideoConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`InstructBlipVideoProcessor`]. See [`InstructBlipVideoProcessor.__call__`] for details. qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipVideoProcessor`]. See [`InstructBlipVideoProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the language model. Input tokens can optionally be provided to serve as text prompt, which the language model can continue. Indices can be obtained using [`InstructBlipVideoProcessor`]. See [`InstructBlipVideoProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary of the language model. Only relevant in case an encoder-decoder language model (like T5) is used. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the pre-trained position encodings. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). a InstructBlipVideo Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. c!eZdZeZdZdZdZdZdgZ deffd Z dZ dZ d Z d ejfd Zd Zd ZdZdZeeeee ddej6dej6deej:deej6deej:deej:deej:deedeedeej:deededeed ee effdZ!ejD d dej6deej:deej:deej:deej:ded ej:fdZ#xZ$S)!)InstructBlipVideoForConditionalGenerationrqTF query_tokensrAc4t||t|j|_t j tjd|j|jj|_ t|j|_t j|jj|j j|_|j$r t'j(|j }nt+j(|j }|j,%|j,j/|j,|j0%|j0j/|j0||_|j5y)Nr)rGrHr r vision_modelrrMr;rnum_query_tokensqformer_configrIrrqformerr text_configlanguage_projectionuse_decoder_only_language_modelr from_configrrextend_keep_in_fp32_moduleslanguage_modelr)r0rArrVs r1rHz2InstructBlipVideoForConditionalGeneration.__init__ s'  89M9MNLLQ8O8OQWQfQfQrQr)st4V5J5JK #%99V-B-B-N-NPVPbPbPnPn#o  1 11==f>P>PQN2>>v?Q?QRN  + + 7  " " ) ).*J*J K  / / ;  & & - -n.R.R S, r6c6|jjSr)rrr5s r1rz>InstructBlipVideoForConditionalGeneration.get_input_embeddings%s""7799r6c:|jj|yr)rrrs r1rz>InstructBlipVideoForConditionalGeneration.set_input_embeddings(s 007r6c:|jj|yr)rset_output_embeddings)r0new_embeddingss r1rz?InstructBlipVideoForConditionalGeneration.set_output_embeddings+s 11.Ar6r)c6|jjSr)rget_output_embeddingsr5s r1rz?InstructBlipVideoForConditionalGeneration.get_output_embeddings.s""88::r6c6|jjSr)r get_encoderr5s r1rz5InstructBlipVideoForConditionalGeneration.get_encoder1""..00r6c6|jjSr)r get_decoderr5s r1rz5InstructBlipVideoForConditionalGeneration.get_decoder4rr6c|jjs_|jj|jj_|jj|jj _yyr)rArrsharedr  embed_tokensdecoderr5s r1 _tie_weightsz6InstructBlipVideoForConditionalGeneration._tie_weights7sT{{::7;7J7J7Q7QD   ' ' 47;7J7J7Q7QD   ' ' 4;r6c|j}t|dkDr:d|vr6tjj dkDrt j dt|jdrd|jj_ yy)z Some pre-processing hacks to make the model `accelerate` compatible. Check https://github.com/huggingface/transformers/pull/21707 for more details. rraThe `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning. Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for more details on creating a `device_map` for large models._hf_hookTN) hf_device_maprjr;cuda device_countrrrrrio_same_device)r0rs r1_preprocess_acceleratez@InstructBlipVideoForConditionalGeneration._preprocess_accelerate<sy ** }  !&6m&KPUPZPZPgPgPilmPm NNM  4&& 3:>D   ( ( 7 4r6rqformer_input_idsqformer_attention_maskrrdecoder_input_idsdecoder_attention_maskrrlabelsrrprc ~ | | n|jj} |j\}}}}}|j||z|||}|j ||| | | }|d}t j |jddt j|j}|jj|jddd}t j |jddt j|j}|t j|}|j|d}|j|d}t j||gd}|j||||||| | }|dddd|jdddf}|j!|}|j||jj"|zd}t j |jddt j|j}|j$j'|}|t j|}t)|jd dd||jj*k(j-dj/|}|j1j3|j||<nyt4j7d t j||j3|jgd}t j||j3|jgd}|jj8r|j%|||| | | }| r |j:n|d}d} | | j3|j} |dd| jd dddf}|d ddddfj=}!| d ddfj=j3|j}"t?d }#|#|!jAd|jjBjD|"jAd} n@|j%|||||| | | |  }| r |jFn|d} | r |j:n|d}| s||||f}$| | f|$zS|$StI| ||||S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Returns: Examples: ```python >>> from transformers import InstructBlipVideoProcessor, InstructBlipVideoForConditionalGeneration >>> import torch >>> from huggingface_hub import hf_hub_download >>> import av >>> import numpy as np >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> model = InstructBlipVideoForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b", device_map="auto") >>> processor = InstructBlipVideoProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample uniformly 4 frames from the videWhy is this video funny?o >>> total_frames = container.streams.video[0].frames >>> indices = np.arange(0, total_frames, total_frames / 4).astype(int) >>> clip = read_video_pyav(container, indices) >>> prompt = "What is happening in the video?" >>> inputs = processor(text=prompt, images=clip, return_tensors="pt").to(model.device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... repetition_penalty=1.5, ... length_penalty=1.0, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) "A person is eating a bowl of pasta, and they are using a fork to eat it. The person is sitting at a table, and the plate of pasta is on the table in front" ```N)rqrrrrprr[r>r`r)rrrrGrHrrrvideo_token_indexKExpanding inputs for video tokens in InstructBLIPVideo should be done in processing. Please follow instruction here (https://gist.github.com/zucchini-nlp/65f22892b054dc0d68228af56fbeaac2) to update your InstructBLIPVideo model. Using processors without these attributes in the config is deprecated and will throw an error in v4.47.)rrrrrr.r) reduction) rrrrrrrrr)r$r%r&r'r()%rArrbrerr;rr]rAr?rry ones_likerepeat_interleaverjrrrrrr,r unsqueeze expand_asrwrvr warning_oncerr%rr rirrr$r#)%r0rqrrrrrrrrrrrprrzframeschannelrXrYr& image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputslanguage_model_attention_maskrspecial_image_maskrr%r$ shift_logits shift_labelsloss_fctrs% r1rz1InstructBlipVideoForConditionalGeneration.forwardPsj&1%->q-A+A1(DE !% 8 8 F!6 = =j$++JfJfioJoqs t(- ! & & (" -UZZH]HdHd) %C++@@B9M  !"__Y7N 4;; 3T : F"+t{{/L/L"L!W!WXZ![!e!efs!t 0E0M0M0O0R0RS`SgSg0hM, -   z  "II'N>NOdOkOk>l&mstuM"YY.0A0AB_BfBf0ghnoN ;; 6 6))+-"3%9'# *G(3W^^ FD!6==1FKKN?#4a 78%c3B3k2==? %c12g99;>>v}}M ,f= 1 1"dkk6M6M6X6X Y[g[l[lmo[pq))+-"3'="3%9'#* G$/7<jAd t j||j=|jgd }t j||j=|jgd }|jBj$jDsM|jGdd|jd zd z |d<|jGdd|jd z|d<||d}|jBj$jDs||d<|jBjHdi||}|S)a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width) or (batch_size, num_frames, num_channels, height, width)): Input images or videos to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. rT)rrpNr[r>rr`r)rrrrGrHrrrr max_length min_length)rrrr>)%rrrbrerrr;rr]rAr?rryr r rjrrrArr bos_token_idr,rrrepeatrr r rwrvrr ris_encoder_decodergetgenerate)r0rqrrrrrpgenerate_kwargsrzrrrXrYrrrrrrrlanguage_attention_mask start_tokensrrinputsrs r1r#z2InstructBlipVideoForConditionalGeneration.generatesb> 4 )  ' ' )6B5G5G2 FGVU#++J,?&RWX (( %=)     %zz,*;*;*=cr*B%**]i]p]pq((// 0B0B10Er2N $zz,*;*;*=cr*B%**]i]p]pq ! )%*__5F%G "-??A?N!7!I!I&VW!I!X!&,@BX+Y_`!a '1%".#7 % %66q:PLJ $ = =>A]A]]`aadpp  l^5::lNaNabI!((Q7I  !"__Y7N3113I>  4;; 3T : F"+t{{/L/L"L!W!WXZ![!e!efs!t 0E0M0M0O0R0RS`SgSg0hM, -   z  "II'N>NOdOkOk>l&mstuM"YY(.*;*;B15598<=A,0/3-1&*).$(G ''G !,,G !))9)9 : G E--. G !!1!12 G $E$4$45G !))9)9 :G $D>G 'tnG ))*G d^G #'G D>G  uJJ KG OG RU]]_9==A0459).q''q$E$4$45q!))9)9 : q E,,- q !!1!12 q#'q   qqr6r)BrC dataclassesrtypingrrrrr;rtorch.nnr activationsr generationr modeling_outputsr rrrmodeling_utilsr pytorch_utilsrrrutilsrrrrrrautorrconfiguration_instructblipvideorr r! get_loggerr7rr#r+r@rrrrrrr rrrdrsryr}rrr!INSTRUCTBLIPVIDEO_START_DOCSTRINGr,rr>r6r1r;s, !.. %!) .ll?   H %  ;   @G GTNNb 299 -BII-`#%#%LS ryyS l-)$@#C@FBBJ  . .b 299  RYY dBIIdN\ bii\ ~00fA $DA H %!B&"J& L0PRaL Lr6