S^(hddlZddlmZmZmZmZddlZddlZddlmZm Z m Z ddl m Z ddl mZddlmZddlmZmZmZdd lmZmZmZmZdd lmZd d lmZej<eZ Gd de jBZ"Gdde jBZ#Gdde jBZ$Gdde jBZ%Gdde jBZ&Gdde jBZ'Gdde jBZ(Gdde jBZ)Gdde jBZ*Gdd e jBZ+Gd!d"e jBZ,Gd#d$e jBZ-Gd%d&eZ.Gd'd(e.Z/Gd)d*e.eZ0y)+N)ListOptionalTupleUnion)Tensordevicenn)CrossEntropyLoss)ACT2FN)GenerationMixin))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModelapply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)logging)BlipTextConfigc eZdZdZfdZ d deejdeejdeejde dejf dZ xZ S) BlipTextEmbeddingsz;Construct the embeddings from word and position embeddings.cPt|tj|j|j |j |_tj|j|j |_ tj|j |j|_ tj|j|_|jdt!j"|jj%ddt'|dd|_||_y) N) padding_idxeps position_ids)rF) persistentposition_embedding_typeabsolute)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandgetattrr!configselfr6 __class__s i/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/blip/modeling_blip_text.pyr$zBlipTextEmbeddings.__init__1s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  (/v7PR\']$ input_idsr inputs_embedspast_key_values_lengthreturncH||j}n|jdd}|d}||jdd|||zf}||j|}|}|jdk(r|j |}||z }|j |}|j |}|S)Nrrr")sizerr)r!r+r,r0) r8r<rr=r> input_shape seq_length embeddingsr+s r:forwardzBlipTextEmbeddings.forwardCs  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL   00;M"  ' ': 5"&":":<"H  - -J^^J/ \\*- r;)NNNr) __name__ __module__ __qualname____doc__r$rr2 LongTensor FloatTensorintrrE __classcell__r9s@r:rr.ssE(153759&' E,,-u//0  1 12  !$   r;rc4eZdZfdZdZdZdZdZdZ dde jde e jd e e jd e e jd e e jd e e e e jd e ede e jfdZxZS)BlipTextSelfAttentionct|||_|j|jzdk7r0t |ds$t d|j|jfz|j|_t|j|jz |_|j|jz|_ tj|j|j|_ |r_tj|j|j|_tj|j|j|_n^tj|j|j|_tj|j|j|_tj |j"|_t'|dd|_|j(dk(s|j(dk(rG|j*|_tj,d|j*zd z |j|_yy) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)r!r" relative_keyrelative_key_queryr)r#r$r6r'num_attention_headshasattr ValueErrorrLattention_head_size all_head_sizer Linearqueryencoder_hidden_sizekeyvaluer.attention_probs_dropout_probr0r5r!r*r%distance_embeddingr8r6is_cross_attentionr9s r:r$zBlipTextSelfAttention.__init__cs     : : :a ?PVXhHi^%%v'A'ABC  $*#=#= #&v'9'9F 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD #>rr;c||_yNattn_gradients)r8rgs r:save_attn_gradientsz)BlipTextSelfAttention.save_attn_gradients~s ,r;c|jSrerfr8s r:get_attn_gradientsz(BlipTextSelfAttention.get_attn_gradientss"""r;c||_yre attention_map)r8rns r:save_attention_mapz(BlipTextSelfAttention.save_attention_maps *r;c|jSrermrjs r:get_attention_mapz'BlipTextSelfAttention.get_attention_maps!!!r;c|jdd|j|jfz}|j|}|j ddddS)NrrrUrr )rArVrYviewpermute)r8x new_x_shapes r:transpose_for_scoresz*BlipTextSelfAttention.transpose_for_scoressNffhsmt'?'?AYAY&ZZ AFFK yyAq!$$r; hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsr?cJ|j|}|du} | rC|j|j|} |j|j|} |}n|y|j|j|} |j|j|} t j |d| gd} t j |d| gd} n@|j|j|} |j|j|} |j|} | | f}t j | | jdd} |jdk(s|jdk(rF|jd}t j|tj|j jdd}t j|tj|j jdd}||z }|j||jzdz }|j!| j" }|jdk(rt j$d | |}| |z} nE|jdk(r6t j$d | |}t j$d | |}| |z|z} | t'j(|j*z } || |j!| jz} t-j.d| }|j1|}|||z}t j || }|j3dddd j5}|jdd|j6fz}|j|}|r||fn|f}||fz}|S)NrrUdimrrrSrT)dtyperrzbhld,lrd->bhlrzbhrd,lrd->bhlrr )r\rwr^r_r2catmatmul transposer!rAr3longrrsrar*toreinsummathsqrtrYr Softmaxr0rt contiguousrZ)r8rxryrzr{r|r}r~mixed_query_layerrc key_layer value_layer query_layerattention_scoresrCposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probsattention_probs_dropped context_layernew_context_layer_shapeoutputss r:rEzBlipTextSelfAttention.forwards~!JJ}5 3$> 11$((;P2QRI33DJJ?T4UVK3N  '11$((=2IJI33DJJ}4MNK >!#4i"@aHI))^A%6 $D!LK11$((=2IJI33DJJ}4MNK//0AB #[1!<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q&++-a0J"\\*EJJ}OcOcdiijlnopN"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||#',,"?  &= &I # %S"S* **,CD 6G=/2mM]^--r;NNNNNF)rFrGrHr$rhrkrorqrwr2rrrKrboolrErMrNs@r:rPrPbsu6-#+"%7;15=A>BDH,1N||N!!2!23NE--. N ((9(9: N !)):): ; N!uU->->'?!@AN$D>N u|| Nr;rPcneZdZfdZdej dej dej fdZxZS)BlipTextSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr) r#r$r r[r'denser,r-r.r/r0r7s r:r$zBlipTextSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r;rx input_tensorr?cr|j|}|j|}|j||z}|Srerr0r,r8rxrs r:rEzBlipTextSelfOutput.forward7 =1  ]3 }|'CD r;rFrGrHr$r2rrErMrNs@r:rr1> U\\RWR^R^r;rceZdZd fd ZdZ d dej deejdeejdeejdeejdee e ejd ee d e ej fd Z xZ S)BlipTextAttentionct|t|||_t ||_t |_yre)r#r$rPr8routputset pruned_headsrbs r:r$zBlipTextAttention.__init__s3 )&2DE (0 Er;c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr)lenrr8rVrYrrr\r^r_rrrZunion)r8headsindexs r: prune_headszBlipTextAttention.prune_headss u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:r;rxryrzr{r|r}r~r?c p|j|||||||}|j|d|} | f|ddz} | S)Nrr)r8r) r8rxryrzr{r|r}r~ self_outputsattention_outputrs r:rEzBlipTextAttention.forward sWyy    ! "    ;;|A F#% QR(88r;)Fr) rFrGrHr$rr2rrrKrrrErMrNs@r:rrs" ;*7;15=A>BDH,1||!!2!23E--.  ((9(9:  !)):): ; !uU->->'?!@A$D> u|| r;rcVeZdZfdZdej dej fdZxZS)BlipTextIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yre) r#r$r r[r'intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr7s r:r$zBlipTextIntermediate.__init__#s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r;rxr?cJ|j|}|j|}|Sre)rrr8rxs r:rEzBlipTextIntermediate.forward+s& =1 00? r;rrNs@r:rr"s#9U\\ellr;rcneZdZfdZdej dej dej fdZxZS)BlipTextOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r#r$r r[rr'rr,r-r.r/r0r7s r:r$zBlipTextOutput.__init__3s` YYv779K9KL f&8&8f>S>STzz&"<"<= r;rxrr?cr|j|}|j|}|j||z}|Srerrs r:rEzBlipTextOutput.forward9rr;rrNs@r:rr2rr;rceZdZfdZ d dej deejdeejdeejdeejdeeeejdee d eej fd Z d Z xZ S) BlipTextLayercFt|||_|j|_d|_t ||_||_|jjr&t ||jj|_ t||_ t||_ y)Nr)rc)r#r$r6chunk_size_feed_forward seq_len_dimr attention layer_num is_decodercrossattentionr intermediaterr)r8r6rr9s r:r$zBlipTextLayer.__init__As}  '-'E'E$*62" ;; ! !"3Ft{{OeOe"fD 08$V, r;rxryrzr{r|r}r~r?c||ddnd}|j|||||} | d} | dd} | d} |$|j| |||||} | d} | | ddz} t|j|j|j | }|f| z} | | fz} | S)NrU)r~r}rrr)r~)rrrfeed_forward_chunkrr)r8rxryrzr{r|r}r~self_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attention_outputs layer_outputs r:rEzBlipTextLayer.forwardMs:H9S>"1#5Y] !%   /3 "0" 2!4(2.226 ,&*&9&9 %&"3 ':' # 7q9  7" ==G0  # #T%A%A4CSCSUe  /G+.00r;cL|j|}|j||}|Sre)rr)r8rintermediate_outputrs r:rz BlipTextLayer.feed_forward_chunkys,"//0@A{{#68HI r;r) rFrGrHr$r2rrrKrrrErrMrNs@r:rr@s -7;15=A>BDH,1*||*!!2!23*E--. * ((9(9: * !)):): ; *!uU->->'?!@A*$D>* u|| *Xr;rcDeZdZfdZ ddej deejdeejdeejdeejdeeeejdee d ee d ee d ee d e eej e ffd Z xZ S)BlipTextEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}w)NF) r#r$r6r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r8r6ir9s r:r$zBlipTextEncoder.__init__sP  ]]eFLdLdFe#fM&!$<#fg &+#$gsA$rxryrzr{r|past_key_values use_cacher~output_hidden_states return_dictr?c |jr%|jr|rtjdd}| rdnd} |rdnd} |r|jj rdnd} |rdnd}t |jjD]}|j|}| r| |fz} |||nd}|||nd}|jr/|jr#|j|j|||||||}n||||||||}|d}|r ||dfz }|s| |dfz} | |dfz} | r| |fz} | std||| | | fDSt||| | | S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FrrrrUc3$K|]}|| ywrer).0vs r: z*BlipTextEncoder.forward..s = s)last_hidden_staterrx attentionscross_attentions) rtrainingloggerwarningr6rrrr_gradient_checkpointing_func__call__tupler)r8rxryrzr{r|rrr~rrall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cacher layer_modulelayer_head_maskr} layer_outputss r:rEzBlipTextEncoder.forwards  & &4==p" "6BD$5b4%64;;;Q;QrW[#,R$t{{445# RA::a=L#$58H$H!.7.CilO3B3N_Q/TXN**t}} $ A A ))!"#)*"% ! !-!"#)*"%! *!,M"}R'8&::" &9]1=MQ'Q$G# RJ 1]4D D  "&%'(    9+.+*1   r;) NNNNNNFFT)rFrGrHr$r2rrrKrrrrrErMrNs@r:rrs ,7;15=A>BEI$(,1/4&*S ||S !!2!23S E--. S ((9(9: S !)):): ; S "%e.?.?(@"ABS D>S $D>S 'tnS d^S  uU\\"$MM NS r;rcVeZdZfdZdej dej fdZxZS)BlipTextPoolerct|tj|j|j|_tj |_yre)r#r$r r[r'rTanh activationr7s r:r$zBlipTextPooler.__init__s9 YYv1163E3EF '')r;rxr?c\|dddf}|j|}|j|}|S)Nr)rr)r8rxfirst_token_tensor pooled_outputs r:rEzBlipTextPooler.forwards6+1a40 #56  6 r;rrNs@r:rrs#$ U\\ellr;rcVeZdZfdZdej dej fdZxZS)BlipTextPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r#r$r r[r'rrrrr transform_act_fnr,r-r7s r:r$z(BlipTextPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr;rxr?cl|j|}|j|}|j|}|Sre)rr r,rs r:rEz'BlipTextPredictionHeadTransform.forwards4 =1 --m< }5 r;rrNs@r:r r s$UU\\ellr;r c*eZdZfdZdZdZxZS)BlipTextLMPredictionHeadcHt|t||_t j |j |jd|_t jtj|j|_ |j|j_ y)NF)bias) r#r$r  transformr r[r'r&decoder Parameterr2zerosrr7s r:r$z!BlipTextLMPredictionHead.__init__sm 8@yy!3!3V5F5FUS LLV->->!?@ !II r;c:|j|j_yre)rrrjs r: _tie_weightsz%BlipTextLMPredictionHead._tie_weightss II r;cJ|j|}|j|}|Sre)rrrs r:rEz BlipTextLMPredictionHead.forwards$}5  ]3 r;)rFrGrHr$rrErMrNs@r:rrs &&r;rcVeZdZfdZdej dej fdZxZS)BlipTextOnlyMLMHeadcBt|t||_yre)r#r$r predictionsr7s r:r$zBlipTextOnlyMLMHead.__init__s 3F;r;sequence_outputr?c(|j|}|Sre)r)r8rprediction_scoress r:rEzBlipTextOnlyMLMHead.forwards ,,_=  r;rrNs@r:rrs#<!u||! !r;rc"eZdZdZeZdZgZdZy)BlipTextPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. bertc.t|tjtjfr<|jj j d|jjnct|tjrI|jj j|jj jdt|tjr2|j%|jj jyyy)zInitialize the weightsg)meanstd?N) rr r[r%weightdatanormal_r6initializer_ranger,rzero_fill_)r8modules r: _init_weightsz%BlipTextPreTrainedModel._init_weights-s fryy",,7 8 MM   & &CT[[5R5R & S  - KK   " " $ MM   $ $S ) fbii (V[[-D KK   " " $.E (r;N) rFrGrHrIr config_classbase_model_prefix_no_split_modulesr.rr;r:r!r!#s "L %r;r!c!eZdZdZdfd ZdZdZdZdede e de d e d ef d Z dd eejdeejd eejdeejdeejdeejdeejdeejdeeej"dee dee dee dee d ee d ee ejeffdZxZS) BlipTextModela The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. argument and `is_decoder` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ct||||_t||_t ||_|r t|nd|_|jyre) r#r$r6rrDrencoderrpooler post_init)r8r6add_pooling_layerr9s r:r$zBlipTextModel.__init__DsI   ,V4&v. 0AnV,t  r;c.|jjSrerDr)rjs r:get_input_embeddingsz"BlipTextModel.get_input_embeddingsNs...r;c&||j_yrer:)r8r_s r:set_input_embeddingsz"BlipTextModel.set_input_embeddingsQs*/'r;c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr5rrr)r8heads_to_prunerrs r: _prune_headszBlipTextModel._prune_headsUsE +002 CLE5 LL  u % / / ; ;E B Cr;ryrBrrr?c|jdk(r|dddddddf}n>|jdk(r|r|\}}tj||}|ddddfj||d|ddddfk} | j |j } | j d|j dkr[|j d| j dz } tjtj||| f|| j | gd} | dddddddf|ddddddfz}n3|ddddddf}n%td j||j |j |j }d |z d z}|S) a= Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`Tuple[int]`): The shape of the input to the model. device (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. r NrUrr)rrr)axiszAWrong shape for input_ids (shape {}) or attention_mask (shape {})rr&g) rr2r3repeatrrshaperonesrXformat) r8ryrBrrextended_attention_mask batch_sizerCseq_ids causal_maskprefix_seq_lens r:get_extended_attention_maskz)BlipTextModel.get_extended_attention_mask]s&    1 $&4Qa]&C #    !Q &)4& J,,z&A%dD!m4;;J TUVZabfhikoboZpp *nn^-A-AB $$Q'.*>*>q*AA%3%9%9!%<{?P?PQR?S%SN"'))!JJ!+Z HQW_j_p_p(   #K+6aq!m*D~VWY]_cefVfGg*g'*8D$9I*J'SZZ!5!5 #:"<"<4::"<"N#&)@#@H"L&&r;r<rrzr=encoder_embedsr{r|rrr~rrc | | n|jj} | | n|jj} | | n|jj} |r| | n|jj} nd} | | t d|4|j |||j}|\}}|j}nY|%|jdd}|\}}|j}n2|%|jdd}|\}}|j}n t d| | ddjdnd}|)tj|||zfj|}|j||||}|t|tr|dj\}}}n|j\}}}||f}t|tr|Dcgc]}|j!|}}n?|)tj||}|j!|}n|j!|}nd}|j#||jj$}||j'|||| }n|}|j)|||||| | | | | }|d}|j*|j+|nd}| s ||f|d dzSt-|||j.|j0|j2|j4 Scc}w) a. encoder_hidden_states (`torch.FloatTensor`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timerzGYou have to specify either input_ids or inputs_embeds or encoder_embedsrrUrC)r<rr=r>) ryrzr{r|rrr~rrr)r pooler_outputrrxrr)r6r~ruse_return_dictrrX%warn_if_padding_and_no_attention_maskrArrFr2rGrrNrlistinvert_attention_mask get_head_maskrrDr5r6rrrxrr)r8r<ryrrzr=rOr{r|rrr~rrrrBrJrCrr>rIencoder_batch_sizeencoder_sequence_length_encoder_hidden_shapemaskencoder_extended_attention_maskembedding_outputencoder_outputsrrs r:rEzBlipTextModel.forwardsUF2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K%0 "J %%F  &',,.s3K%0 "J "))F  '(--/4K%0 "J #**Ffg gDSC^!3A!6!"$;QAVA[A[A]>"$;Q$68O#P 0$7`v2wX\43M3Md3S2w/2w'/).4HQW)X&262L2LMc2d/262L2LMc2d/.2 +&&y$++2O2OP  !##)+'= /   . ,, 2"7#B+/!5#'  *!,8< 8O O4UY #]3oab6II I;-'+;;)77&11,==   Y3xs:K)T)NNNNNNNNNNNNNF)rFrGrHrIr$r;r=rArrrLrrrNrr2rrKrrrErMrNs@r:r3r3;s/0C?'$?'38:?'GM?'[_?' ?'F-115/3,004158<9==A$(,0/3&*%*I ELL)I !.I u||, I ELL) I   - I !.I  ( 5I !) 6I "$u'8'8"9:I D>I $D>I 'tnI d^I TNI uU\\"$PP Q!I r;r3c%eZdZfdZdZdZdZdZ ddee jdee jdee jd ee jd ee jd ee jd ee jd ee jdee e jdee dee dee dee dee dee dee deee jeff"dZddZdZxZS)BlipTextLMHeadModelct||t|d|_t ||_|j |_y)NF)r8)r#r$r3r"rclslabel_smoothingr7s r:r$zBlipTextLMHeadModel.__init__,s8  !&EB &v.%55r;c6|jjSre)r"r;rjs r:r;z(BlipTextLMHeadModel.get_input_embeddings3syy--//r;c:|jj|yre)r"r=r8new_embeddingss r:r=z(BlipTextLMHeadModel.set_input_embeddings6s &&~6r;cB|jjjSre)rbrrrjs r:get_output_embeddingsz)BlipTextLMHeadModel.get_output_embeddings9sxx##+++r;c||jj_|j|jj_yre)rbrrrrfs r:set_output_embeddingsz)BlipTextLMHeadModel.set_output_embeddings<s,'5$$2$7$7!r;r<ryrrzr=r{r|labelsrrr~rr return_logitsr reductionr?c2| | n|jj} |d} |j|||||||| | | | | | }|d}|j|}|r|ddddddfj Sd}||ddddddfj }|ddddfj j |j }t||j}||jd|jj|jd}|dk(r0|j|jddjd}| s|f|d dz}||f|zS|St|||j|j|j |j" S) a encoder_hidden_states (`torch.FloatTensor`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (`torch.LongTensor`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NF) ryrrzr=r{r|rrr~rrrrrr)rnrcnonerU)losslogitsrrxrr)r6rRr"rbrrrr rcrsr&rAsumrrrxrr)r8r<ryrrzr=r{r|rlrrr~rrrmrrnrrrlm_lossshifted_prediction_scoresloss_fctrs r:rEzBlipTextLMHeadModel.forward@sR&1%.s.nU_j--aZ=N=N1OPns58)r)r8rrreordered_past layer_pasts ` r:_reorder_cachez"BlipTextLMHeadModel._reorder_caches=) J ncmnn N r;)NNNNNNNNNNNNNFTr$)NN)rFrGrHr$r;r=rirkrr2rrrrrrrrEr}rrMrNs@r:r`r`+s607,8 -115/3,0048<9=)-8<$(,0/3&*(-%)#)#X ELL)X !.X u||, X ELL) X   - X  ( 5X !) 6X &X "$u||"45X D>X $D>X 'tnX d^X  ~X TN!X "C=#X $ uU\\"$EE F%X t <r;r`)1rtypingrrrrr2torch.utils.checkpointrrr torch.nnr activationsr generationr modeling_outputsrrrmodeling_utilsrrrrutilsrconfiguration_blipr get_loggerrFrModulerrPrrrrrrrr rrr!r3r`rr;r:rsK" // $$%!)  .   H %00h{BII{~  . .d 299  RYY <BII<@Z biiZ | RYY  bii$ryy0!"))!%o%0l +l `S1?Sr;