T^(h36ddlmZmZmZmZddlZddlmZddlmZddl m Z m Z ddl m Z mZdd lmZd d lmZe j&eZGd d eZGddej.ZGddej.ZGdde ZGddeZddgZy))ListOptionalTupleUnionN)nn)ACT2FN)is_torchdynamo_compilinglogging)LlavaCausalLMOutputWithPastLlavaForConditionalGeneration)MistralRMSNorm)Mistral3Configc eZdZy)Mistral3RMSNormN__name__ __module__ __qualname__k/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/mistral3/modular_mistral3.pyrrrrcxeZdZdZdeffd ZdejdejdejfdZxZ S)Mistral3PatchMergerz< Learned merging of spatial_merge_size ** 2 patches configc"t|||_|jj}|j |_|jjj |_tj||j dzz|d|_ y)Nr Fbias) super__init__r vision_config hidden_sizespatial_merge_size patch_sizerLinear merging_layer)selfrr% __class__s rr#zMistral3PatchMerger.__init__(sr  **66 "(";";++33>>YY{T5L5La5O'OQ\chirimage_features image_sizesreturnc|Dcgc]&}|d|jz|d|jzf(}}|Dcgc] \}}||z }}}|jd}g}t|j|D]\} } || \}}| j |||j dddj d} tjjj| |j|j} | j ||jdzzdj} |j| tj|d}|j|}|Scc}wcc}}w)Nrrr ) kernel_sizestridedim)r'shape enumeratesplitviewpermute unsqueezetorchr functionalunfoldr&tappendcatr)) r*r,r- image_sizehwtokens_per_imagedpermuted_tensor image_index image_tokens image_gridgrids rforwardzMistral3PatchMerger.forward1slcn U_Z]doo -z!}/O P  /::daAE::   $)2>3G3GHX3Y)Z ) %K{+DAq%**1a3;;Aq!DNNqQJ88&&--(?(?H_H_.D99Q!8!8!!;;R@BBDD  " "4 ( )?:++N;) ;s +E"E') rrr__doc__rr#r;TensorrK __classcell__r+s@rrr#s?j~jellRWR^R^rrc\eZdZdeffd Zdej dej fdZxZS)Mistral3MultiModalProjectorrc^t|t|jj|_t ||_t|jtrdnt|j}tj|jj|z|jj|j|_t"|j$|_tj|jj|jj|j|_y)Nrr )r"r#rr$r%normr patch_merger isinstancevision_feature_layerintlenrr( text_configmultimodal_projector_biaslinear_1r projector_hidden_actactlinear_2)r*rnum_feature_layersr+s rr#z$Mistral3MultiModalProjector.__init__Js #F$8$8$D$DE /7",V-H-H#"NQTWX^XsXsTt  , ,/A A    * *11  &556    * *F,>,>,J,JQWQqQq  rr,r-c|j|}|j||}|j|}|j|}|j |}|S)N)rSrTr[r]r^)r*r,r- hidden_statess rrKz#Mistral3MultiModalProjector.forwardZsR>2**>;G n5 /  m4 r) rrrrr#r;rMrKrNrOs@rrQrQIs* ~ ellrrQc eZdZy)Mistral3CausalLMOutputWithPastNrrrrrcrccrrrcc#eZdZdejdeeeefdejfdZ dde ejde ejde ejde ejd e eejd e ejde eeeefd e ejd e e d e e de e de e de ejdeeejfde ejdee eff dZy) Mistral3ForConditionalGeneration pixel_valuesrVr-c ||jDcic] \}}| || }}}|j|f|dd|}t|tr|j|}n3|D cgc]} |j| } } t j | d}|j|jd|} | Scc}}wcc} w)a= Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`): The tensors corresponding to the input images. vision_feature_layer (`Union[int, List[int]]`): The index of the layer to select the vision feature. If multiple indices are provided, the vision feature of the corresponding indices will be concatenated to form the vision features. image_sizes (`torch.Tensor`): Tensor containing the image sizes as returned by the processor. Returns: image_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`). T)r-output_hidden_statesr0r3r) items vision_towerrUrWrar;r@multi_modal_projectorsqueeze) r*rfrVr-kwargskv image_outputsselected_image_feature layer_idxhs_poolr,s rget_image_featuresz3Mistral3ForConditionalGeneration.get_image_featureshs,$*<<>C41aQ]!Q$CC))),uKfjuntu  *C 0%2%@%@AU%V "Ocd)}229=dGd%*YYwB%? "334J4R4RST4UWbcDes B3B3!B9N input_idsattention_mask position_idspast_key_values inputs_embedslabels use_cacheoutput_attentionsrh return_dictcache_positionlogits_to_keepr.c | | n|jj} | | n|jj} | | n|jj} ||n|jj}|du|duz r t d| | t d||j |}|#|j|||}||jjk(jd}|j|j|j}ts{||j|jk7rW||jjk(j}|j d|j dz}t d|d ||j|j|j"}|j%||}|j&d ||||| | | | | |d |}|d}d}|<||dd|j ddz dfj|j}|d ddddf|j|jdk7j)}|d ddf|j|jdk7j)}n1|d ddddfj)}|d ddfj)}t+j,}||j/d|j1d|j/dj|j}| s|f|ddz}||f|zS|St3|||j4|j6|j8| Sd S)a< labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. logits_to_keep (`int` or `torch.Tensor`, *optional*): If an `int`, compute logits for the last `logits_to_keep` tokens. If `0`, calculate logits for all `input_ids` (special case). Only last token logits are needed for generation, and calculating them only for that token can save memory, which becomes pretty significant for long sequences or large vocabulary size. If a `torch.Tensor`, must be 1D corresponding to the indices to keep in the sequence length dimension. This is useful when using packed tensor format (single dimension for batch and sequence length). Returns: Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Mistral3ForConditionalGeneration >>> model = Mistral3ForConditionalGeneration.from_pretrained("mistralai/Mistral-Small-3.1-24B-Instruct-2503") >>> processor = AutoProcessor.from_pretrained("mistralai/Mistral-Small-3.1-24B-Instruct-2503") >>> prompt = "[INST][IMG]What is the image?[/INST]" >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=15) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "What is the image?The image depicts two cats lying on a pink blanket." ```Nz:You must specify exactly one of input_ids or inputs_embedszdYou cannot specify both pixel_values and inputs_embeds at the same time, and must specify either one)rfrVr-r0rrz6Image features and image tokens do not match: tokens: z , features ) rvrwrxryr{r|rhr}r~r.)losslogitsrxra attentionsimage_hidden_statesr)rr|rhuse_return_dictrV ValueErrorget_input_embeddingsrtimage_token_indexr: expand_astodevicer numelsumr5dtypemasked_scatterlanguage_model contiguousrCrossEntropyLossr8sizercrxrar)r*rurfrvrwrxryrVrzr{r|rhr}r~rr- lm_kwargsr,special_image_maskn_image_tokensn_image_featuresoutputsrrshift_attention_mask shift_logits shift_labelsloss_fctoutputs rrKz(Mistral3ForConditionalGeneration.forwardsr2C1N-TXT_T_TqTq$8$D $++JjJj &1%O>OPR>S>V>VWcWjWj>kDY,F'+'7D7V# CV C-#33!//))2>2J   QU   r)NNNNNNNNNNNNNrN)rrrr; FloatTensorrrWrrMrtr LongTensorboolrrcrKrrrreregs"''"$CcN3"\\ "L15481537=A59@D-1$(,0/3&*5934.2!L E,,-L u001L !. L u//0 L "$u'8'8"9: L   1 12L 'uS$s)^'<=L ))*L D>L $D>L 'tnL d^L !!1!12L c5<</0L ell+!L $ u44 5%L rreMistral3PreTrainedModel)typingrrrrr;r activationsr utilsr r llava.modeling_llavar rmistral.modeling_mistralrconfiguration_mistral3r get_loggerrloggerrModulerrQrcre__all__rrrrs 0/ !6]52   H % n #"))#L"))4 %@ q 'Dq j& r