T^(hQbdZddlZddlmZddlmZmZmZmZm Z ddl m Z ddl m Z mZmZmZddlmZmZmZmZdd lmZmZdd lmZmZd d lmZmZmZm Z erdd lm!Z!ejDe#Z$erddl%m%Z%ndZ%dZ&dZ'dZ(GddedZ)GddedZ*GddeZ+dgZ,y)z Processor class for SmolVLM. N) timedelta) TYPE_CHECKINGDictListOptionalUnion) BatchFeature) ImageInput VideoInputmake_batched_videosmake_nested_list_of_images) ImagesKwargsProcessingKwargsProcessorMixinUnpack) BatchEncoding TextInput)is_num2words_availablelogging)DEFAULT_MEDIA_OUTTRODEFAULT_VIDEO_INTROFRAME_TIMESTAMP_MESSAGEsmolvlm_sample_indices_fn)PreTokenizedInput) num2wordsc d}t|D]4}t|D]}||d|dzd|dzdz||zzz }!|dz }6|d||z||zz|zz }|S)zKPrompt with expanded image tokens for when the image is split into patches.z )range) image_seq_len image_rows image_colsfake_token_around_image image_tokenglobal_image_tokentext_split_imagesn_hn_ws l/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/smolvlm/processing_smolvlm.py_prompt_split_imager.5sZ "$ C *+sQwiuS1WIQ/OOU`TaerRrr    T! " $ %&   " M] * +% % ' c&||z||zz|zS)z5Prompt with expanded image tokens for a single image.)r$r'r(r)s r-_prompt_single_imager2Js6 # #   " M] * +% % 'r/cT|dk(r|dk(rt||||St||||||S)Nr)r'r(r))r2r.)r%r&r$r'r(r)s r-get_image_prompt_stringr4TsHQ:?# $;#1   z:/F Ug r/c8eZdZUeeed<eeeefed<y)SmolVLMImagesKwargsreturn_row_col_infomax_image_sizeN) __name__ __module__ __qualname__rbool__annotations__rstrintr1r/r-r6r6cs!$'T#s(^,,r/r6F)totalc.eZdZUeed<ddddddidZy)SmolVLMProcessorKwargs images_kwargsTF)add_special_tokenspaddingis_split_into_wordsr7) text_kwargsrCN)r9r:r;r6r= _defaultsr1r/r-rBrBhs+&&#'#( "4 Ir/rBc <eZdZdZddgZddgZdZdZ ddede e ffd Z dd Z dd e eeeeeefd e ed eeed fd edeedef dZdeeee e fdeedeedeeee effdZdZdZedZ dfd ZxZS)SmolVLMProcessora Constructs a SmolVLM processor which wraps a LLama tokenizer and SmolVLM image processor into a single processor. [`SmolVLMProcessor`] offers all the functionalities of [`SmolVLMImageProcessor`] and [`SmolVLMTokenizerFast`]. See the docstring of [`~IdeficsProcessor.__call__`] and [`~IdeficsProcessor.decode`] for more information. Args: image_processor (`SmolVLMImageProcessor`): An instance of [`SmolVLMImageProcessor`]. The image processor is a required input. tokenizer (`PreTrainedTokenizerBase`, *optional*): An instance of [`PreTrainedTokenizerBase`]. This should correspond with the model's text model. The tokenizer is a required input. image_seq_len (`int`, *optional*, defaults to 169): The length of the image sequence i.e. the number of tokens per image in the input. This parameter is used to build the string from the input prompt and image tokens and should match the value the model used. It is computed as: image_seq_len = int(((image_size // patch_size) ** 2) / (scale_factor**2)) chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. image_processor tokenizerr$ chat_templateSmolVLMImageProcessor AutoTokenizerc t|dd|_t|dd|_t|dd|_t|dd|_||_|j d |_|j|_ |j|_ |j jd d |_ |j d |_ |j d |_ts t!dt#|H||fd|i|y)Nfake_image_tokenzr(zend_of_utterance_tokenzr)z video_sizedo_image_splittingF max_framesfpszbPackage `num2words` is required to run SmolVLM processor. Install it with `pip install num2words`.rM)getattrrQr(rRr)r$video_samplingrSsize image_sizerTgetdo_video_splittingdefault_max_frames default_fpsr ImportErrorsuper__init__)selfrKrLr$rMkwargs __class__s r-razSmolVLMProcessor.__init__s!( 3EGb c"9mYG&-i9QSg&h#"))5I>"Z*)88F).."1"D"D"1"@"@"D"DEY[`"a"1"@"@"N*99%@t  )[=[TZ[r/c T|(|Dcgc]}|j|j}}|Dcgc] }t|} }|j|f||d|d} |d| fS| k7rt d|d| d| j ddgt|zg} | j ddgt|zg} g} t || | D]\}}}g}t ||D]O\}}t|||j|j|j|j }|j|Q|j|j}t|dk(r t d |d}t|D]\}}||||d zzz }| j|| | fScc}wcc}w) N)rTrYrCz!The number of images in the text z and images z should be the same.rowsrcols)r(r'r)z.The image token should be present in the text.r)countr(lenrK ValueErrorpopzipr4r$rQr)appendsplit enumerate)rbtextimages output_kwargsrTimage_processor_sizesamplen_images_in_textsublistn_images_in_images image_inputsr%r&prompt_strings sample_rows sample_colsimage_prompt_stringsn_rowsn_colsimage_prompt_string split_sampleis r-process_visionzSmolVLMProcessor.process_visions  MQR6 T-=-= >R R:@Awc'lAA+t++  '9@T XefuXv  <% % !1 134D3E\RdQeeyz "%%fsSY.?@ !%%fsSY.?@ 03D*j0Q * ,FK#% "%k;"? A&=&& $ 0 0,0,A,A'+'>'> '#%++,?@ A"<<(8(89L< A% !QRR"!_F*34H*I D&&- QU0CCC D  ! !& )- *0|++U SAs "F F%rqrprvideosrcreturnc j|| | td||du|duz r td|jtfd|jji|}|t |t r|g}n.t |tst |dt s tdt|Dcgc]}|j|jc}}|dkDr||td|dt} |Jt|}|j||||j|j \}} | j!| nK|It#|}|j||||j|j$ \}} | j!| |(|jd d |i|d } | j!| | Scc}w) a Processes the input prompts and returns a BatchEncoding. Example: ```python >>> import requests >>> from transformers import SmolVLMProcessor >>> from transformers.image_utils import load_image >>> processor = SmolVLMProcessor.from_pretrained("HuggingFaceM4/SmolVLM2-256M-Video-Instruct") >>> processor.image_processor.do_image_splitting = False # Force as False to simplify the example >>> url1 = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" >>> url2 = "https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg" >>> image1, image2 = load_image(url1), load_image(url2) >>> images = [[image1], [image2]] >>> text = [ ... "In this image, we see", ... "bla bla bla", ... ] >>> outputs = processor(images=images, text=text, return_tensors="pt", padding=True) >>> input_ids = outputs.input_ids >>> input_tokens = processor.tokenizer.batch_decode(input_ids) >>> print(input_tokens) ['<|begin_of_text|>(()*169) In this image, we see', '<|reserved_special_token_0|><|reserved_special_token_0|><|reserved_special_token_0|><|begin_of_text|>bla bla bla(()*169)'] ``` Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`, *optional*): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. If is of type `List[ImageInput]`, it's assumed that this is for a single prompt i.e. of batch size 1. text (`Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). Wherever an image token, `` is encountered it is expanded to `` + `` + `` * `image_seq_len` * `. return_tensors (`Union[str, TensorType]`, *optional*): If set, will return tensors of a particular framework. See [`PreTrainedTokenizerFast.__call__`] for more information. Nz5You must provide one of `text`, `images` or `videos'.z4You must specify exactly one of `images` or `videos`tokenizer_init_kwargsrzAInvalid input text. Please provide a string, or a list of stringsz We detected z4 tokens in the text but no images/videos were passed)rTrsrprGr1)rj _merge_kwargsrBrL init_kwargs isinstancer>listsumrhr(r rrrTrZupdater rS) rbrqrpaudiorrcrrrtruinputs vision_inputs text_inputss r-__call__zSmolVLMProcessor.__call__sh \}}||z }t |dz}t |dz}|j |dd|d@|j ||j t | |D]"}|D]}d|vr g}|dD]}|jdd k(r|jd }|jd }tt   }|j d tjt|t| dt|D]@\}}|j d tj|d|j ddiB|j d t d|j |||d<%|S)a Used within `apply_chat_template` when a model has special way to process conversation history. For example, video models might want to specify in the prompt the duration of video or which frame indices at which timestamps were sampled. This information cannot be accessed before the video is loaded. For most models it is a no-op, must be overriden by model processors which require special processing. Args: conversation (`List[Dict, str, str]`): The conversation to process. Always comes in batched format. batch_images (`List[List[ImageInput]]`): Batch of images that were loaded from url/path defined in the conversation. The images are ordered in the same way as in the conversation. Comes in nested list format, one list of `PIL` images per batch. batch_videos (`List[List[ImageInput]]`): Batch of videos that were loaded from url/path defined in the conversation. The videos are ordered in the same way as in the conversation. Comes in nested list format, one list of 4D video arrays per batch. batch_video_metadata (`List[List[Dict[[str, any]]]]`): Batch of metadata returned from loading videos. That includes video fps, duration and total number of framer in original video. Metadata are ordered in the same way as `batch_videos`. Comes in nested list format, one list of 4D video arrays per batch. durationframes_indicesrV<02d:contenttypevideor)secondsrp) frame_countvideo_duration)rrp) timestampimage)copydeepcopyrlrWr?rmrir[rkrrformatrr>rorr)rbrrrrchat_template_kwargsbatch_num_framesbatch_timestamps metadata_list video_listmetadatar duration_sec frames_idxrV timestampsidxframe_npsecmmss conversationmsg new_contentblockcurr_timestampscurr_num_framestdrtss r-#_process_messages_for_chat_templatez4SmolVLMProcessor._process_messages_for_chat_template?s=> m4 -/*),-A<)P 4 %M:#&}j#A 4%&x< $X/?@ h. %(U%;*>q*A*:*>*>q*A's|jj|i|}|S)z This method forwards all its arguments to SmolVLMTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )rL batch_decode)rbargsrcbatched_decode_outputs r-rzSmolVLMProcessor.batch_decodes& !< ; ;T LV L$$r/c>|jj|i|}|S)z This method forwards all its arguments to SmolVLMTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rLdecode)rbrrc decode_outputs r-rzSmolVLMProcessor.decodes% .--t>v> r/c|jj}|jj}ttj ||zS)N)rLmodel_input_namesrKrdictfromkeys)rbtokenizer_input_namesimage_processor_input_namess r-rz"SmolVLMProcessor.model_input_namess? $ @ @&*&:&:&L&L#DMM"=@U"UVWWr/c  |jn |jnfd}|s|}t | |f||d|S)Nc$t|fd|S)N)rU target_fps skip_secs)r)r fn_kwargsrUrrs r-sample_indices_fn_funczDSmolVLMProcessor.apply_chat_template..sample_indices_fn_funcs(,%/JR[_h r/)video_load_backendsample_indices_fn)r]r^r`apply_chat_template) rbrrUrrrrrcrrds ``` r-rz$SmolVLMProcessor.apply_chat_templatesa1;0BT,, )3);T%%  ! 6 w*  -?Sd hn  r/)NN)FN)NNNN)NNrpyavN)r9r:r;__doc__ attributes valid_kwargsimage_processor_classtokenizer_classr?rr>rarrr rrr rrBrrranyrrrpropertyrr __classcell__)rds@r-rJrJwst&$[1J#_5L3%Oim\>A\X`adXe\6,,`OSbf! dj$z"2Dj9I4JJKdI2DOTJ]E^^_d  d /0 d dLUDc3h01U:&U:& U #4S#X#78 Un%XX!  r/rJ)-rrdatetimertypingrrrrrfeature_extraction_utilsr image_utilsr r r rprocessing_utilsrrrrtokenization_utils_baserrutilsrrvideo_processing_smolvlmrrrrr get_loggerr9loggerrr.r2r4r6rBrJ__all__r1r/r-rs ==4 WV?4<   H %#I* -,e- -U N ~N b  r/