T^(hG<dZddlmZmZmZddlmZddlmZm Z m Z ddl m Z m Z mZmZmZmZddlmZmZmZddlmZej0eZd Zed Dcgc] }d |d d  c}edDcgc] }d|dd  c}zZGddeZGdde ZGdde dZ de!fdZ"dZ#dZ$dZ%GddeZ&dgZ'y cc}wcc}w)!z Processor class for PaliGemma. )ListOptionalUnion) BatchFeature) ImageInputis_valid_imagemake_flat_list_of_images) ImagesKwargsProcessingKwargsProcessorMixin TextKwargsUnpack!_validate_images_text_input_order) AddedTokenPreTokenizedInput TextInput)loggingziz4>z3c8eZdZUeeeeeeeefed<y)PaliGemmaTextKwargssuffixN) __name__ __module__ __qualname__rrrrr__annotations__p/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/paligemma/processing_paligemma.pyrr-s& U9&7i$O`Jaab ccrrceZdZUeeed<y)PaliGemmaImagesKwargsdo_convert_rgbN)rrrrboolrrrr r"r"1s TN"rr"c4eZdZUeed<eed<ddiddidZy) PaliGemmaProcessorKwargs text_kwargs images_kwargspaddingF data_formatchannels_first)r'r(N)rrrrrr" _defaultsrrr r&r&5s.$$(( u  + Irr&F)totalreturncHt|txr|jdS)Nhttp) isinstancestr startswith)vals r is_urlr5Cs c3  :CNN6$::rc2t|xs t|SN)r5r elems r is_image_or_image_urlr:Hs $< />$//rc<t|txs t|Sr7)r1r2r:r8s r _is_str_or_imager<Ls dS " A&;D&AArc ||z|z||dS)aZ Builds a string from the input prompt and image tokens. For example, for the call: build_string_from_input( prompt="Prefix str" bos_token="", image_seq_len=3, image_token="", ) The output will be: "Initial str" Args: prompt (`List[Union[str, ImageInput]]`): The input prompt. bos_token (`str`): The beginning of sentence token. image_seq_len (`int`): The length of the image sequence. image_token (`str`): The image token. num_images (`int`): Number of images in the prompt.  rprompt bos_token image_seq_len image_token num_imagess r build_string_from_inputrEPs$&M)J6 7 {6(" MMrc eZdZdZddgZdgZdZdZ dfd Z dde d e e e e e e e fd eed efd Zd ZdZedZxZS)PaliGemmaProcessora Constructs a PaliGemma processor which wraps a PaliGemma image processor and a PaliGemma tokenizer into a single processor. [`PaliGemmaProcessor`] offers all the functionalities of [`SiglipImageProcessor`] and [`GemmaTokenizerFast`]. See the [`~PaliGemmaProcessor.__call__`] and [`~PaliGemmaProcessor.decode`] for more information. Args: image_processor ([`SiglipImageProcessor`], *optional*): The image processor is a required input. tokenizer ([`GemmaTokenizerFast`], *optional*): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. image_processor tokenizer chat_template)SiglipImageProcessorSiglipImageProcessorFast)GemmaTokenizerGemmaTokenizerFastc | td| tdt|ds td|j|_t|dsCttdd}d |gi}|j ||j t|_n|j|_|jtd|_ d|_ t|5||| y) Nz)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.image_seq_lengthz;Image processor is missing an `image_seq_length` attribute.rCFT) normalizedspecialadditional_special_tokens)rJ) ValueErrorhasattrrPr IMAGE_TOKENadd_special_tokensconvert_tokens_to_idsimage_token_id add_tokens EXTRA_TOKENS add_bos_token add_eos_tokensuper__init__)selfrHrIrJkwargsrC tokens_to_add __class__s r r_zPaliGemmaProcessor.__init__{s  "HI I  AB B(:;Z[ [ / @ @y-0$[UDQK8;-HM  ( ( 7"+"A"A+"ND "+":":D \*"' "'  )=Qrimagestextrar.c t||\}}|jtfd|jji|}|dj dd}|dnd}| t d|tjdd }t|r|g}nt|trt|d r |.|+td |Dsutjd t|trKt|tr;t|t|k7r$t d t|dt|dt!|r|gg}nt|tt"frt!|d r|D cgc]} | g}} nKt|tt"fr*t|d tt"frt!|d d s t dt%||D cgc]R\} } t'| |jj(|j*t,t| tr t| ndT} } } t/|}ng} |D]}|j1t,t,|j*z}|j3t,}|dk7r|tt,znd }|d||jj(z||dz}| j5|| Dcgc]}|d } }|t|r|g}|&|Dcgc]}||jj6z}}|j8|fi|dd}|dj;dd|ddxx|j*z cc<|j f||d|d}i|d|i}|r.|dj=|dd k(d}|j?d|itA|Scc} wcc} } wcc}wcc}w)ah Main method to prepare for the model one or several sequences(s) and image(s). This method forwards the `text` and `kwargs` arguments to GemmaTokenizerFast's [`~GemmaTokenizerFast.__call__`] if `text` is not `None` to encode the text. To prepare the image(s), this method forwards the `images` and `kwrags` arguments to SiglipImageProcessor's [`~SiglipImageProcessor.__call__`] if `images` is not `None`. Please refer to the docstring of the above two methods for more information. The usage for PaliGemma fine-tuning preparation is slightly different than usual. suffix passed are suffixes to the prompt in `text`, and will be placed after the prompt. This is because attention is handled differently for the prefix and the suffix. For instance, ```python image = PIL_cow_image prompt = "answer en Where is the cow standing?" suffix = "on the beach" inputs = processor(text=prompt, images=image, suffix=suffix) ``` Here `inputs` will contain the `input_ids` and `token_type_ids` that follow ```python inputs["input_ids"][:, 256:] # tensor([[ 2, 6006, 603, 573, 13910, 9980, 235336, 108, 477, 573, 8318]]) inputs["token_type_ids"][:, 256:] tensor([[0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1]]) ``` Meaning the last three tokens are of "label" ("suffix") type while the other ones are of "prefix" type. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. text (`str`, `List[str]`, `List[List[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. suffix (`str`, `List[str]`, `List[List[str]]`): The suffixes or batch of suffixes to be encoded. Only necessary for finetuning. See https://github.com/google-research/big_vision/blob/main/big_vision/configs/proj/paligemma/README.md for more information. If your prompt is " What is on the image", the suffix corresponds to the expected prediction "a cow sitting on a bench". Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. If `suffix` is provided, the `input_ids` will also contain the suffix input ids. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. - **labels** -- Labels compatible with training if `suffix` is not None tokenizer_init_kwargsr'rNTFzF`images` are expected as arguments to a `PaliGemmaProcessor` instance.z]You are using PaliGemma without a text prefix. It will perform as a picture-captioning model.rc3,K|] }t|vywr7)rV).0samples r z.PaliGemmaProcessor.__call__..s@{f,@saLYou are passing both `text` and `images` to `PaliGemmaProcessor`. The processor expects special image tokens in the text, as many tokens as there are images per each text. It is recommended to add `` tokens in the very beginning of your text. For this call, we will infer how many images each text has and add special tokens.z Received z images for zK prompts. Each prompt should be associated with an image or list of images.zAimages must be an image, list of images or list of list of imagesr?r>r( pixel_values max_length) text_pairreturn_token_type_ids input_idstoken_type_idsilabels)data)!r _merge_kwargsr&rI init_kwargspoprTlogger warning_oncer<r1listanywarningrlenr tupleziprErArPrVr replacerfindappend eos_tokenrHget masked_fillupdater)r`rdreaudiovideosra output_kwargsrrrimager@ image_list input_stringsexpanded_samplesrkexpanded_samplebos_rfind_index bos_indexsfxroinputs return_datarus r __call__zPaliGemmaProcessor.__call__sD9F *** $ "&.."<"<  }-11(DA(.(: >ef f <   o D D !6D d #(8a(A    2@4@@<dD)j.F6{c$i/('F }LT LWX "&)%hZFu 6>&QR);T39:%ug:F:ve}5"6!9tUm<&vay|4$%hii/2$.? !+ ,%"&..":":&*&;&;$/6@T6R3z?XY  ! !2&9#% "=F&,nn[+PTPePeBe&fO&5&;&;K&HOFUY[F[#k2B BabI' 3dnn6N6NNQ`ajakQll$%++O<=>N N6F82 N N  "26":XF  @FGcDNN444GFG+t++FUmO6TUVde   ' + +L$ ? K - ( 6$:O:O O 6  "7 M*  ?>> K(44Vs +t~~**D;F;;rc:|jj|i|S)z This method forwards all its arguments to GemmaTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rIdecoders r rzPaliGemmaProcessor.decodeFs %t~~$$d5f55rc|jj}|jj}ttj ||zSr7)rImodel_input_namesrHr|dictfromkeys)r`tokenizer_input_namesimage_processor_input_namess r rz$PaliGemmaProcessor.model_input_namesMsA!% @ @&*&:&:&L&L#DMM"7:U"UVWWr)NNN)NNNN)rrr__doc__ attributes valid_kwargsimage_processor_classtokenizer_classr_rrrrrrr&rrrrpropertyr __classcell__)rcs@r rGrGfs $[1J#$LP>O R@"^b b.b.I0$y/4HYCZZ[b. 12 b. b.J<6XXrrGN)(rtypingrrrfeature_extraction_utilsr image_utilsrr r processing_utilsr r r rrrtokenization_utils_baserrrutilsr get_loggerrrzrVranger[rr"r&r$r5r:r<rErG__all__)is0r rs)(4OO    H % ).t5A$qgQ5RWX[R\8]Q4#wa8]] d*d#L# /u ;4; 0BN,lXlX^ W 68]s C 5C