T^(h\AVdZddlZddlmZddlmZmZmZmZm Z ddl m Z ddl m Z mZmZddlmZmZmZmZdd lmZmZmZdd lmZerdd lmZej8eZd efd Z dZ!dZ"dZ#dZ$GddedZ%GddedZ&e%e&jNd<GddeZ(dgZ)y)z Processor class for Idefics3. N) accumulate) TYPE_CHECKINGDictListOptionalUnion) BatchFeature) ImageInputis_valid_image load_image) ImagesKwargsProcessingKwargsProcessorMixinUnpack) AddedToken BatchEncoding TextInput)logging)PreTokenizedInputreturncHt|txr|jdS)Nhttp) isinstancestr startswith)vals n/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/idefics3/processing_idefics3.pyis_urlr$s c3  :CNN6$::c2t|xs t|SN)rr )elems ris_image_or_image_urlr$(s $< />$//r c d}t|D]4}t|D]}||d|dzd|dzdz||zzz }!|dz }6|d||z||zz|zz }|S)zKPrompt with expanded image tokens for when the image is split into patches.z )range) image_seq_len image_rows image_colsfake_token_around_image image_tokenglobal_img_tokentext_split_imagesn_hn_ws r_prompt_split_imager5,sZ "$ C *+sQwiuS1WIQ/OOU`TaerRrr    T! " $ %&   M] * +% % ' r c&||z||zz|zS)z5Prompt with expanded image tokens for a single image.)r,r/r0r1s r_prompt_single_imager8?s6 # #   M] * +% % 'r cT|dk(r|dk(rt||||St||||||S)Nr)r/r0r1)r8r5)r-r.r,r/r0r1s rget_image_prompt_stringr:IsHQ:?# $;#-   z:/F Ue r c8eZdZUeeed<eeeefed<y)Idefics3ImagesKwargsreturn_row_col_infomax_image_sizeN) __name__ __module__ __qualname__rbool__annotations__rrintr7r rr<r<Xs!$'T#s(^,,r r<F)totalc.eZdZUeed<ddddddidZy)Idefics3ProcessorKwargs images_kwargsTF)add_special_tokenspaddingis_split_into_wordsr=) text_kwargsrHN)r?r@rAr<rC _defaultsr7r rrGrG]s+''#'#( "4 Ir rGrHc eZdZdZddgZddgZdZdZ ddede e ffd Z d Z dd e eeeeeefd e ed eeed fde ed eedef dZdZdZedZxZS)Idefics3Processora Constructs a Idefics3 processor which wraps a LLama tokenizer and Idefics3 image processor into a single processor. [`Idefics3Processor`] offers all the functionalities of [`Idefics3ImageProcessor`] and [`Idefics3TokenizerFast`]. See the docstring of [`~IdeficsProcessor.__call__`] and [`~IdeficsProcessor.decode`] for more information. Args: image_processor (`Idefics3ImageProcessor`): An instance of [`Idefics3ImageProcessor`]. The image processor is a required input. tokenizer (`PreTrainedTokenizerBase`, *optional*): An instance of [`PreTrainedTokenizerBase`]. This should correspond with the model's text model. The tokenizer is a required input. image_seq_len (`int`, *optional*, defaults to 169): The length of the image sequence i.e. the number of tokens per image in the input. This parameter is used to build the string from the input prompt and image tokens and should match the value the model used. It is computed as: image_seq_len = int(((image_size // patch_size) ** 2) / (scale_factor**2)) chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. image_processor tokenizerr, chat_templateIdefics3ImageProcessor AutoTokenizerc | td| tdtddd|_tddd|_tddd|_d |_||_tjd |_ d |j|j|jgi}|j|t|0||fd |i|y) Nz)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.zFT) normalizedspecialzz z*(\n?\n?|\n?)+additional_special_tokensrR) ValueErrorrfake_image_tokenr0end_of_utterance_tokenglobal_image_tagr,recompile%_regex_to_remove_extra_special_tokensrIsuper__init__)selfrPrQr,rRkwargs tokens_to_add __class__s rrbzIdefics3Processor.__init__s  "HI I  AB B *+FSXbf g%iE4P&01ERWae&f# .*68ZZ@m5n2 (%%  ++*  $$]3 )[=[TZ[r cg}|D]_}g}|D]E}t|r|j| t|s,|jt|G|j|a|Sr")r appendrr )rcprompts prompt_imagespromptimagesr#s r_extract_images_from_promptsz.Idefics3Processor._extract_images_from_promptssn  )FF 4!$'MM$'D\MM*T"23  4   ( )r rltextrrdrc | | td|jtfd|jji|}||n |j }g}g} t } |tt|tr|g}n.t|tst|dts td|D cgc]'} | j|jj)}} |t|r|gg}n$t|ttfrt|dr|t|t!|k7rRtd|jjdt|d|jjdt!|d dgtt#|z} t%t!|D cgc]} || | | | d z}} nO|g}nKt|ttfs5t|dttfst|dds td |D cgc] } t!| } } |D cgc](} | Dcgc]}t'|r t)|n|c}*}} }|j*|fi|d }| j-||h| |k7rtd |d| d| j/ddgt!|zg}| j/ddgt!|zg}|j0j}|jj}|j2}g}t5|||D]\} }}g}t5||D]'\}}t7||||||}|j9|)| j;|}t!|dk(r td|d} t=|D]\} }| ||| d zzz } |j9| |jdd|i|d}| j-|| S|bt?|r/tdt|d|jjd|jdd|i|d}| j-|| Scc} wcc} wcc} wcc}wcc}} w)a Processes the input prompts and returns a BatchEncoding. Example: ```python >>> import requests >>> from transformers import Idefics3Processor >>> from transformers.image_utils import load_image >>> processor = Idefics3Processor.from_pretrained("HuggingFaceM4/Idefics3-8B-Llama3") >>> processor.image_processor.do_image_splitting = False # Force as False to simplify the example >>> url1 = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" >>> url2 = "https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg" >>> image1, image2 = load_image(url1), load_image(url2) >>> images = [[image1], [image2]] >>> text = [ ... "In this image, we see", ... "bla bla bla", ... ] >>> outputs = processor(images=images, text=text, return_tensors="pt", padding=True) >>> input_ids = outputs.input_ids >>> input_tokens = processor.tokenizer.batch_decode(input_ids) >>> print(input_tokens) ['<|begin_of_text|>(()*169) In this image, we see', '<|reserved_special_token_0|><|reserved_special_token_0|><|reserved_special_token_0|><|begin_of_text|>bla bla bla(()*169)'] ``` Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`, *optional*): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. If is of type `List[ImageInput]`, it's assumed that this is for a single prompt i.e. of batch size 1. text (`Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). Wherever an image token, `` is encountered it is expanded to `` + `` + `` * `image_seq_len` * `. image_seq_len (`int`, *optional*): The length of the image sequence. If not provided, the default value of self.image_seq_len is used. image_seq_len should be equal to int(((image_size // patch_size) ** 2) / (scale_factor**2)) return_tensors (`Union[str, TensorType]`, *optional*): If set, will return tensors of a particular framework. See [`PreTrainedTokenizerFast.__call__`] for more information. z+You must provide either `text` or `images`.tokenizer_init_kwargsrzAInvalid input text. Please provide a string, or a list of stringszThe total number of zP tokens in the prompts should be the same as the number of images passed. Found  z tokens and z images.r'zdInvalid input images. Please provide a single image or a list of images or a list of list of images.rHz!The number of images in the text z and images z should be the same.rowscols)r0r/r1z.The image token should be present in the text.rnrLzFound z. tokens in the text but no images were passed.r7) rZ _merge_kwargsrGrQ init_kwargsr,r rrlistcountr0contentr$tuplesumlenrr+rr rPupdatepopr[r]zipr:rhsplit enumerateany)rcrlrnaudiovideosr,rd output_kwargsn_images_in_textn_images_in_imagesinputssamplecumsum_images_in_textiim image_inputsr-r.r[r0r1prompt_strings sample_rows sample_colsimage_prompt_stringsn_rowsn_colsimage_prompt_string split_sample text_inputss r__call__zIdefics3Processor.__call__sp N>N>V>V=WWcdghndocppxz ./C$zBR7S2T,T)"'s+;'>y!Lllls#Acr|jj|i|}|jjd|S)z This method forwards all its arguments to Idefics3TokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. rX)rQdecoder`r)rcrrd decode_outputs rrzIdefics3Processor.decode\s8 .--t>v> 99==iWWr c|jj}|jj}ttj ||zSr")rQmodel_input_namesrPrvdictfromkeys)rctokenizer_input_namesimage_processor_input_namess rrz#Idefics3Processor.model_input_namesds? $ @ @&*&:&:&L&L#DMM"=@U"UVWWr )NN)NNNNN)r?r@rA__doc__ attributes valid_kwargsimage_processor_classtokenizer_classrDrrrbrmrr rrrrGrrrrpropertyr __classcell__)rfs@rrOrOos&$[1J#_5L4%Oim\>A\X`adXe\: OSbf'+ aj$z"2Dj9I4JJKaI2DOTJ]E^^_a  } a01a aFmXXXr rO)*rr^ itertoolsrtypingrrrrrfeature_extraction_utilsr image_utilsr r r processing_utilsrrrrtokenization_utils_baserrrutilsrr get_loggerr?loggerrBrr$r5r8r:r<rGrCrO__all__r7r rrs ==4AAVVKK<   H %;4;0& -