S^(h#dZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z m Z mZddlmZmZdd lmZGd d e d Zej*eZGdde ZdgZy)z Processor class for Donut. N)contextmanager)ListOptionalUnion) ImageInput)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)loggingceZdZiZy)DonutProcessorKwargsN)__name__ __module__ __qualname__ _defaultsh/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/donut/processing_donut.pyrrsIrrF)totalc eZdZdZddgZdZdZdfd Z ddede e e e e e efd eefd Zd Zd Zed ZddZedZedZxZS)DonutProcessora Constructs a Donut processor which wraps a Donut image processor and an XLMRoBERTa tokenizer into a single processor. [`DonutProcessor`] offers all the functionalities of [`DonutImageProcessor`] and [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. See the [`~DonutProcessor.__call__`] and [`~DonutProcessor.decode`] for more information. Args: image_processor ([`DonutImageProcessor`], *optional*): An instance of [`DonutImageProcessor`]. The image processor is a required input. tokenizer ([`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`], *optional*): An instance of [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. The tokenizer is a required input. image_processor tokenizerAutoImageProcessor AutoTokenizerc d}d|vr+tjdt|jd}||n|}| t d| t dt ||||j|_d|_ y)Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.z)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.F) warningswarn FutureWarningpop ValueErrorsuper__init__rcurrent_processor_in_target_context_manager)selfrrkwargsr __class__s rr'zDonutProcessor.__init__9s  & ( MM  !' +> ? -<-H/N_  "HI I  AB B )4!%!5!5*/'rimagestextr+c ||jr|j||fi|S| | td|jtfd|j j i|}||j|fi|d}|-||djdd|j |fi|d}|S|Sdd<|d|d<|S) a When used in normal mode, this method forwards all its arguments to AutoImageProcessor's [`~AutoImageProcessor.__call__`] and returns its output. If used in the context [`~DonutProcessor.as_target_processor`] this method forwards all its arguments to DonutTokenizer's [`~DonutTokenizer.__call__`]. Please refer to the docstring of the above two methods for more information. zBYou need to specify either an `images` or `text` input to process.tokenizer_init_kwargs images_kwargs text_kwargsadd_special_tokensF input_idslabels) r)r(r% _merge_kwargsrr init_kwargsr setdefault) r*r-r.audiovideosr+ output_kwargsinputs encodingss r__call__zDonutProcessor.__call__Ms  * *)4))&$A&A A >dlab b***  "&.."<"<   )T))&SM/4RSF  !m,778LeT&tL}]/KLI <M ^ (5F8 "+K"8F; Mrc:|jj|i|S)z This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoder*argsr+s rr@zDonutProcessor.batch_decodews +t~~**D;F;;rc:|jj|i|S)z This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rdecoderAs rrDzDonutProcessor.decode~s %t~~$$d5f55rc#Ktjdd|_|j|_d|j |_d|_yw)z Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning TrOCR. z`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your images inputs, or in a separate call.TNF)r!r"r)rr(rr*s ras_target_processorz"DonutProcessor.as_target_processorsH   9 +/'!% !%!5!5*/'sA A c||jj}i}|r4tjd|tj}|n |j d}tj |}tjd|d|tj}|j }||j|d}n|j }tj |} tj |} tj| d| |tjtjz} | | j dj} d| vr3d| vr/|j| d | } | rt| dk(r| d } | ||<ntg||<| jd D]?} | j} | |vr| d d k(r | dddk(r| dd} ||j| At||dk(r ||d ||<||j|t|zdj}|ddd k(r|g|j|ddd | zS|r4t|r|r|gS|S|rgSd|iS)zS Convert a (generated) token sequence into an ordered JSON format. Nz zz(.*?)z text_sequence)rget_added_vocabresearch IGNORECASEgroupescapereplaceDOTALLstrip token2jsonlensplitappendfind)r*tokensrLrMoutput start_tokenkey key_escaped end_tokenstart_token_escapedend_token_escapedcontentvalueleafs rr[zDonutProcessor.token2jsons{  ..88:K))L&"--HK"##A&C))C.K T+a"8&"--PI%++-K  R8%OO- &(ii &<#$&IIi$8!))*+51B0CDfbmm^`^g^gNg&%mmA.446G(W-? $Ze f "5zQ(-a*/F3K&(s $+MM)$<5D#'::r@rDrrGr[propertyrmr __classcell__)r,s@rrr%s $[1J0%O0,"NR ((uS$s)Y8IIJK( -. (T<6 0 04Gl**$$rr)rorSr! contextlibrtypingrrr image_utilsrprocessing_utilsr r r tokenization_utils_baser r utilsrr get_loggerrloggerr__all__rrrr~sf %((%HHC+5   H %t$^t$n  r