T^(hdZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlZddlmZddlmZmZdd lmZmZmZmZmZmZmZmZmZej@e!Z"e r#erddl#Z#erddl$Z%erddl&mZ'd d d Z(d Z)e e e*ee+e*ffZ,eGddeZ-GddeZ.GddeZ/ddgZ0y)z Tokenization class for Wav2Vec2.N) dataclass)groupby) TYPE_CHECKINGDictListOptionalTupleUnion)PreTrainedTokenizer) AddedToken BatchEncoding) ModelOutputPaddingStrategy TensorTypeadd_end_docstringsis_flax_availableis_tf_availableis_torch_availablelogging to_py_objz vocab.jsonztokenizer_config.json vocab_filetokenizer_config_filea padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Controls the maximum length to use by one of the truncation/padding parameters. If left unset or set to `None`, this will use the predefined model maximum length if a maximum length is required by one of the truncation/padding parameters. If the model has no specific maximum input length (like XLNet) truncation/padding to a maximum length will be deactivated. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. cheZdZUdZeeeefed<dZeee e fed<dZ eee e fed<y)Wav2Vec2CTCTokenizerOutputaN Output type of [` Wav2Vec2CTCTokenizer`], with transcription. Args: text (list of `str` or `str`): Decoded logits in text from. Usually the speech transcription. char_offsets (list of `List[Dict[str, Union[int, str]]]` or `List[Dict[str, Union[int, str]]]`): Offsets of the decoded characters. In combination with sampling rate and model downsampling rate char offsets can be used to compute time stamps for each charater. Total logit score of the beam associated with produced text. word_offsets (list of `List[Dict[str, Union[int, str]]]` or `List[Dict[str, Union[int, str]]]`): Offsets of the decoded words. In combination with sampling rate and model downsampling rate word offsets can be used to compute time stamps for each word. textN char_offsets word_offsets) __name__ __module__ __qualname____doc__r rstr__annotations__r ListOfDictrp/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/wav2vec2/tokenization_wav2vec2.pyrr]sN  S 3 8"`): The beginning of sentence token. eos_token (`str`, *optional*, defaults to `""`): The end of sentence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. word_delimiter_token (`str`, *optional*, defaults to `"|"`): The token used for defining the end of a word. do_lower_case (`bool`, *optional*, defaults to `False`): Whether or not to accept lowercase input and lowercase the output when decoding. target_lang (`str`, *optional*): A target language the tokenizer should set by default. `target_lang` has to be defined for multi-lingual, nested vocabulary such as [facebook/mms-1b-all](https://huggingface.co/facebook/mms-1b-all). **kwargs Additional keyword arguments passed along to [`PreTrainedTokenizer`] input_idsattention_maskc @||_||_||_| |_t |d5} t j | |_ddd| |j| |_n|j|_|jjD cic]\} } | |  c} } |_ t|0d|||||||| d| |jjD]/}t|dkDs|jt!|ddd1y#1swYxYwcc} } w) Nutf-8encoding) unk_token bos_token eos_token pad_token do_lower_caseword_delimiter_tokenreplace_word_delimiter_char target_langTFrstriplstrip normalizedr')_word_delimiter_tokenr6r8r9openjsonloadvocabencoderitemsdecodersuper__init__keyslen add_tokensr )selfrr3r4r2r5r7r8r6r9kwargs vocab_handlekvtoken __class__s r)rHzWav2Vec2CTCTokenizer.__init__s&:"*+F(& *w / 1<<0DJ 1  "::k2DL::DL)-););)=>A1>   '!5(C#  \\&&( _E5zA~ 5dW\ ]^ _5 1 1?sD DDr9c P|j|jk(rt|jd||jvr8t|ddj|jj d||_||j d<|j||_|jjDcic]\}}|| c}}|_|jj D]/}t|dkDs|jt|ddd 1y cc}}w) zN Set the target language of a nested multi-lingual dictionary zF is not a multi-lingual, nested tokenizer. Cannot set target language.z does not exist. Choose one of z, .r9r:TFr;N) rCrD ValueErrorjoinrIr9 init_kwargsrErFrJrKr )rLr9rOrPrQs r)set_target_langz$Wav2Vec2CTCTokenizer.set_target_langs :: % |+qrs s djj ( },KDIIVZV`V`VeVeVgLhKiijkl l&*5'zz+. )-););)=>A1> \\&&( _E5zA~ 5dW\ ]^ _ ?s? D"returnc|j"|jrtjdyt |jS)z^ `str`: Word delimiter token. Log an error if used while not having been set. N2Using word_delimiter_token, but it is not set yet.r?verboseloggererrorr$rLs r)r7z)Wav2Vec2CTCTokenizer.word_delimiter_token7  % % -$,, LLM N4--..r(cR|jy|j|jSz `Optional[int]`: Id of the word_delimiter_token in the vocabulary. Returns `None` if the token has not been set. Nr?convert_tokens_to_idsr7r`s r)word_delimiter_token_idz,Wav2Vec2CTCTokenizer.word_delimiter_token_id*  % % -))$*C*CDDr(c||_yNr?rLvalues r)r7z)Wav2Vec2CTCTokenizer.word_delimiter_token %*"r(c0|j||_yrirer?rks r)rfz,Wav2Vec2CTCTokenizer.word_delimiter_token_id%)%?%?%F"r(c,t|jSrirJrFr`s r) vocab_sizezWav2Vec2CTCTokenizer.vocab_size4<<  r(cft|j}|j|j|Sri)dictrDupdateadded_tokens_encoder)rLrCs r) get_vocabzWav2Vec2CTCTokenizer.get_vocabs'T\\" T../ r( new_tokensspecial_tokensc g}|D]B}t|tr|jt|ddd2|j|Dt|||S)NFr;) isinstancer$appendr rG _add_tokens)rLrzr{to_addrQrRs r)rz Wav2Vec2CTCTokenizer._add_tokenssY %E%% juUW\]^ e$  % w"6>::r(c |jr|j}t|jd|jS)z\ Converts a string into a sequence of tokens (string), using the tokenizer.  )r6upperlistreplacer7)rLrrMs r) _tokenizezWav2Vec2CTCTokenizer._tokenize s4   ::z@Wav2Vec2CTCTokenizer.convert_tokens_to_string..,s%+tO`uV`UCZ8H4I,J+ts#%r:c"|jk7Srir5)charrLs r)z?Wav2Vec2CTCTokenizer.convert_tokens_to_string..2s44>>3Ir(Nz`char_offsets`: z and `processed_tokens`: z9 have to be of the same length, but are: `len(offsets)`: z and `len(processed_tokens)`: rr)rJziprrfilterr7r8_compute_offsetsr5rU enumerate_get_word_offsetsrVstripr6lower)rLrrrrrcharschar_repetitionsprocessed_charsrrri join_charstrings` r)convert_tokens_to_stringz-Wav2Vec2CTCTokenizer.convert_tokens_to_strings v;! BG G &)+tdklrds+t&u #E#E"6{aS0 v&I5QRix `d8Q8Q0QD , ,W[ [  '+* | "5001A5$..YL< C$88 &|n4MoM^''*<'8&9:O,-/%_5 /4*. Q' / L"#55lDDdDde '# 9Cb 0668   \\^F l[[K s!$E2rr ctc_tokenc tj|j}tjdg|ddf}t |||Dcgc] \}}}|||d}}}}t t fd|}|Scc}}}w)Nr)r start_offset end_offsetc|dk7S)Nrr')offsetsrs r)rz7Wav2Vec2CTCTokenizer._compute_offsets..hsgfo.Jr()npasarraycumsum concatenaterrr) rrr end_indices start_indicestsers ` r)rz%Wav2Vec2CTCTokenizer._compute_offsets\sjj!1299; ["-='>? LOuVcepKq  @G1aQ ;  vJGTU  sBrword_delimiter_charcg}d}d}d}d}t|D]L\}}|d} | |k(rdnd} | |k(r |d}|| z }n'| dk(r|j|||dn |d}|d}| }| }N|dk(r|j|||d|S) NSPACErrrWORDr)wordrrr)rr~) rrr last_staterrrroffsetrstates r)rz&Wav2Vec2CTCTokenizer._get_word_offsetsks    "7+ IAv&>D#'::GE "#L1  G# ''|cm(no$*.#9L!' !5JDJ' (     |[e f gr(c |rd|z}||fS)Nrr')rLris_split_into_wordsrMs r)prepare_for_tokenizationz-Wav2Vec2CTCTokenizer.prepare_for_tokenizations :Df~r( token_idsskip_special_tokensclean_up_tokenization_spacesch|j||}g} |D]A} |r,| |jvs| |jk7r| |jvr1| j | C|j | ||||} | d} ||n |j }|r|j| } |s|rt| | d| dS| S)7 special _decode function is needed for Wav2Vec2Tokenizer because added tokens should be treated exactly the same as tokens of the base vocabulary and therefore the function `convert_tokens_to_string` has to be called on the whole token list and not individually on added tokens r)rrrrrrrr) convert_ids_to_tokensall_special_idsr5all_special_tokensr~rrclean_up_tokenizationr) rLrrrrrrrfiltered_tokensrrQ string_outputrs r)_decodezWav2Vec2CTCTokenizer._decodes44YTg4h$ !E"---%4>>2IeW[WnWnNn MM%  !55 %*G 3 3 6 V$,7 )22 % (--d3D "5-*>:*>:  Kr( sequencesz np.ndarrayz torch.Tensorz tf.Tensorc |Dcgc]}|j|f||||d|}}|s|r0t|dD cic]} | |D cgc]} | |  c} c} } S|Scc}wcc} wcc} } w)a Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. output_char_offsets (`bool`, *optional*, defaults to `False`): Whether or not to output character offsets. Character offsets can be used in combination with the sampling rate and model downsampling rate to compute the time-stamps of transcribed characters. Please take a look at the Example of [`~Wav2Vec2CTCTokenizer.decode`] to better understand how to make use of `output_char_offsets`. [`~Wav2Vec2CTCTokenizer.batch_decode`] works the same way with batched output. output_word_offsets (`bool`, *optional*, defaults to `False`): Whether or not to output word offsets. Word offsets can be used in combination with the sampling rate and model downsampling rate to compute the time-stamps of transcribed words. Please take a look at the Example of [`~Wav2Vec2CTCTokenizer.decode`] to better understand how to make use of `output_word_offsets`. [`~Wav2Vec2CTCTokenizer.batch_decode`] works the same way with batched output. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `List[str]` or [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`]: The list of decoded sentences. Will be a [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`] when `output_char_offsets == True` or `output_word_offsets == True`. )rrrrr)decoder) rLrrrrrrMseq batch_decodedrOds r) batch_decodez!Wav2Vec2CTCTokenizer.batch_decodesx!   DKK $7-I$7$7         "5-YfghYi.jTUq2OA1Q42O/O.jk k  3P.jsA A&  A! A& !A& c Ht|}|jd|||||d|S)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. output_char_offsets (`bool`, *optional*, defaults to `False`): Whether or not to output character offsets. Character offsets can be used in combination with the sampling rate and model downsampling rate to compute the time-stamps of transcribed characters. Please take a look at the example below to better understand how to make use of `output_char_offsets`. output_word_offsets (`bool`, *optional*, defaults to `False`): Whether or not to output word offsets. Word offsets can be used in combination with the sampling rate and model downsampling rate to compute the time-stamps of transcribed words. Please take a look at the example below to better understand how to make use of `output_word_offsets`. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str` or [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`]: The list of decoded sentences. Will be a [`~models.wav2vec2.tokenization_wav2vec2.Wav2Vec2CTCTokenizerOutput`] when `output_char_offsets == True` or `output_word_offsets == True`. Example: ```python >>> # Let's see how to retrieve time steps for a model >>> from transformers import AutoTokenizer, AutoFeatureExtractor, AutoModelForCTC >>> from datasets import load_dataset >>> import datasets >>> import torch >>> # import model, feature extractor, tokenizer >>> model = AutoModelForCTC.from_pretrained("facebook/wav2vec2-base-960h") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/wav2vec2-base-960h") >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base-960h") >>> # load first sample of English common_voice >>> dataset = load_dataset("mozilla-foundation/common_voice_11_0", "en", split="train", streaming=True, trust_remote_code=True) >>> dataset = dataset.cast_column("audio", datasets.Audio(sampling_rate=16_000)) >>> dataset_iter = iter(dataset) >>> sample = next(dataset_iter) >>> # forward sample through model to get greedily predicted transcription ids >>> input_values = feature_extractor(sample["audio"]["array"], return_tensors="pt").input_values >>> logits = model(input_values).logits[0] >>> pred_ids = torch.argmax(logits, axis=-1) >>> # retrieve word stamps (analogous commands for `output_char_offsets`) >>> outputs = tokenizer.decode(pred_ids, output_word_offsets=True) >>> # compute `time_offset` in seconds as product of downsampling ratio and sampling_rate >>> time_offset = model.config.inputs_to_logits_ratio / feature_extractor.sampling_rate >>> word_offsets = [ ... { ... "word": d["word"], ... "start_time": round(d["start_offset"] * time_offset, 2), ... "end_time": round(d["end_offset"] * time_offset, 2), ... } ... for d in outputs.word_offsets ... ] >>> # compare word offsets with audio `en_train_0/common_voice_en_19121553.mp3` online on the dataset viewer: >>> # https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0/viewer/en >>> word_offsets[:3] [{'word': 'THE', 'start_time': 0.7, 'end_time': 0.78}, {'word': 'TRICK', 'start_time': 0.88, 'end_time': 1.08}, {'word': 'APPEARS', 'start_time': 1.2, 'end_time': 1.64}] ```)rrrrrr')rr)rLrrrrrrMs r)rzWav2Vec2CTCTokenizer.decodes@|i( t||  3)E 3 3     r(save_directoryfilename_prefixc tjj|stj d|dytjj ||r|dzndt dz}t|dd5}|jtj|jd d d d zddd|fS#1swY|fSxYwNzVocabulary path (z) should be a directory-rrwr/r0TF)indent sort_keys ensure_ascii ) ospathisdirr^r_rVVOCAB_FILES_NAMESr@writerAdumpsrCrLrrrfs r)save_vocabularyz$Wav2Vec2CTCTokenizer.save_vocabularyysww}}^, LL,^,<|rFN)F)TFFF)r)FNTFFF)FNFFri)'r r!r"r#rvocab_files_namesmodel_input_namesrHr$rXpropertyr7rintrfsetterrsrryr rr boolrrrrfloatr staticmethodrrrrrrr r __classcell__rRs@r)r+r+ss<*$&67  $'/_b_3_*/c//E#EE  +!+##G$G!C!!4 ;eDItJ7G,G&H ;Z^ ;kn ;BI#I#I##".3$)$) =\S =\=\(, =\ " =\ " =\ c5e$$ %=\~ s) ,0I BE  d3c3h'( )  JM c5e,,- DG  c5e$$ %  D%*7;!.3.3.3292"2'/tn 2  2 (, 2&d^2&d^2 2t%*7;$)$) BcDcO\>S^^_B"B'/tn B " B " B cBR%*7;$)$) g d3i~{RSg "g '/tn g " g " g  g R c HSM ]bcf]g r(r+c8eZdZdZeZddiddidZddgZ d%fd Ze d e fd Z e d e e fd Ze jd Z ejd Zee d&deej(eeeej(eeefdeee efde e de e de e de ee efded efdZe d e fdZd efdZde d e fdZde d e fdZ dee d e fdZ! d'dee ded e ed e fd!Z"d(d"e d#e e d e#e fd$Z$xZ%S))Wav2Vec2Tokenizera8 Constructs a Wav2Vec2 tokenizer. This tokenizer inherits from [`PreTrainedTokenizer`] which contains some of the main methods. Users should refer to the superclass for more information regarding such methods. Args: vocab_file (`str`): File containing the vocabulary. bos_token (`str`, *optional*, defaults to `""`): The beginning of sentence token. eos_token (`str`, *optional*, defaults to `""`): The end of sentence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. word_delimiter_token (`str`, *optional*, defaults to `"|"`): The token used for defining the end of a word. do_lower_case (`bool`, *optional*, defaults to `False`): Whether or not to lowercase the output when decoding. do_normalize (`bool`, *optional*, defaults to `False`): Whether or not to zero-mean unit-variance normalize the input. Normalizing can help to significantly improve the performance for some models, *e.g.*, [wav2vec2-lv60](https://huggingface.co/models?search=lv60). return_attention_mask (`bool`, *optional*, defaults to `False`): Whether or not [`~Wav2Vec2Tokenizer.__call__`] should return `attention_mask`. Wav2Vec2 models that have set `config.feat_extract_norm == "group"`, such as [wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base-960h), have **not** been trained using `attention_mask`. For such models, `input_values` should simply be padded with 0 and no `attention_mask` should be passed. For Wav2Vec2 models that have set `config.feat_extract_norm == "layer"`, such as [wav2vec2-lv60](https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self), `attention_mask` should be passed for batched inference. **kwargs Additional keyword arguments passed along to [`PreTrainedTokenizer`] zfacebook/wav2vec2-base-960hzJhttps://huggingface.co/facebook/wav2vec2-base-960h/resolve/main/vocab.jsonzNhttps://huggingface.co/facebook/wav2vec2-base-960h/resolve/main/tokenizer.jsonr input_valuesr-c tjdt||_||_| |_||_t|d5} tj| |_ ddd|jjD cic]\} } | |  c} } |_ t|8d||||||| |d| y#1swYUxYwcc} } w)NzThe class `Wav2Vec2Tokenizer` is deprecated and will be removed in version 5 of Transformers. Please use `Wav2Vec2Processor` or `Wav2Vec2CTCTokenizer` instead.r/r0)r2r3r4r5r6 do_normalizereturn_attention_maskr7r')warningswarn FutureWarningr?r6rr r@rArBrDrErFrGrH)rLrr3r4r2r5r7r6r rrMrNrOrPrRs r)rHzWav2Vec2Tokenizer.__init__s   F  &:"*%:"( *w / 3<99\2DL 3*.););)=>A1>   '%"7!5   3 3?sB4 C4B=rYc|j"|jrtjdyt |jS)zW `str`: Padding token. Log an error if used while not having been set. Nr[r\r`s r)r7z&Wav2Vec2Tokenizer.word_delimiter_tokenrar(cR|jy|j|jSrcrdr`s r)rfz)Wav2Vec2Tokenizer.word_delimiter_token_idrgr(c||_yrirjrks r)r7z&Wav2Vec2Tokenizer.word_delimiter_tokenrmr(c0|j||_yrirorks r)rfz)Wav2Vec2Tokenizer.word_delimiter_token_idrpr( raw_speechpadding max_lengthpad_to_multiple_of padding_sidereturn_tensorsr]c dt|tjxrt|jdkD} | r&t|jdkDrt d|| xs@t|t tfxr(t|dtjtt f} | r@t|dtjs#|D cgc]} tj| }} n1| s/t|tjstj|}| s|g}|jrQ|D cgc]F} | tj| z tjtj| dzz H}} td|i} |j| |||||j||}|Scc} wcc} w)a Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of sequences. Args: raw_speech (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`): The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy array or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep. padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. r:rz2Only mono-channel audio is supported for input to rgh㈵>r )rrrrrrr])r}rndarrayrJshaperUrtuplerr meansqrtvarrpadr)rLrrrrrrr]rMis_batched_numpy is_batchedspeechxencoded_inputs padded_inputss r)__call__zWav2Vec2Tokenizer.__call__ s6&j"**=[#jFVFVBWZ[B[ J$4$4 5 9QRVQWXY Y% zD%= 1 lz*Q-RTR\R\^ceiQj7k  jA C;EF"**V,FJFJz2::$FJ/J$J   PZ[11rwwqz>RWWRVVAY5E-FF[J[' 'CD !1%"&"<"<)!  5G\s F()A F-c,t|jSrirrr`s r)rszWav2Vec2Tokenizer.vocab_sizeIrtr(cBt|jfi|jSri)rvrDrxr`s r)ryzWav2Vec2Tokenizer.get_vocabMsDLL>D$=$=>>r(rQc|jj||jj|jSrrrs r)rz&Wav2Vec2Tokenizer._convert_token_to_idPrr(rcR|jj||j}|Srrrs r)rz&Wav2Vec2Tokenizer._convert_id_to_tokenTrr(rc6t|Dcgc]}|d }}ttfd|}dj|Dcgc]}|jk(rdn|c}j }j r|j}|Scc}wcc}w)rrc"|jk7Srir)rQrLs r)rz.asET^^4Kr(rr)rrrrVr7rr6r)rLr token_groupgrouped_tokensrrQrs` r)rz*Wav2Vec2Tokenizer.convert_tokens_to_stringYs =DFOL[+a.LLv&K^\]\klSX$*C*C!C#Nlmssu   \\^F M ms BBrrrc (|j||}g}|D]A}|r,||jvs||jk7r||jvr1|j |C|j |}||n |j }|r|j|} | S|S)rr)rrr5rr~rrr) rLrrrrMrrrQr clean_texts r)rzWav2Vec2Tokenizer._decodeks44YTg4h$ !E"---%4>>2IeW[WnWnNn MM%  !,,V4,7 )22 % (33D9J Kr(rrc tjj|stj d|dytjj ||r|dzndt dz}t|dd5}|jtj|jd d d d zddd|fS#1swY|fSxYwr) rrrr^r_rVrr@rrArrDrs r)rz!Wav2Vec2Tokenizer.save_vocabularysww}}^, LL,^,<rFs' !DD5@      H %4:$sE#s(O+, -  == =*Q.QhR+Rj "#6 7r(