T^(hJddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z mZddlmZej"eZdZdd iZgd ZGd d eZd gZy) N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)loggingu▁ vocab_filezsentencepiece.bpe.model)ace_Arabace_Latnacm_Arabacq_Arabaeb_Arabafr_Latnajp_Arabaka_Latnamh_Ethiapc_Arabarb_Arabars_Arabary_Arabarz_Arabasm_Bengast_Latnawa_Devaayr_Latnazb_Arabazj_Latnbak_Cyrlbam_Latnban_Latnbel_Cyrlbem_Latnben_Bengbho_Devabjn_Arabbjn_Latnbod_Tibtbos_Latnbug_Latnbul_Cyrlcat_Latnceb_Latnces_Latncjk_Latnckb_Arabcrh_Latncym_Latndan_Latndeu_Latndik_Latndyu_Latndzo_Tibtell_Grekeng_Latnepo_Latnest_Latneus_Latnewe_Latnfao_Latnpes_Arabfij_Latnfin_Latnfon_Latnfra_Latnfur_Latnfuv_Latngla_Latngle_Latnglg_Latngrn_Latnguj_Gujrhat_Latnhau_Latnheb_Hebrhin_Devahne_Devahrv_Latnhun_Latnhye_Armnibo_Latnilo_Latnind_Latnisl_Latnita_Latnjav_Latnjpn_Jpankab_Latnkac_Latnkam_Latnkan_Kndakas_Arabkas_Devakat_Georknc_Arabknc_Latnkaz_Cyrlkbp_Latnkea_Latnkhm_Khmrkik_Latnkin_Latnkir_Cyrlkmb_Latnkon_Latnkor_Hangkmr_Latnlao_Laoolvs_Latnlij_Latnlim_Latnlin_Latnlit_Latnlmo_Latnltg_Latnltz_Latnlua_Latnlug_Latnluo_Latnlus_Latnmag_Devamai_Devamal_Mlymmar_Devamin_Latnmkd_Cyrlplt_Latnmlt_Latnmni_Bengkhk_Cyrlmos_Latnmri_Latnzsm_Latnmya_Mymrnld_Latnnno_Latnnob_Latnnpi_Devanso_Latnnus_Latnnya_Latnoci_Latngaz_Latnory_Oryapag_Latnpan_Gurupap_Latnpol_Latnpor_Latnprs_Arabpbt_Arabquy_Latnron_Latnrun_Latnrus_Cyrlsag_Latnsan_Devasat_Bengscn_Latnshn_Mymrsin_Sinhslk_Latnslv_Latnsmo_Latnsna_Latnsnd_Arabsom_Latnsot_Latnspa_Latnals_Latnsrd_Latnsrp_Cyrlssw_Latnsun_Latnswe_Latnswh_Latnszl_Latntam_Tamltat_Cyrltel_Telutgk_Cyrltgl_Latntha_Thaitir_Ethitaq_Latntaq_Tfngtpi_Latntsn_Latntso_Latntuk_Latntum_Latntur_Latntwi_Latntzm_Tfnguig_Arabukr_Cyrlumb_Latnurd_Arabuzn_Latnvec_Latnvie_Latnwar_Latnwol_Latnxho_Latnydd_Hebryor_Latnyue_Hantzho_Hanszho_Hantzul_Latnc PeZdZUdZeZddgZgZee e d<gZ ee e d< d+de e eefffd Zd Zd Zed Zed efd Zej,ded dfdZ d,dee de ee ded ee ffd Z d-dee de ee d ee fdZ d-dee de ee d ee fdZdede ede efdZdZded eefdZdZdZdZ d-d ed!e ed e!efd"Z" d.d#eeded$e eeded e#f fd% Z$d&Z%d'Z&d/d(Z'd)ed dfd*Z(xZ)S)0 NllbTokenizeraN Construct an NLLB tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import NllbTokenizer >>> tokenizer = NllbTokenizer.from_pretrained( ... "facebook/nllb-200-distilled-600M", src_lang="eng_Latn", tgt_lang="fra_Latn" ... ) >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_french = "Le chef de l'ONU affirme qu'il n'y a pas de solution militaire en Syrie." >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_french, return_tensors="pt") ``` Args: vocab_file (`str`): Path to the vocabulary file. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. cls_token (`str`, *optional*, defaults to `""`): The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. mask_token (`str`, *optional*, defaults to `""`): The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. tokenizer_file (`str`, *optional*): The path to a tokenizer file to use instead of the vocab file. src_lang (`str`, *optional*): The language to use as source language for translation. tgt_lang (`str`, *optional*): The language to use as target language for translation. sp_model_kwargs (`Dict[str, str]`): Additional keyword arguments to pass to the model initialization. input_idsattention_mask prefix_tokens suffix_tokensNsp_model_kwargsc \| t} t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|dddn|}| in| |_||_t jd i|j|_|jjt|||_ ||||d|_ d|_ t|j|_t|@d |||||||| | | | |j|d || | nd|_|j%|j"|_| |_|j+|j"y) NFT) normalizedspecial)rlstripr)rr r) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokentokenizer_filesrc_langtgt_langadditional_special_tokensrlegacy_behaviourr=)FAIRSEQ_LANGUAGE_CODES isinstancestrr rrspmSentencePieceProcessorsp_modelLoadr_added_tokens_decoderfairseq_offsetlen sp_model_sizesuper__init__ _src_langconvert_tokens_to_idscur_lang_code_idrset_src_lang_special_tokens)selfrrrrrrrrrrrrrrkwargs __class__s h/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/nllb/tokenization_nllb.pyrzNllbTokenizer.__init__qs$ % ,(> %MWXacfMgJyUDImv MWXacfMgJyUDImv MWXacfMgJyUDImv MWXacfMgJyUDImv *c* zd4 N  &5%> J   ((8c~|jj}d|d<|jj|d<|S)Nrsp_model_proto)__dict__copyrserialized_model_proto)rstates r __getstate__zNllbTokenizer.__getstate__s; ""$ j"&--"F"F"H rc||_t|dsi|_tjdi|j|_|j j |jy)Nrr)r hasattrrrrrLoadFromSerializedProtor )rds r __setstate__zNllbTokenizer.__setstate__sQ t./#%D 22JT5I5IJ  --d.A.ABrcFt|j|jzSN)rrrrs r vocab_sizezNllbTokenizer.vocab_sizes4==!D$7$777rreturnc|jSr)rrs rrzNllbTokenizer.src_langs ~~r new_src_langcH||_|j|jyr)rr)rrs rrzNllbTokenizer.src_langs% ((8r token_ids_0 token_ids_1already_has_special_tokensc|rt|||dSdgt|jz}dgt|jz}||dgt|zz|zS|dgt|zzdgt|zz|zS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrrr)rget_special_tokens_maskrrr)rrrr prefix_ones suffix_onesrs rr!z%NllbTokenizer.get_special_tokens_masks& &72'[]a3 cC 2 233 cC 2 233  1#K(8"89KG GqcC $445!s;?O9OPS^^^rc|||j|z|jzS|j|z|z|jzS)aa Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An NLLB sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )rr)rrrs r build_inputs_with_special_tokensz.NllbTokenizer.build_inputs_with_special_tokenssG,  %% 3d6H6HH H!!K/+=@R@RRRrc|jg}|jg}|t||z|zdgzSt||z|z|z|z|zdgzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. nllb does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. r) sep_token_id cls_token_idr)rrrsepclss r$create_token_type_ids_from_sequencesz2NllbTokenizer.create_token_type_ids_from_sequencessm$  !  !  s[(3./1#5 53$s*S0;>DEKKrreturn_tensorsrrc v|| td||_||fd|d|}|j|}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensr,forced_bos_token_id) ValueErrorrr)r raw_inputsr,rr extra_kwargsinputs tgt_lang_ids r_build_translation_inputsz'NllbTokenizer._build_translation_inputs'sY  x/`a a  jiT.i\hi00: (3$% rct|jDcic]}|j||}}|j|j|Scc}wr)rangerconvert_ids_to_tokensupdateadded_tokens_encoder)rivocabs r get_vocabzNllbTokenizer.get_vocab3sK;@;QRa++A.1RR T../ SsAtextcD|jj|tS)N)out_type)rencoder)rr>s r _tokenizezNllbTokenizer._tokenize8s}}##D3#77rcr|jj|}|r||jzS|jS)z0Converts a token (str) in an id using the vocab.)r PieceToIdr unk_token_id)rtokenspm_ids r_convert_token_to_idz"NllbTokenizer._convert_token_to_id;s4((//5v+++L4;L;LLrcR|jj||jz S)z=Converts an index (integer) in a token (str) using the vocab.)r IdToPiecer)rindexs r_convert_id_to_tokenz"NllbTokenizer._convert_id_to_tokenAs"}}&&ut/B/B'BCCrcldj|jtdj}|S)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)rtokens out_strings rconvert_tokens_to_stringz&NllbTokenizer.convert_tokens_to_stringEs,WWV_,,-=sCIIK rsave_directoryfilename_prefixctjj|stj d|dytjj ||r|dzndt dz}tjj|jtjj|k7rBtjj|jrt|j||fStjj|jsCt|d5}|jj}|j|ddd|fS|fS#1swY|fSxYw)NzVocabulary path (z) should be a directory-rNrwb)ospathisdirloggererrorrPVOCAB_FILES_NAMESabspathrisfileropenrr write)rrWrXout_vocab_fileficontent_spiece_models rsave_vocabularyzNllbTokenizer.save_vocabularyJs%ww}}^, LL,^,<>rc8|j|jSr)set_tgt_lang_special_tokensrrs r_switch_to_target_modez$NllbTokenizer._switch_to_target_modejrprc|j||_|jr%g|_|j|jg|_y|jg|_|jg|_y)zReset the special tokens to the source lang setting. - In legacy mode: No prefix and suffix=[eos, src_lang_code]. - In default mode: Prefix=[src_lang_code], suffix = [eos] Nr cur_lang_coderr eos_token_idr)rrs rrz)NllbTokenizer.set_src_lang_special_tokensmsc "77A  !#D "&"3"3T5G5G!HD "&"4"4!5D "&"3"3!4D rlangc|j||_|jr%g|_|j|jg|_y|jg|_|jg|_y)zReset the special tokens to the target lang setting. - In legacy mode: No prefix and suffix=[eos, tgt_lang_code]. - In default mode: Prefix=[tgt_lang_code], suffix = [eos] Nru)rrxs rrrz)NllbTokenizer.set_tgt_lang_special_tokenszsc "77=  !#D "&"3"3T5G5G!HD "&"4"4!5D "&"3"3!4D r) r{rzzzzNNNNNF)NFr)r=NrG)rN)*__name__ __module__ __qualname____doc__ravocab_files_namesmodel_input_namesrrint__annotations__rrrrrrrrpropertyrrsetterboolr!r%r+r5r=rBrHrLrVrrir rmrorsrrr __classcell__)rs@rrr$sDL*$&67!M49!!M49! 48"&E9"$sCx.1E9N C88#__9S9T99 sx_9_3;DI3F_ko_ c_>JNS9S3;DI3FS cS8JNL9L3;DI3FL cL2 *- 9A# RZ[^R_  8c8d3i8M D !c!HSM!]bcf]g!(#)-" M9 M MDI& M  M  M?? 5 5 5 5rr)r\shutilrtypingrrrrr sentencepiecertokenization_utilsr r r utilsr get_loggerr|r_rRrarr__all__rrrrsh  33PP   H %!#<=R&a5'a5H  r