S^(hQdZddlZddlZddlZddlmZmZmZddlm Z m Z m Z m Z ddl mZejeZddiZd Zd ZGd d e ZGd dZGddZgdZy)zTokenization classes for Bert.N)ListOptionalTuple)PreTrainedTokenizer _is_control_is_punctuation_is_whitespace)logging vocab_filez vocab.txtctj}t|dd5}|j}dddt D]\}}|j d}|||<|S#1swY4xYw)z*Loads a vocabulary file into a dictionary.rutf-8encodingN ) collections OrderedDictopen readlines enumeraterstrip)r vocabreadertokensindextokens h/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/bert/tokenization_bert.py load_vocabrsw  # # %E j# 0$F!!#$!&) u T"e  L $$s A''A0cN|j}|sgS|j}|S)z@Runs basic whitespace cleaning and splitting on a piece of text.)stripsplit)textrs rwhitespace_tokenizer$*s% ::>@ r% token_ids_0 token_ids_1returnc||jg|z|jgzS|jg}|jg}||z|z|z|zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) cls_token_id sep_token_id)rBrdreclsseps r build_inputs_with_special_tokensz.BertTokenizer.build_inputs_with_special_tokenssb&  %%&48I8I7JJ J  !  ![ 3&4s::r%already_has_special_tokensc|rt|||dS|+dgdgt|zzdgzdgt|zzdgzSdgdgt|zzdgzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rdrermr)r@get_special_tokens_maskrK)rBrdrermrFs rrpz%BertTokenizer.get_special_tokens_masks& &72'[]a3   "31#K 001QC7A3[AQ;QRVWUXX XsqcC ,,-33r%c|jg}|jg}|t||z|zdgzSt||z|zdgzt||zdgzzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). rro)rirhrK)rBrdrerkrjs r$create_token_type_ids_from_sequencesz2BertTokenizer.create_token_type_ids_from_sequencesst.  !  !  s[(3./1#5 53$s*+qc1C c8I4JaS4PPPr%save_directoryfilename_prefixcd}tjj|r2tjj||r|dzndtdz}n |r|dznd|z}t |dd5}t |jjd D]>\}}||k7rtjd |d |}|j|d z|d z }@ ddd|fS#1swY|fSxYw)Nr-r_r wrrc |dS)Nror4)kvs rz/BertTokenizer.save_vocabulary..s PRSTPUr%)keyzSaving vocabulary to z\: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted!rro) r5r6isdirr`VOCAB_FILES_NAMESrsortedrr9loggerwarningwrite)rBrsrtrr writerr token_indexs rsave_vocabularyzBertTokenizer.save_vocabulary s 77== (/3!6rUfgsUt tJ4C/C/n\J *cG 4 &,TZZ-=-=-?EU&V "{K'NN/ |rr __classcell__)rFs@rr'r'3sL*X* #%)/ b22= E= JN;9;3;DI3F; c;4sx4943;DI3F4ko4 c4:JNQ9Q3;DI3FQ cQ:cHSM]bcf]gr%r'cJeZdZdZ d dZd dZdZd dZdZdZ d Z y) r;a Constructs a BasicTokenizer that will run basic tokenization (punctuation splitting, lower casing, etc.). Args: do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). do_split_on_punc (`bool`, *optional*, defaults to `True`): In some instances we want to skip the basic punctuation splitting so that later tokenization can capture the full context of the words, such as contractions. Ncd|g}||_t||_||_||_||_yrH)r)setr*r+r,do_split_on_punc)rBr)r*r+r,rs rrAzBasicTokenizer.__init__8s<  K*{+&<#* 0r%cJ|r$|jjt|n |j}|j|}|jr|j |}t jd|}t|}g}|D]}||vrY|jr0|j}|jdur/|j|}n|jr|j|}|j|j||tdj|}|S)aj Basic Tokenization of a piece of text. For sub-word tokenization, see WordPieceTokenizer. Args: never_split (`List[str]`, *optional*) Kept for backward compatibility purposes. Now implemented directly at the base class level (see [`PreTrainedTokenizer.tokenize`]) List of token not to split. NFCFr^)r*unionr _clean_textr+_tokenize_chinese_chars unicodedata normalizer$r)lowerr,_run_strip_accentsextend_run_split_on_puncr`)rBr#r*unicode_normalized_text orig_tokensrVr output_tokenss rrRzBasicTokenizer.tokenizeHsCNd&&,,S-=>SWScSc %  & &//5D"-"7"7t"D)*AB    MEK'%%!KKME))6 $ 7 7 >'' 33E:E    7 7{ K L M,CHH\,BC r%ctjd|}g}|D].}tj|}|dk(r|j|0dj |S)z$Strips accents from a piece of text.NFDMnr_)rrcategoryrTr`)rBr#outputcharcats rrz!BasicTokenizer._run_strip_accentsns^$$UD1 D&&t,Cd{ MM$   wwvr%cv|jr|||vr|gSt|}d}d}g}|t|krb||}t|r|j |gd}n)|r|j gd}|dj ||dz }|t|krb|Dcgc]}dj |c}Scc}w)z&Splits punctuation on a piece of text.rTFror_)rlistrKr rTr`) rBr#r*charsistart_new_wordrrxs rrz!BasicTokenizer._run_split_on_puncys$$)@T[EX6MT  #e*n8Dt$ tf%!%!MM"%!&r !!$' FA#e*n%++q +++sB6cg}|D]c}t|}|j|r4|jd|j||jdS|j|edj|S)z)Adds whitespace around any CJK character.r^r_)ord_is_chinese_charrTr`rBr#rrcps rrz&BasicTokenizer._tokenize_chinese_charssm $DTB$$R( c" d# c" d# $wwvr%c|dk\r|dksF|dk\r|dks<|dk\r|dks2|dk\r|dks(|d k\r|d ks|d k\r|d ks|d k\r|dks |dk\r|dkryy)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr4)rBrs rrzBasicTokenizer._is_chinese_charsr6\bFlf vg "-g "-g "-g "-f vg "-r%cg}|D]Q}t|}|dk(s|dk(s t|r$t|r|jdA|j|Sdj |S)zBPerforms invalid character removal and whitespace cleanup on text.rir^r_)rrr rTr`rs rrzBasicTokenizer._clean_textsf $DTBQw",+d*;d# c" d# $wwvr%)TNTNTrH) rrrrrArRrrrrrr4r%rr;r;!s<0# 1 $L ,, 0 r%r;ceZdZdZddZdZy)r=zRuns WordPiece tokenization.c.||_||_||_yrH)rr-max_input_chars_per_word)rBrr-rs rrAzWordpieceTokenizer.__init__s "(@%r%cg}t|D]}t|}t||jkDr|j |j Bd}d}g}|t|krst|}d} ||kr:dj |||} |dkDrd| z} | |jvr| } n |dz}||kr:| d}n"|j | |}|t|krs|r|j |j |j||S)a Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the given vocabulary. For example, `input = "unaffable"` wil return as output `["un", "##aff", "##able"]`. Args: text: A single token or whitespace separated tokens. This should have already been passed through *BasicTokenizer*. Returns: A list of wordpiece tokens. FrNr_z##roT) r$rrKrrTr-r`rr) rBr#rrris_badstart sub_tokensend cur_substrsubstrs rrRzWordpieceTokenizer.tokenizes& (. 1EKE5zD999$$T^^4FEJ#e*$%j! ckWWU5%56Fqy!%+%+ 1HCck%!F!!*-#e*$"$$T^^4$$Z0; 1<r%N)d)rrrrrArRr4r%rr=r=s&A .r%r=)r;r'r=)rrr5rtypingrrrtokenization_utilsrrr r utilsr get_loggerrrr}rr$r'r;r=__all__r4r%rrs{% ((cc   H %!;/k'k\^^B66r Dr%