T^(hQdZddlZddlZddlZddlmZmZmZddlm Z m Z m Z m Z ddl mZejeZddiZd Zd ZGd d e ZGd dZGddZd gZy)z"Tokenization classes for Splinter.N)ListOptionalTuple)PreTrainedTokenizer _is_control_is_punctuation_is_whitespace)logging vocab_filez vocab.txtctj}t|dd5}|j}dddt D]\}}|j d}|||<|S#1swY4xYw)z*Loads a vocabulary file into a dictionary.rutf-8encodingN ) collections OrderedDictopen readlines enumeraterstrip)r vocabreadertokensindextokens p/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/splinter/tokenization_splinter.py load_vocabr sw  # # %E j# 0$F!!#$!&) u T"e  L $$s A''A0cN|j}|sgS|j}|S)z@Runs basic whitespace cleaning and splitting on a piece of text.)stripsplit)textrs rwhitespace_tokenizer$+s% ::>r%c.|jjSN)r<r)rIs rr)zSplinterTokenizer.do_lower_cases##111r%c,t|jSrL)lenrrIs r vocab_sizezSplinterTokenizer.vocab_sizes4::r%cBt|jfi|jSrL)dictradded_tokens_encoderrIs r get_vocabzSplinterTokenizer.get_vocabsDJJ<$";";<>@ r% token_ids_0 token_ids_1returnc||jg|z|jgzS|jg}|jg}|jg|jdgz}|jdk(r||z|z|z|z|zS||z|z|z|z|zS)a Build model inputs from a pair of sequence for question answering tasks by concatenating and adding special tokens. A Splinter sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences for question answering: `[CLS] question_tokens [QUESTION] . [SEP] context_tokens [SEP]` Args: token_ids_0 (`List[int]`): The question token IDs if pad_on_right, else context tokens IDs token_ids_1 (`List[int]`, *optional*): The context token IDs if pad_on_right, else question token IDs Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. .right) cls_token_id sep_token_idrJrH padding_side)rBrfrgclssepquestion_suffixs r build_inputs_with_special_tokensz2SplinterTokenizer.build_inputs_with_special_tokenss&  %%&48I8I7JJ J  !  !112d6P6PQT6U5VV    '$6<{JSP P$s*[8?JSP Pr%already_has_special_tokensc|rt|||dS|+dgdgt|zzdgzdgt|zzdgzSdgdgt|zzdgzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rfrgrsr)r@get_special_tokens_maskrN)rBrfrgrsrFs rrvz)SplinterTokenizer.get_special_tokens_masks& &72'[]a3   "31#K 001QC7A3[AQ;QRVWUXX XsqcC ,,-33r%cn|jg}|jg}|jg|jdgz}|t ||z|zdgzS|j dk(r+t ||z|z|zdgzt ||zdgzzSt ||z|zdgzt ||z|zdgzzS)a Create the token type IDs corresponding to the sequences passed. [What are token type IDs?](../glossary#token-type-ids) Should be overridden in a subclass if the model has a special way of building those. Args: token_ids_0 (`List[int]`): The first tokenized sequence. token_ids_1 (`List[int]`, *optional*): The second tokenized sequence. Returns: `List[int]`: The token type ids. rjrrkru)rmrlrJrHrNrn)rBrfrgrprorqs r$create_token_type_ids_from_sequencesz6SplinterTokenizer.create_token_type_ids_from_sequencess   !  !112d6P6PQT6U5VV  s[(3./1#5 5    's[(?:S@AQCG#k\_N_J`decfJff fs[(3./1#5K/\}}||k7rtjd |d |}|j|d z|d z }@ ddd|fS#1swY|fSxYw)Nr-rar wrrc |dS)Nrur4)kvs rz3SplinterTokenizer.save_vocabulary..!s PRSTPUr%)keyzSaving vocabulary to z\: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted!rru) r5r6isdirrbVOCAB_FILES_NAMESrsortedrr9loggerwarningwrite)rBryrzrr writerr token_indexs rsave_vocabularyz!SplinterTokenizer.save_vocabularys 77== (/3!6rUfgsUt tJ4C/C/n\J *cG 4 &,TZZ-=-=-?EU&V "{K'NN/ |rr __classcell__)rFs@rr'r'4se)V* ##. `??22= E= JNQ9Q3;DI3FQ cQBsx4943;DI3F4ko4 c4:JNg9g3;DI3Fg cg:cHSM]bcf]gr%r'c@eZdZdZd dZd dZdZd dZdZdZ d Z y) r;a Constructs a BasicTokenizer that will run basic tokenization (punctuation splitting, lower casing, etc.). Args: do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). NcV|g}||_t||_||_||_yrL)r)setr*r+r,)rBr)r*r+r,s rrAzBasicTokenizer.__init__As2  K*{+&<#*r%c|r$|jjt|n |j}|j|}|jr|j |}t |}g}|D]}||vrY|jr0|j}|jdur/|j|}n|jr|j|}|j|j||t dj|}|S)a Basic Tokenization of a piece of text. Split on "white spaces" only, for sub-word tokenization, see WordPieceTokenizer. Args: **never_split**: (*optional*) list of str Kept for backward compatibility purposes. Now implemented directly at the base class level (see [`PreTrainedTokenizer.tokenize`]) List of token not to split. Fr`)r*unionr _clean_textr+_tokenize_chinese_charsr$r)lowerr,_run_strip_accentsextend_run_split_on_puncrb)rBr#r* orig_tokensrXr output_tokenss rrUzBasicTokenizer.tokenizeIsCNd&&,,S-=>SWScSc %  & &//5D)$/    MEK'%%!KKME))6 $ 7 7 >'' 33E:E    7 7{ K L M,CHH\,BC r%ctjd|}g}|D].}tj|}|dk(r|j|0dj |S)z$Strips accents from a piece of text.NFDMnra) unicodedata normalizecategoryrWrb)rBr#outputcharcats rrz!BasicTokenizer._run_strip_accentsns^$$UD1 D&&t,Cd{ MM$   wwvr%c^|||vr|gSt|}d}d}g}|t|krb||}t|r|j|gd}n)|r|jgd}|dj||dz }|t|krb|Dcgc]}dj |c}Scc}w)z&Splits punctuation on a piece of text.rTFrura)listrNr rWrb) rBr#r*charsistart_new_wordrrxs rrz!BasicTokenizer._run_split_on_puncys  "t{':6MT  #e*n8Dt$ tf%!%!MM"%!&r !!$' FA#e*n%++q +++sB*cg}|D]c}t|}|j|r4|jd|j||jdS|j|edj|S)z)Adds whitespace around any CJK character.r`ra)ord_is_chinese_charrWrbrBr#rrcps rrz&BasicTokenizer._tokenize_chinese_charssm $DTB$$R( c" d# c" d# $wwvr%c|dk\r|dksF|dk\r|dks<|dk\r|dks2|dk\r|dks(|d k\r|d ks|d k\r|d ks|d k\r|dks |dk\r|dkryy)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr4)rBrs rrzBasicTokenizer._is_chinese_charsr6\bFlf vg "-g "-g "-g "-f vg "-r%cg}|D]Q}t|}|dk(s|dk(s t|r$t|r|jdA|j|Sdj |S)zBPerforms invalid character removal and whitespace cleanup on text.rir`ra)rrr rWrbrs rrzBasicTokenizer._clean_textsf $DTBQw",+d*;d# c" d# $wwvr%)TNTNrL) rrrrrArUrrrrrr4r%rr;r;-s+&+#J ,, 0 r%r;ceZdZdZddZdZy)r=zRuns WordPiece tokenization.c.||_||_||_yrL)rr-max_input_chars_per_word)rBrr-rs rrAzWordpieceTokenizer.__init__s "(@%r%cg}t|D]}t|}t||jkDr|j |j Bd}d}g}|t|krst|}d} ||kr:dj |||} |dkDrd| z} | |jvr| } n |dz}||kr:| d}n"|j | |}|t|krs|r|j |j |j||S)a Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the given vocabulary. For example, `input = "unaffable"` wil return as output `["un", "##aff", "##able"]`. Args: text: A single token or whitespace separated tokens. This should have already been passed through *BasicTokenizer*. Returns: A list of wordpiece tokens. FrNraz##ruT) r$rrNrrWr-rbrr) rBr#rrris_badstart sub_tokensend cur_substrsubstrs rrUzWordpieceTokenizer.tokenizes& (. 1EKE5zD999$$T^^4FEJ#e*$%j! ckWWU5%56Fqy!%+%+ 1HCck%!F!!*-#e*$"$$T^^4$$Z0; 1<r%N)d)rrrrrArUr4r%rr=r=s&A .r%r=)rrr5rtypingrrrtokenization_utilsrrr r utilsr get_loggerrrrrr$r'r;r=__all__r4r%rrs} ) ((cc   H %!;/v+vrRRj66r  r%