S^(h3dZddlZddlZddlmZmZmZddlmZddl m Z e je Z ddd Zd ZGd d eZd gZy) z Tokenization classes for BioGPT.N)ListOptionalTuple)PreTrainedTokenizer)loggingz vocab.jsonz merges.txt) vocab_file merges_filecbt}|d}|ddD]}|j||f|}|S)z Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rN)setadd)wordpairs prev_charchars l/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/biogpt/tokenization_biogpt.py get_pairsr!sF EEQIQR 9d#$  Lc BeZdZdZeZddgZ dfd ZedZ dZ dZ dZ d Z dd Zd Zd Zd Z ddeedeeedeefdZ ddeedeeededeeffd Z ddeedeeedeefdZddedeedeefdZdZdZxZS)BioGptTokenizera: Construct an FAIRSEQ Transformer tokenizer. Moses tokenization followed by Byte-Pair Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Merges file. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc  ddl} d|_| |_i|_i|_ t |d5} tj| |_ ddd|jjD cic]\} } | |  c} } |_ t |d5} | jjddd}dddDcgc]}t|jdd }}tt!|t#t%||_i|_t+|Xd |||||d |y#t$r tdwxYw#1swYxYwcc} } w#1swYxYwcc}w) NrzqYou need to install sacremoses to use BioGptTokenizer. See https://pypi.org/project/sacremoses/ for installation.enutf-8encoding ) bos_token eos_token sep_token unk_token pad_token) sacremoses ImportErrorlangsmcache_moses_tokenizercache_moses_detokenizeropenjsonloadencoderitemsdecoderreadsplittupledictziprangelen bpe_rankscachesuper__init__)selfr r r%r"r#r$r&kwargsr( vocab_handlekv merges_handlemergesmerge __class__s rr>zBioGptTokenizer.__init__\sb   %'"')$ *w / 3<99\2DL 3)-););)=>A1> + 0 ;M"'')//5cr:F ;8>?u% bq)*??c&%F *<=>       - M   3 3> ; ;?s.D1E 1 E#E#E'1E EE$c,t|jS)zReturns vocab size)r:r1r?s r vocab_sizezBioGptTokenizer.vocab_sizes4<<  rcBt|jfi|jSN)r7r1added_tokens_encoderrIs r get_vocabzBioGptTokenizer.get_vocabsDLL>D$=$=>>rc||jvr+|jj|}||j|<|j|j|dddS)Nr*TF)aggressive_dash_splits return_strescape)r,r+MosesTokenizertokenize)r?textr*moses_tokenizers rmoses_tokenizezBioGptTokenizer.moses_tokenizesc t11 1"gg44$4?O/>D & &t ,))$/88 %9  rc||jvr+|jj|}||j|<|j|j|S)NrP)r-r+MosesDetokenizer detokenize)r?tokensr*moses_detokenizers rmoses_detokenizez BioGptTokenizer.moses_detokenizesT t33 3 $ 8 8d 8 C 1BD ( ( .++D1<cNjj|tdS)Ninf)r;getfloat)pairr?s rz%BioGptTokenizer.bpe..s1C1CD%PU,1Wrkeyrr r! z z ) r6r<rminr;r:indexextend ValueErrorappendjoin) r?tokenrrbigramfirstsecondnew_wordijs ` rbpezBioGptTokenizer.bpesU3BZ E"I$6#88 DJJ ::e$ $$6> !$WXFT^^+"ME6HAc$i- 5!,A OOD1I.A7e#CIM(9d1q5kV>SOOEFN3FAOODG,FAc$i- XHD4yA~!$9:xx~ : D  5 /"OODH-sE F?Fc|r|j}n|j||j}g}|D]=}|s|jt |j |jd?|S)zReturns a tokenized string.ri)r5rXr*rllistrw)r?rVbypass_tokenizer split_tokensrps r _tokenizezBioGptTokenizer._tokenizesm ::)>s)C$DE Frc|jj||jj|jS)z0Converts a token (str) in an id using the vocab.)r1rcr%)r?rps r_convert_token_to_idz$BioGptTokenizer._convert_token_to_ids,||t||'7'7'GHHrcN|jj||jS)z=Converts an index (integer) in a token (str) using the vocab.)r3rcr%)r?rks r_convert_id_to_tokenz$BioGptTokenizer._convert_id_to_tokens||t~~66rc|Dcgc]$}|jddjdd&}}dj|j}|j||j}|Scc}w)z:Converts a sequence of tokens (string) in a single string.rir`)replaceror5r^r*)r?r\trVs rconvert_tokens_to_stringz(BioGptTokenizer.convert_tokens_to_stringshDJJa!))C$,,VS9JJ&&($$VTYY7 Ks)A- token_ids_0 token_ids_1returncV||jg|zS|jg}||z|z|zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BioGPT sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) sep_token_idr?rrseps r build_inputs_with_special_tokensz0BioGptTokenizer.build_inputs_with_special_tokenssA&  %%&4 4  ![ 3&44ralready_has_special_tokensc|rt|||dS|'dgdgt|zzdgzdgt|zzSdgdgt|zzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrr r)r=get_special_tokens_maskr:)r?rrrrGs rrz'BioGptTokenizer.get_special_tokens_masksy$ &72'[]a3   "31#K 001QC7A3[AQ;QR RsqcC ,,--rc|jg}|t||zdgzSt||zdgzt||zdgzzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. A FAIRSEQ Transformer sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). rr )rr:rs r$create_token_type_ids_from_sequencesz4BioGptTokenizer.create_token_type_ids_from_sequencess^.  !  {S()QC/ /;$%+c+2C.Ds.JJJrsave_directoryfilename_prefixc .tjj|stj d|dytjj ||r|dzndt dz}tjj ||r|dzndt dz}t|dd 5}|jtj|jd d d dzdddd}t|dd 5}t|jjdD]M\}} || k7rtjd|d| }|jdj |dz|dz }O ddd||fS#1swYxYw#1swY||fSxYw)NzVocabulary path (z) should be a directory-rr r wrrr!TF)indent sort_keys ensure_asciirrc |dS)Nr r')kvs rrfz1BioGptTokenizer.save_vocabulary..Js Y[\]Y^rrgzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rir )ospathisdirloggererrorroVOCAB_FILES_NAMESr.writer/dumpsr1sortedr;r2warning) r?rrr merge_filefrkwriter bpe_tokens token_indexs rsave_vocabularyzBioGptTokenizer.save_vocabulary:sww}}^, LL,^,<     M  s*)zzrz)FrL)NF)__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesr>propertyrJrNrXr^rwr|r~rrrintrrboolrrstrrrrr __classcell__)rGs@rrr.sM(T*$&67 * X!!? E *X I7JN5953;DI3F5 c52sx.9.3;DI3F.ko. c.8JNK9K3;DI3FK cK<&c&HSM&]bcf]g&8 rr)rr/rtypingrrrtokenization_utilsrutilsr get_loggerrrrrr__all__r'rrrs_' ((5   H % x)xv  r