T^(h0 d Z ddlZddlmZ ddlmZmZmZmZm Z ddl
Z ddl m
Z
ddlmZ ej e ZdZd d
iZ G d d e
Zd gZy)
zTokenization classes for . N)copyfile)AnyDictListOptionalTuple )PreTrainedTokenizer)loggingu ▁
vocab_filezsentencepiece.bpe.modelc
\ e Zd ZdZeZddgZ ddeee e
f ddf fd
Z d Z d Z
dd
ee d eee dee fd Z d d
ee d eee d
edee f fd
Z dd
ee d eee dee fdZed Zd Zde dee fdZd Zd Zd Zdde dee dee fdZ xZ S )
XGLMTokenizera
Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on
[SentencePiece](https://github.com/google/sentencepiece).
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
sp_model_kwargs (`dict`, *optional*):
Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
to set:
- `enable_sampling`: Enable subword regularization.
- `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
- `nbest_size = {0,1}`: No sampling is performed.
- `nbest_size > 1`: samples from the nbest_size results.
- `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
- `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
Attributes:
sp_model (`SentencePieceProcessor`):
The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
input_idsattention_maskNsp_model_kwargsreturnc R |i n|| _ d| _ t | j D
cg c] }
d|
d
} }
| j dg xs g | d< | dxx | D cg c] } | | d vs | c} z
cc< t j
d i | j | _ | j j t | || _ d| _
ddddd | _ t | j }
t | j D
ci c] }
d|
d|
|
z | j z } }
| j j | | j j D ci c] \ }}||
c}}| _ t! | D d ||||||| j d
| y c c}
w c c} w c c}
w c c}}w ) N z additional_special_tokens r r )`
- pair of sequences: ` A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) sep_token_idr4 rK rL seps r= build_inputs_with_special_tokensz.XGLMTokenizer.build_inputs_with_special_tokens sF ( %%&4 4 ![ 3&,{::rE already_has_special_tokensc |rt | ||d S |dgdgt | z z S dgdgt | z z ddgz dgt | z z S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rK rL rR r r )r2 get_special_tokens_maskr. )r4 rK rL rR r< s r= rT z%XGLMTokenizer.get_special_tokens_mask s{ & &72 '[]a 3
31#K 001 1sqcC ,,-A61#K@P:PQQrE c z | j g}|t ||z dgz S t ||z |z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. XLM-RoBERTa does
not make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
r )rN r. rO s r= $create_token_type_ids_from_sequencesz2XGLMTokenizer.create_token_type_ids_from_sequences sU $ ! s[()QC/ /3$s*S0;>?1#EErE c ` t | j | j z | j z S N)r. r) r, r$ )r4 s r=
vocab_sizezXGLMTokenizer.vocab_size s' 4==!D$7$77$:O:OOOrE c t | j D ci c] }| j | | }}|j | j |S c c}w rX )r% rY convert_ids_to_tokensr/ added_tokens_encoder)r4 r6 vocabs r= get_vocabzXGLMTokenizer.get_vocab sK ;@ ;QRa++A.1RR
T../ Ss Atextc D | j j |t S )N)out_type)r) encoder+ )r4 r_ s r= _tokenizezXGLMTokenizer._tokenize s }}##D3#77rE c || j v r| j | S | j j | }|r|| j z S | j S )z0Converts a token (str) in an id using the vocab.)r- r) PieceToIdr, unk_token_id)r4 tokenspm_ids r= _convert_token_to_idz"XGLMTokenizer._convert_token_to_id sU D.. .--e4 4 ((/ 06v+++L4;L;LLrE c || j v r| j | S | j j || j z
S )z=Converts an index (integer) in a token (str) using the vocab.)r1 r) IdToPiecer, )r4 indexs r= _convert_id_to_tokenz"XGLMTokenizer._convert_id_to_token sA D.. .--e4 4}}&&ut/B/B'BCCrE c l dj | j t d j }|S )zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r4 tokens
out_strings r= convert_tokens_to_stringz&XGLMTokenizer.convert_tokens_to_string s, WWV_,,-=sCIIK
rE save_directoryfilename_prefixc t j j | st j d| d y t j j ||r|dz ndt d z }t j j | j t j j | k7 rBt j j | j rt | j | |fS t j j | j sCt |d 5 }| j j }|j | d d d |fS |fS # 1 sw Y |fS xY w)NzVocabulary path (z) should be a directory-ro r wb)ospathisdirloggererrorrq VOCAB_FILES_NAMESabspathr isfiler openr) rB write)r4 rx ry out_vocab_fileficontent_spiece_models r= save_vocabularyz XGLMTokenizer.save_vocabulary s% ww}}^, LL,^,<r s_ ! 3 3 5
H % !#<= G!' G!T
rE