T^(hJ d Z ddlZddlmZ ddlmZmZmZmZm Z ddl
m Z ddl m
Z
mZmZ ddlmZmZmZmZmZmZmZmZ dd lmZ dd
lm Z d d l m Z m Z e j@ e! Z"d
dddZ# e d Z$d Z% G d de Z&dgZ'y)z
Fast tokenization class for MarkupLM. It overwrites 2 methods of the slow tokenizer class, namely _batch_encode_plus
and _encode_plus, in which the Rust tokenizer is used.
N) lru_cache)DictListOptionalTupleUnion)
processors )PaddingStrategy
TensorTypeadd_end_docstrings)ENCODE_KWARGS_DOCSTRING
AddedToken
BatchEncoding EncodedInputPreTokenizedInput TextInput
TextInputPairTruncationStrategy)PreTrainedTokenizerFast)logging )0MARKUPLM_ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRINGMarkupLMTokenizerz
vocab.jsonz
merges.txtztokenizer.json)
vocab_file merges_filetokenizer_filec t t t d t d dz t t t d t d dz z t t t d t d dz z } | dd }d }t d
D ]1 }|| vs| j | |j d
|z |dz
}3 |D cg c]
}t | }}t t
| | S c c}w ) a3
Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control
characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large #
of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset
you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe
vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings.
!~r ¡ ¬ ® ÿNr )listrangeordappendchrdictzip)bscsnbs u/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/markuplm/tokenization_markuplm_fast.pybytes_to_unicoder2 / s
U3s8SX\
*+d5TCIPQM3R.SSVZ[`adeiajloptluxyly[zV{{
AB A
4[ B; IIaL IIdQh
FA
Q#a& B B
s C4c b t }| d }| dd D ] }|j ||f |} |S )z
Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length
strings).
r r N)setadd)wordpairs prev_charchars r1 get_pairsr: F sF
EEQIQR
9d#$ L c * e Zd ZdZeZeZ d0 fd Zd Z e
e e d1de
eeee ee f d ee
eee f d
e
eee eeee f d ee
ee eee f d ed
e
eeef de
eeef dee dedee dee dee
eef dee dee dedededededef(d Z e
e e d1d e
ee ee ee f d ee d
eeeee d ee
ee eee f d ed
e
eeef de
eeef dee dedee dee dee
eef dee dee dedededededef(d Zd2ded ee d edee fd!Z e
e e d1de
eef d ee d
eeee d eee d ed
e
eeef de
eeef dee dedee dee dee
eef dee dee dedededededef(d" Z ddddej< ej> ddddddddddddfd e
ee ee ee f d ee d
eeeee d eeee d ed#ed$edee dedee dee dee dee dee dedededededef(d%Z ddddej< ej> ddddddddddddfde
eef d ee d
eeee d eee d ed#ed$edee dedee dee dee dee dee dedededededef(d&Z!dej< dddfd'e
e"ee#f ef dee d#edee dee dee de$fd(Z% d3d)ee d*eee dee fd+Z& d3d)ee d*eee dee fd,Z'd3d-ed.ee de(e fd/Z) xZ*S )4MarkupLMTokenizerFasta
Construct a MarkupLM tokenizer. Based on byte-level Byte-Pair-Encoding (BPE).
[`MarkupLMTokenizerFast`] can be used to turn HTML strings into to token-level `input_ids`, `attention_mask`,
`token_type_ids`, `xpath_tags_seq` and `xpath_tags_seq`. This tokenizer inherits from [`PreTrainedTokenizer`] which
contains most of the main methods.
Users should refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
errors (`str`, *optional*, defaults to `"replace"`):
Paradigm to follow when decoding bytes to UTF-8. See
[bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
mask_token (`str`, *optional*, defaults to `""`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
add_prefix_space (`bool`, *optional*, defaults to `False`):
Whether or not to add an initial space to the input. This allows to treat the leading word just as any
other word. (RoBERTa tokenizer detect beginning of words by the preceding space).
NFTc ( t |t rt |dd n|}t |t rt |dd n|}t |t rt |dd n|}t | t rt | dd n| } t |
t rt |
dd n|
}
t | t rt | dd n| } t | t rt | dd n| } t | d i d|d|d|d|d|d |d
|d |
d |d
| d| d| d|
d|d|d|d|d|d|| |r t d || _ d}t | j |d }|rt j |j }d|v rt |d |d< d|v rt |d |d< d}|j d|
|
k7 r|
|d< d}|r>t t |j d } |d i |}t! | j || || _ || _ || _ t) | j | _ | j* d z | _ | j, g| j" z | _ | j& g| j" z | _ || _ || _ y ) NF)lstriprstripTr r tags_dictr errors bos_token eos_token unk_token sep_token cls_token pad_token
mask_tokenadd_prefix_space trim_offsets max_depth max_width pad_widthpad_token_labelonly_label_first_subwordzY`trim_offsets=True` is not implemented for MarkupLMTokenizerFast. Please set it to False.post_processorsepclstyper )
isinstancestrr super__init__NotImplementedErrorrA getattrbackend_tokenizerjsonloads __getstate__tuplegetr popsetattrrL rM rN len
unk_tag_id
pad_tag_idpad_xpath_tags_seqpad_xpath_subs_seqrO rP ) selfr r rA r rB rC rD rF rG rE rH rI rJ rL rM rN rO rP rK kwargstokenizer_component tokenizer_component_instancestatechanges_to_applycomponent_class new_value __class__s r1 rY z MarkupLMTokenizerFast.__init__ s . JTT]_bIcJyuEir IST]_bIcJyuEir IST]_bIcJyuEir IST]_bIcJyuEir IST]_bIcJyuEir IST]_bIcJyuEir KUU_adJeZ
4Fku
!
#
*
"
.
&
!
" #
$ ,%
& &>)
,
&k
# .'.t/E/EGZ\`'a$ 'JJ ; H H JKE ~ $U5\ 2e ~ $U5\ 2e $ yy+-=>BRR,<()#' ")*eii6G"H + 4e 4 ..0CYO #"" dnn- //A-#'??"3dnn"D #'>>"2T^^"C . (@%r; c g }g }|j d }|D ] }|j s|j j d }|d }t | dk( rdnt |d dd }|j | j
j
|| j |j t | j | |d| j }|d| j }|| j g| j t | z
z z
}|| j g| j t | z
z z
}||fS )z
Given the xpath expression of one particular node (like "/html/body/div/li[1]/div/span[2]"), return a list of
tag IDs and corresponding subscripts, taking into account max depth.
/[r r N)
splitstriprd intr) rA ra re minrM rL rf rN ) ri xpathxpath_tags_listxpath_subs_list xpath_unitsunit name_substag_namesubs r1
get_xpath_seqz#MarkupLMTokenizerFast.get_xpath_seq s+
kk#& =D::<
**3/I |H9~*!IaL"4E0FC " "4>>#5#5h#P Q " "3t~~s#; < = **:DNN;)*:DNN;DOO,_AU0UVVDNN+t~~O@T/TUU //r; r text text_pairxpaths node_labelsadd_special_tokenspadding
truncation
max_lengthstridepad_to_multiple_of padding_sidereturn_tensorsreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping
return_lengthverbosereturnc H d }|4 || s t d t |t t f s,t d t |t t f s t d |t |t t f }n5t |t t f xr |xr t |d t t f }||n|}|J d |rQt | t | k( sJ d t || D ]# \ }}t | t | k( r J d n t | t | k( sJ d |r|;t | t | k7 r$t d t | d
t | d |t t || n|}t
|du } | j d!i d |d
|d|d|d|d|d|d|d| d|
d| d| d|
d|d|d|d |d |d ||S | j d!i d |d |d|d|d|d|d|d|d| d|
d| d| d|
d|d|d|d |d |d ||S )"a
Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of
sequences with nodes, xpaths and optional labels.
Args:
text (`str`, `List[str]`, `List[List[str]]`):
The sequence or batch of sequences to be encoded. Each sequence can be a string, a list of strings
(words of a single example or questions of a batch of examples) or a list of list of strings (batch of
words).
text_pair (`List[str]`, `List[List[str]]`):
The sequence or batch of sequences to be encoded. Each sequence should be a list of strings
(pretokenized string).
xpaths (`List[List[int]]`, `List[List[List[int]]]`):
Node-level xpaths. Each bounding box should be normalized to be on a 0-1000 scale.
node_labels (`List[int]`, `List[List[int]]`, *optional*):
Node-level integer labels (for token classification tasks).
c t | t ryt | t t f rft | dk( ryt | d t ryt | d t t f r)t | d dk( xs t | d d t S yy)NTr F)rV rW r& r` rd )ts r1 _is_valid_text_inputz._is_valid_text_input7 sz !S!Ae}-q6Q; !c* !tUm4 qt9>EZ!Q-EE r; NzStext input must of type `str` (single example) or `List[str]` (batch of examples). zwNodes must be of type `List[str]` (single pretokenized example), or `List[List[str]]` (batch of pretokenized examples).r z%You must provide corresponding xpathszAYou must provide nodes and xpaths for an equal amount of examplesz2You must provide as many nodes as there are xpathszbatch length of `text`: z- does not match batch length of `text_pair`: .batch_text_or_text_pairsis_pairr r r r r r r r r r r r r r r r r r r rU )
ValueErrorrV r& r` rd r, boolbatch_encode_plus encode_plus) ri r r r r r r r r r r r r r r r r r r r rj r
is_batchednodes
nodes_examplexpaths_exampler r s r1 __call__z MarkupLMTokenizerFast.__call__ sb V ( '- !vww i$ 7 M dT5M2 M
#D4-8J#D4-8hThjQUVWQX[_afZgFhJ!)y!J#JJ! u:V, q.q q,14UF1C
w-
~=)S -@@vBvv@
w u:V, b.b b, $Tc)n)D .s4yk :I'q* FOEZtCi,@'A`d $9D01G)4)) )A (
$6
& & $6 * . '< '< +D ,F!" (>#$ ,%& )
. $4## # (
$6
& & $6 * . '< '< +D ,F!" (>#$ ,%& )
r; r r c | j d||||
|d|\ }}}} | j di d|d|d|d|d|d|d|d |d
| d |
d | d
| d|
d|d|d|d|d|d||S )Nr r r r r r r r r r padding_strategytruncation_strategyr r r r r r r r r r r r rU )"_get_padding_truncation_strategies_batch_encode_plus)ri r r r r r r r r r r r r r r r r r r r rj r r s r1 r z'MarkupLMTokenizerFast.batch_encode_plus s : ElDDkDk E
!! 1 E
E
A-z6 't&&
%=
$
2
.
!4
"
2
&
*
#8
#8
'@
(B!
" $:#
$ (%
& )
r; pairc t |r||fgn|g} | j j |f|dd|}|d j S )NFr is_pretokenizedr )
_tokenizer encode_batchtokens)ri r r r rj
batched_input encodingss r1 tokenizez MarkupLMTokenizerFast.tokenize sQ *.$ TF
0DOO00
.@RW
[a
|"""r; c | j d||||
|d|\ }}}} | j di d|d|d|d|d|d|d|d |d
| d |
d | d
| d|
d|d|d|d|d|d||S )aj
Tokenize and prepare for the model a sequence or a pair of sequences. .. warning:: This method is deprecated,
`__call__` should be used instead.
Args:
text (`str`, `List[str]`, `List[List[str]]`):
The first sequence to be encoded. This can be a string, a list of strings or a list of list of strings.
text_pair (`List[str]` or `List[int]`, *optional*):
Optional second sequence to be encoded. This can be a list of strings (words of a single example) or a
list of list of strings (words of a batch of examples).
r r r r r r r r r r r r r r r r r r r r rU )r _encode_plus)ri r r r r r r r r r r r r r r r r r r r rj r r s r1 r z!MarkupLMTokenizerFast.encode_plus s J ElDDkDk E
!! 1 E
E
A-z6 !t
$
2
.
!4
"
2
&
*
#8
#8
'@
(B!
" $:#
$ (%
& )
r; r r c
t |t st dt | d | j |||| |
| |r|D cg c]
\ }}|g|f }}}| j
j
||d }|D cg c] }| j ||
||||dn||| ! }}i }|d d j D ]' }|D cg c] \ }} || D ] } | } } }} | ||< ) |D cg c] \ } }|D ] } | } }} } |r2g } t | D ] \ }!\ }"} | |!gt |"d z z
} | |d < |d D ] }#| j |#|| g }$g }%t t |d D ]Q }&|r |d |& }'n|&}'g }(g })t |d |& | |& j | |& j D ] \ }*}+},|,}|r<|+dk( r7|(j! | j" |)j! | j$ G| j' ||' |, \ }-}.|(j) |-g |)j) |.g |*| j* | j, | j. fv r7|(j! | j" |)j! | j$ t1 d
|$j! |( |%j! |) T |$|d < |%|d < |g }/t t |d D ] }&|r |d |& }'n|&}'g }0t |d |& |d
|& | |& j D ] \ }*}1},|,`| j2 r<|1d dk( r|0j! ||' |, 5|0j! | j4 Q|0j! ||' |, i|0j! | j4 |/j! |0 |/|d< |s|d
= t7 || | S c c}}w c c}w c c} } }w c c} }} w )Nz/batch_text_or_text_pairs has to be a list (got ))r r r r r r Tr )encodingr r r r r r r r input_idsoverflow_to_sample_mappingzId not recognizedxpath_tags_seqxpath_subs_seqoffset_mappinglabels) tensor_type) rV r& TypeErrorrT set_truncation_and_paddingr r _convert_encodingkeys enumeraterd &_eventual_warn_about_too_long_sequencer' r, sequence_idsword_idsr) rg rh r extend cls_token_id sep_token_id pad_token_idr rP rO r )2ri r r r r r r r r r r r r r r r r r r r r r r r tokens_and_encodingssanitized_tokenskeyitem_estacksanitized_encodingsr itoksr r r batch_indexoriginal_indexxpath_tags_seq_examplexpath_subs_seq_exampleid sequence_idword_idr{ r| r labels_exampleoffsets2 r1 r z(MarkupLMTokenizerFast._batch_encode_plus* s 4 2D9 MdSkNlMmmn op p
'' - 3! 1%
(
Sk'li$(;'l $'lOO00 $ 1 1
, &
" "!&;&;*C+E * (,+ +
#
, '*1-224 *C&:NN74DINqQNQNEN$) S ! * 1E S SWQd Sq Sq S S %)+ & )*> ?
K 9D!*qcC[8I4J.JJ*
K=W 9 :)+6 XI 7 7 :w W X %5k%B!CD :K(!12N!OP[!\!,%' "%' ",/ -k:#K0==#K099-
>(K
& ;!#3.55d6M6MN.55d6M6MN;?;M;MfUcNdelNm;n8 .556GH.556GHd / /1B1BDDUDU VV.55d6M6MN.55d6M6MN ()< ==%
>&
! !"8 9 ! !"8 97 :: .<)*-;)* "F $S)9+)F%G H
. ,%56R%ST_%`N%0N!#+.$[1+>$%56{C' 4==, D'B
* 88 %ayA~ . 5 5k.6QRY6Z [ . 5 5d6J6J K * 1 1+n2Mg2V W&--d.B.BC D
n--
.0 *0 X &)$%56 -/BP^__k (m
0 O Ss O;$OO
)O!c |r||fgn|g}|g}||gnd } | j |fi dt |d u d|d|d|d|d|d|d| d |
d
| d | d |
d
|d|d|d|d|d||}| `|s^t |j D ci c]. \ }}|t | dkD rt |d t r|d n|0 c}}|j }| j |d || |S c c}}w )Nr r r r r r r r r r r r r r r r r r r r ) r r r itemsrd rV r& r r )ri r r r r r r r r r r r r r r r r r r r rj r batched_xpathsbatched_node_labelsbatched_outputr values r1 r z"MarkupLMTokenizerFast._encode_plus s 6 09$ *+tf
/:/F{mD000
$./
"
,
2
.
!4
"
2
&
*
#8
#8
'@
(B!
" $:#
$ (%
& )
2 !*C * '5&:&:&<"U SZ!^
58T8RqX]] (( N
33N;4OQ[]de s 3C#
encoded_inputsc |d| j v }|| j d }|t j k( r t | }||||z dk7 r ||z dz |z }|t j k7 xr t | |k7 }|rd|vrdgt | z |d< |r|t | z
} ||n | j
}|dk( r|r|d dg| z z |d< d|v r|d | j g| z z |d< d|v r|d | j g| z z |d< d|v r|d | j g| z z |d< d|v r|d | j g| z z |d< d |v r|d dg| z z |d < || j g| z z || j d < |S |d
k( r|rdg| z |d z |d< d|v r| j g| z |d z |d< d|v r| j g| z |d z |d< d|v r| j g| z |d z |d< d|v r| j g| z |d z |d< d |v rdg| z |d z |d < | j g| z |z || j d < |S t d t | z |S ) a
Args:
Pad encoded inputs (on left/right and up to predefined length or max length in the batch)
encoded_inputs:
Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`).
max_length: maximum length of the returned list and optionally padding length (see below).
Will truncate by taking into account the special tokens.
padding_strategy: PaddingStrategy to use for padding.
- PaddingStrategy.LONGEST Pad to the longest sequence in the batch
- PaddingStrategy.MAX_LENGTH: Pad to the max length (default)
- PaddingStrategy.DO_NOT_PAD: Do not pad
The tokenizer padding sides are defined in self.padding_side:
- 'left': pads on the left of the sequences
- 'right': pads on the right of the sequences
pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value.
This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability
`>= 7.5` (Volta).
padding_side:
The side on which the model should have padding applied. Should be selected between ['right', 'left'].
Default value is picked from the class attribute of the same name.
return_attention_mask:
(optional) Set to False to avoid returning attention mask (default: set to model specifics)
attention_maskr r righttoken_type_idsr r r special_tokens_maskleftzInvalid padding strategy:)
model_input_namesr LONGESTrd
DO_NOT_PADr pad_token_type_idrg rh rO r r rW )
ri r r r r r r required_inputneeds_to_be_padded
differences
r1 _padzMarkupLMTokenizerFast._pad
s B ! ($48N8N$N !'(>(>q(AB 6 6 6 ^,J !&8&D*WiJimnJn%);;q@DVVJ - 1K1K K qPSTbPcgqPq !%5^%K01sS 5H/HN+ , #c.&99J+7+C<IZIZLw&(7EFV7W[\Z]`jZj7jN#34#~5&'78DC* X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
)r r )ri r r rS rR s r1 build_inputs_with_special_tokensz6MarkupLMTokenizerFast.build_inputs_with_special_tokenso sb " %%&48I8I7JJ J ! ![ 3&4s::r; c | j g}| j g}|t ||z |z dgz S t ||z |z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. RoBERTa does not
make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
r )r r rd )ri r r rR rS s r1 $create_token_type_ids_from_sequencesz:MarkupLMTokenizerFast.create_token_type_ids_from_sequences sh ! ! s[(3./1#5 53$s*[83>?1#EEr; save_directoryfilename_prefixc f | j j j || }t | S )N)name)r modelsaver` )ri r r filess r1 save_vocabularyz%MarkupLMTokenizerFast.save_vocabulary s+ %%**>*PU|r; )Nreplace