S^(hC d Z ddlZddlZddlZddlZddlmZ ddlmZm Z m
Z
m Z m Z m
Z
ddlZddlZddlmZmZ ddlmZ ddlmZmZmZmZmZ dd l m Z m Z ej> e Z!d
d d d
Z" G d de Z#y)z(Tokenization classes for OpenAI Jukebox. N)INFINITY)AnyDictListOptionalTupleUnion )
AddedTokenPreTrainedTokenizer)
BatchEncoding)
TensorTypeis_flax_availableis_tf_availableis_torch_availablelogging)_is_jax _is_numpyz artists.jsonz lyrics.jsonz genres.json) artists_file lyrics_file genres_filec " e Zd ZdZeZddgZg ddddf fd Zed Z d
Z
d Z d Z d
Z
d dededededeeeeeeef f f
dZd ZdedefdZdee defdZ d!deeeef defdZd"de fdZ d#d ed ee dee fd Z d Z xZ S )$JukeboxTokenizeraq
Constructs a Jukebox tokenizer. Jukebox can be conditioned on 3 different inputs :
- Artists, unique ids are associated to each artist from the provided dictionary.
- Genres, unique ids are associated to each genre from the provided dictionary.
- Lyrics, character based tokenization. Must be initialized with the list of characters that are inside the
vocabulary.
This tokenizer does not require training. It should be able to process a different number of inputs:
as the conditioning of the model can be done on the three different queries. If None is provided, defaults values will be used.:
Depending on the number of genres on which the model should be conditioned (`n_genres`).
```python
>>> from transformers import JukeboxTokenizer
>>> tokenizer = JukeboxTokenizer.from_pretrained("openai/jukebox-1b-lyrics")
>>> tokenizer("Alan Jackson", "Country Rock", "old town road")["input_ids"]
[tensor([[ 0, 0, 0, 6785, 546, 41, 38, 30, 76, 46, 41, 49,
40, 76, 44, 41, 27, 30]]), tensor([[ 0, 0, 0, 145, 0]]), tensor([[ 0, 0, 0, 145, 0]])]
```
You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer or when you
call it on some text, but since the model was not pretrained this way, it might yield a decrease in performance.
If nothing is provided, the genres and the artist will either be selected randomly or set to None
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to:
this superclass for more information regarding those methods.
However the code does not allow that and only supports composing from various genres.
Args:
artists_file (`str`):
Path to the vocabulary file which contains a mapping between artists and ids. The default file supports
both "v2" and "v3"
genres_file (`str`):
Path to the vocabulary file which contain a mapping between genres and ids.
lyrics_file (`str`):
Path to the vocabulary file which contains the accepted characters for the lyrics tokenization.
version (`List[str]`, `optional`, default to `["v3", "v2", "v2"]`) :
List of the tokenizer versions. The `5b-lyrics`'s top level prior model was trained using `v3` instead of
`v2`.
n_genres (`int`, `optional`, defaults to 1):
Maximum number of genres to use for composition.
max_n_lyric_tokens (`int`, `optional`, defaults to 512):
Maximum number of lyric tokens to keep.
unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
input_idsattention_mask)v3v2r i z
<|endoftext|>c t |t rt |dd n|}|| _ || _ || _ d|i| _ t |d 5 } t j | | _
d d d t |d 5 } t j | | _ d d d t |d 5 } t j | | _ d d d d}
t | j dk( r|
j dd }
t j |
| _ | j j% D ci c] \ } } | |
c} } | _ | j j% D ci c] \ } } | |
c} } | _ | j j% D ci c] \ } } | |
c} } | _ t-
| \ d ||||d
| y # 1 sw Y ^xY w# 1 sw Y ;xY w# 1 sw Y xY wc c} } w c c} } w c c} } w ) NF)lstriprstripr utf-8encoding#[^A-Za-z0-9.,:;!?\-'\"()\[\] \t\n]+O z\-'z\-+') unk_tokenn_genresversionmax_n_lyric_tokens )
isinstancestrr r) r* r( _added_tokens_decoderopenjsonloadartists_encodergenres_encoderlyrics_encoderlenreplaceregexcompile out_of_vocabitemsartists_decodergenres_decoderlyrics_decodersuper__init__)selfr r r r) r* r( r' kwargs vocab_handleoovkv __class__s y/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/deprecated/jukebox/tokenization_jukebox.pyr? zJukeboxTokenizer.__init__d s JTT]_bIcJyuEir "4
&'^"
,
1 ;\#'99\#:D ; +
0 :L"&))L"9D : +
0 :L"&))L"9D : 5 t"" #r )++fg.C !MM# .151E1E1K1K1M NA1 N 040C0C0I0I0K L1q!t L040C0C0I0I0K L1q!t L
1
% ; ; : : : : O L Ls6
G=G-G 3
G+&
G1
G7GG G(c t | j t | j z t | j z S N)r5 r2 r3 r4 r@ s rG
vocab_sizezJukeboxTokenizer.vocab_size s3 4''(3t/B/B+CCc$J]J]F^^^ c J | j | j | j dS )Nr2 r3 r4 rN rJ s rG get_vocabzJukeboxTokenizer.get_vocab s' # 3 3 " 1 1 " 1 1
rL c |D cg c] }| j j |d }}t t | D ]Z }|| D cg c] }| j j |d c}||< || dg| j
t || z
z z ||< \ |d D cg c] }| j j |d c}g g g} ||| fS c c}w c c}w c c}w )zConverts the artist, genre and lyrics tokens to their index using the vocabulary.
The total_length, offset and duration have to be provided in order to select relevant lyrics and add padding to
the lyrics token sequence.
r )r2 getranger5 r3 r( r4 )
r@ list_artists list_genres list_lyricsartist
artists_idgenresgenre character lyric_idss
rG _convert_token_to_idz%JukeboxTokenizer._convert_token_to_id s
IUUfd**..vq9U
UC ,- jFR]^dRe"f4#6#6#:#:5!#D"fK "-f"5
PST_`fTgPh@h8i"iK j NYYZ^\ d))--i;\^`bde ; 11
V"f ]s #C#C"#Cc t | S )aS
Converts a string into a sequence of tokens (string), using the tokenizer. Split in words for word-based
vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces).
Do NOT take care of added tokens. Only the lyrics are split into character for the character-based vocabulary.
)listr@ lyricss rG _tokenizezJukeboxTokenizer._tokenize s F|rL c \ | j ||| \ }}}| j | }|||fS )zV
Converts three strings in a 3 sequence of tokens using the tokenizer
)prepare_for_tokenizationrb )r@ rW rZ ra rA s rG tokenizezJukeboxTokenizer.tokenize s: !% = =feV T v 'uf$$rL artistsrY ra is_split_into_wordsreturnc t t | j D ] }| j | dk( r.|| j ||< || j g||< C| j || dz ||< || j d D cg c] }| j | dz c}||< | j d dk( rt
j d | _ d}t t | D ci c] }|| |dz
c}| _ d| j d < t | dz | _
| j | _ | j j D
ci c] \ } }
|
|
c}
} | _
d
| j d< nt
j d | _ | j | }|j d d
}| j j! d
| g g f}|||fS c c}w c c}w c c}
} w )a
Performs any necessary transformations before tokenization.
Args:
artist (`str`):
The artist name to prepare. This will mostly lower the string
genres (`str`):
The genre name to prepare. This will mostly lower the string.
lyrics (`str`):
The lyrics to prepare.
is_split_into_words (`bool`, *optional*, defaults to `False`):
Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the
tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)
which it will tokenize. This is useful for NER or token classification.
r z.v2_r r r% zOABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789.,:;!?-+'"()[]
z z$[^A-Za-z0-9.,:;!?\-+'\"()\[\] \t\n]+\
)rS r5 r) lower
_normalizesplitr7 r8 r9 vocabn_vocabr4 r: r= _run_strip_accentsr6 sub) r@ rf rY ra rg idxrZ rr indexrD rE s rG rd z)JukeboxTokenizer.prepare_for_tokenization s $ T\\*+ C||C D( &s| 1 1 3 %c{ 0 0 2 3s #ws| DL"&**D 48JJ4D4D4F"GDAq1a4"GD %'D " %
.U VD ((0 d+""&&r62B:&&' R #Hs G(9G-!
G2c t j d| }g }|D ]. }t j | }|dk( r |j | 0 dj | S )z$Strips accents from a piece of text.NFDMnrl ) unicodedata normalizecategoryappendjoin)r@ textoutputcharcats rG rt z#JukeboxTokenizer._run_strip_accents s^ $$UD1 D &&t,Cd{ MM$
wwv rL r c r t t d t d dz D cg c]
}t | c}t t d t d dz D cg c]
}t | c}z t t d t d dz D cg c]
}t | c}z dgz }t | }t j
d }d
j
|j D cg c]
}||v r|nd c} }|j d | j d }|S c c}w c c}w c c}w c c}w ) z
Normalizes the input text. This process is for the genres and the artist
Args:
text (`str`):
Artist or Genre string to normalize
azrk AZ09.z_+rl rj )
rS ordchr frozensetrer8 r ro ru strip)r@ r iacceptedpatterncs rG rp zJukeboxTokenizer._normalize s #3s8SX\ : ;SV ; $SXs3x!| <=!s1v=
> $SXs3x!| <=!s1v=
> e
X&**U#wwTZZ\JQ(]3JK{{3%++C0
<==
Ks D%D*D/,D4c $ dj | S )N )r r` s rG convert_lyric_tokens_to_stringz/JukeboxTokenizer.convert_lyric_tokens_to_string s xx rL tensor_typeprepend_batch_axisc J t |t s t | }|t j k( r2t s t d ddl}|j }|j }n|t j k( r2t s t d ddl
}|j }|j }nW|t j k( r.t s t d ddlm} |j }t" }nt$ j& }t( } |r|g} || s || }|S # t+ d xY w)a
Convert the inner content to tensors.
Args:
tensor_type (`str` or [`~utils.TensorType`], *optional*):
The type of tensors to use. If `str`, should be one of the values of the enum [`~utils.TensorType`]. If
unset, no modification is done.
prepend_batch_axis (`int`, *optional*, defaults to `False`):
Whether or not to add the batch dimension during the conversion.
zSUnable to convert output to TensorFlow tensors format, TensorFlow is not installed.r NzMUnable to convert output to PyTorch tensors format, PyTorch is not installed.zEUnable to convert output to JAX tensors format, JAX is not installed.zUnable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length.)r, r
TENSORFLOWr ImportError
tensorflowconstant is_tensorPYTORCHr torchtensorJAXr jax.numpynumpyarrayr npasarrayr
ValueError) r@ inputsr r tf as_tensorr r jnps rG convert_to_tensorsz#JukeboxTokenizer.convert_to_tensors s +z2$[1K *// /"$!i
$ I I
J . .
.%'!"qrr I I
JNN
*$&!"ijj # I I
I!I
! V$"6*
f
s >D
D"c g d}|gt | j z }|gt | j z }| j ||| \ }}}| j ||| \ } }
} t gt | d z } t t | j D
cg c]) }
| j
|| |
gz |
|
z | |
z g| + }}
t || d S c c}
w )a\ Convert the raw string to a list of token ids
Args:
artist (`str`):
Name of the artist.
genres (`str`):
List of genres that will be mixed to condition the audio
lyrics (`str`, *optional*, defaults to `""`):
Lyrics used to condition the generation
)r r r rQ )r )r attention_masks)r5 r) re r] r rS r r
)r@ rW rY ra return_tensorsr artists_tokens
genres_tokens
lyrics_tokensrX
genres_ids full_tokensr r s rG __call__zJukeboxTokenizer.__call__F s C --C --7;}}VVU[7\4
}.2.G.GXegt.u+
J $9+KO(<<
3t||,-
# #jm_,z!}<{1~MN\j
$
9 YZZ
s .Csave_directoryfilename_prefixc t j j | st j d| d yt j j ||r|dz ndt d z }t |dd 5 }|j t j | j d
ddd t j j ||r|dz ndt d z }t |dd 5 }|j t j | j d
ddd t j j ||r|dz ndt d
z }t |dd 5 }|j t j | j d
ddd |||fS # 1 sw Y xY w# 1 sw Y xY w# 1 sw Y &xY w)a
Saves the tokenizer's vocabulary dictionary to the provided save_directory.
Args:
save_directory (`str`):
A path to the directory where to saved. It will be created if it doesn't exist.
filename_prefix (`Optional[str]`, *optional*):
A prefix to add to the names of the files saved by the tokenizer.
zVocabulary path (z) should be a directoryN-rl r wr" r# F) ensure_asciir r )ospathisdirloggererrorr VOCAB_FILES_NAMESr/ writer0 dumpsr2 r3 r4 )r@ r r r fr r s rG save_vocabularyz JukeboxTokenizer.save_vocabularya s ww}}^, LL,^,<JKU$%%))%0KKFQR$%%)))4RRvv%% LRs "A1"A6)F)NF)rl ptrI )!__name__
__module__ __qualname____doc__r vocab_files_namesmodel_input_namesr? propertyrK rO r] rb re r- boolr r r rd rt rp r r r r r r r
r r r
__classcell__)rF s @rG r r * s9 4l *$&67 # !)
V _ _
2 % SX+'+'$'+'14+'KO+' sCd38n, -+'Z s s * T#Y 3 hm;#+E#z/,B#C;`d;z[- [6"8c "8HSM "8]bcf]g "8H&rL r )$r r0 r r r{ json.encoderr typingr r r r r r r r r7 tokenization_utilsr r tokenization_utils_baser
utilsr r r r r
utils.genericr r
get_loggerr r r r r+ rL rG r sj / ! : : B 5 a a 0
H % # j&* j&rL