S^(h9dZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZmZmZmZmZd d lmZej,eZd ZeGd d eZeGddeZeGddeZGdde j:ZGdde j:ZGdde j:Z Gdde j:Z!Gdde j:Z"Gdde j:Z#Gdde j:Z$Gd d!e j:Z%Gd"d#e j:Z&Gd$d%e Z'd&Z(d'Z)ed(e(Gd)d*e'Z*d*d%gZ+y)+zPyTorch EnCodec model.N) dataclass)ListOptionalTupleUnion)nn)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) EncodecConfigrcbeZdZUdZdZeejed<dZ eejed<y) EncodecOutputah Args: audio_codes (`torch.LongTensor` of shape `(batch_size, nb_chunks, chunk_length)`, *optional*): Discret code embeddings computed using `model.encode`. audio_values (`torch.FlaotTensor` of shape `(batch_size, sequence_length)`, *optional*) Decoded audio values, obtained using the decoder part of Encodec. N audio_codes audio_values) __name__ __module__ __qualname____doc__rrtorch LongTensor__annotations__r FloatTensorj/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/encodec/modeling_encodec.pyrr+3/3K%**+204L(5,,-4rrcbeZdZUdZdZeejed<dZ eejed<y)EncodecEncoderOutputa Args: audio_codes (`torch.LongTensor` of shape `(batch_size, nb_chunks, chunk_length)`, *optional*): Discret code embeddings computed using `model.encode`. audio_scales (`torch.Tensor` of shape `(batch_size, nb_chunks)`, *optional*): Scaling factor for each `audio_codes` input. This is used to unscale each chunk of audio when decoding. Nr audio_scales) rrrrrrrrrr$rrrr r#r#9r!rr#c:eZdZUdZdZeejed<y)EncodecDecoderOutputz Args: audio_values (`torch.FloatTensor` of shape `(batch_size, segment_length)`, *optional*): Decoded audio values, obtained using the decoder part of Encodec. Nr) rrrrrrrrrrrr r&r&Gs 15L(5,,-4rr&c eZdZdZ ddededededef fd Zdejd ejfd Ze ddejd e eefd e d e fdZ dZxZS) EncodecConv1dz;Conv1d with asymmetric or causal padding and normalization. in_channels out_channels kernel_sizestridedilationc t||j|_|j|_|j |_|j dvrt d|j |dkDr$|dkDrtjd|d|d|dtj||||||_ tjj}ttjjd r$tjjj}|j d k(r||j|_ n*|j d k(rtj d||_|jj$d }t'j(|jj*d t&j, }|jj.d }t'j(|dz |zdzt&j, }|j1d |d|j1d|d|j1d||z dy)N weight_normtime_group_normIself.norm_type must be one of `"weight_norm"`, `"time_group_norm"`), got rzQEncodecConv1d has been initialized with stride > 1 and dilation > 1 (kernel_size=z stride=z , dilation=z).r-r0r1r)dtyper,F) persistentr+ padding_total)super__init__use_causal_convcausalpad_mode norm_type ValueErrorloggerwarningrConv1dconvutilsr0hasattrparametrizations GroupNormnormr+rtensorr,int64r-register_buffer) selfconfigr)r*r+r,r-r0 __class__s r r8zEncodecConv1d.__init__Us ,,  )) >>!C C[\`\j\j[kl  A:(Q, NN!!, XfX[ RTV  IIk<fW_` hh** 288,,m <((33??K >>] *#DII.DI ^^0 0 Q 5DIii++A. dii..q1E99%%a(llK!Ox#?!#C5;;W  Xv%@ ]KEJ _kF.BuUr hidden_statesreturnc>|jd}||jz |jz|jz dz}t j |j tjdz }||jz|jz|jz }||z S)zSee `pad_for_conv1d`.r)shaper+r6r,rceiltorH)rJrMlengthn_frames ideal_lengths r _get_extra_padding_for_conv1dz+EncodecConv1d._get_extra_padding_for_conv1d~s $$R(T---0B0BBdkkQTUU::h'**5;;7!;$++-0@0@@4CUCUU f$$rpaddingsmodevaluecl|jd}|\}}|dk(s"tjj||||St ||}d}||kr*||z dz}tjj|d|f}tjj||||} | jd|z } | dd| fS)zTiny wrapper around torch.nn.functional.pad, just to allow for reflect padding on small input. If this is the case, we insert extra 0 padding to the right before the reflection happens. rPreflectrr.N)rQr functionalpadmax) rMrXrYrZrT padding_left padding_rightmax_pad extra_padpaddedends r _pad1dzEncodecConv1d._pad1ds $$R(&.# my ==$$]HdEJ JlM2 W &(1,IMM--ma^LM""=(D%Hll2*c4C4i  rcz|j|}|jr+|j||j|f|j}nA|jdz}|j|z }|j||||zf|j}|j |}|j dk(r|j|}|S)N)rYr1)rWr:rfr6r;rAr<rF)rJrM extra_paddingrar`s r forwardzEncodecConv1d.forwards::=I ;; KK 8J8JM7ZaeananKoM!..!3M-- =L KK mm.KLSWS`S`(M -0 >>. . IIm4Mr)rr)zero)rrrrintr8rTensorrW staticmethodrstrfloatrfrj __classcell__rLs@r r(r(RsEop'V#&'V69'VHK'VUX'Vhk'VR %|| %  %!ell!eCHo!S!bg!!$rr(c <eZdZdZddedededeffd ZdZxZS) EncodecConvTranspose1dzDConvTranspose1d with asymmetric or causal padding and normalization.r)r*r+r,ct||j|_|j|_|j |_|j dvrt d|j tj|||||_ tjj}ttjjdr$tjjj}|j dk(r||j|_ n*|j dk(rtjd||_|js|jdk(s t dyy)Nr/r2r0r1r?zB`trim_right_ratio` != 1.0 only makes sense for causal convolutions)r7r8r9r:trim_right_ratior<r=rConvTranspose1drArBr0rCrDrErF)rJrKr)r*r+r,r0rLs r r8zEncodecConvTranspose1d.__init__s ,, & 7 7)) >>!C C[\`\j\j[kl &&{L+vV hh** 288,,m <((33??K   } ,#DII.DI   !2 2 Q 5DI t44;ab b < rc~|jjd}|jjd}||z }|j|}|jdk(r|j |}|j r#t j||jz}n|dz}||z }|jd|z }|d||f}|S)Nrr1rhrP.) rAr+r,r<rFr:mathrRrxrQ)rJrMr+r,r6rar`res r rjzEncodecConvTranspose1d.forwardsii++A. !!!$#f,  -0 >>. . IIm4M ;;!IImd6K6K&KLM*Q.M$}4 !!"% 5%c<+;&;< r)r)rrrrrmr8rjrrrss@r rurus3NcCcscQTc^ac0rruc(eZdZdZfdZdZxZS) EncodecLSTMzz LSTM without worrying about the hidden state, nor the layout of the data. Expects input as convolutional layout. cnt|tj|||j|_yN)r7r8rLSTMnum_lstm_layerslstm)rJrK dimensionrLs r r8zEncodecLSTM.__init__s( GGIy&2H2HI rc|jddd}|j|d|z}|jddd}|S)Nrhrr)permuter)rJrMs r rjzEncodecLSTM.forwardsH%--aA6  -03mC %--aA6 r)rrrrr8rjrrrss@r r}r}sJrr}c<eZdZdZdededeeffd ZdZxZ S)EncodecResnetBlockz> Residual block from SEANet model as used by EnCodec. rKdim dilationsc  t ||jdf}t|t|k7r t d||j z}g}t t||D]R\}\}} |dk(r|n|} |t|dz k(r|n|} |tjgz }|t|| | || gz }Ttj||_ |jrt|||d|_ytj|_y)Nrz7Number of kernel sizes should match number of dilationsrr3)r+)r7r8residual_kernel_sizelenr=compress enumerateziprELUr( ModuleListblockuse_conv_shortcutshortcutIdentity) rJrKrr kernel_sizeshiddenrir+r-in_chsout_chsrLs r r8zEncodecResnetBlock.__init__s 33Q7 | I .VW W'*3C i4P*Q ^ &A& XFSF#l"3a"77cVG bffhZ E mFFG[S[\] ]E  ^ ]]5)  # #)&#sJDMKKMDMrc`|}|jD] }||} |j||zSr)rr)rJrMresiduallayers r rjzEncodecResnetBlock.forwards: ZZ 1E!-0M 1}}X&66r) rrrrrrmrr8rjrrrss@r rrs+*}*3*49*(7rrc.eZdZdZdeffd ZdZxZS)EncodecEncoderz"SEANet encoder as used by EnCodec.rKc t|t||j|j|j g}d}t |jD]}||jz}t|jD]"}|t|||j|zdggz }$|tjgz }|t|||dz|dz|gz }|dz}|t|||jzgz }|tjgz }|t|||jz|j|j gz }tj"||_y)Nrrhr+r,)r7r8r(audio_channels num_filtersr+reversedupsampling_ratiosrangenum_residual_layersrdilation_growth_raterrr} hidden_sizelast_kernel_sizerlayers)rJrKmodelscalingratio current_scalejrLs r r8zEncodecEncoder.__init__ sa vv'<'P>PRXRdRdeff667 E#f&8&88M6556 j,V]VE`E`bcEcefDghii j bffhZ E mFM=1;LZ_bcZclqrs sE qLG  +fg0B0B&BCDD "&&( -&2D2D(DfFXFXZ`ZqZqrssmmE* rc8|jD] }||} |SrrrJrMrs r rjzEncodecEncoder.forward6%[[ 1E!-0M 1rrrrrrr8rjrrrss@r rrs,+}+,rrc.eZdZdZdeffd ZdZxZS)EncodecDecoderz"SEANet decoder as used by EnCodec.rKc t|tdt|jz}t ||j ||jz|jg}|t|||jzgz }|jD]}||jz}|tjgz }|t|||dz|dz|gz }t|jD]%}|t||dz|j |zdfgz }'|dz}|tjgz }|t ||j|j"|j$gz }tj&||_y)Nrhrr)r7r8rmrrr(rrr+r}rrrurrrrrrrr)rJrKrrrrrrLs r r8zEncodecDecoder.__init__?s{ a3v77889vv'9'97VEWEW;WY_YkYklm +fg0B0B&BCDD-- E#f&8&88M bffhZ E &v}mq>P^cfg^gpuv E6556 o,V]a5G&JeJeghJhjkIlmnn o MG  "&&( -(:(:F MM-0 "INNE#2J/ rcZtjj||j}|Sr)rr] embeddingrrJrrs r decodezEncodecEuclideanCodebook.decode}s!==**9djjAr) rrrrrr8rrrrrrss@r rr^s!+ 9} 9rrc4eZdZdZdeffd ZdZdZxZS)EncodecVectorQuantizationzY Vector quantization implementation. Currently supports only euclidean distance. rKcBt|t||_yr)r7r8rcodebookrJrKrLs r r8z"EncodecVectorQuantization.__init__s 08 rcb|jddd}|jj|}|SNrrhr)rrr)rJrMembed_ins r rz EncodecVectorQuantization.encodes/%--aA6 =='' 6rcb|jj|}|jddd}|Sr)rrrrs r rz EncodecVectorQuantization.decodes/=='' 2##Aq!,r) rrrrrr8rrrrrss@r rrs9}9 rrceZdZdZdeffd Zd deedefdZ d de jdeede jfdZ d e jde jfd Z xZS) EncodecResidualVectorQuantizerzResidual Vector Quantizer.rKct||j|_|j|_|j|_t j t|jDcgc] }t|c}|_ ycc}wr) r7r8r frame_ratenum_quantizersrrrrr)rJrK_rLs r r8z'EncodecResidualVectorQuantizer.__init__sh #11 ++$33mmPUV\VkVkPl$m1%>v%F$mn $ms)B bandwidthrNc tj|j|jz}|j}|3|dkDr.t t dtj|dz|z }|S)z:Return num_quantizers based on specified target bandwidth.rlri)r{log2rrrrmr_floor)rJrbw_per_qrs r get_num_quantizers_for_bandwidthz?EncodecResidualVectorQuantizer.get_num_quantizers_for_bandwidths`99T//04??B,,  Y_ Q 9t3Ch3N(O!PQNr embeddingsc|j|}|}g}|jd|D]:}|j|}|j|}||z }|j |<t j |} | S)z Encode a given input tensor with the specified frame rate at the given bandwidth. The RVQ encode method sets the appropriate number of quantizers to use and returns indices for each quantizer. N)rrrrappendrstack) rJrrrr all_indicesrr quantized out_indicess r rz%EncodecResidualVectorQuantizer.encodes >>yI [[.1 (Ell8,G W-I)+H   w '  ( kk+. rcodesctjd|j}t|D]*\}}|j|}|j |}||z},|S)z7Decode the given codes to the quantized representation.rl)device)rrGrrrr)rJr quantized_outrrrrs r rz%EncodecResidualVectorQuantizer.decodes[ S> #E* 6JAwKKNE W-I)I5M 6rr)rrrrrr8rrqrmrrrnrrrrrss@r rrsp$o}o(5/UX(5/UZUaUa ELLU\\rrc"eZdZdZeZdZdZdZy)EncodecPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. encodec input_valuesct|tjrm|jjj d|j j|j%|jjjyyt|tjtjfrJ|jjj|jjjdyt|tjrtjj|j|jjt!j"|j$|j&|j(dzz }tjj+|j| |yyt|tj,rz|jjj d|j j|j.2|jj|j.jyyt|tj0rb|j3D]N\}}d|vr tjj5|*d|vs/tjj7|dPyy) zInitialize the weightsrl)meanstdNrwr)abweightbias) isinstancerLinearrdatanormal_rKinitializer_rangerzero_ LayerNormrEfill_r@initkaiming_normal_r{sqrtgroupsr)r+uniform_ Embedding padding_idxrnamed_parametersxavier_uniform_ constant_)rJmoduleknameparams r _init_weightsz$EncodecPreTrainedModel._init_weightss fbii ( MM   & &CT[[5R5R & S{{&   &&(' r|| < = KK   " " $ MM   $ $S )  * GG # #FMM 2{{&IIfmmv/A/AFDVDVWXDY/YZ[  a 8' - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  (%668 2 et#GG++E2t^GG%%eS1  2)rN) rrrrr config_classbase_model_prefixmain_input_namerrrr rrs !L!$O2rraL This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`EncodecConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a( Args: input_values (`torch.FloatTensor` of shape `(batch_size, channels, sequence_length)`, *optional*): Raw audio input converted to Float and padded to the approriate length in order to be encoded using chunks of length self.chunk_length and a stride of `config.chunk_stride`. padding_mask (`torch.BoolTensor` of shape `(batch_size, channels, sequence_length)`, *optional*): Mask to avoid computing scaling factors on padding token indices (can we avoid computing conv on these+). Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. `padding_mask` should always be passed, unless the input was truncated or not padded. This is because in order to process tensors effectively, the input audio should be padded so that `input_length % stride = step` with `step = chunk_length-stride`. This ensures that all chunks are of the same shape bandwidth (`float`, *optional*): The target bandwidth. Must be one of `config.target_bandwidths`. If `None`, uses the smallest possible bandwidth. bandwidth is represented as a thousandth of what it is, e.g. 6kbps bandwidth is represented as `bandwidth == 6.0` audio_codes (`torch.LongTensor` of shape `(batch_size, nb_chunks, chunk_length)`, *optional*): Discret code embeddings computed using `model.encode`. audio_scales (`torch.Tensor` of shape `(batch_size, nb_chunks)`, *optional*): Scaling factor for each `audio_codes` input. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z%The EnCodec neural audio codec model.cpeZdZdeffd ZdZdZdejde de de eje ejffd Z ddejde ejde e d e edee eje ejfeff d Zed eejd e fdZddejde ejdejfdZ ddejdejde ejd e edee ejejfeff dZeeeee ddejde ejde e de ejde ejd e edee ejejfeffdZxZS) EncodecModelrKct||||_t||_t ||_t||_ttj|jj|_ d|jz|jjk7r td|jy)Nrhz'The codebook_size must be a power of 2.)r7r8rKrencoderrdecoderr quantizerrmr{rrbits_per_codebookr= post_initrs r r8zEncodecModel.__init__s   %f- %f- 7?!$TYYt{{/H/H%I!J d$$ $ (A(A AFG G rc|jSr)r rJs r get_encoderzEncodecModel.get_encoder- ||rc|jSr)r!r&s r get_decoderzEncodecModel.get_decoder0r(rrr padding_maskrNc|jd}||jjz }|jjA|d|jjzkDr%t d|d|jjd}|jj rt||j dz}tj|dd|jdz }|jd jdd jd z}||z }|j|}|jj||} | jd d} | |fS) z Encodes the given input using the underlying VQVAE. If `config.normalize` is set to `True` the input is first normalized. The padding mask is required to compute the correct scale. rPNgh㈵>zDuration of frame (z) is longer than chunk rTrrh)rrg:0yE>r)rQrK sampling_ratechunk_length_s RuntimeError normalize unsqueezerrrrr r r"r transpose) rJrrr+rTdurationscalemonorrs r _encode_framezEncodecModel._encode_frame3s3##B'DKK555 ;; % % 1h HbHbAb6b!4XJ>UVZVaVaVpVpUqrs s ;; ',*@*@*CCL99\1d;l>P>PQR>SSDHHQK$$T$:??ADHE'%/L\\,/ %%j)<1%e|r return_dictcd||n|jj}||jjd}||jjvr&td|d|jjd|j\}}}|dks|dkDrtd||jj }||}|} n|jj } |#tj|j}g} g} || z } || z| z dk7r td td|| z | D]e} |d | | |zfj}|dddd| | |zf}|j|||\}}| j|| j|gtj| } |s| | fSt| | S) a Encodes the input audio waveform into discrete codes. Args: input_values (`torch.Tensor` of shape `(batch_size, channels, sequence_length)`): Float values of the input audio waveform. padding_mask (`torch.Tensor` of shape `(batch_size, channels, sequence_length)`): Padding mask used to pad the `input_values`. bandwidth (`float`, *optional*): The target bandwidth. Must be one of `config.target_bandwidths`. If `None`, uses the smallest possible bandwidth. bandwidth is represented as a thousandth of what it is, e.g. 6kbps bandwidth is represented as bandwidth == 6.0 Returns: A list of frames containing the discrete encoded codes for the input audio waveform, along with rescaling factors for each chunk when `normalize` is True. Each frames is a tuple `(codebook, scale)`, with `codebook` of shape `[batch_size, num_codebooks, frames]`. Nrz)This model doesn't support the bandwidth z. Select one of .rrhz1Number of audio channels must be 1 or 2, but got zkThe input length is not properly padded for batched chunked decoding. Make sure to pad the input correctly..)rKr7target_bandwidthsr=rQ chunk_length chunk_strider ones_likeboolrr6rrr#)rJrr+rr7rchannels input_lengthr;r,encoded_framesscalesstepoffsetmaskframe encoded_framer4s r rzEncodecModel.encodeMs2&1%>!Q q(8uUVWXZ[3++--[[F%H kkI5I*IV5I E ;;r?L Vf|333 4} 8MPU8U U 4 v 5 6&,:O O 6 f F   >> q VWaVbbcde eZrrr4c|jdd}|jj|}|j|}|||j dddz}|S)NrrrP)r2r"rr!r)rJrr4routputss r _decode_framezEncodecModel._decode_framesV1%^^**51 ,,z*   2q! 44Grrr$c6||n|jj}|jj}|>t|dk7rt dt||j |d|d}ncg}t ||D](\}} |j || } |j| *|j||jjxsd}|5|jd|jdkr|dd|jdf}|s|fSt|S)aM Decodes the given frames into an output audio waveform. Note that the output might be a bit bigger than the input. In that case, any extra steps at the end can be trimmed. Args: audio_codes (`torch.LongTensor` of shape `(batch_size, nb_chunks, chunk_length)`, *optional*): Discret code embeddings computed using `model.encode`. audio_scales (`torch.Tensor` of shape `(batch_size, nb_chunks)`, *optional*): Scaling factor for each `audio_codes` input. padding_mask (`torch.Tensor` of shape `(batch_size, channels, sequence_length)`): Padding mask used to pad the `input_values`. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NrzExpected one frame, got rrP.) rKr7r;rr=rVrrrSr<rQr&) rJrr$r+r7r;rdecoded_framesrFr4rHs r rzEncodecModel.decodes.0&1% . u++E59%%f- . 33NDKKD\D\Da`abL  # (:(:2(>ASASTVAW(W'-E|/A/A"/E-E(EFL ? "#L11r) output_typercH||n|jj}|#tj|j }| | t d| | t d|||j |||d\}}|j||||d}|s||fSt||S)a Returns: Examples: ```python >>> from datasets import load_dataset >>> from transformers import AutoProcessor, EncodecModel >>> dataset = load_dataset("hf-internal-testing/ashraq-esc50-1-dog-example") >>> audio_sample = dataset["train"]["audio"][0]["array"] >>> model_id = "facebook/encodec_24khz" >>> model = EncodecModel.from_pretrained(model_id) >>> processor = AutoProcessor.from_pretrained(model_id) >>> inputs = processor(raw_audio=audio_sample, return_tensors="pt") >>> outputs = model(**inputs) >>> audio_codes = outputs.audio_codes >>> audio_values = outputs.audio_values ```zBYou specified `audio_codes` but did not specify the `audio_scales`zBYou specified `audio_scales` but did not specify the `audio_codes`F)r7r)rr) rKr7rr=r>r=rrr)rJrr+rrr$r7rs r rjzEncodecModel.forwardsB&1%rr#rrorrSrVr&rr ENCODEC_INPUTS_DOCSTRINGrr_CONFIG_FOR_DOCrjrrrss@r rrsm } !LL5:JM u||Xell33 4:04%)&* F<llF<u||,F<E? F< d^ F< uU\\8ELL#99:r\rr#r&Moduler(rur}rrrrrrrENCODEC_START_DOCSTRINGr[r__all__rrr rfs !// -1   H %"  5K 5  5  5; 5  5 5;5 5^BII^B6RYY6r "))  77BRYY>RYYD!ryy!H ()RYY)X!2_!2H"B+SQ)SQ SQl 3 4r