S^(hvdZddlZddlmZddlmZddlZddlZddl m Z ddl m cm Z ddlmZddlmZmZmZmZdd lmZd ZeGd d eZeGd deZeGddeZGdde j6ZGdde j6ZGdde j6ZGdde j6ZGdde j6Z Gdde j6Z!Gdde j6Z"Gdd e j6Z#Gd!d"eZ$d#Z%d$Z&ed%e%Gd&d'e$Z'd'd"gZ(y)(zTransformers DAC model.N) dataclass)Optional)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardreplace_return_docstrings) DacConfigr ceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<y) DacOutputa. Args: loss (`torch.Tensor`): Loss from the encoder model, comprising the weighted combination of the commitment and codebook losses. audio_values (`torch.Tensor` of shape `(batch_size, input_length)`): Reconstructed audio data. quantized_representation (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`): Quantized continuous representation of input. audio_codes (`torch.LongTensor` of shape `(batch_size, num_codebooks, time_steps)`): Codebook indices for each codebook (quantized discrete representation of input). projected_latents (`torch.Tensor` of shape `(batch_size, num_codebooks * dimension, time_steps)`): Projected latents (continuous representation of input before quantization). Nloss audio_valuesquantized_representation audio_codesprojected_latents)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrr LongTensorrb/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/dac/modeling_dac.pyrr(st )-D(5$$ %,04L(5,,-4<@hu'8'89@.2K%**+259x 1 129rrceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<y)DacEncoderOutputa Args: loss (`torch.Tensor`): Loss from the encoder model, comprising the weighted combination of the commitment and codebook losses. quantized_representation (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`, *optional*): Quantized continuous representation of input. audio_codes (`torch.Tensor` of shape `(batch_size, num_codebooks, time_steps)`, *optional*): Codebook indices for each codebook (quantized discrete representation of input). projected_latents (`torch.Tensor` of shape `(batch_size, num_codebooks * dimension, time_steps)`, *optional*): Projected latents (continuous representation of input before quantization). Nrrrr) rrrrrrrrrrrrrrrr r ?s_ )-D(5$$ %,<@hu'8'89@/3K%++,359x 1 129rr c:eZdZUdZdZeejed<y)DacDecoderOutputz Args: audio_values (`torch.FloatTensor` of shape `(batch_size, input_length)`, *optional*): Decoded audio values, obtained using the decoder part of Dac. Nr) rrrrrrrrrrrrr"r"Ss 15L(5,,-4rr"c(eZdZdZfdZdZxZS)Snake1dz; A 1-dimensional Snake activation function module. ct|tjt j d|d|_y)Nr )super__init__nn Parameterronesalpha)self hidden_dim __class__s rr'zSnake1d.__init__ds+ \\%**Q A">? rc|j}|j|d|dd}||jdzjt j |j|zj dzz}|j|}|S)Nrr g& .>)shapereshaper+ reciprocalrsinpow)r, hidden_statesr2s rforwardzSnake1d.forwardhs##%--eAha"E %d):(F(F(H599UYU_U_boUoKpKtKtuvKw(ww %--e4 r)rrrrr'r8 __classcell__r.s@rr$r$_s@rr$c4eZdZdZdeffd ZdZdZxZS)DacVectorQuantizea Implementation of VQ similar to Karpathy's repo (https://github.com/karpathy/deep-vector-quantization) Additionally uses following tricks from improved VQGAN (https://arxiv.org/pdf/2110.04627.pdf): 1. Factorized codes: Perform nearest neighbor lookup in low-dimensional space for improved codebook usage 2. l2-normalized codes: Converts euclidean distance to cosine similarity which improves training stability configcDt|tj|j|j d|_tj|j |jd|_tj|j|j |_ y)Nr  kernel_size) r&r'r(Conv1d hidden_size codebook_dimin_projout_proj Embedding codebook_sizecodebookr,r=r.s rr'zDacVectorQuantize.__init__|sn yy!3!3V5H5HVWX  &"5"5v7I7IWXY  V%9%96;N;NO rc@|j|}|j|\}}tj||j d}tj||j d}|||z j z}|j |}|||||fS)aJ Quantizes the input tensor using a fixed codebook and returns the corresponding codebook vectors. Args: hidden_state (`torch.FloatTensor` of shape `(batch_size, dimension, time_steps)`): Input tensor. Returns: quantized_representation (`torch.Tensor`of shape `(batch_size, dimension, time_steps)`): Quantized continuous representation of input. commitment_loss (`torch.FloatTensor`of shape `(1)`): Commitment loss to train encoder to predict vectors closer to codebook entries. codebook_loss (`torch.FloatTensor`of shape `(1)`): Codebook loss to update the codebook. audio_codes (`torch.LongTensor` of shape `(batch_size, time_steps)`): Codebook indices for each codebook, quantized discrete representation of input. projected_latents (torch.FloatTensor of shape `(batch_size, num_codebooks * dimension, time_steps)`): Projected latents (continuous representation of input before quantization). mean) reduction)rDdecode_latentsFmse_lossdetachrE)r, hidden_staterrrcommitment_loss codebook_losss rr8zDacVectorQuantize.forwards*!LL6040C0CDU0V- +**%68P8W8W8Yekl #;=N=U=U=Wcij #48PSd8d7l7l7n#n #'==1I#J '-Vgggrc||j\}}}|jdddj||z|}|jj}t j |}t j |}|jdjdd}|d|z|jzz |jdjddjz}|jdd} | j|jdd} |j| jdd} | | fS)Nrr1r T)keepdimr0) r2permuter3rHweightrN normalizer6sumtmaxsize transpose) r,r7 batch_sizer-sequence_length encodingsrHl2_normdistindicesrs rrMz DacVectorQuantize.decode_latentss2?2E2E/ J!))!Q2:::;WYcd ==''KK * ;;x(--"&&q$&71y=8::<7788<<?;N;NqZ^;N;_;a;a;cc((1+a.//-"4"4Q"7<#'==#9#C#CAq#I '00r) rrrrr r'r8rMr9r:s@rr<r<ps" PyPh@1rr<c4eZdZdZddedeffd ZdZxZS)DacResidualUnitza A residual unit composed of Snake1d and weight-normalized Conv1d layers with dilations. dimensiondilationct|d|zdz}t||_t j ||d|||_t||_t j ||d|_y)Nr1)r@rgpaddingr r?) r&r'r$snake1r(rAconv1snake2conv2)r,rfrgpadr.s rr'zDacResidualUnit.__init__sb !a'i( YYy)X_bc i( YYy)C rc|}|j|j|}|j|j|}|jd|jdz dz}|dkDr |d|| f}||z}|S)ar Forward pass through the residual unit. Args: hidden_state (`torch.Tensor` of shape `(batch_size, channels, time_steps)`): Input tensor . Returns: output_tensor (`torch.Tensor` of shape `(batch_size, channels, time_steps)`): Input tensor after passing through the residual unit. r0r1r.)rmrlrornr2)r,rQ output_tensorrks rr8zDacResidualUnit.forwards%  4;;}#=>  4;;}#=> %%b)M,?,?,CCI Q;'WgX-=(=>L$}4 r)r )rrrrintr'r8r9r:s@rreres#D#DcDrrec8eZdZdZddededeffd ZdZxZS)DacEncoderBlockz"Encoder block used in DAC encoder.r=stride stride_indexc `t||jd|zz}t|dzd|_t|dzd|_t|dzd|_t|dz|_tj|dz|d|z|tj|dz |_ y)Nr1r rgr r@rwrk)r&r'encoder_hidden_sizere res_unit1 res_unit2 res_unit3r$rlr(rAmathceilrm)r,r=rwrxrfr.s rr'zDacEncoderBlock.__init__s ..L@ (a!D(a!D(a!Di1n- YY NI1v:fVZV_V_`fij`jVk  rc|j|}|j|}|j|j|}|j |}|SN)r~rrlrrmr,rQs rr8zDacEncoderBlock.forwardsI~~l3 ~~l3 {{4>>,#?@ zz,/ rr r rrrrr rtr'r8r9r:s@rrvrvs%,  y  #    rrvc8eZdZdZddededeffd ZdZxZS)DacDecoderBlockz"Decoder block used in DAC decoder.r=rwrxc lt||jd|zz}|jd|dzzz}t||_t j ||d|z|tj|dz |_ t|d|_ t|d|_ t|d|_ y)Nr1r r|rzrr{)r&r'decoder_hidden_sizer$rlr(ConvTranspose1drrconv_t1rer~rr)r,r=rwrx input_dim output_dimr.s rr'zDacDecoderBlock.__init__s ..!\/A //19I3JJ i( ))  F IIfqj)   )a@(a@(a@rc|j|}|j|}|j|}|j|}|j |}|Sr)rlrr~rrrs rr8zDacDecoderBlock.forwardsN{{<0 ||L1 ~~l3 ~~l3 ~~l3 rrrr:s@rrrs),AyA#AA$rrc|eZdZdZdeffd Zd deefdZde jfdZ de jfd Z xZ S) DacResidualVectorQuantizez ResidualVectorQuantize block - Introduced in SoundStream: An end2end neural audio codec (https://arxiv.org/abs/2107.03312) r=ct||j}|j}||_t j t |jDcgc] }t|c}|_||_ycc}wr) r&r' n_codebooksquantizer_dropoutr( ModuleListranger< quantizers)r,r=rrir.s rr'z"DacResidualVectorQuantize.__init__sh (( "44&--ERXRdRdLe(fq):6)B(fg!2)gsA; n_quantizerscd}|}d}d}g}g}||n |j}|jrtj|jdf|jzdz}tj d|jdz|jdf} t |jd|jz} | d| |d| |j|j}t|jD]\} } |jdur| |k\rn| |\} }}}}tj|jdf| |j|k}|| |ddddfzz}|| z }|||zz }|||zz }|j||j|tj|d}tj|d}|||||fS)aQ Quantizes the input tensor using a fixed set of codebooks and returns corresponding codebook vectors. Args: hidden_state (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`): Input tensor to be quantized. n_quantizers (`int`, *optional*): Number of quantizers to use. If specified and `self.quantizer_dropout` is True, this argument is ignored during training, and a random number of quantizers is used. Returns: quantized_representation (`torch.Tensor` of shape `(batch_size, dimension, time_steps)`): Quantized continuous representation of input. audio_codes (`torch.Tensor` of shape `(batch_size, num_codebooks, time_steps)`): Codebook indices for each codebook (quantized discrete representation of input). projected_latents (`torch.Tensor` of shape `(batch_size, num_codebooks * dimension, time_steps)`): Projected latents (continuous representation of input before quantization). commitment_loss (`torch.Tensor` of shape `(1)`): Commitment loss to train the encoder to predict vectors closer to codebook entries. codebook_loss (`torch.Tensor` of shape `(1)`): Codebook loss to update the codebook. rNr F) fill_valuedevicedim)rtrainingrr*r2randintrtrtor enumeraterfullappendstackcat)r,rQrrresidualrRrSrrdropout n_dropoutr quantizerquantized_representation_icommitment_loss_icodebook_loss_i indices_iprojected_latents_imasks rr8z!DacResidualVectorQuantize.forward"s.$%   '3'?|TEUEU == ::|'9'9!'<&>?$BRBRRUVVLmmAt'7'7!';l>P>PQR>S=UVGL..q1D4J4JJKI'.z ':L) $'??<+>+>?L%doo6 :LAy}}%!|*;mvn j &(9?IWj ::|11!461\M`M`adppD'?B\_cdegkmqdq_rBr'r $"<>2[,[\||0a8hhtw}}Q'778;??QQU?VWXY {# ]A)-a$q1u+,L+/??1+=+L+LWUVXdeqXqstUtMu+v (   $ $%8 9 LL !)-);)D)DEX)Y &'?B\'\ $ ](3D!)LLL3\sD= r)rrrrr r'rrtr8rTensorrrr9r:s@rrrsK 3y 3>h(3->h@ZellZ4MELLMrrc.eZdZdZdeffd ZdZxZS) DacDecoderz DAC Decoderr=ct ||j}|j}|j}t j ||dd|_g}t|D]\}}|t|||gz }t j||_ |jddzzz}t||_ t j |ddd|_t j|_y)Nrjrr@rkr1r )r&r'rBrupsampling_ratiosr(rArmrrrblockr$rlroTanhtanh) r,r= input_channelchannelsstridesrrxrwrr.s rr'zDacDecoder.__init__s ** --**YY}hAqQ $-g$6 E L& offlCD DE E]]5) //19I3JJ j) YYz1!QG GGI rc|j|}|jD] }||} |j|}|j|}|j |}|Sr)rmrrlror)r,rQlayers rr8zDacDecoder.forwards_zz,/ ZZ /E .L /{{<0 zz,/ yy. rrrrrr r'r8r9r:s@rrrsy* rrc.eZdZdZdeffd ZdZxZS) DacEncoderz DAC Encoderr=ct||j}tjd|j dd|_g|_t|D],\}}|dz}|xjt|||gz c_.tj|j|_|j dzz}t||_ tj||jdd|_y)Nr rjrr)rwrxr1)r&r'downsampling_ratiosr(rAr}rmrrrvrr$rlrBro)r,r=rrxrwd_modelr.s rr'zDacEncoder.__init__s ,,YYq&"<"g& YYw(:(:STU rc|j|}|jD] }||} |j|}|j|}|Sr)rmrrlro)r,rQmodules rr8zDacEncoder.forwardsQzz,/ jj 0F!,/L 0{{<0 zz,/ rrr:s@rrrsVyV$ rrc.eZdZdZeZdZdZdZdZ dZ y)DacPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. dac input_valuesct|tjrVtjj |j dtjj |jdyy)Ng{Gz?)stdr) isinstancer(rAinit trunc_normal_rW constant_bias)r,rs r _init_weightsz DacPreTrainedModel._init_weightssH fbii ( GG ! !&--T ! : GG  fkk1 - )rcztjj}ttjjdr$tjjj}|j j D]&}||j||j(||jj||jj|jjD]}||j||jj||jj||jj||jj||jj||jj||j j||j j|j jD]}||j"||jj||jj||jj||jj||jj||jjy)N weight_norm)r(utilsrhasattrparametrizationsrrrDrEencoderrmrorr~rrdecoderr)r,rrs rapply_weight_normz$DacPreTrainedModel.apply_weight_normshh** 288,,m <((33??K^^.. (E  &  ' ( DLL&&'DLL&&'\\'' /E  $ -- . -- . -- . -- . -- . -- . / DLL&&'DLL&&'\\'' /E  & -- . -- . -- . -- . -- . -- . /rcV|jjD]T}tjj |j tjj |j Vtjj |jjtjj |jj|jjD]^}tjj |jtjj |jjtjj |jjtjj |jjtjj |jjtjj |jjtjj |jjatjj |jjtjj |jj|jjD]^}tjj |jtjj |jjtjj |jjtjj |jjtjj |jjtjj |jjtjj |jjayr)rrr(rremove_weight_normrDrErrmrorr~rrrr)r,rs rrz%DacPreTrainedModel.remove_weight_normsi^^.. 8E HH ' ' 6 HH ' ' 7 8 ##DLL$6$67 ##DLL$6$67\\'' ?E HH ' ' 4 HH ' '(=(= > HH ' '(=(= > HH ' '(=(= > HH ' '(=(= > HH ' '(=(= > HH ' '(=(= > ? ##DLL$6$67 ##DLL$6$67\\'' ?E HH ' ' 6 HH ' '(=(= > HH ' '(=(= > HH ' '(=(= > HH ' '(=(= > HH ' '(=(= > HH ' '(=(= > ?rN) rrrrr config_classbase_model_prefixmain_input_namerrrrrrrrs)L$O. /B?rraH This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`DacConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_values (`torch.Tensor` of shape `(batch_size, 1, time_steps)`). Audio data to encode, n_quantizers (`int`, *optional*): Number of quantizers to use. If `None`, all quantizers are used. Default is `None`. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z%The DAC (Descript Audio Codec) model.c ReZdZdeffd Zeee d dejde e de e fdZ eee d de ejd e ejde e fd Zeeeee d dejde e de e fd ZxZS)DacModelr=ct||||_t||_t ||_t||_ttj|jj|_ d|jz|jjk7r td|jy)Nr1z'The codebook_size must be a power of 2.)r&r'r=rrrrrrrtrlog2rGbits_per_codebook ValueError post_initrIs rr'zDacModel.__init__Os   !&) !&) 26:!$TYYt{{/H/H%I!J d$$ $ (A(A AFG G r) output_typerrr return_dictc||n|jj}|j|}|j||\}}}}}|jj|z|jj |zz} |s| |||fSt | |||S)a Encode given audio data and return quantized latent codes Args: input_values (`torch.Tensor of shape `(batch_size, 1, time_steps)`): Input audio data to encode, n_quantizers (int, *optional*): Number of quantizers to use. If None, all quantizers are used. Default is None. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Returns: )r=rrrcommitment_loss_weightcodebook_loss_weightr ) r,rrrrrrrRrSrs rencodezDacModel.encode_s(&1% M^__rrrc| | td||n|jj}||jj |d}|j |j d}|s|fSt|S)aDecode given latent codes and return audio data Args: quantized_representation (torch.Tensor of shape `(batch_size, dimension, time_steps)`, *optional*): Quantized continuous representation of input. audio_codes (`torch.Tensor` of shape `(batch_size, num_codebooks, time_steps)`, *optional*): The codebook indices for each codebook, representing the quantized discrete representation of the input. This parameter should be provided if you want to decode directly from the audio codes (it will overwrite quantized_representation). return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Returns: zDEither `quantized_representation` or `audio_codes` must be provided.rr )rr=rrrrsqueezer")r,rrrrs rdecodezDacModel.decodes. $ + 0Ccd d%0%>> from datasets import load_dataset, Audio >>> from transformers import DacModel, AutoProcessor >>> librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> model = DacModel.from_pretrained("descript/dac_16khz") >>> processor = AutoProcessor.from_pretrained("descript/dac_16khz") >>> librispeech_dummy = librispeech_dummy.cast_column("audio", Audio(sampling_rate=processor.sampling_rate)) >>> audio_sample = librispeech_dummy[-1]["audio"]["array"] >>> inputs = processor(raw_audio=audio_sample, sampling_rate=processor.sampling_rate, return_tensors="pt") >>> encoder_outputs = model.encode(inputs["input_values"]) >>> # Get the intermediate audio codes >>> audio_codes = encoder_outputs.audio_codes >>> # Reconstruct the audio from its quantized representation >>> audio_values = model.decode(encoder_outputs.quantized_representation) >>> # or the equivalent with a forward pass >>> audio_values = model(inputs["input_values"]).audio_values ```Nr0F)rr.)r=rr2rrr) r,rrrlengthrrrrrs rr8zDacModel.forwards@&1%rsq ! - ) : : :, :{: :& 5{5 5bii"C1 C1L"bii"Jbii0bii>GM GMT""JBJ?J?Z +Cg!Cg CgL + ,r