T^(h|+ ddlZddlZddlmZmZmZddlZddlZddl m Z ddl m cm Z ddl mZddlmZddlmZddlmZddlmZmZmZmZmZmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#d d l$m%Z%e#jLe'Z(d Z)dZ*Gdde jVZ,Gdde jVZ-Gdde jVZ.Gdde jVZ/Gdde jVZ0Gdde jVZ1Gdde jVZ2Gdde jVZ3Gdd e jVZ4Gd!d"e jVZ5Gd#d$eZ6Gd%d&e jVZ7Gd'd(e jVZ8Gd)d*e jVZ9Gd+d,e jVZ:Gd-d.e jVZ;Gd/d0e jVZ< dUd1ee=e=fd2e>d3e=d4eej~d5e=d6ejf d7ZAgd8ZBd9ZCd:ZDeZEe d;eCGd<d=e6ZFd>ZGd?ZHd@ZIe dAeCGdBdCe6ZJe dDeCGdEdFe6ZKdGZLddgZMe dHeCGdIdJe6ZNGdKdLe jVZOGdMdNe jVZPdOZQdPZRe dQeCGdRdSe6ZSgdTZTy)VN)OptionalTupleUnion)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)BaseModelOutputCausalLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardis_peft_availablelogging) WavLMConfigz1patrickvonplaten/wavlm-libri-clean-100h-base-plusrc$eZdZfdZdZxZS)WavLMSamePadLayercPt||dzdk(rd|_yd|_yNrr)super__init__num_pad_remove)selfnum_conv_pos_embeddings __class__s f/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/wavlm/modeling_wavlm.pyrzWavLMSamePadLayer.__init__/s) #:Q#>!#CacV|jdkDr|ddddd|j f}|SNr)r r! hidden_statess r$forwardzWavLMSamePadLayer.forward3s6    ")!Q0F43F3F2F0F*FGMr%__name__ __module__ __qualname__rr* __classcell__r#s@r$rr.sKr%rc$eZdZfdZdZxZS)WavLMPositionalConvEmbeddingct|tj|j|j|j |j dz|j |_tjj}ttjjdr$tjjj}tr(ddl }|jj|jj d5||jdd|_dddt|jdrU|jjj j"}|jjj j$}n,|jj&}|jj(}|jj+|||jj+||n||jdd|_t-|j |_t0|j2|_y#1swY'xYw) Nr) kernel_sizepaddinggroups weight_normr) modifier_rankweight)namedimparametrizations)rrnnConv1d hidden_sizer"num_conv_pos_embedding_groupsconvutilsr7hasattrr<r deepspeedzeroGatheredParametersr9 original0 original1weight_gweight_vregister_external_parameterrr5rfeat_extract_activation activation)r!configr7rDrIrJr#s r$rz%WavLMPositionalConvEmbedding.__init__:s II      6622a777   hh** 288,,m <((33??K % ' 224993C3CST2U I' aH  Ityy"459955<<FF9955<<FF99--99-- NN 6 6tX F NN 6 6tX F#DIIH!DDI()G)GH  !?!?@ I Is IIc|jdd}|j|}|j|}|j|}|jdd}|SNrr) transposerAr5rMr(s r$r*z$WavLMPositionalConvEmbedding.forward[sV%//15  -0  ]3  6 %//15 r%r+r0s@r$r2r29sABr%r2c$eZdZfdZdZxZS)WavLMFeatureProjectionc4t|tj|jd|j |_tj|jd|j|_ tj|j|_ y)Neps) rrr= LayerNormconv_dimlayer_norm_eps layer_normLinearr? projectionDropoutfeat_proj_dropoutdropoutr!rNr#s r$rzWavLMFeatureProjection.__init__gsf ,,vr':@U@UV))FOOB$79K9KLzz&":":; r%cp|j|}|j|}|j|}||fSN)r[r]r`)r!r)norm_hidden_statess r$r*zWavLMFeatureProjection.forwardms:!__];(:;  ]3 000r%r+r0s@r$rSrSfs < 1r%rSceZdZdZ ddedededededef fd Z dd ejd e ejd e ejd ed e eje eje e ejff dZ d ejd eejej fdejd ed ejejff dZdeded ejfdZdejd ejfdZxZS)WavLMAttentionz=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsr` num_buckets max_distancehas_relative_position_biasct|||_||_||_||z|_|j |z|jk7rt d|jd|d|j dz|_tj|||_ tj|||_ tj|||_ tj|||_ ||_||_tj t#j$d|jdd|_tj|j d|_|r0tj*|j|j|_yy)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gr)rrrgrhr`head_dim ValueErrorscalingr=r\k_projv_projq_projout_projrirj Parametertorchonesgru_rel_pos_constgru_rel_pos_linear Embeddingrel_attn_embed)r!rgrhr`rirjrkr#s r$rzWavLMAttention.__init__xs7 "" !Y. MMI %$.. 8MdnnM]$YKr3 }}d* ii 95 ii 95 ii 95  )Y7 &(!#ejjDNNAq.Q!R"$))DMM1"= %"$,,t/?/?"PD  &r%r)attention_mask position_biasoutput_attentionsreturnc|j\}}}|S|j||}|jdj|dddj ||j z||}|j |j dd|j dfz} | jdddd} |j| } | j | j dddzjd} tj| jdd\} } | | |jzd z zd z} | j ||j zdd|z}|j d||f}|j||||\}}|||fS) z'Attention layer with relative attentionNrrrUrr)rr;?g@)size compute_bias unsqueezerepeatviewrhshapepermuterysumrvsigmoidchunkrxtorch_multi_head_self_attention)r!r)r|r}r~indexbsztgt_len_gated_hidden_statesrelative_position_projgate_agate_b gate_outputgated_position_bias attn_output attn_weightss r$r*zWavLMAttention.forwards(,,.Wa   --gw?M''*11#q!Q?DDS4>>EY[bdkl  ,001D1DSb1IT^^]_L`1`a199!Q1E"&!8!89L!M!7!DDQBDO)?)? ?# EFL *..sT^^/CRKm[166GW7MN$($H$H >+>@Q% ! \L-77r%rcX|jddx}x}}||jdnd}dx} } d} tj||||j|j t jdgt j|jj|jj|jjf| | | |j|jj|jj|j |||d|jj|jj|jj\} } | jdd} | C| dddfj#| j$dd|j fz| j$ddz} | | fS)zCsimple wrapper around torch's multi_head_attention_forward functionrrNFT)use_separate_proj_weight q_proj_weight k_proj_weight v_proj_weight)rQneFmulti_head_attention_forwardrgrhrvemptycatrsbiasrqrrr`rtr9training broadcast_tor)r!r)r|rr~querykeyvaluekey_padding_maskbias_kbias_v add_zero_attnrrs r$rz.WavLMAttention.torch_multi_head_self_attentions,55a;;;e3A3M>,,Q/SW %&$B$B    NN NN KK  IIt{{'')9)94;;;K;KL M    LL MM MM   MM   %)++,,++,,++,,+% ! \2"++Aq1  #(40==""2A&$..)::\=O=OPQPR=SSLL((r% query_length key_lengthctj|tjdddf}tj|tjdddf}||z }|j|}|j |j j j}|j |}|jgd}|S)Ndtype)rrr) rvarangelong_relative_positions_buckettor{r9devicer)r!rrcontext_positionmemory_positionrelative_positionrelative_position_bucketvaluess r$rzWavLMAttention.compute_biass << EJJG4P,,zDT1WM+.>>#'#B#BCT#U #;#>#>t?R?R?Y?Y?`?`#a $$%=> * r%relative_positionsc$|jdz}|dkDjtj|z}tj|}|dz}||k}tj |j |z }|tj |j|z z }|||z z}||zjtj}tj|tj||dz }|tj|||z }|Sr) rirrvrabslogfloatmathrjmin full_likewhere)r!rrirelative_buckets max_exactis_smallrelative_positions_if_largerelative_position_if_larges r$rz)WavLMAttention._relative_positions_buckets &&!+ .266uzzB[P"YY'9:1$ % 1&+ii0B0H0H0JY0V&W#&ADHHTM^M^ajMjDk&k#&A[S\E\&]#&/2M&M%Q%QRWR\R\%]"%*YY &8RT_bcTc(d& " EKK2DF`aar%)i@i TNNFr)r,r-r.__doc__intrboolrrvTensorrrr* FloatTensorr LongTensor BoolTensorrrrr/r0s@r$rfrfusG +/"Q"Q"Q "Q  "Q  "Q%)"QN2604"' '8||'8!.'8  - '8  '8 u||Xell3XeELL>Q5RR S'8R5)((5)e..0@0@@A5)#.. 5)  5)   U.. / 5)n#%BSBS U=N=N SXSdSd r%rfc$eZdZfdZdZxZS)WavLMFeedForwardct|tj|j|_tj |j|j|_ t|jtrt|j|_n|j|_tj |j|j|_tj|j |_yrc)rrr=r^activation_dropoutintermediate_dropoutr\r?intermediate_sizeintermediate_dense isinstance hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutras r$rzWavLMFeedForward.__init__s $&JJv/H/H$I!"$))F,>,>@X@X"Y f'' -'-f.?.?'@D $'-'8'8D $IIf&>&>@R@RS jj)>)>?r%c|j|}|j|}|j|}|j|}|j |}|Src)rrrrrr(s r$r*zWavLMFeedForward.forward)sX// > 00? 11-@ ))-8 ++M: r%r+r0s@r$rrs @r%rc2eZdZddedeffd ZddZxZS)WavLMEncoderLayerrNrkct|t|j|j|j |j |j||_tj|j|_ tj|j|j|_t!||_tj|j|j|_yN)rgrhr`rirjrkrVrrrfr?num_attention_headsattention_dropoutrimax_bucket_distance attentionr=r^rr`rXrZr[r feed_forwardfinal_layer_normr!rNrkr#s r$rzWavLMEncoderLayer.__init__4 '((00,,**33'A  zz&"7"78 ,,v'9'9v?T?TU,V4 " V-?-?VEZEZ [r%c|}|j|||||\}}}|j|}||z}|j|}||j|z}|j |}||f}|r||fz }|S)Nr|r}r~r)rr`r[rr) r!r)r|r}r~r attn_residualroutputss r$r*zWavLMEncoderLayer.forwardCs% 59^^ )'/ 6D6 2 |] ]3 % 5  6 %(9(9-(HH --m<  -0    &Gr%Trr,r-r.rrrr*r/r0s@r$rr3s \{ \ \r%rc2eZdZddedeffd ZddZxZS) WavLMEncoderLayerStableLayerNormrNrkct|t|j|j|j |j |j||_tj|j|_ tj|j|j|_t!||_tj|j|j|_yrrrs r$rz)WavLMEncoderLayerStableLayerNorm.__init__]rr%c|}|j|}|j||||\}}}|j|}||z}||j|j |z}||f}|r||fz }|S)N)r|r}r~)r[rr`rr)r!r)r|r}r~rrrs r$r*z(WavLMEncoderLayerStableLayerNorm.forwardls%  6 59^^ )'/ 6D6 2 |]  ]3 % 5 %(9(9$:O:OP]:^(__  -0    &Gr%r)NNFrr0s@r$rr\s \{ \ \r%rc.eZdZfdZ ddZxZS) WavLMEncoderc t|||_t||_t j |j|j|_ t j|j|_ t jt|jDcgc]}t!||dk(c}|_d|_ycc}wNrVr)rkF)rrrNr2pos_conv_embedr=rXr?rZr[r^rr` ModuleListrangenum_hidden_layersrlayersgradient_checkpointingr!rNir#s r$rzWavLMEncoder.__init__s  :6B,,v'9'9v?T?TUzz&"7"78 mmUZ[a[s[sUt uPQ v16 K u  ',# v!C c|rdnd}|rdnd}|5|jdjdd|jd}d||<|j|} || z}|j |}|j |}t xs t|} d} t|jD]\} } |r||fz}tjg}|jxr | dkDxr||jjk}|r| rM|jr,|jr |j!| j"||| |}n | ||| || }|dd\}} |rd}|s|dfz}|r||fz}|st%d|||fDSt'||| S) NrUrrrrNNNc3&K|] }|| ywrcr .0vs r$ z'WavLMEncoder.forward..mq_`_lmlast_hidden_stater) attentions)rrrrr[r`r r enumeraterrvrandrrN layerdropr_gradient_checkpointing_func__call__tupler r!r)r|r~output_hidden_states return_dictall_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusr}rlayerdropout_probabilityskip_the_layer layer_outputss r$r*zWavLMEncoder.forwards#7BD$5b4  %$2$<$."rrr)rrrrr`r r rrrvrrrNrrrrr[rr rs r$r*z#WavLMEncoderStableLayerNorm.forwards#7BD$5b4  %$2$<$rU)meanrvexprr)probsmarginal_probs perplexitys r$_compute_perplexityz.WavLMGumbelVectorQuantizer._compute_perplexityCsR*YY .599^VZEZ;[*[ac ddeiik r%c|j\}}}|j|}|j||z|jzd}|jrt j j|j|jd}|j|}tj|j||z|jdjd}|j|}n}|jd}|j|jj!d|jddd}|j||z|jd}|j|}|j||zd}|j#d|j$z} | j||z|j|j&d} | j)dj||d} | |fS)NrUT)tauhardrrr)rr6rr1rr= functionalgumbel_softmaxrr7type_asrvsoftmaxr>argmax new_zerosscatter_rr5r3r) r!r) batch_sizesequence_lengthr?codevector_probscodevector_soft_distr=codevector_idxcodevectors_per_groupr5s r$r*z"WavLMGumbelVectorQuantizer.forwardIs3@3F3F0 O[((7 %**:+G$//+Y[]^ ==!}};;MAQAQ Q+00o1Mt`d`m`moqr !oob)..z?BO J&&r%) r,r-r.rr staticmethodr>r*r/r0s@r$r.r.(s& * "'r%r.ceZdZdZeZdZdZdZdZ dZ dZ dde e jefd eefd Z dd ed e jfd Zy)WavLMPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. wavlm input_valuesTFc zt|tr|jjjj dd|jj jjtjj|jyt|trtjj |jjddtjd|jj d|jj"zz ztjj%|jj dyt|t&rtjd|j(j*z }tjj|j(j| |tjj|j(j | |yt|tj,rm|jjj d|j.j0|j %|j jjyyt|tj2tj4frJ|j jj|jjj7dyt|tj8rtjj;|j|j jtj|j<|j"|j dzz }tjj|j | |yyy) zInitialize the weightsrr)r9stdrr)abNr)rr.r6r9datanormal_rzero_r=inituniform_r5r2rArsqrtr4 in_channels constant_rSr] in_featuresr\rNinitializer_rangerX GroupNormfill_r>kaiming_normal_r6)r!moduleks r$ _init_weightsz"WavLMPreTrainedModel._init_weights{s f8 9    % % * * 2 2 2 C    # # ( ( . . 0 GG  V// 0  < = GGOO "" !v{{'>'>q'AFKKD[D['["\]]   GG  fkk.. 2  6 7 !f//;;;tj||z |ddzS)Nfloor) rounding_moder)rvdiv input_lengthr4strides r$_conv_out_lengthzOWavLMPreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengths"99\K7wWZ[[ [r%r)rNrjzip conv_kernel conv_stridernum_adapter_layersadapter_stride)r!rirjrsr4rrrs r$ _get_feat_extract_output_lengthsz5WavLMPreTrainedModel._get_feat_extract_output_lengthss2=1Ddkk--+  \ $'t{{'>'> @W@W#X Q K,]KPM Q 4;;99: _ 04;;C]C] ^  _r%feature_vector_lengthr|c|jddddf}|j||}|jtj}|j d}tj ||f|j|j}d|tj|j d|j|dz f<|jdgjdjdgj}|S)NrUrrjr)rrr)r) cumsumryrrvrrzerosrrrflipr)r!rzr|rjnon_padded_lengthsoutput_lengthsrJs r$"_get_feature_vector_attention_maskz7WavLMPreTrainedModel._get_feature_vector_attention_masks ,22r2:1b5A>>?Q_j>k'**5::6#))!,  . /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr%rc)r,r-r.rr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_supports_flash_attn_2_supports_sdparhrrvrrrrryrr r%r$rRrRns L$O&*#"N9DZ^"5#3#3S#89HPQU0Y]%(:?:J:Jr%rRc&eZdZdfd ZdZxZS)WavLMNoLayerNormConvLayercdt||dkDr|j|dz nd|_|j||_t j |j|j|j||j||j|_ t|j|_ y)Nrrr4rrr)rrrY in_conv_dim out_conv_dimr=r>rurv conv_biasrArrLrMr!rNlayer_idr#s r$rz"WavLMNoLayerNormConvLayer.__init__s rurvrrArXr[rrLrMrs r$rz WavLMLayerNormConvLayer.__init__s rurvrrArrLrMrcr[rs r$rz WavLMGroupNormConvLayer.__init__s output_hidden_sizeadapter_kernel_sizerxrAras r$rzWavLMAdapterLayer.__init__=sJ II  % % )) )  & &((   r%cj|j|}tjj|d}|S)Nrr)rAr=rCglur(s r$r*zWavLMAdapterLayer.forwardGs/ -0  ))-Q)? r%r+r0s@r$rr<s  r%rc$eZdZfdZdZxZS) WavLMAdapterct|jjk7rTt j jj|_t jj|_ndx|_|_t jfdtjD|_ j|_ y)Nc34K|]}tywrc)r)r rrNs r$rz(WavLMAdapter.__init__..Ys#h!$5f$=#hs)rrrr?r=r\projrXproj_layer_normrrrwrrras `r$rzWavLMAdapter.__init__Os   $ $(:(: : &"4"4f6O6OPDI#%<<0I0I#JD /3 3DI,mm#huVMfMfGg#hh ))r%ch|j.|j"|j|}|j|}|jdd}|jD]D}tj j }|j r||jkDs=||}F|jdd}|SrP)rrrQrnprandomrr)r!r)r#layerdrop_probs r$r*zWavLMAdapter.forward\s 99 T%9%9%E IIm4M 00?M%//15 [[ 5EYY--/N==^dnn%D %m 4  5 &//15 r%r+r0s@r$rrNs  *r%rr mask_prob mask_lengthr| min_masksrc|\}dkr tdkDrtdddtjjdj fd}|-|j j djnt|Dcgc]}c}}tj|ft } g} |} | d k(r| S|D]} || } tjjtj| dz z | d }t|d k(rdz }n|d }tj|tj| | z tj |zg}| j#|tj$| } tj&| dddddf|| f} | j)|| z} tjddddf}tj&||| fj)|| z}| |z} | j+dz kDrdz | | dz kD<tj,| | dd| Scc}w) af Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://arxiv.org/abs/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `ct|zz z}t|}|zkDrz}|dz z |krt|dz z d}|S)z;Given input length, compute how many spans should be maskedrr)rmax)rqnum_masked_spanepsilonrrrrKs r$compute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spansoi,6DwNOoy9 [ (? :- `attention_mask` should only be passed if the corresponding processor has `config.return_attention_mask == True`. For all models whose processor has `config.return_attention_mask == False`, `attention_mask` should **not** be passed to avoid degraded performance when doing batched inference. For such models `input_values` should simply be padded with 0 and passed without `attention_mask`. Be aware that these models also yield slightly different results depending on whether `input_values` is padded or not. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z_The bare WavLM Model transformer outputting raw hidden-states without any specific head on top.cbeZdZdeffd ZdZdZ ddejde ejde ejfdZ e e eeeed e  dd e ej&de ej&de ejd e ed e ede edeeeffdZxZS) WavLMModelrNct||||_t||_t ||_|jdkDs|jdkDrEtjtj|jj|_|j rt#||_nt'||_|j(r t+|nd|_|j/y)Nr)rrrNrfeature_extractorrSfeature_projectionmask_time_probmask_feature_probr=rurvrr?r]masked_spec_embeddo_stable_layer_normr)encoderrrjradapter post_initras r$rzWavLMModel.__init__&s   !4V!<"8"@  3 &&*B*BS*H%'\\%,,v?Q?Q2R2[2[2]%^D "  & &6v>DL'/DL/5/A/A|F+t  r%cXtjdt|jyz Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.Nwarningswarn FutureWarningfreeze_feature_encoderr!s r$freeze_feature_extractorz#WavLMModel.freeze_feature_extractor:'   Q  ##%r%c8|jjy Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)rrrs r$rz!WavLMModel.freeze_feature_encoderFs 113r%r)mask_time_indicesr|ct|jdds|S|j\}}}|)|jj |j ||<n|jj dkDr|jrt||f|jj |jj||jj}tj||jtj}|jj |j ||<|jjdkDr|jrt||f|jj|jj |jj"}tj||jtj}|dddfj%d|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://arxiv.org/abs/1904.08779). apply_spec_augmentTNr)rrr|r)rr)rrrrU)getattrrNrrrrrrrmask_time_lengthmask_time_min_masksrvtensorrrrmask_feature_lengthmask_feature_min_masksexpand)r!r)rr|rJrKr?mask_feature_indicess r$_mask_hidden_stateszWavLMModel._mask_hidden_statesMst{{$8$? 4A3E3E3G0 O[  (/3/E/E/H/HI\I\/]M+ , [[ ' '! + 5_-++44 KK88-++99 ! !& ->}G[G[chcmcm n /3/E/E/H/HI\I\/]M+ , ;; ( (1 ,#8[)++77 KK;;++<< $ $)<<0D]MaMainisis#t #74#@#G#GO]_#` 23M. /r%audio checkpoint output_typermodalityexpected_outputrTr~rrrcH||n|jj}||n|jj}||n|jj}|j |}|j dd}|!|j |jd|d}|j|\}}|j|||}|j|||||} | d}|j|j|}|s ||f| ddzSt||| j| jS) NrrFr|)rr|r|r~rrr)rextract_featuresr)r)rNr~ruse_return_dictrrQrrrrrrWavLMBaseModelOutputr)r) r!rTr|rr~rrr r)encoder_outputss r$r*zWavLMModel.forward{sb"2C1N-TXT_T_TqTq$8$D $++JjJj &1%uEN+/*A*ABR*S' '00 ->~1 ,, )/!5# ' (* << # LL7M!#34qr7JJ J#+-)77&11   r%)NNNNNNN)r,r-r.rrrrrvrrrrrWAVLM_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErrrrr*r/r0s@r$rr!s {( &4:>59 ,((,$E$5$56,!!1!12 ,\++AB&($. 269=,0/3&*2 u||,2 !.2 $E$5$56 2 $D> 2 'tn 2 d^2  u** +2 C2 r%rrzZ'mister quilter is the aposle of the middle classes and we are glad to welcome his gospel'gQ)@zcWavLM Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC).ceZdZddeeffd ZdZdZdZdZ e e e e eeee ddeej&d eej&d eed eed eed eej&deeeffdZxZS) WavLMForCTC target_langct||t||_t j |j |_||_|jtd|jdt|dr|jr |jn |j}t j ||j|_|j%y)NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `WavLMForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.rj)rrrrSr=r^ final_dropoutr`r vocab_sizeror#rCrjrr?r\lm_headr)r!rNrrr#s r$rzWavLMForCTC.__init__s  ' zz&"6"67 &    $00@AHH *1)GFL^L^F % %djdvdv yy!3V5F5FG  r%c|j}|&t|jddtd|d|-t|jddtj dy||j |dyy)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. Nadapter_attn_dimzCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load)rrrNrologgerinfo load_adapter)r!rs r$ tie_weightszWavLMForCTC.tie_weightss&&  "wt{{ D 'tn D d^ D &D  un$ %D CD r%rz WavLM Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. ceZdZfdZdZdZdZeee e e e d dde ejde ejd e ed e ed e ed e ejd eee ffdZxZS)WavLMForSequenceClassificationct||t|dr|jr t dt ||_|jdz}|jr0tjtj||z |_ tj|j|j |_tj|j |j$|_|j)y)Nrjz\Sequence classification does not support the use of WavLM adapters (config.add_adapter=True)r)rrrCrjrorrSruse_weighted_layer_sumr=rurvrw layer_weightsr\r?classifier_proj_size projector num_labels classifierrr!rN num_layersr#s r$rz'WavLMForSequenceClassification.__init__as  6= )f.@.@n  ' --1  ( (!#ejj.Dz.Q!RD 6#5#5v7R7RS))F$?$?ARARS r%cXtjdt|jyrrrs r$rz7WavLMForSequenceClassification.freeze_feature_extractorrrr%cL|jjjyrr%rs r$rz5WavLMForSequenceClassification.freeze_feature_encoder~r&r%cP|jjD] }d|_ yr(r)rs r$r*z0WavLMForSequenceClassification.freeze_base_modelr+r%r)rrrrrTr|r~rrr-rc<||n|jj}|jjrdn|}|j|||||}|jjrr|t}t j |d}tjj|jd} || jdddzjd}n|d}|j|}||jd} n|j|j d|} | j#dj%dd|j d} d || <|jd| jdjddz } |j'| } d}|Ft)}|| jd|jj*|jd}|s| f|tdz}||f|zS|St-|| |j.|j0 S)  labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTr rrrUrrrr3)rNr rLrSrArvstackr=rCrFrMrrrOr9rrrrrQrrPr r)r)r!rTr|r~rrr-rr) norm_weights pooled_output padding_maskexpand_padding_maskr5r4loss_fctrFs r$r*z&WavLMForSequenceClassification.forwards.&1%->r1a-HHMMRSMTM#AJM}5  !)..1.5MBB=CVCVWXCY[ijL"."8"8"<"C"CAq-J]J]^_J`"a 25M.. /)--!-4|7G7GA7G7N7S7STVXY7ZZM/  ')HFKKDKK,B,BCV[[QS_UDY)F)G!HHF)-)9TGf$ Ev E'!//))   r%r)r,r-r.rrrr*rrrrr rrrvrrrrr*r/r0s@r$rJrJYs" &:(++AB&,$ 26,0/3&*)-< u||,< !.< $D> < 'tn < d^ < &<  u.. /< C< r%rJzmicrosoft/wavlm-base-plus-sdza WavLM Model with a frame classification head on top for tasks like Speaker Diarization. ceZdZfdZdZdZdZeee e e e de  ddeej deej d eej d eed eed eed eee ffdZxZS) WavLMForAudioFrameClassificationct||t|dr|jr t dt ||_|jdz}|jr0tjtj||z |_ tj|j|j |_|j |_|j%y)Nrjz_Audio frame classification does not support the use of WavLM adapters (config.add_adapter=True)r)rrrCrjrorrSrrLr=rurvrwrMr\r?rPrQ init_weightsrRs r$rz)WavLMForAudioFrameClassification.__init__s  6= )f.@.@q  ' --1  ( (!#ejj.Dz.Q!RD ))F$6$68I8IJ ++ r%cXtjdt|jyr#rrs r$rz9WavLMForAudioFrameClassification.freeze_feature_extractorrr%cL|jjjyrr%rs r$rz7WavLMForAudioFrameClassification.freeze_feature_encoderr&r%cP|jjD] }d|_ yr(r)rs r$r*z2WavLMForAudioFrameClassification.freeze_base_modelr+r%rrrTr|r-r~rrrc ||n|jj}|jjrdn|}|j|||||}|jjrr|t}t j |d}tjj|jd} || jdddzjd}n|d}|j|} d} |\t} | | jd|jt j |jd|jd} |s| f|tdz} | St#| | |j$|j& S) rXNTr rrrUr)axisr3)rNr rLrSrArvrYr=rCrFrMrrrQrrPrGrr)r)r!rTr|r-r~rrrr)rZr5r4r^rFs r$r*z(WavLMForAudioFrameClassification.forward sh0&1%->r1a-HHMMRSMTM#AJM/  ')HFKKDOOtuDY)F)G!HHFM$!//))   r%r)r,r-r.rrrr*rrr_FRAME_CLASS_CHECKPOINTrr_FRAME_EXPECTED_OUTPUTrrvrrrrr*r/r0s@r$r`r`s &:(++AB*)$. 26)-,0/3&*3 u||,3 !.3 & 3 $D> 3 'tn 3 d^3  u++ ,3 C3 r%r`c&eZdZdfd ZdZxZS) AMSoftmaxLossctt| ||_||_||_t jtj||d|_ t j|_ y)NT)r) rrkrscalemarginrPr=rurvrandnr9rr4)r! input_dimrPrmrnr#s r$rzAMSoftmaxLoss.__init__HsS mT+-  $ll5;;y*#EUYZ '') r%c|j}tjj|jd}tjj|d}t j ||}||jz }tjj||j}|jt j|j||z}|j||}|S)Nrrr)flattenr=rC normalizer9rvmmrnone_hotrPrmrrr4) r!r)r-r9 cos_thetapsionehotr5r4s r$r*zAMSoftmaxLoss.forwardPs!((!(< // 1/E HH]F3 $++%&&vt?ekk&++-iHHyy( r%)g>@g?r+r0s@r$rkrkGs * r%rkcXeZdZdfd Zdej dej fdZxZS) TDNNLayerct||dkDr|j|dz n|j||_|j||_|j ||_|j||_tj|j|j z|j|_ tj|_ y)Nrr)rrtdnn_dimrr tdnn_kernelr4 tdnn_dilationdilationr=r\kernelReLUrMrs r$rzTDNNLayer.__init___s +BFD]D]!^))F$=$=v?X?XY&v'@'@&BSBST Rs>FcXtjdt|jyr#rrs r$rz(WavLMForXVector.freeze_feature_extractorrr%cL|jjjyrr%rs r$rz&WavLMForXVector.freeze_feature_encoderr&r%cP|jjD] }d|_ yr(r)rs r$r*z!WavLMForXVector.freeze_base_modelr+r%ricVd}|jjD] }|||d}|S)z? Computes the output length of the TDNN layers c||z |zdzS)Nrr rps r$rszBWavLMForXVector._get_tdnn_output_lengths.._conv_out_lengths!;.69A= =r%r)rNr})r!rirsr4s r$_get_tdnn_output_lengthsz(WavLMForXVector._get_tdnn_output_lengthss:  >  ;;22 LK,]KKM Lr%rrrTr|r~rrr-rc||n|jj}|jjrdn|}|j|||||}|jjrr|t}t j |d}tjj|jd} || jdddzjd}n|d}|j|}|jD] } | |} |%|jd} |j!d} n|j#|jd} |j%| }g} g} t'|D]U\}}| j)||d|fjd| j)||d|fj!dWt j | } t j | } t j*| | gd}|j-|}|j/|}d}||j1||}|s||f|tdz}||f|zS|St3||||j4|j6S) rXNTr rrrUr)r4r5 embeddingsr)r)rNr rLrSrArvrYr=rCrFrMrrrOrr9rVryrrrrrrQrrr)r)r!rTr|r~rrr-rr)rZ tdnn_layer mean_features std_featuresfeat_extract_output_lengthstdnn_output_lengthsrlengthstatistic_poolingoutput_embeddingsr5r4rFs r$r*zWavLMForXVector.forwards0&1%->r1a-HHMMRSMTM#AJM}5 )) 6J&}5M 6  !)..1.5M(,,,3L*.*O*OP^PbPbghPbPi*j '"&"?"?@["\ ML&':; J 6$$]1gvg:%>%C%C%C%JK##M!WfW*$=$A$Aa$A$HI J"KK 6M ;;|4L!II}l&CL 223DE!23  >>&&1D/07;X;Y3ZZF)-)9TGf$ Ev E(!//))   r%r)r,r-r.rrrr*rrvrrrrrr_XVECTOR_CHECKPOINTrr_XVECTOR_EXPECTED_OUTPUTrrrrr*r/r0s@r$rrs& &:( eE I 'tn I d^ I &I  um# $I CI r%r)r`rrJrrrRr')Urrtypingrrrnumpyrrvtorch.nnr=torch.nn.functionalrCrr activationsrintegrations.deepspeedr integrations.fsdpr modeling_outputsr r r rrrmodeling_utilsrrBrrrrrconfiguration_wavlmr get_loggerr,rrrModulerr2rSrfrrrrr)r.rRrrrrrrrrrndarrayrrWAVLM_START_DOCSTRINGrr rrArGrHrrJrhrir`rkrzrrr__all__r r%r$rsO )) %!@7.-   H %I *299*Z 1RYY 1c RYYc Lryy0& &R"ryy"JP 299P fQ "))Q hC'C'LW?Wt *bii6bii0)")))X $299F26 t c?tttU--. t  t ZZ tn'&"H/eP %P P f!"smT &T T n p %9p p f9Q  g ';g  g TBII. @5  O *O  O d r%