T^(h8 &dZddlZddlZddlmZddlmZmZmZddl Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZmZmZmZmZmZm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,ddl-m.Z.dZ/dZ0e*rddl1m2Z3erddlm4Z4e+jje6Z7dZ8dZ9dZ:gdZ;dZdZ?dZ@dZAddgZBdZCd ZDeGd!d"e$ZE dvd#eeFeFfd$eGd%eFd&ee jd'eFd(e jf d)ZJ dwd*ed+eFd,ee jfd-ZKGd.d/e jZMGd0d1e jZNGd2d3e jZOGd4d5e jZPGd6d7e jZQGd8d9e jZRGd:d;eRZSGd<d=e jZTGd>d?e jZUGd@dAeUZVGdBdCeUZWeUeWeVdDZXGdEdFe jZYGdGdHe jZZGdIdJe jZ[GdKdLe jZ\GdMdNe jZ]GdOdPe jZ^GdQdRe jZ_GdSdTe jZ`GdUdVe jZaGdWdXe"ZbdYZcdZZde&d[ecGd\d]ebZee&d^ecGd_d`ebZfe&daecGdbdcebZge&ddecdeGdfdgebZhe&dhecGdidjebZie&dkecGdldmebZjGdndoe jZkGdpdqe jZle&drecGdsdtebZmgduZny)xzPyTorch Wav2Vec2 model.N) dataclass)OptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)!flash_attn_supports_top_left_maskis_flash_attn_available)BaseModelOutputCausalLMOutputMaskedLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)PreTrainedModel) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forward cached_fileis_peft_availableis_safetensors_availableloggingreplace_return_docstrings)Wav2Vec2Configzadapter.{}.binzadapter.{}.safetensors) load_file)_flash_attention_forwardr!zfacebook/wav2vec2-base-960h)r i$iz['MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL'g= ףpJ@zsuperb/wav2vec2-base-superb-ksz '_unknown_'g)\(@zanton-l/wav2vec2-base-superb-sdzanton-l/wav2vec2-base-superb-svg\(\?c^eZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ ee ejed<dZee ejed<dZeejed <dZeejed <y) Wav2Vec2ForPreTrainingOutputa1 Output type of [`Wav2Vec2ForPreTraining`], with potential hidden states and attentions. Args: loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://arxiv.org/pdf/2006.11477.pdf) . (classification) loss. projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. contrastive_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The contrastive loss (L_m) as stated in the [official paper](https://arxiv.org/pdf/2006.11477.pdf) . diversity_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The diversity loss (L_d) as stated in the [official paper](https://arxiv.org/pdf/2006.11477.pdf) . Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentionscontrastive_lossdiversity_loss)__name__ __module__ __qualname____doc__r'rtorch FloatTensor__annotations__r(r)r*r+rr,r-r.l/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.pyr&r&as<)-D(5$$ %,48hu0018>B):): ;B9=8E$5$56=8.compute_num_masked_spansoi,6DwNOoy9 [ (? :- Sample `num_negatives` vectors from feature vectors. )r9rKNrJr r)size) rNrYrVr]astyperWr\rUrSr`rOrandint) rmrnrorcrGsequence_length_rangesampled_negative_indices batch_idxhighmapped_masked_indicesfeature_indicessampled_indicess r8_sample_negative_indicesr{sT #1JIIo6 "xxz?M.Zbdbjbjk+<*G  &RWWUckoMp:& K  +//1A5 56G 6R S//"))D1H*=ag*FPQS`Hab))++At4!8]:S+T?:;q@;MbbqLr +,=i,HI !+y?/JJ+ K $#r7c&eZdZdfd ZdZxZS)Wav2Vec2NoLayerNormConvLayercdt||dkDr|j|dz nd|_|j||_t j |j|j|j||j||j|_ t|j|_ y)Nrr  kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__s r8rz%Wav2Vec2NoLayerNormConvLayer.__init__'s !#Car7cV|jdkDr|ddddd|j f}|SNr)rrs r8rzWav2Vec2SamePadLayer.forwards6    ")!Q0F43F3F2F0F*FGMr7rrs@r8rrsKr7rc.eZdZdZfdZdZdZxZS)Wav2Vec2FeatureEncoderz.Construct the features from raw audio waveformc t||jdk(rDt|dgt |j dz Dcgc]}t ||dzc}z}nV|jdk(r.t |j Dcgc]}t||}}ntd|jdtj||_ d|_ d |_ ycc}wcc}w) Ngroupr)rr layerz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT)rrfeat_extract_normrrUnum_feat_extract_layersr}rrMr ModuleList conv_layersgradient_checkpointing_requires_grad)rrirrs r8rzWav2Vec2FeatureEncoder.__init__s   # #w .5fqIJNSTZTrTruvTvNwNIJ,Va!eDNK % % 0HMfNlNlHmCD*6A>K01I1I0JJst ==5&+#"Ns C" C'cJ|jD] }d|_ d|_yNF) parameters requires_gradrrparams r8_freeze_parametersz)Wav2Vec2FeatureEncoder._freeze_parameterss(__& (E"'E  (#r7c |dddf}|jr|jrd|_|jD]K}|jr5|jr)|jr|j |j |}D||}M|S)NT)rtrainingrrr_gradient_checkpointing_func__call__)r input_valuesr+ conv_layers r8rzWav2Vec2FeatureEncoder.forwards$QW-    4==*.M '** :J""t'B'Bt}} $ A A''!! !+= 9  :r7)r/r0r1r2rrrrrs@r8rrs8#&$ r7rceZdZfdZxZS)Wav2Vec2FeatureExtractorct||tjd|jj d|jj dj dty)Nz The class `zD` has been depreciated and will be removed in Transformers v5. Use `rz ` instead.)rrwarningswarnrr/ __bases__ FutureWarningrrrs r8rz!Wav2Vec2FeatureExtractor.__init__s[   $..1123NN,,Q/889 E   r7)r/r0r1rrrs@r8rrs   r7rc$eZdZfdZdZxZS)Wav2Vec2FeatureProjectionc4t|tj|jd|j |_tj|jd|j|_ tj|j|_ y)NrIeps) rrrrrlayer_norm_epsrLinearr projectionDropoutfeat_proj_dropoutdropoutrs r8rz"Wav2Vec2FeatureProjection.__init__sf ,,vr':@U@UV))FOOB$79K9KLzz&":":; r7cp|j|}|j|}|j|}||fSr)rrr)rr+norm_hidden_statess r8rz!Wav2Vec2FeatureProjection.forwards:!__];(:;  ]3 000r7rrs@r8rrs < 1r7rceZdZdZ ddededededededeeffd Z d e jd ed efd Z dde jdee jdee e jdee jdee jdede e jee jee e jffdZxZS)Wav2Vec2Attentionz=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsr is_decoderr is_causalrc t|||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ tj||||_ tj||||_tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g)r)rrrrrhead_dimrrMscalingrrrrk_projv_projq_projout_proj) rrrrrrrrrs r8rzWav2Vec2Attention.__init__s "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB r7tensorseq_lenbszc|j|||j|jjddj Sr)viewrrr contiguousrrr r s r8_shapezWav2Vec2Attention._shapes7{{3GQQRSUVWbbddr7r+key_value_statespast_key_valuer<layer_head_maskoutput_attentionsr>c |du}|j\}} } |j||jz} |r0|.|djd|jdk(r |d} |d} n |rE|j |j |d|} |j |j |d|} n|}|j |j |d|} |j |j |d|} tj|d| gd} tj|d| gd} nD|j |j |d|} |j |j |d|} |jr| | f}||jzd|jf}|j | | |j|} | j|} | j|} | jd}tj| | jdd}|j||jz| |fk7r/t!d||jz| |fd|j|{|j|d| |fk7r#t!d |d| |fd|j|j||j| ||z}|j||jz| |}t"j$j'|d}||j|jfk7r*t!d |jfd|j|jdddd|j||j| |z}|j||jz| |}|r?|j||j| |}|j||jz| |}nd}t"j$j)||j(|j* }tj|| }|j||jz| |jfk7r9t!d ||jz| |jfd|j|j||j| |j}|jdd}|j|| |j,}|j/|}|||fS) #Input shape: Batch x Time x ChannelNrr$r rIrz$Attention weights should be of size , but is z!Attention mask should be of size z/Head mask for a single layer should be of size )pr `attn_output` should be of size )rqrrr9rrrr3catrrrr rabmmrrMr functionalsoftmaxrrrr)rr+rrr<rris_cross_attentionr tgt_lenrd query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r8rzWav2Vec2Attention.forwards.T9',,.Wa{{=1DLL@ *q!''*.>.D.DQ.GG(*J)!,L T[[1A%BBLJ;;t{{3C'Db#NL  'T[[%?SIJ;;t{{='A2sKLN1$5z#BJJ 99nQ&7%FANLT[[%?SIJ;;t{{='A2sKL ??),7NDNN*B > Ct{{<#>CCZP 'Z''4 +|++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL',,S4>>-A7GTL}},,\r,B  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVZVdVdfmov?wwL',,S4>>-A7GTL  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<AAr7)FTFNNNNNF)r/r0r1r2rBfloatrWrr!rr3Tensorrrrrrs@r8rrsMG  +/CCC C  C  CC(C>eU\\eCece 488<1526"'vB||vB#5<<0vB!u||!45 vB !. vB "%,,/ vB vB u||Xell3XeELL>Q5RR SvBr7rcVeZdZdZfdZdej dedefdZ ddej de ej d e e ej d e ej d e ej d e d e ej e ej e e ej ffdZ xZ S)Wav2Vec2FlashAttention2aL Wav2Vec2 flash attention module. This module inherits from `Wav2Vec2Attention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cBt||i|t|_yr)rrr _flash_attn_uses_top_left_mask)rargskwargsrs r8rz Wav2Vec2FlashAttention2.__init__s# $)&) /P.Q+r7rr r cR|j|||j|jSr)r rrrs r8_reshapez Wav2Vec2FlashAttention2._reshapes{{3GGr7r+rrr<rrr>c |r td|du}|j\}} } |j|j|d|} |rP|N|djd|jdk(r,|dj dd} |dj dd} n*|rE|j|j |d|} |j|j|d|} n||j|j |d|} |j|j|d|} tj|dj dd| gd} tj|dj dd| gd} nD|j|j |d|} |j|j|d|} |jr$| j dd| j ddf}| jd}|||djdz }| j}|tjk(rtjrtj}nMt|j dr|j j"}n |jj$j}t&j)d |d | j+|} | j+|} | j+|} t-| | | || |j.r |j0nd |j2|j4 }|j7|| d}|j9|}|sd}||fS) NzDWav2Vec2FlashAttention2 attention does not support output_attentionsrIrr$r rr_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .r))rruse_top_left_mask)rMrqr4rr9rrrr3rrrKfloat32is_autocast_enabledget_autocast_gpu_dtyperrr6rlogger warning_oncetor#rrrr0rar)rr+rrr<rrrr q_lenrdr r!r" kv_seq_len input_dtype target_dtyper(r%s r8rzWav2Vec2FlashAttention2.forwards0 cd d.T9%**, UA}}T[[%?SI *q!''*.>.D.DQ.GG(*44Q:J)!,66q!$  (??<8L#|4J'??<8L.     $(MMDLLsnn"AA  "))#ub9 mmK0  LL.88r7r*)r/r0r1r2rr3r,rBr4rrrWrrrs@r8r.r.s RHu||HcHH 488<1526"'i9||i9#5<<0i9!u||!45 i9 !. i9 "%,,/ i9 i9 u||Xell3XeELL>Q5RR Si9r7r.c$eZdZ d dejdeejdeeejdeejdeejdedeejeejeeejfffd ZxZ S) Wav2Vec2SdpaAttentionr+rrr<rrr>cz|s|*tjdt| ||||||S|du}|j \}} } |j |} |r0|.|dj d|j dk(r |d} |d} n |rE|j|j|d|} |j|j|d|} n|}|j|j|d|} |j|j|d|} tj|d| gd} tj|d| gd} nD|j|j|d|} |j|j|d|} |jr| | f}|j| | |} |jr || dkDrd nd }tjjj!| | | ||j"r |j$nd | }|j ||j&| |j(fk7r7t+d ||j&| |j(fd|j |j-dd}|j/|| |j0}|j3|}|d|fS)rNaWav2Vec2Model is using Wav2Vec2SdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `layer_head_mask` not None. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rrr<rrrr$r rIrTFr)) attn_mask dropout_prrr)r<r=rrrqrr9rrrr3rrrrrscaled_dot_product_attentionrrrrrMrrarr)rr+rrr<rrrr rrdr r!r"rr(rs r8rzWav2Vec2SdpaAttention.forwards  ;   l 7?!1-- /"3 # .T9',,.Wa{{=1 *q!''*.>.D.DQ.GG(*J)!,L T[[1A%BBLJ;;t{{3C'Db#NL  'T[[%?SIJ;;t{{='A2sKLN1$5z#BJJ 99nQ&7%FANLT[[%?SIJ;;t{{='A2sKL ??),7N{{<#> !NN~/E'TU+D[` hh))FF   $&*mmdll G     #t~~w !N N2CRVR_R_3`2ab$$&')  "++Aq1 "))#wG mmK0 D.00r7r*) r/r0r1r3r,rrrWrrrs@r8rDrD s 488<1526"'f1||f1#5<<0f1!u||!45 f1 !. f1 "%,,/ f1 f1 u||Xell3XeELL>Q5RR Sf1f1r7rD)eagersdpaflash_attention_2c$eZdZfdZdZxZS)Wav2Vec2FeedForwardct|tj|j|_tj |j|j|_ t|jtrt|j|_n|j|_tj |j|j|_tj|j |_yr)rrrractivation_dropoutintermediate_dropoutrrintermediate_sizeintermediate_dense isinstance hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropoutrs r8rzWav2Vec2FeedForward.__init__s $&JJv/H/H$I!"$))F,>,>@X@X"Y f'' -'-f.?.?'@D $'-'8'8D $IIf&>&>@R@RS jj)>)>?r7c|j|}|j|}|j|}|j|}|j |}|Sr)rRrVrPrWrYrs r8rzWav2Vec2FeedForward.forwardsX// > 00? 11-@ ))-8 ++M: r7rrs@r8rMrM~s @r7rMc&eZdZfdZddZxZS)Wav2Vec2EncoderLayerct|t|j|j|j |j d|_tj|j|_ tj|j|j|_t||_tj|j|j|_y)NFrrrrr)rrWAV2VEC2_ATTENTION_CLASSES_attn_implementationrnum_attention_headsattention_dropout attentionrrrXrrrrrM feed_forwardfinal_layer_normrs r8rzWav2Vec2EncoderLayer.__init__s 3F4O4OP((00,,  zz&"7"78 ,,v'9'9v?T?TU/7 " V-?-?VEZEZ [r7c|}|j|||\}}}|j|}||z}|j|}||j|z}|j |}|f}|r||fz }|SNr<r)rcrrrdrerr+r<r attn_residualr%rdoutputss r8rzWav2Vec2EncoderLayer.forwards% )- .L]*8* & |Q ]3 % 5  6 %(9(9-(HH --m<  "    &Gr7rrrs@r8r\r\s \r7r\cfeZdZfdZ ddej deej defdZxZ S)#Wav2Vec2EncoderLayerStableLayerNormct|t|j|j|j |j d|_tj|j|_ tj|j|j|_t||_tj|j|j|_t%|ddt'||_yd|_y)NFr^radapter_attn_dim)rrr_r`rrarbrcrrrXrrrrrMrdregetattrWav2Vec2AttnAdapterLayer adapter_layerrs r8rz,Wav2Vec2EncoderLayerStableLayerNorm.__init__s 3F4O4OP((00,,   zz&"7"78 ,,v'9'9v?T?TU/7 " V-?-?VEZEZ [ 6-t 4 @!9&!AD !%D r7r+r<rc$|}|j|}|j|||\}}}|j|}||z}||j|j |z}|j ||j |z}|f}|r||fz }|Srg)rrcrrdrerrris r8rz+Wav2Vec2EncoderLayerStableLayerNorm.forwards &  6 )- .L]*8* & |Q ]3 % 5 %(9(9$:O:OP]:^(__    ))D,>,>},MMM "    &Gr7r) r/r0r1rr3r,rrWrrrs@r8rmrms>&*26"' ||!. r7rmc reZdZfdZ ddej deejdededef dZ xZ S) Wav2Vec2Encoderct|||_t||_t j |j|j|_ t j|j|_ t jt|jDcgc] }t!|c}|_d|_|j&dk(|_ycc}wNrFrK)rrrrpos_conv_embedrrrrrrrXrrrUnum_hidden_layersr\layersrr`_use_flash_attention_2rrrdrs r8rzWav2Vec2Encoder.__init__s  =fE,,v'9'9v?T?TUzz&"7"78 mm5QWQiQiKj$ka%9&%A$kl &+#&,&A&AEX&X#%l!Cr+r<routput_hidden_states return_dictc4|rdnd}|rdnd}||jdjdd|jd}d||<|jr |d|vr|nd}nd|ddddddfj |j z }|t j|j jz}|j|jdd|jd|jd}|j|} || z}|j|}|j|}txs t|} |jD]} |r||fz}t j g} |j"r| |j$j&krdnd } | r| rG|j(r+|j"r|j+| j,|||}n | ||| }|d}| rd }|s|dfz}|r||fz}|st/d |||fDSt1||| S)Nr6rIr r$r?rJTFrhNNc3&K|] }|| ywrr6.0vs r8 z*Wav2Vec2Encoder.forward..3mq_`_lmlast_hidden_stater+r,) unsqueezerepeatr9r{r>rKr3finfominexpandrxrrr r rzrPrr layerdroprrrtuplerrr+r<rr~rall_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusrdropout_probabilityskip_the_layer layer_outputss r8rzWav2Vec2Encoder.forwards[#7BD$5b4  %$2$<$Q>Q2R2V2V!V!/!6!6"((+Q0D0DR0H.J^J^_aJb"#11-@%(;;  6  ]3 02R6LT6R [[ PE#$58H$H!#(**R. %)]]8KdkkNcNc8cTjoN![..4==$($E$E%&) %M%*%nXi%M!.a 0  ,  &9]1=M.rrr)rrr9r>rKr{r3rrrrxrr r rzrPrrrrrrrrrrs r8rz&Wav2Vec2EncoderStableLayerNorm.forwardHsn#7BD$5b4  %$2$<$Q>Q2R2V2V!V!/!6!6"((+Q0D0DR0H.J^J^_aJb"#11-@%(;;  ]3 02R6LT6R [[ PE#$58H$H!#(**R. %)]]8KdkkNcNc8cTjoN![..4==$($E$E%&) %M%*%nXi%M!.a 0  ,  &9]1=MrI) flattenrr9r3where zeros_likerSmeanexplog)probsmask mask_extendedmarginal_probs perplexitys r8_compute_perplexityz1Wav2Vec2GumbelVectorQuantizer._compute_perplexitys   LLN1dD=9@@MMKK ue6F6Fu6MNE"YY1Y- :N"ZZAZ.NYY .599^VZEZ;[*[ac ddeiik r7c|j\}}}|j|}|j||z|jzd}|jrt j j|j|jdj|}tj|j||z|jdjd}|j||}n|jd} |j|jj!d| jddd}|j||z|jd}|j||}|j||zd}|j#d|j$z} | j||z|j|j&d} | j)dj||d} | |fS)NrIT)tauhardrr rr)r9rr rrrrgumbel_softmaxr+rtype_asr3rrargmax new_zerosscatter_rrrrS) rr+rorcrGrcodevector_probscodevector_soft_distrcodevector_idxcodevectors_per_grouprs r8rz%Wav2Vec2GumbelVectorQuantizer.forwards3@3F3F0 O[((7 %**:+G$//+Y[]^ ==!}};;##%4+;+;$ < gm$  $)=="":#?RTU[[]ce$ 112FHYZJ+11b19N,66}7J7JKTTN''A.   044Z/5QSWSbSbdfg 112BDUVJ+00o1MrR 0 : :2 >AQAQ Q+00o1Mt`d`m`moqr !oob)..z?BO J&&r7r) r/r0r1r2r staticmethodrrrrs@r8rrs& (  #'r7rc$eZdZfdZdZxZS)Wav2Vec2Adapterct|jjk7rTt j jj|_t jj|_ndx|_|_t jfdtjD|_ j|_ y)Nc34K|]}tywr)Wav2Vec2AdapterLayer)rrdrs r8rz+Wav2Vec2Adapter.__init__..s#kQ$8$@#ks)rroutput_hidden_sizerrrprojrproj_layer_normrrUnum_adapter_layersrzrrs `r8rzWav2Vec2Adapter.__init__s   $ $(:(: : &"4"4f6O6OPDI#%<<0I0I#JD /3 3DI,mm#k%PVPiPiJj#kk ))r7ch|j.|j"|j|}|j|}|jdd}|jD]D}tj j }|j r||jkDs=||}F|jdd}|Sr)rrrrzrNrOrr)rr+rlayerdrop_probs r8rzWav2Vec2Adapter.forwards 99 T%9%9%E IIm4M 00?M%//15 [[ 5EYY--/N==^dnn%D %m 4  5 &//15 r7rrs@r8rrs  *r7rc$eZdZfdZdZxZS)rct|tj|jd|jz|j |j d|_y)Nr$r )rr)rrrrradapter_kernel_sizeadapter_striderrs r8rzWav2Vec2AdapterLayer.__init__sJ II  % % )) )  & &((   r7cj|j|}tjj|d}|S)Nr r)rrrglurs r8rzWav2Vec2AdapterLayer.forward s/ -0  ))-Q)? r7rrs@r8rrs  r7rc>eZdZfdZdej fdZxZS)rqct||j|_|j|_t j|j |_t j|j |j|_ t j|_ t j|j|j |_ y)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)rrro input_dimr hidden_dimrrnormrlinear_1ReLUact_fnlinear_2rs r8rz!Wav2Vec2AttnAdapterLayer.__init__s 00 ,,LL1  $//4>>B ggi  $..$//B r7r+c|j|}|j|}|j|}|j|}|Sr)rrrrrs r8rz Wav2Vec2AttnAdapterLayer.forward s@ -0  m4  M2  m4 r7)r/r0r1rr3r4rrrs@r8rqrqs CU%6%6r7rqceZdZdZeZdZdZdZdZ dZ dZ dde e jefdeefd Z dd ed e jfd Zd ZdZddefdZy)Wav2Vec2PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. wav2vec2rTc Ht|trW|jj|jjd|j_d|j_yt|t r|jjjjdd|jjjjtjj|j yt|t"rtjj|j$jddt'j(d|j$j*d|j$j,zz ztjj/|j$jdyt|t0rt'j(d|j2j4z }tjj|j2j| |tjj|j2j| |yt|tj6rm|jjjd|j8j:|j%|jjjyyt|tj<tj>frJ|jjj|jjjAd yt|tjBrtjjE|j|jjt'j(|jF|j,|j*dzz }tjj|j| |yyy) zInitialize the weightsTr)r )rstdrr$)abNr)$rSWav2Vec2ForPreTraining project_hidreset_parameters project_q_is_hf_initializedrrrdatanormal_rzero_rinituniform_rrrmathsqrtr in_channels constant_rr in_featuresrrinitializer_rangerrfill_rkaiming_normal_r)rmoduleks r8 _init_weightsz%Wav2Vec2PreTrainedModel._init_weights7s f4 5    / / 1    - - /48F   126F   /  = >    % % * * 2 2 2 C    # # ( ( . . 0 GG  V// 0  ? @ GGOO "" !v{{'>'>q'AFKKD[D['["\]]   GG  fkk.. 2  9 : !f//;;;tj||z |ddzS)Nfloor) rounding_moder )r3divrDrrs r8_conv_out_lengthzRWav2Vec2PreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengthgs"99\K7wWZ[[ [r7r )rrziprrrUrr)rrerrrrrds r8 _get_feat_extract_output_lengthsz8Wav2Vec2PreTrainedModel._get_feat_extract_output_lengths^s2=1Ddkk--+  \ $'t{{'>'> @W@W#X Q K,]KPM Q 4;;99: _ 04;;C]C] ^  _r7feature_vector_lengthr<c|jddddf}|j||}|jtj}|j d}tj ||f|j|j}d|tj|j d|j|dz f<|jdgjdjdgj}|S)NrIrrr)rKdevicer )r ) cumsumrr>r3longr9rVrKr rYfliprW)rrr<rnon_padded_lengthsoutput_lengthsrcs r8"_get_feature_vector_attention_maskz:Wav2Vec2PreTrainedModel._get_feature_vector_attention_maskus ,22r2:1b5A>>?Q_j>k'**5::6#))!,  . /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr7c|jjt|jdi}|j D]D\}}t |t s|jD]\}}||dj||g<Ft |tr8|jjD]\}}||djd|g<|S)NzF has no adapter layers. Make sure to define `config.adapter_attn_dim`.r7lm_head) rrorMr named_modulesrSrqnamed_parametersjoinWav2Vec2ForCTCr)radapter_weightsrr param_namers r8 _get_adaptersz%Wav2Vec2PreTrainedModel._get_adapterss ;; ' ' ///uvw w ..0 JLD&&":;)/)@)@)BJ%JDIOCHHdJ-?$@AJ J dN +#||<<> E e?D)T): ;< Er7c|jD]$}t|ts|j|&t|tr|j|j yy)zc (Re-)initialize attention adapter layers and lm head for adapter-only fine-tuning N)modulesrSrqrrr)rrs r8init_adapter_layersz+Wav2Vec2PreTrainedModel.init_adapter_layerssU lln +F&":;""6* + dN +   t|| , ,r7 target_langc |jjtd|d||jk(r|stj d|dy|j dd}|j dd}|j d d}|j d d}|j d d}|j d d} |j d d} |j dd} |j dtrdnd} | )tjdt| td| } |jj} d}| dur5tj|} t| |||||| | | }t|}|Bt$j|} t| |||||| | | }t'j(|dd}|j+}t-|j/t-|j/z }t-|j/t-|j/z }t1|dkDr!tdddj3|dt1|dkDr!tdddj3|d|dj4d}||jj6k7rWt9j:|jj<||j>|j@|_!||j_|jEDcic]\}}||jG||}}}|jI|d ||_y#t $r| rYt"$r| rt!d| d| d|dYwxYw#t $rt"$rt!d| d| d|dwxYwcc}}w)!a Load a language adapter model from a pre-trained adapter model. Parameters: target_lang (`str`): Has to be a language id of an existing adapter weight. Adapter weights are stored in the format adapter..safetensors or adapter..bin force_load (`bool`, defaults to `True`): Whether the weights shall be loaded even if `target_lang` matches `self.target_lang`. cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (i.e., do not try to download the model). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. mirror (`str`, *optional*): Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information. Activate the special ["offline-mode"](https://huggingface.co/transformers/installation.html#offline-mode) to use this method in a firewalled environment. Examples: ```python >>> from transformers import Wav2Vec2ForCTC, AutoProcessor >>> ckpt = "facebook/mms-1b-all" >>> processor = AutoProcessor.from_pretrained(ckpt) >>> model = Wav2Vec2ForCTC.from_pretrained(ckpt, target_lang="eng") >>> # set specific language >>> processor.tokenizer.set_target_lang("spa") >>> model.load_adapter("spa") ``` NzCannot load_adapter for - if `config.adapter_attn_dim` is not defined.z#Adapter weights are already set to r7 cache_dirforce_downloadFresume_downloadproxieslocal_files_onlytokenuse_auth_tokenrevisionuse_safetensorszrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.zV`token` and `use_auth_token` are both specified. Please set only the argument `token`.)filenamer"r#r$r%r&r(r!zCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named cpuT) map_location weights_onlyrzThe adapter weights z has unexpected keys: z, z has missing keys: zlm_head.weightr rK)strict)%rrorMrr<warningpoprrrr _name_or_pathWAV2VEC2_ADAPTER_SAFE_FILEformatrsafe_load_fileEnvironmentError ExceptionWAV2VEC2_ADAPTER_PT_FILEr3loadrsetkeysrZrr9 vocab_sizerrrr rKritemsr>load_state_dict)rr force_loadr2r!r"r#r$r%r&r'r(r)model_path_or_id state_dictfilepath weight_pathrunexpected_keys missing_keystarget_vocab_sizerrs r8 load_adapterz$Wav2Vec2PreTrainedModel.load_adapters.~ ;; ' ' /7 }Dqrs s $** *: NN@ QO P JJ{D1 $4e< **%6=**Y-!::&8%@ 7D)$4d;::j$/ **%6@X@Z`ef  % MME   l#E;;44  % '188EH )$%#1$3#%5%'  ,K8 &  /66{CH )$%#1$3#%5%'  #ZZ!&!% (,,.joo/037K7K7M3NN?//12S9J5KK  ! #3K=@VW[W`W`apWqVrrstu u   "3K=@STXT]T]^jTkSllmno o''78>>qA  6 6 699 ..0A$++]a]g]gDL&7DKK "?I>N>N>PQdaaoa011Q Q Z6'W$ "#  "*45E4FG==M>FZqJ# B$  &01A0BC99I8JK::B1F 6Rs*M%,N(O% N4NN(Or)T)r/r0r1r2r! config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_supports_flash_attn_2_supports_sdparrr3 LongTensorrBrrWrrrrrUrGr6r7r8rr*s "L"$O&*#!N%9PZ^"5#3#3S#89HPQU0Y]%(:?:J:J( -|'|'r7ra Wav2Vec2 was proposed in [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://arxiv.org/abs/2006.11477) by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving etc.). This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`Wav2Vec2Config`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`Wav2Vec2Processor.__call__`] for details. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) `attention_mask` should only be passed if the corresponding processor has `config.return_attention_mask == True`. For all models whose processor has `config.return_attention_mask == False`, such as [wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base-960h), `attention_mask` should **not** be passed to avoid degraded performance when doing batched inference. For such models `input_values` should simply be padded with 0 and passed without `attention_mask`. Be aware that these models also yield slightly different results depending on whether `input_values` is padded or not. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zbThe bare Wav2Vec2 Model transformer outputting raw hidden-states without any specific head on top.cbeZdZdeffd ZdZdZ ddejde ejde ejfdZ e e eeeed e  dd e ej&de ej&de ejd e ed e ede edeeeffdZxZS) Wav2Vec2Modelrct||||_t||_t ||_|jdkDs|jdkDrEtjtj|jj|_|j rt#||_nt'||_|j(r t+|nd|_|j/y)Nr))rrrrfeature_extractorrfeature_projectionmask_time_probmask_feature_probrrr3r,rrmasked_spec_embeddo_stable_layer_normrencoderrurradapter post_initrs r8rzWav2Vec2Model.__init__s   !7!?";F"C  3 &&*B*BS*H%'\\%,,v?Q?Q2R2[2[2]%^D "  & &9&ADL*62DL282D2Dv.$  r7cXtjdt|jyz Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.Nrrrfreeze_feature_encoderrs r8freeze_feature_extractorz&Wav2Vec2Model.freeze_feature_extractor'   Q  ##%r7c8|jjy Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)rRrr`s r8r_z$Wav2Vec2Model.freeze_feature_encoders 113r7r+ror<ct|jdds|S|j\}}}|)|jj |j ||<n|jj dkDr|jrt||f|jj |jj||jj}tj||jtj}|jj |j ||<|jjdkDr|jrt||f|jj|jj |jj"}tj||jtj}|dddfj%d|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://arxiv.org/abs/1904.08779). apply_spec_augmentTNr)r:r;r<r=r.)r:r;r=rI)rprrqrVr>rKrTrrlmask_time_lengthmask_time_min_masksr3rr rWrUmask_feature_lengthmask_feature_min_masksr)rr+ror<rcrGrmask_feature_indicess r8_mask_hidden_statesz!Wav2Vec2Model._mask_hidden_statesst{{$8$? 4A3E3E3G0 O[  (/3/E/E/H/HI\I\/]M+ , [[ ' '! + 5_-++44 KK88-++99 ! !& ->}G[G[chcmcm n /3/E/E/H/HI\I\/]M+ , ;; ( (1 ,#8[)++77 KK;;++<< $ $)<<0D]MaMainisis#t #74#@#G#GO]_#` 23M. /r7audio checkpoint output_typerHmodalityexpected_outputrrr~rr>cH||n|jj}||n|jj}||n|jj}|j |}|j dd}|!|j |jd|d}|j|\}}|j|||}|j|||||} | d}|j|j|}|s ||f| ddzSt||| j| jS) Nr r$Fr )ror<r<rr~rr)rextract_featuresr+r,)rrr~use_return_dictrRrrr9rSrmrXrYrr+r,) rrr<rorr~rrvr+encoder_outputss r8rzWav2Vec2Model.forwardsb"2C1N-TXT_T_TqTq$8$D $++JjJj &1%uEN+/*A*ABR*S' '00 ->~1 ,, )/!5# ' (* << # LL7M!#34qr7JJ J&+-)77&11   r7rNNNNN)r/r0r1r!rrar_r3r4rrNrmrWAV2VEC2_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPEr,rWrrrrrs@r8rPrPs ~( &4:>59 ,((,$E$5$56,!!1!12 ,\++DE&+$. 269=,0/3&*2 u||,2 !.2 $E$5$56 2 $D> 2 'tn 2 d^2  u-- .2 F2 r7rPz5Wav2Vec2 Model with a quantizer and `VQ` head on top.ceZdZdeffd ZdefdZdZdZe dde jde jd e jdefd Z e eeee  dd ee j&d ee j&dee j(dee j(deedeedeedeeeffdZxZS)rrct||t||_t j |j |_t||_ t j|j|j|_ t j|j|j|_|j!yr)rrrPrrrfeat_quantizer_dropoutdropout_featuresr quantizerrrproj_codevector_dimrrrrZrs r8rzWav2Vec2ForPreTraining.__init__7s  %f- " 6+H+H I6v>99V%7%79S9ST6#8#8&:T:TU r7rc&||j_y)zb Set the Gumbel softmax temperature to a given value. Only necessary for training N)rr)rrs r8set_gumbel_temperaturez-Wav2Vec2ForPreTraining.set_gumbel_temperatureDs&1"r7cXtjdt|jyr\r^r`s r8raz/Wav2Vec2ForPreTraining.freeze_feature_extractorJrbr7cL|jjjyrdrrRrr`s r8r_z-Wav2Vec2ForPreTraining.freeze_feature_encoderV ''::c ||n|jj}||jtj}|j ||||||}|j |d} |j|d} |!|j| jd|d}|j| |\} } | j|jjj} |j| } dx} x}}|| j\}}}| jd||jjd}|j||d|j!d ddd }|j#| dddf|| |jj$}| |k(j'd}|j)rt+d |dd|<|j-dd j/d|j1d}d|jz d zj-ddj3}t4j6j9|j+|d }|jj:|jj<z}|| z |z |j?z}||jj@|zz} |s| | | | | f|d dzS| | | f|d dzStC| | | | |jD|jF||S)a mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. sampled_negative_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length, num_negatives)`, *optional*): Indices indicating which quantized target vectors are used as negative sampled vectors in contrastive loss. Required input for pre-training. Returns: Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, Wav2Vec2ForPreTraining >>> from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices, _sample_negative_indices >>> from datasets import load_dataset >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base") >>> model = Wav2Vec2ForPreTraining.from_pretrained("facebook/wav2vec2-base") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values # Batch size 1 >>> # compute masked indices >>> batch_size, raw_sequence_length = input_values.shape >>> sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item() >>> mask_time_indices = _compute_mask_indices( ... shape=(batch_size, sequence_length), mask_prob=0.2, mask_length=2 ... ) >>> sampled_negative_indices = _sample_negative_indices( ... features_shape=(batch_size, sequence_length), ... num_negatives=model.config.num_negatives, ... mask_time_indices=mask_time_indices, ... ) >>> mask_time_indices = torch.tensor(data=mask_time_indices, device=input_values.device, dtype=torch.long) >>> sampled_negative_indices = torch.tensor( ... data=sampled_negative_indices, device=input_values.device, dtype=torch.long ... ) >>> with torch.no_grad(): ... outputs = model(input_values, mask_time_indices=mask_time_indices) >>> # compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states) >>> cosine_sim = torch.cosine_similarity(outputs.projected_states, outputs.projected_quantized_states, dim=-1) >>> # show that cosine similarity is much higher than random >>> cosine_sim[mask_time_indices.to(torch.bool)].mean() > 0.5 tensor(True) >>> # for contrastive loss training model should be put into train mode >>> model = model.train() >>> loss = model( ... input_values, mask_time_indices=mask_time_indices, sampled_negative_indices=sampled_negative_indices ... ).loss ```N)r<rr~rorrr Fr )rorIr$r z-infirS) reduction)r'r(r)r*r+r,r-r.)$rrwr>r3rWrrrrr9rrrrKr r permutercontrastive_logits_temperatureallanyr+rrarqrrr cross_entropyrrrSdiversity_loss_weightr&r+r,)rrr<rorurr~rrktransformer_featuresrvquantized_featuresr*r'r-r.rcrGrnegative_quantized_featuresr neg_is_postargetnum_codevectorss r8rzWav2Vec2ForPreTraining.forwardrsIJ&1%uEN59NN 0A5C5 110224>>3H3H3N3NO!^^,>?3777. # /7I7O7O 4J +=*A*A"k*R(--/44R8+ '+F*J*JOR+gaAq! ( 44"47++$ :: F-0KKPPQSTJ~~).vqr :&%%a+33B AGF,1133t;FFq!LTTVF!}}::6<<>6]b:c "kkCCdkkFgFggO.1FF/Y]n]r]r]ttN$dkk&G&G.&XXD24FH]^ahijikalll(*<>STW^_`_aWbb b+1'9"7!//))-)  r7)g?)NNNNNN)r/r0r1r!rrBrrar_rr3r4rrrzrr&r|rr, BoolTensorrWrrrrrs@r8rr5sL ~ 1#1 &=  ** ,,"-- (++DE+GVef268 ^ 'tn^ d^^  u22 3^ gF^ r7rz6Wav2Vec2 Model with a `language modeling` head on top.ceZdZfdZee d dejdeejdee dee dee deejde e effd ZxZS) Wav2Vec2ForMaskedLMc>t||tjdtt ||_tj|j|_ tj|j|j|_|jy)NzSThe class `Wav2Vec2ForMaskedLM` is deprecated. Please use `Wav2Vec2ForCTC` instead.)rrrrrrPrrr final_dropoutrrrr<rrZrs r8rzWav2Vec2ForMaskedLM.__init__sp   acp &f- zz&"6"67 yy!3!3V5F5FG  r7rr<rr~rlabelsr>c ||n|jj}|j||||}|d}|j|}|j |} |s | f|ddz} | St | |j |jS)N)rr~rrr$)rr+r,)rrwrrrrr+r,) rrr<rr~rrrkr+routputs r8rzWav2Vec2ForMaskedLM.forward%s&1% q 'tn q d^ q&q un$ %qFqr7rzfWav2Vec2 Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC).a. target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`Wav2Vec2ForCTC`] with adapters. Uses 'eng' by default. ceZdZddeeffd ZdZdZdZdZ e e e e eeee ddeej&d eej&d eed eed eed eej&deeeffdZxZS)rrct||t||_t j |j |_||_|jtd|jdt|dr|jr |jn |j}t j ||j|_|j%y)NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `Wav2Vec2ForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.r)rrrPrrrrrrr<rMrrrrrrrrZ)rrrrrs r8rzWav2Vec2ForCTC.__init__Ns  %f- zz&"6"67 &    $00@AHH *1)GFL^L^F % %djdvdv yy!3V5F5FG  r7c|j}|&t|jddtd|d|-t|jddtj dy||j |dyy)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. NrozCannot pass `target_lang`: r z)By default `target_lang` is set to 'eng'.T)r?)rrprrMr<inforG)rrs r8 tie_weightszWav2Vec2ForCTC.tie_weightses&&  "wt{{c ||n|jj}|I|j|jjk\r"t d|jj|j |||||}|d}|j |}|j|} d} |b||n$tj|tj}|j|jdjtj} |dk\} | jd} |j| }tj j#| dtj$j'dd}tj(j*j-d 5tj j/||| | |jj0|jj2|jj4 } ddd|s| f|t6dz}| | f|zS|St9| | |j:|j< S#1swYExYw) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rurrJrI)rrKr F)enabled)blankr zero_infinityr'rr+r,)rrwrCr<rMrrrr3 ones_liker rrSr> masked_selectrr log_softmaxr9rbackendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONrr+r,)rrr<rr~rrrkr+rr're labels_masktarget_lengthsflattened_targets log_probsrs r8rzWav2Vec2ForCTC.forwards'0&1% D 'tn D d^ D &D  un$ %D FD r7rz Wav2Vec2 Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. ceZdZfdZdZdZdZeee e e e de e ddeej"deej"d eed eed eed eej"d eee ffdZxZS)!Wav2Vec2ForSequenceClassificationct||t|dr|jr t dt ||_|jdz}|jr0tjtj||z |_ tj|j|j |_tj|j |j$|_|j)y)Nrz_Sequence classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r )rrrrrMrPrryuse_weighted_layer_sumrrr3r\ layer_weightsrrclassifier_proj_size projector num_labels classifierrZrr num_layersrs r8rz*Wav2Vec2ForSequenceClassification.__init__s  6= )f.@.@q &f- --1  ( (!#ejj.Dz.Q!RD 6#5#5v7R7RS))F$?$?ARARS r7cXtjdt|jyr\r^r`s r8raz:Wav2Vec2ForSequenceClassification.freeze_feature_extractorrbr7cL|jjjyrdrr`s r8r_z8Wav2Vec2ForSequenceClassification.freeze_feature_encoder rr7cP|jjD] }d|_ yrrrs r8rz3Wav2Vec2ForSequenceClassification.freeze_base_model rr7rn)rprqrHrrrsrrr<rr~rrr>c<||n|jj}|jjrdn|}|j|||||}|jjrr|t}t j |d}tjj|jd} || jdddzjd}n|d}|j|}||jd} n|j|j d|} | j#dj%dd|j d} d || <|jd| jdjddz } |j'| } d}|Ft)}|| jd|jj*|jd}|s| f|tdz}||f|zS|St-|| |j.|j0 S)  labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrur rrIrr$r)r)rrwrrrr3stackrrrrr rSrrrr9rrrrrrr+r,)rrr<rr~rrrkr+ norm_weights pooled_output padding_maskexpand_padding_maskrr'loss_fctrs r8rz)Wav2Vec2ForSequenceClassification.forward s2&1%->r1a-HHMMRSMTM#AJM}5  !)..1.5MBB=CVCVWXCY[ijL"."8"8"<"C"CAq-J]J]^_J`"a 25M.. /)--!-4|7G7GA7G7N7S7STVXY7ZZM/  ')HFKKDKK,B,BCV[[QS_UDY)F)G!HHF)-)9TGf$ Ev E'!//))   r7ry)r/r0r1rrar_rrrzr_SEQ_CLASS_CHECKPOINTrr|_SEQ_CLASS_EXPECTED_OUTPUT_SEQ_CLASS_EXPECTED_LOSSrr3r,rWrrrrrs@r8rrs" &=(++DE(,$2. 26,0/3&*)-< u||,< !.< $D> < 'tn < d^ < &<  u.. /< F< r7rzd Wav2Vec2 Model with a frame classification head on top for tasks like Speaker Diarization. ceZdZfdZdZdZdZeee e e e de  ddeej deej d eej d eed eed eed eee ffdZxZS)#Wav2Vec2ForAudioFrameClassificationct||t|dr|jr t dt ||_|jdz}|jr0tjtj||z |_ tj|j|j |_|j |_|j%y)NrzbAudio frame classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r )rrrrrMrPrryrrrr3r\rrrrr init_weightsrs r8rz,Wav2Vec2ForAudioFrameClassification.__init__g s  6= )f.@.@t &f- --1  ( (!#ejj.Dz.Q!RD ))F$6$68I8IJ ++ r7cXtjdt|jyrr^r`s r8razc ||n|jj}|jjrdn|}|j|||||}|jjrr|t}t j |d}tjj|jd} || jdddzjd}n|d}|j|} d} |\t} | | jd|jt j |jd|jd} |s| f|tdz} | St#| | |j$|j& S) rNTrur rrIr)axisr)rrwrrrr3rrrrrr rSrrrrrr+r,)rrr<rrr~rrkr+rrr'rrs r8rz+Wav2Vec2ForAudioFrameClassification.forward sh0&1%->r1a-HHMMRSMTM#AJM/  ')HFKKDOOtuDY)F)G!HHFM$!//))   r7ry)r/r0r1rrar_rrrzr_FRAME_CLASS_CHECKPOINTrr|_FRAME_EXPECTED_OUTPUTrr3r,rWrrrrrs@r8rr` s &=(++DE*)$. 26)-,0/3&*3 u||,3 !.3 & 3 $D> 3 'tn 3 d^3  u++ ,3 F3 r7rc&eZdZdfd ZdZxZS) AMSoftmaxLossctt| ||_||_||_t jtj||d|_ t j|_ y)NT)r) rrrscalemarginrrrr3randnrrr')rrrrrrs r8rzAMSoftmaxLoss.__init__ sS mT+-  $ll5;;y*#EUYZ '') r7c|j}tjj|jd}tjj|d}t j ||}||jz }tjj||j}|jt j|j||z}|j||}|S)Nrrr )rrr normalizerr3mmrone_hotrrrrWr') rr+rr cos_thetapsionehotrr's r8rzAMSoftmaxLoss.forward s!((!(< // 1/E HH]F3 $++%&&vt?ekk&++-iHHyy( r7)g>@g?rrs@r8rr s * r7rcXeZdZdfd Zdej dej fdZxZS) TDNNLayerct||dkDr|j|dz n|j||_|j||_|j ||_|j||_tj|j|j z|j|_ tj|_ y)Nrr )rrtdnn_dimrr tdnn_kernelr tdnn_dilationdilationrrkernelrrrs r8rzTDNNLayer.__init__ s c&trddlm}tr+t|jrt j d|jdd}|jjj|j|j|jjdd}tjj|||jj |j"}|jdd}|j%|}|S)Nr) LoraLayerzDetected LoRA on TDNNLayer. LoRA weights won't be applied due to optimization. You should exclude TDNNLayer from LoRA's target modules.r r$)r)rpeft.tuners.lorarrSrrrrrr rrrrrconv1drrr)rr+rrs r8rzTDNNLayer.forward s   2  $++y1 O &//15 ##(():):D+BFD]D]!^))F$=$=v?X?XY&v'@'@&BSBST Rs>FcXtjdt|jyrr^r`s r8raz+Wav2Vec2ForXVector.freeze_feature_extractor! rbr7cL|jjjyrdrr`s r8r_z)Wav2Vec2ForXVector.freeze_feature_encoder- rr7cP|jjD] }d|_ yrrrs r8rz$Wav2Vec2ForXVector.freeze_base_model4 rr7recVd}|jjD] }|||d}|S)z? Computes the output length of the TDNN layers c||z |zdzS)Nr r6rs r8rzEWav2Vec2ForXVector._get_tdnn_output_lengths.._conv_out_lengthA s!;.69A= =r7r )rr)rrerrs r8_get_tdnn_output_lengthsz+Wav2Vec2ForXVector._get_tdnn_output_lengths< s:  >  ;;22 LK,]KKM Lr7rnrorr<rr~rrr>c||n|jj}|jjrdn|}|j|||||}|jjrr|t}t j |d}tjj|jd} || jdddzjd}n|d}|j|}|jD] } | |} |%|jd} |j!d} n|j#|jd} |j%| }g} g} t'|D]U\}}| j)||d|fjd| j)||d|fj!dWt j | } t j | } t j*| | gd}|j-|}|j/|}d}||j1||}|s||f|tdz}||f|zS|St3||||j4|j6S) rNTrur rrIr)r'r embeddingsr+r,)rrwrrrr3rrrrrr rSrr rrrr enumerater^rrRrr rr+r,)rrr<rr~rrrkr+r tdnn_layer mean_features std_featuresfeat_extract_output_lengthstdnn_output_lengthsrlengthstatistic_poolingoutput_embeddingsrr'rs r8rzWav2Vec2ForXVector.forwardK s0&1%->r1a-HHMMRSMTM#AJM}5 )) 6J&}5M 6  !)..1.5M(,,,3L*.*O*OP^PbPbghPbPi*j '"&"?"?@["\ ML&':; J 6$$]1gvg:%>%C%C%C%JK##M!WfW*$=$A$Aa$A$HI J"KK 6M ;;|4L!II}l&CL 223DE!23  >>&&1D/07;X;Y3ZZF)-)9TGf$ Ev E(!//))   r7ry)r/r0r1rrar_rrr3rNrBrrrzr_XVECTOR_CHECKPOINTrr|_XVECTOR_EXPECTED_OUTPUTrr,rWrrrrs@r8r r  s& &=( eE I 'tn I d^ I &I  um# $I FI r7r )rrrrrr rPrrr)or2rr dataclassesrtypingrrrnumpyrNr3torch.utils.checkpointrtorch.nnr activationsr integrations.deepspeedr integrations.fsdpr modeling_flash_attention_utilsr rmodeling_outputsrrrrrrrmodeling_utilsrrrrrrrrrrrconfiguration_wav2vec2r!r8r3safetensors.torchr"r5r# get_loggerr/r<rr|r{r}rrrrrrrr r!r&rBr+rNndarrayrlr{Moduler}rrrrrrrrr.rDr_rMr\rmrurrrrrqrWAV2VEC2_START_DOCSTRINGrzrPrrrrrrrr __all__r6r7r8r4s !)) %!@7h.   3,5=J   H %!"#4&u9*<Q8 &7;&7 &7Z26 t c?tttU--. t  t ZZ tpZ^!$!$*-!$BJ2::BV!$H299*60*bii*Z299+RYY+\ 5  1 1 [B [B~{9/{9|h1-h1X !0"))0 299 F*"))*ZR biiR jV RYYV rI'BIII'Xbii>299$ryy2x'ox'v &#LhP +P P fQSkl\ 4\ m\ ~RTlm*q1*qn*qZp T ,T  T n r (?r r j  g *Ag  g TBII. @  O 0O  O d r7