T^(h@2ddlmZmZmZddlZddlmZddlmZddl m Z ddl m Z m Z mZddlmZddlmZmZmZmZd d lmZmZmZmZmZmZmZd d lmZd Z d Z!dZ"gdZ#dZ$dZ%dZ&dZ'dZ(GddejRZ*GddeZ+GddeZ,GddejRZ-GddeZ.Gdd eZ/Gd!d"eZ0d#Z1d$Z2ed%e1Gd&d'ee0Z3ed(e1Gd)d*eZ4ed+e1Gd,d-eZ5gd.Z6y)/)OptionalTupleUnionN)ACT2FN)is_deepspeed_zero3_enabled)BaseModelOutputCausalLMOutputSequenceClassifierOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardreplace_return_docstrings)Wav2Vec2EncoderWav2Vec2EncoderStableLayerNormWav2Vec2FeatureEncoderWav2Vec2ForCTC!Wav2Vec2ForSequenceClassification Wav2Vec2ModelWav2Vec2SamePadLayer) HubertConfigrzfacebook/hubert-large-ls960-ft)ri$iz['MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL'gGz6@zsuperb/hubert-base-superb-ksz '_unknown_'g(\!@c$eZdZfdZdZxZS)HubertPositionalConvEmbeddingct|tj|j|j|j |j dz|j |_d|_|jr&tj|j|_ntjj}ttjjdr$tjjj}tr(ddl}|j"j%|jj&d5||jdd|_dddt|jdrU|jjj&j(}|jjj&j*}n,|jj,}|jj.}|j"j1|||j"j1||n||jdd|_t3|j |_t6|j8|_y#1swY'xYw) Nr) kernel_sizepaddinggroups weight_normr modifier_rankweight)namedimparametrizations)super__init__nnConv1d hidden_sizenum_conv_pos_embeddingsnum_conv_pos_embedding_groupsconv batch_normconv_pos_batch_norm BatchNorm1dutilsr!hasattrr'r deepspeedzeroGatheredParametersr$ original0 original1weight_gweight_vregister_external_parameterHubertSamePadLayerrrfeat_extract_activation activation)selfconfigr!r5r:r; __class__s g/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/hubert/modular_hubert.pyr)z&HubertPositionalConvEmbedding.__init__0s II      6622a777     % % nnV-?-?@DO((..Krxx00-@ hh77CC )+ ^^66tyy7G7GWX6YM +DIIH! LDIM499&89#yy99@@JJH#yy99@@JJH#yy11H#yy11H::4J::4J' aH )&*H*HI  !?!?@MMs ?I??J c|jdd}|j|j|}|j|}|j|}|j |}|jdd}|S)Nrr) transposer0r/rr?r@ hidden_statess rCforwardz%HubertPositionalConvEmbedding.forwardUsn%//15 ?? & OOM:M -0  ]3  6 %//15 __name__ __module__ __qualname__r)rH __classcell__rBs@rCrr/s#AJ rIrc eZdZy)r=NrKrLrMrIrCr=r=arIr=c eZdZy)HubertFeatureEncoderNrQrRrIrCrUrUerSrIrUc$eZdZfdZdZxZS)HubertFeatureProjectioncnt||j|_|jr3tj|j d|j |_tj|j d|j|_ tj|j|_ y)N)eps)r(r)feat_proj_layer_normr* LayerNormconv_dimlayer_norm_eps layer_normLinearr, projectionDropoutfeat_proj_dropoutdropoutr@rArBs rCr)z HubertFeatureProjection.__init__js} $*$?$?!  $ $ ll6??2+>FDYDYZDO))FOOB$79K9KLzz&":":; rIc|jr|j|}|j|}|j|}|S)N)r[r_rardrFs rCrHzHubertFeatureProjection.forwardrs;  $ $ OOM:M 6  ]3 rIrJrOs@rCrWrWis <rIrWc eZdZy) HubertEncoderNrQrRrIrCrhrh{rSrIrhc eZdZy)HubertEncoderStableLayerNormNrQrRrIrCrjrjrSrIrjc|eZdZdZeZdZdZdZdZ dZ dZ de e jeffdZded e jfd Zy ) HubertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. hubert input_valuesTczt|tjrm|jjj d|j j|j%|jjjyyt|tjtjtjfrJ|jjj|jjjdyt|tjr_trddl}t#|dr|t#|drp|j$j'|j(|j*gd5tj,j/|jjdddn|j$j'|jd5tj,j/|jjdddn3tj,j/|jj|j%|jjjyyt|t0r2t#|d r%|j2jj5yyt|t6rMt#|d r@|j8jjd|j j:d zz yyy#1swYxYw#1swYxYw) zInitialize the weights)meanstdNg?rr;r:r"masked_spec_embed layer_weightsr) isinstancer*r`r$datanormal_rAinitializer_rangebiaszero_r\ GroupNormr2fill_r+rr5r4r6r7r;r:initkaiming_normal_ HubertModelrsuniform_HubertForSequenceClassificationrtnum_hidden_layers)r@moduler5s rC _init_weightsz#HubertPreTrainedModel._init_weightssP fbii ( MM   & &CT[[5R5R & S{{&   &&(' r||R^^ L M KK   " " $ MM   $ $S )  *)+ 6:.76:3N"::FOOV__;]mn:oD// 0B0BCDD#::6==XY:ZD// 0B0BCDD'' (:(:;{{&   &&('  ,v23((--6684  ? @v/$$))//t{{7T7TWX7X0YZ0ADDDDs?4L%#4L1%L.1L: input_lengthscd}t|jj|jjD]\}}||||}|S)zH Computes the output length of the convolutional layers c>tj||z |ddzS)Nfloor) rounding_moder)torchdiv) input_lengthrstrides rC_conv_out_lengthzPHubertPreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengths"99\K7wWZ[[ [rI)ziprA conv_kernel conv_stride)r@rrrrs rC _get_feat_extract_output_lengthsz6HubertPreTrainedModel._get_feat_extract_output_lengthssQ  \ $'t{{'>'> @W@W#X Q K,]KPM QrIfeature_vector_lengthattention_maskc|j|jdjtj}|j d}tj ||f|j|j}d|tj|j d|j|dz f<|jdgjdjdgj}|S)NrYr)dtypedevicer)r) rsumtorlongshapezerosrrarangeflipcumsumbool)r@rroutput_lengths batch_sizes rC"_get_feature_vector_attention_maskz8HubertPreTrainedModel._get_feature_vector_attention_masks>>~?Q?QRT?UVYYZ_ZdZde#))!,  . /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOrIN)rKrLrM__doc__r config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_supports_flash_attn_2_supports_sdparrr LongTensorintrrrRrIrCrlrlsh  L $O&*#!N[B eEDTDTVYDY>Z   ]b]m]m rIrla! Hubert was proposed in [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://arxiv.org/abs/2106.07447) by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, Abdelrahman Mohamed. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving etc.). This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`HubertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`Wav2Vec2Processor.__call__`] for details. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) `attention_mask` should only be passed if the corresponding processor has `config.return_attention_mask == True`. For all models whose processor has `config.return_attention_mask == False`, such as [hubert-base](https://huggingface.co/facebook/hubert-base-ls960), `attention_mask` should **not** be passed to avoid degraded performance when doing batched inference. For such models `input_values` should simply be padded with 0 and passed without `attention_mask`. Be aware that these models also yield slightly different results depending on whether `input_values` is padded or not. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z`The bare Hubert Model transformer outputting raw hidden-states without any specific head on top.ceZdZdeffd ZdZdZeee e e  dde e jde e jde e jd e ed e ed e ed eee ffd ZxZS)rrAct||||_t||_t ||_|jdkDs|jdkDrEtjtj|jj|_|j rt#||_nt'||_|j)|`y)Nrp)r(r)rArUfeature_extractorrWfeature_projectionmask_time_probmask_feature_probr* ParameterrTensorr,rrsdo_stable_layer_normrjencoderrh post_initadapterres rCr)zHubertModel.__init__ s   !5f!="9&"A  3 &&*B*BS*H%'\\%,,v?Q?Q2R2[2[2]%^D "  & &7?DL(0DL  LrIctdNzNot needed for HubertAttributeErrorr@s rCfreeze_feature_extractorz$HubertModel.freeze_feature_extractor455rIctdrrrs rCfreeze_feature_encoderz"HubertModel.freeze_feature_encoder!rrI) output_typerrnrmask_time_indicesoutput_attentionsoutput_hidden_states return_dictreturnc||n|jj}||n|jj}||n|jj}|j |}|j dd}||j |jd|}|j|}|j||}|j|||||} | d}|s |f| ddzSt|| j| jS)aZ Returns: Example: ```python >>> from transformers import AutoProcessor, HubertModel >>> from datasets import load_dataset >>> import soundfile as sf >>> processor = AutoProcessor.from_pretrained("facebook/hubert-large-ls960-ft") >>> model = HubertModel.from_pretrained("facebook/hubert-large-ls960-ft") >>> def map_to_array(batch): ... speech, _ = sf.read(batch["file"]) ... batch["speech"] = speech ... return batch >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.map(map_to_array) >>> input_values = processor(ds["speech"][0], return_tensors="pt").input_values # Batch size 1 >>> hidden_states = model(input_values).last_hidden_state ```Nrr)r)rrrrr)last_hidden_staterG attentions)rArruse_return_dictrrErrr_mask_hidden_statesrr rGr) r@rnrrrrrextract_featuresrGencoder_outputss rCrHzHubertModel.forward$s,L2C1N-TXT_T_TqTq$8$D $++JjJj &1% E 'tn E d^E  uo% &E ZDE rIrzdHubert Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC).cVeZdZ eeeeeee e fdZ xZ S) HubertForCTC) checkpointrrexpected_output expected_lossc $t|di|yNrRr(rHr@ super_kwargsrBs rCrHzHubertForCTC.forwardus ','rI) rKrLrMrrr _CHECKPOINT_FOR_DOCr r_CTC_EXPECTED_OUTPUT_CTC_EXPECTED_LOSSrHrNrOs@rCrrns=  *+BC&"$,( (D(rIrz Hubert Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. c XeZdZ eeeeeede e fdZ xZ S)raudio)rrrmodalityrrc $t|di|yrrrs rCrHz'HubertForSequenceClassification.forwards ','rI) rKrLrMrrr _SEQ_CLASS_CHECKPOINTr r_SEQ_CLASS_EXPECTED_OUTPUT_SEQ_CLASS_EXPECTED_LOSSrHrNrOs@rCrrs@ *+BC(,$2. (D(rIr)rrrrl)7typingrrrrtorch.nnr* activationsrintegrations.deepspeedrmodeling_outputsr r r modeling_utilsr r3r rrrwav2vec2.modeling_wav2vec2rrrrrrrconfiguration_hubertr_HIDDEN_STATES_START_POSITIONrr_EXPECTED_OUTPUT_SHAPErrrrrModulerr=rUrWrhrjrlHUBERT_START_DOCSTRINGrrrr__all__rRrIrCrsx)) !@YY- /!"!7&u7*/BII/d -  1 bii$ O  #A GOGT&#Lfa -!6a a Hn (> (  (  (&G ( ( frI