T^(h= d Z ddlmZmZmZ ddlZddlmZ ddl m
Z
m Z m Z ddl
mZ ddlmZ dd lmZmZ e rddlZ ej* e Z G d
d e Zd gZy) z%
Feature extractor class for Whisper
)ListOptionalUnionN )is_torch_available)mel_filter_bank spectrogramwindow_function)SequenceFeatureExtractor) BatchFeature)
TensorTypeloggingc e Zd ZdZdgZ d fd Zdej dedej fdZ
ddej dedej fd Z e dd
e
ej d e
ej d ede
ej fd
Z d deej e
e e
ej e
e
e f dedee deeeef dee dee dee dee dee dee dee defdZ xZS ) WhisperFeatureExtractora
Constructs a Whisper feature extractor.
This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains
most of the main methods. Users should refer to this superclass for more information regarding those methods.
This class extracts mel-filter bank features from raw speech using a custom numpy implementation of the `Short Time
Fourier Transform` which should match pytorch's `torch.stft` equivalent.
Args:
feature_size (`int`, *optional*, defaults to 80):
The feature dimension of the extracted features.
sampling_rate (`int`, *optional*, defaults to 16000):
The sampling rate at which the audio files should be digitalized expressed in hertz (Hz).
hop_length (`int`, *optional*, defaults to 160):
Length of the overlaping windows for the STFT used to obtain the Mel Frequency coefficients.
chunk_length (`int`, *optional*, defaults to 30):
The maximum number of chuncks of `sampling_rate` samples used to trim and pad longer or shorter audio
sequences.
n_fft (`int`, *optional*, defaults to 400):
Size of the Fourier transform.
padding_value (`float`, *optional*, defaults to 0.0):
Padding value used to pad the audio. Should correspond to silences.
dither (`float`, *optional*, defaults to 0.0):
Adds dithering. In other words, adds a small Gaussian noise to each frame.
E.g. use 0.0001 to add dithering with a normal distribution centered
around 0.0 with standard deviation 0.0001 (assuming [-1,+1] range of raw_speech).
The value 0.0 means no dithering.
Dithering has similar effect as `spectrogram(mel_floor=...)`. It reduces
the high log_mel_fbank values for signals with hard-zero sections,
when VAD cutoff is present in the signal.
input_featuresc t
| d||||d| || _ || _ || _ ||z | _ | j
|z | _ || _ || _ t d|dz z |dd|dd | _
y ) N) feature_size
sampling_rate
padding_valuereturn_attention_mask g @@slaney)num_frequency_binsnum_mel_filters
min_frequency
max_frequencyr norm mel_scale ) super__init__n_fft
hop_length chunk_length n_samples
nb_max_framesr ditherr mel_filters) selfr r r% r& r$ r r) r kwargs __class__s t/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/whisper/feature_extraction_whisper.pyr# z WhisperFeatureExtractor.__init__H s
%''"7
$ (%
5 !^^z 9 * * 5A:~ ( '
waveform_batchdevicereturnc |dk7 rt d| d g }|D ] }t |t | j d | j | j d| j
| j d }|dddd f }t j ||j d
z
}|d z d z }|j | t j | }|S ) z
Compute the log-mel spectrogram of the provided audio, gives similar results to Whisper's original torch
implementation with 1e-5 tolerance.
cpuz Got device `z` for feature extraction, but feature extraction on CUDA accelerator devices requires torch, which is not installed. Either set `device='cpu'`, or install torch according to the official instructions: https://pytorch.org/get-started/locally/hanng @log10) frame_lengthr% powerr) r* log_melN @ @)
ValueErrorr r
r$ r% r) r* npmaximummaxappendarray)r+ r0 r1 log_spec_batchwaveformlog_specs r. _np_extract_fbank_featuresz2WhisperFeatureExtractor._np_extract_fbank_featuresl s
U? vh 'q q
& ,H"
F3 !ZZ ??{{ , , H 3B3'Hzz(HLLNS,@AH 3#-H ! !( + , .1 r/ rD c 8 t j | j |t j }t j | j
| }| j dk7 rF|| j t j |j |j |j z z
}t j || j
| j |d }|dddf j d z }t j | j j |t j }|j |z }t j |d
j# }|j% d k( rD|j' d d d
j' dd d
} t j( || dz
}n't j( ||j' dz
}|dz dz }|dk7 r |j+ j- }|j/ S )z
Compute the log-mel spectrogram of the audio using PyTorch's GPU-accelerated STFT implementation with batching,
yielding results similar to cpu computing with 1e-5 tolerance.
)r1 r )dtyper1 T)windowreturn_complex.Nr: r g|=)min)dimkeepdimr r r; r< r4 )torch
from_numpytofloat32 hann_windowr$ r) randnshaperH r1 stftr% absr* Tclampr6 rL r@ r? detachr4 numpy)
r+ rD r1 rI rU
magnitudesr* mel_specrE max_vals
r. _torch_extract_fbank_featuresz5WhisperFeatureExtractor._torch_extract_fbank_features s
##H-00 G""4::f=
;;# ekk(.._g_n_n&oo oHzz(DJJ_cd#ss(^'')Q.
&&t'7'78;;FEMMR ==:-;;xU399; <<>Q llq$l7:>>1d>STUVG }}Xw}=H }}Xx||~/CDH sNc) U? (,,.H~~ r/ input_valuesattention_maskr c |t j |t j }g }t | |j d D ]m \ }}||d| j z
t j |d| j dz z }||j d k r|||d |j | o |S | D cg c]< }||j z
t j |j dz z > }}|S c c}w )z[
Every array in the list is normalized to have zero mean and unit variance
Nr: gHz>r )
r> rB int32zipsummeansqrtvarrT rA )r_ r` r normed_input_valuesvectorlength normed_slicexs r. zero_mean_unit_var_normz/WhisperFeatureExtractor.zero_mean_unit_var_norm s % XXnbhh ?N"$ "%lN4F4Fr4J"K
9 & )=)=)? ?2776RYSY?K^K^K`cgKgCh h L . .q 11,9L)#**<8
9 #" Vb"bPQALBGGAEEGdN4K#K"b "b"" #cs :AC?
max_length
raw_speech
truncationpad_to_multiple_ofreturn_tensorsr paddingr do_normalizereturn_token_timestampsc h |O|| j k7 rmt d| j j d| j d| j d| d t j d| j j d t
|t j xr t |j d kD }
|
r&t |j d
kD rt d | |
xs@ t
|t t f xr( t
|d t j t t f }|r>|D cg c]2 }t j |gt j
j 4 }}n|s@t
|t j s&t j |t j
}nht
|t j rN|j t j t j" u r |j% t j }|s!t j |g j g}t' d|i }| j) |||r|n | j* |||xs | }| rD| j- |d |d | j. |d< t j0 |d d |d< |j3 d j5 d
d d }t7 r | j8 n | j: } ||d |
}t
|d t< r7|D cg c]' }t j |t j
) c}|d< n||d< |r |d dddd| j> f |d< | )|D cg c] }t | | j> z c}|d< ||jA | }|S c c}w c c}w c c}w )a
Main method to featurize and prepare for the model one or several sequence(s). Implementation uses PyTorch for
the STFT computation if available, otherwise a slower NumPy based one.
Args:
raw_speech (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`):
The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float
values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not
stereo, i.e. single float per timestep.
truncation (`bool`, *optional*, default to `True`):
Activates truncation to cut input sequences longer than *max_length* to *max_length*.
pad_to_multiple_of (`int`, *optional*, defaults to None):
If set will pad the sequence to a multiple of the provided value.
This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
`>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128.
return_attention_mask (`bool`, *optional*):
Whether to return the attention mask. If left to the default, will return the attention mask according
to the specific feature_extractor's default.
[What are attention masks?](../glossary#attention-mask)
For Whisper models, `attention_mask` should always be passed for batched inference, to avoid subtle
bugs.
return_tensors (`str` or [`~utils.TensorType`], *optional*):
If set, will return tensors instead of list of python integers. Acceptable values are:
- `'tf'`: Return TensorFlow `tf.constant` objects.
- `'pt'`: Return PyTorch `torch.Tensor` objects.
- `'np'`: Return Numpy `np.ndarray` objects.
sampling_rate (`int`, *optional*):
The sampling rate at which the `raw_speech` input was sampled. It is strongly recommended to pass
`sampling_rate` at the forward call to prevent silent errors and allow automatic speech recognition
pipeline.
padding_value (`float`, *optional*, defaults to 0.0):
The value that is used to fill the padding values / vectors.
do_normalize (`bool`, *optional*, defaults to `False`):
Whether or not to zero-mean unit-variance normalize the input. Normalizing can help to significantly
improve the performance of the model.
device (`str`, *optional*, defaults to `'cpu'`):
Specifies the device for computation of the log-mel spectrogram of audio signals in the
`_torch_extract_fbank_features` method. (e.g., "cpu", "cuda")
return_token_timestamps (`bool`, *optional*, defaults to `None`):
Whether or not to return the number of frames of the input raw_speech.
These num_frames can be used by the model to compute word level timestamps.
Nz3The model corresponding to this feature extractor: z& was trained using a sampling rate of zI. Please make sure that the provided `raw_speech` input was sampled with z and not .zDIt is strongly recommended to pass the `sampling_rate` argument to `zN()`. Failing to do so can result in silent errors that might be hard to debug.r r z2Only mono-channel audio is supported for input to r )rH r )rs rn rp rq r r` )r` r )axis
num_frames)!r r= r- __name__loggerwarning
isinstancer> ndarraylenrT listtupleasarrayrQ rW rH float64astyper padr' rm r stackget transposer r^ rF r r% convert_to_tensors)r+ ro rp rq rr r rs rn r rt r1 ru r, is_batched_numpy
is_batchedspeechbatched_speech
padded_inputsr extract_fbank_featuresfeature raw_speech_is r. __call__z WhisperFeatureExtractor.__call__ s D $ 2 22 I$..JaJaIb c))-););(< =))-););( " 0 0 /K /M* +
/1hh}EU7V]^._M* + '**+;<FFq!QO 3E2FD . .DLkLk 0q0A6J nQ' .dr.sY`rzz'/T.sM* + /=M* + .; i r r F)r4 )r )
TNNNrn NNNr4 N)rz
__module__ __qualname____doc__model_input_namesr# r> rB strr~ rF r^ staticmethodr floatrm r boolr intr
r r
__classcell__)r- s @r. r r $ s B ** #"
H 3 SUS]S] < bhh XZXbXb > be#2:: r audio_utilsr r r
!feature_extraction_sequence_utilsr feature_extraction_utilsr utilsr
r rN
get_loggerrz r{ r __all__r! r/ r. r s^ ) ( " H H I 4 ( H %k 6 k \ %
%r/