T^(h d dl Z d dlmZ d dlmZmZmZmZ d dlZ d dl
Z
ddl m Z ddl
mZmZmZ ddlmZ dd lmZ erd d
lmZ dd lmZ dd lmZ ej6 e Z e r
d dl Z dd
l m Z d Z!ddZ"d Z#d Z$ G d de Z%d Z&y) N) defaultdict)
TYPE_CHECKINGDictOptionalUnion )PreTrainedTokenizer)is_torch_availableis_torchaudio_availablelogging ) ffmpeg_read)
ChunkPipeline)BeamSearchDecoderCTC)SequenceFeatureExtractor)PreTrainedModel)(MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMESc g }| D ]g \ }}}t t ||z }t t ||z |z }t t ||z |z }|||f}|j | i |S )z
Rescales the stride values from audio space to tokens/logits space.
(160_000, 16_000, 16_000) -> (2000, 200, 200) for instance.
)introundappend)strideratio new_stridesinput_nleftrighttoken_n
new_strides q/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/pipelines/automatic_speech_recognition.pyrescale_strider! ) s K & ' ueGeO,-5 '123E%'/G345 tU+
:& ' c # R K | j d }||z
|z
}t d|| D ]| }||z } | || }
||
|j d } || j | } |dk( rdn|} | |k\ }
|
rdn|}|
j d }|| |f}|
j d | kD r |
|d| |
s| y y w)Nr pt)
sampling_ratereturn_tensorsdtype)is_lastr )shaperanger% to)inputsfeature_extractor chunk_len stride_left stride_rightr(
inputs_lenstepchunk_start_idx
chunk_end_idxchunk processed _stride_leftr)
_stride_rightr s r
chunk_iterr: = s a J { "\ 1D J5 ')3
}5%e;L;Z;Zkop ! 5 1I +q 0qk :- $,
KKN \=9 ;;q>L ( %E9E E s B B'#B'c t | }t | }t |dz D cg c] }dg|dz z
}}d}t | D ]? }t | D ]/ }| | || k( s|| | dz } | ||dz |dz < | |kD s.| }1 A t j | }|dk7 r!t j ||k( d d |z
nd}
|dk7 r!t j ||k( d d |z
nd} |
| |fS c c}w )Nr
r )lenr+ nparrayargwhere)
sequence_leftsequence_right seq_len_left
seq_len_right_counterlongestijprevious_counter
index_left index_rights r "_fast_find_longest_common_sequencerM R s/ }%L'M27 q8H2IJQsma'(JGJG
<
/}% /A Q >!#44#*1:a=1#4 (8Aq1u%#g- .G // hhw GELPQ\W /04Q7'AWYJFMQRl"++g 01"5a87BXZK {G ++ Ks C2c
D | d d j D cg c] }||j vs| }}| dd D ] }|d j D cg c] }||j vs| }}d}d}t dt | dz D ]d }|dz } t j
t j || d t j |d | k( }
|
|z | z } |
dkD s[| |kD sa|}| }f |j ||d t j | S c c}w c c}w )Nr r
g g @)tolistall_special_idsr+ r= r> sumr? extend) sequences tokenizertok_idsequencenew_seq new_sequenceindexmax_rH epsmatchesmatchings r _find_longest_common_sequencer^ f s3 &/q\!_%;%;%=i6yOhOhAhiHi QR=
.-4QZ->->-@l6FR[RkRkDkl lq#l+a/0 Ag+CffRXXhrsm 4 bqAQ8R RSG {S(H{x$ UV,-
. 88H jls DDD $D c e Zd ZdZ ddddedef dee deedef d eed
f d eeed f f fd
Z dee
j e ef f fd Z
ddZddZddZ ddee fdZ xZS )"AutomaticSpeechRecognitionPipelinea
Pipeline that aims at extracting spoken text contained within some audio.
The input can be either a raw waveform or a audio file. In case of the audio file, ffmpeg should be installed for
to support multiple audio formats
Example:
```python
>>> from transformers import pipeline
>>> transcriber = pipeline(model="openai/whisper-base")
>>> transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac")
{'text': ' He hoped there would be stew for dinner, turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick, peppered flour-fatten sauce.'}
```
Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial)
Arguments:
model ([`PreTrainedModel`] or [`TFPreTrainedModel`]):
The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from
[`PreTrainedModel`] for PyTorch and [`TFPreTrainedModel`] for TensorFlow.
feature_extractor ([`SequenceFeatureExtractor`]):
The feature extractor that will be used by the pipeline to encode waveform for the model.
tokenizer ([`PreTrainedTokenizer`]):
The tokenizer that will be used by the pipeline to encode data for the model. This object inherits from
[`PreTrainedTokenizer`].
decoder (`pyctcdecode.BeamSearchDecoderCTC`, *optional*):
[PyCTCDecode's
BeamSearchDecoderCTC](https://github.com/kensho-technologies/pyctcdecode/blob/2fd33dc37c4111417e08d89ccd23d28e9b308d19/pyctcdecode/decoder.py#L180)
can be passed for language model boosted decoding. See [`Wav2Vec2ProcessorWithLM`] for more information.
chunk_length_s (`float`, *optional*, defaults to 0):
The input length for in each chunk. If `chunk_length_s = 0` then chunking is disabled (default).
For more information on how to effectively use `chunk_length_s`, please have a look at the [ASR chunking
blog post](https://huggingface.co/blog/asr-chunking).
stride_length_s (`float`, *optional*, defaults to `chunk_length_s / 6`):
The length of stride on the left and right of each chunk. Used only with `chunk_length_s > 0`. This enables
the model to *see* more context and infer letters better than without this context but the pipeline
discards the stride bits at the end to make the final reconstitution as perfect as possible.
For more information on how to effectively use `stride_length_s`, please have a look at the [ASR chunking
blog post](https://huggingface.co/blog/asr-chunking).
framework (`str`, *optional*):
The framework to use, either `"pt"` for PyTorch or `"tf"` for TensorFlow. The specified framework must be
installed. If no framework is specified, will default to the one currently installed. If no framework is
specified and both frameworks are installed, will default to the framework of the `model`, or to PyTorch if
no model is provided.
device (Union[`int`, `torch.device`], *optional*):
Device ordinal for CPU/GPU supports. Setting this to `None` will leverage CPU, a positive will run the
model on the associated CUDA device id.
torch_dtype (Union[`int`, `torch.dtype`], *optional*):
The data-type (dtype) of the computation. Setting this to `None` will use float32 precision. Set to
`torch.float16` or `torch.bfloat16` to use half-precision in the respective dtypes.
modelr r. r rT decoderr devicez torch.device torch_dtypez torch.dtypec T |j j dk( rd| _ nq|j j t j v rd| _ n?|j r,|j j d r||| _ d| _ nd| _ t | , |||f||d| y )Nwhisperseq2seq_whisperseq2seqWithLM ctc_with_lmctc)rc rd ) config
model_typetype __class____name__r values_processor_classendswithrb super__init__) selfra r. rT rb rc rd kwargsro s r ru z+AutomaticSpeechRecognitionPipeline.__init__ s << " "i /)DI
__
%
%)Q)X)X)Z
Z!DI . .!22;;HE#"DL%DI DI
+<oVYdohnor" r- c $ t | |fi |S )ae
Transcribe the audio sequence(s) given as inputs to text. See the [`AutomaticSpeechRecognitionPipeline`]
documentation for more information.
Args:
inputs (`np.ndarray` or `bytes` or `str` or `dict`):
The inputs is either :
- `str` that is either the filename of a local audio file, or a public URL address to download the
audio file. The file will be read at the correct sampling rate to get the waveform using
*ffmpeg*. This requires *ffmpeg* to be installed on the system.
- `bytes` it is supposed to be the content of an audio file and is interpreted by *ffmpeg* in the
same way.
- (`np.ndarray` of shape (n, ) of type `np.float32` or `np.float64`)
Raw audio at the correct sampling rate (no further check will be done)
- `dict` form can be used to pass raw audio sampled at arbitrary `sampling_rate` and let this
pipeline do the resampling. The dict must be in the format `{"sampling_rate": int, "raw":
np.array}` with optionally a `"stride": (left: int, right: int)` than can ask the pipeline to
treat the first `left` samples and last `right` samples to be ignored in decoding (but used at
inference to provide more context to the model). Only use `stride` with CTC models.
return_timestamps (*optional*, `str` or `bool`):
Only available for pure CTC models (Wav2Vec2, HuBERT, etc) and the Whisper model. Not available for
other sequence-to-sequence models.
For CTC models, timestamps can take one of two formats:
- `"char"`: the pipeline will return timestamps along the text for every character in the text. For
instance, if you get `[{"text": "h", "timestamp": (0.5, 0.6)}, {"text": "i", "timestamp": (0.7,
0.9)}]`, then it means the model predicts that the letter "h" was spoken after `0.5` and before
`0.6` seconds.
- `"word"`: the pipeline will return timestamps along the text for every word in the text. For
instance, if you get `[{"text": "hi ", "timestamp": (0.5, 0.9)}, {"text": "there", "timestamp":
(1.0, 1.5)}]`, then it means the model predicts that the word "hi" was spoken after `0.5` and
before `0.9` seconds.
For the Whisper model, timestamps can take one of two formats:
- `"word"`: same as above for word-level CTC timestamps. Word-level timestamps are predicted
through the *dynamic-time warping (DTW)* algorithm, an approximation to word-level timestamps
by inspecting the cross-attention weights.
- `True`: the pipeline will return timestamps along the text for *segments* of words in the text.
For instance, if you get `[{"text": " Hi there!", "timestamp": (0.5, 1.5)}]`, then it means the
model predicts that the segment "Hi there!" was spoken after `0.5` and before `1.5` seconds.
Note that a segment of text refers to a sequence of one or more words, rather than individual
words as with word-level timestamps.
generate_kwargs (`dict`, *optional*):
The dictionary of ad-hoc parametrization of `generate_config` to be used for the generation call. For a
complete overview of generate, check the [following
guide](https://huggingface.co/docs/transformers/en/main_classes/text_generation).
Return:
`Dict`: A dictionary with the following keys:
- **text** (`str`): The recognized text.
- **chunks** (*optional(, `List[Dict]`)
When using `return_timestamps`, the `chunks` will become a list containing all the various text
chunks identified by the model, *e.g.* `[{"text": "hi ", "timestamp": (0.5, 0.9)}, {"text":
"there", "timestamp": (1.0, 1.5)}]`. The original full text can roughly be recovered by doing
`"".join(chunk["text"] for chunk in output["chunks"])`.
)rt __call__)rv r- rw ro s r ry z+AutomaticSpeechRecognitionPipeline.__call__ s z w 1&11r" c i } |+| j dk( r|st j d || d< ||| d< t t }
| t j dt ||
d< |"|d|v r t d |
j | i } ||| d< || j dk( r
|r t d | j d
k( r|d k7 r t d | j d
k( r|dvr t d | j dk( r|dk( r t d ||
d< || d< | | j dk7 r t d || d< | j | j |
d< | j | j |
d< | j |
d< | |
| fS )Nrh a Using `chunk_length_s` is very experimental with seq2seq models. The results will not necessarily be entirely accurate and will have caveats. More information: https://github.com/huggingface/transformers/pull/20104. Ignore this warning with pipeline(..., ignore_warning=True)chunk_length_sstride_length_sz`max_new_tokens` is deprecated and will be removed in version 4.49 of Transformers. To remove this warning, pass `max_new_tokens` as a key inside `generate_kwargs` instead.max_new_tokenszp`max_new_tokens` is defined both as an argument and inside `generate_kwargs` argument, please use only 1 versiondecoder_kwargszEWe cannot return_timestamps yet on non-CTC models apart from Whisper!rj wordzRCTC with LM can only predict word level timestamps, set `return_timestamps='word'`rk )charr zCTC can either predict character level timestamps, or word level timestamps. Set `return_timestamps='char'` or `return_timestamps='word'` as required.rg r zWhisper cannot return `char` timestamps, only word level or segment level timestamps. Use `return_timestamps='word'` or `return_timestamps=True` respectively.return_timestampsz)Only Whisper can return language for now.return_languageassistant_modelrT assistant_tokenizer)
rn loggerwarningr dictwarningswarn
FutureWarning
ValueErrorupdater r rT ) rv r{ r| ignore_warningr~ r r generate_kwargsr} preprocess_paramsforward_paramspostprocess_paramss r _sanitize_parametersz7AutomaticSpeechRecognitionPipeline._sanitize_parameters s %yyI%n, 3A . / &3B / 0$T* % MM
0>N+ , & ).>/.Q &
! !/ 2 %3A / 0 (yyI%*; !hiiyyM).?6.I !uvvyyE!&7?O&O ` yy --2Cv2M _ 3DN. /6G 2 3 &yy -- !LMM4C 0 1 +040D0DN, - # # /*...N; '484L4LN0 1 .2DDDr" c # K t |t rg|j d s|j d r t j | j
}n%t
|d 5 }|j }d d d t |t r t || j j }d }i }t |t ry|j dd }d|v rd|v sd|v s t d |j dd }|$|j d d |j dd }|j d }|}|}|| j j k7 r~t rd
d lm} n t% d | j' t) j* | || j j j- }| j j |z }
nd
}
|h|d
|d
z |j. d
kD r t d |j. d
t1 t3 |d
|
z t1 t3 |d
|
z f}t |t4 j6 st9 dt; | d t= |j. d
k7 r t d |r ||dz }t |t0 t> f r||g}tA | jB jD dd
} t1 t3 || j j z | z | z } t1 t3 |d
| j j z | z | z }
t1 t3 |d
| j j z | z | z }| |
|z k r t d tG || j | |
|| jH D ]
}i || y | j: dk( rR|j. d
| j jJ kD r,| j || j j dddd }ny| j: dk( rA|?| j || j j ddd }|j d |d < n)| j || j j dd }| jH |jM | jH }| | j: d k( r t d ||d< d!di|| y # 1 sw Y nxY ww)"Nzhttp://zhttps://rbr r% rawr? zWhen passing a dictionary to AutomaticSpeechRecognitionPipeline, the dict needs to contain a "raw" key containing the numpy array representing the audio and a "sampling_rate" key, containing the sampling_rate associated with that arraypathr )
functionalztorchaudio is required to resample audio samples in AutomaticSpeechRecognitionPipeline. The torchaudio package can be installed through: `pip install torchaudio`.r
z Stride is too large for inputz)We expect a numpy ndarray as input, got ``zMWe expect a single channel audio input for AutomaticSpeechRecognitionPipeline inputs_to_logits_ratioz.Chunk length must be superior to stride lengthrg FrG r$ T)r%
truncationpaddingr& return_attention_mask)r% r& return_token_timestampsr
num_frames)r% r& r r' rh z8Stride is only usable with CTC models, try removing it !r) )'
isinstancestr
startswithrequestsgetcontentopenreadbytesr r. r% r popr r
torchaudior ImportErrorresampletorch
from_numpynumpyr* r r r> ndarray TypeErrorrn r= floatgetattrra rl r: rd n_samplesr, )rv r- r{ r| fr extra_inputsin_sampling_rateFr align_tor/ r0 r1 itemr7 s r
preprocessz-AutomaticSpeechRecognitionPipeline.preprocessg s fc " +v/@/@/L " f-55&$' &1 VVXF& fe $ )?)?)M)MNF fd #ZZ$/F $v-5F?gQWFW N jj-G
64( **Wd3 %zz/ : E F 4#9#9#G#GG*,:%e
$$V,.>@V@V@d@d%' ..<9=O$56 )%fe4( ( l"[77+=#>"[\($ ( &,K%8N^_C (yy --%$*CM
**L,<, rk rj rh rg )r r time_precisionr
)axisr r )r start_offset
end_offsetrk )skip_special_tokensT)r output_char_offsets char_offsetsr r r r )text timestampchunksr) r r )!rn frameworkr( r bfloat16float16r, float32r r r r^ rT r. chunk_lengthra rl max_source_positionsr% _decode_asrr> concatenatesqueezerb decode_beamsdecode_get_word_offsetsreplace_word_delimiter_charr r listr items)$rv
model_outputsr~ r r optional final_itemskeyr r r total_nr r right_nr r% outputr/ r0 r1 r beams chunk_offsetoffsetsr r r r r r startstopr kvs$ r postprocessz.AutomaticSpeechRecognitionPipeline.postprocess7 s\ ))}4h($ &G~~%'#,*<*< QVQ^Q^@_*_
6<<> **, [[40F!dii3I&I'-$u
"E/ ag o. u % & dii9,1+t~~NE
YY+
+ ! 3 3 @ @4::CTCTCiCi iN 2 2 @ @M'
Lv%;A(;K8I{L .I =0K M1L'0+|'KF8$
L "^^ 7 7 "3 / - 8 ND( NN;Q7EMM!$E 99
% %!#-DLL--eF~FE8A;D %Qx{ 8D k4D4<NND,^h#ijk
YY+
+"&))u"4 >>((DW(XD ..///BX\ 0 " %. "nn > >wHrHr sG 2P!PF
] ^,tzz/@/@/W/WW//===L)DJJ,=,=,T,TT..<<<
t,='>eUY] [\
] "(HX D!# #F JJx & JJx & JJy$ ' JJx & JJ)4 0
#1a"
#
# 22E22r" )NNNNN)NNNNNNNN)r N)F)NNN)rp
__module__ __qualname____doc__r r r r r ru r> r r ry r r r r r
__classcell__)ro s @r r` r` s AL EI37@D-1;?p p !!;S!@Ap /0 p
% 6 ;<= p c>)*
p eC $6 78p6=2bjj%,-=2B HETw:rU4p mq]3-5d^]3r" r` c |j d dz }g }|j |z }d}t | D ] \ }} | \ }
} t |
t rt j |
}
| \ } }
}|
j d }
||
v r t j |
|k( d d nd}|
|d }
|
|k\ }|dk7 rJt | dkD r;t j |dd |dd z d dz }t j | d d }||vrt j || n|}||
|z z }t ||j z |z }t |
|j z |z }t j |
| ||z k\ d }|j d dkD rn|d dkD r ||d dz
n|d }d}g }t t | D ] \ }}|dd }|d ||z |z
k r|dk7 r nt | dkD s0t! |
d| | \ }} } | dkD sI| |kD sO| }|} t j |
|dz d |k\ d d dz } | dz |z } |dk( r7| t | k( r)t j" |
|dz | d|d }|d |d< |dk\ s|
|dz | }| dkD r|d| dz n|d g} t j" |d| }|dxx |z
cc< t | dkD r*||t | z
dz
< |dt | | z
}|
d }
|
|k\ }t j |dd |dd z d dz }t | dkD r:t j | d d }||vrt j ||dz n|}t | dkD rPd}!|D ]I }"|dk7 s|!dk7 r|d d n|
d }#|
|!|" }$|$d |$d z
}%|#|$d< |#|%z |$d< |j |$ |"}!K || z
} g }&t% t | D ] }'|&||' j' z
}& |&S )a
Computes the final sequences by merging the end of the nth sequence with the beginning of the n+1th sequence. Since
`WhisperForConditionalGeneration` produces the timestamps pairwise, we filter the consecutive timestamps and only
iterate over them. We keep track of the `time` which indicates the actual starting time of the chunk that is
processed. We need to make sure to offset the timestamps tokens by the `time` in order for the tokenizer to
properly compute the final `offset`.
z<|notimestamps|>r
r Nr< )convert_tokens_to_idsr enumerater r r> r? r whererQ r r r% r* reversedr= rM insertr+ rO )(rS rT r. r timestamp_beginr r timeseq_idxr rV r r/ r0 r1 begin_idxtimestamp_tokens consecutivelast_timestampoffset overlap_timerelevant_timestamp
best_matchsliced_sequenceidxprevious_sequenceprevious_tokensrK rL match_lengthbest_idxend_of_curr_sequence_idxprevious_slice
last_slice
current_slice
actual_offset
sliced_tokensdurationresultrH s( r _find_timestamp_sequencer$ s7 556HIAMO E&336JJN D"9- U
& h %xx)H/5, ; ##A&CRV^C^BHHX 89!NDRZeDe"))K @kvK K,. .D$!2!@!@@NRSF .?.M.M MQ_ _`L!#(;*??UaCa*a!bcd!e !''*Q.>PQR>SVW>WK 21 5 9:]hij]k #
"$.7.H >*C*&7"&=O(+/G,/VW\_cd\d ?+a/@b $Q'9 :OA=
K (!+ z0I)5J'*H "*q.2B)C)V WXY Z[\ ]`a a 5 8PRS7SV`7` 4 )Q<3CW3W24))$,Z!^>V$WYZ\mno\p3" 7H6K 3!+q 2::>Ld2e MXZ[O$56G a$Hbstubvaw !/ 35))OQP^2_ / 3v = 3? >B '!+7FE#e*x/!34 !"9CJ$9 :E '(@(A BH $6hh / 47G7K KLQORSS 1 $ XX&6 7 :2 >N>LT_>_ +~'9:ep
{ a J!,
+
18A q b "
V^_`Va
(M B
(,}Q/??#0
a $1H$<
b! ]+ *
+
kUl F
3u:
$%(//##$ Mr" )N)'r collectionsr typingr r r r r r> r tokenization_utilsr utilsr
r r audio_utilsr baser pyctcdecoder !feature_extraction_sequence_utilsr modeling_utilsr
get_loggerrp r r models.auto.modeling_autor r! r: rM r^ r` r$ r r" r r0 s} # 7 7 4 H H $ 0L0 H %T(*,( 2U3 U3pgr"