T^(hdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZmZm Z ddl!m"Z"ejFe$Z%e rddl&m'Z'ndZ'erddl(m)Z)m*Z*ddl+m,Z,nd\Z,Z*Z)er ddl-m.Z.m/Z/nd\Z/Z.e0e,e*e.e/e)fZ1dZ2dZ3Gdde jhZ5Gdde jhZ6Gdde jhZ7GddeZ8eGd d!eZ9eGd"d#eZ:d$Z;d%Z<ed&e;Gd'd(e8Z=ed)e;Gd*d+e8eZ>gd,Z?y)-zPyTorch MAMBA model.N) dataclass)AnyDictOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN) MambaCache)GenerationMixin)PreTrainedModel) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging)is_causal_conv1d_availableis_mamba_ssm_availableis_mambapy_available) MambaConfig)pscan)mamba_inner_fnselective_scan_fn)selective_state_updateNNN)causal_conv1d_fncausal_conv1d_update)NNzstate-spaces/mamba-130m-hfrc @eZdZdZdedeffd Z d dejde e de ejde ejfd Z d de e de ejde ejfd Z d de e de ejde ejfd ZxZS) MambaMixeru Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) config layer_idxc |t|||_|j|_|j|_|j |_|j|_t|j|_ ||_ |j|_ tj|j|j|j|j |j|j dz |_|j |_t$|j |_|j(|_tj*|j|jdz|j,|_tj*|j|j|j dzzd|_tj*|j|jd|_t5j6d|j dzt4j8dddf}|j;|jdj=}tj>t5j@||_!tj>t5jD|j|_#tj*|j|j|j,|_$|j,|_tJsM|j(r+tMrtNjQd ytSd tNjQd yy) Nr) in_channels out_channelsbias kernel_sizegroupspaddingr(FTdtypea7The fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. Falling back to the mamba.py backend. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1dzuse_mambapy is set to True but the mambapy package is not installed. To install it follow https://github.com/alxndrTL/mamba.py.aThe fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. Falling back to the sequential implementation of Mamba, as use_mambapy is set to False. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. For the mamba.py backend, follow https://github.com/alxndrTL/mamba.py.)*super__init__r# hidden_size state_sizessm_state_size conv_kernelconv_kernel_sizeintermediate_sizeinttime_step_rankr$ use_conv_biasr Conv1dconv1d hidden_act activationr act use_mambapyLinearuse_biasin_projx_projdt_projtorcharangefloat32expand contiguous ParameterlogA_logonesDout_projis_fast_path_availablerlogger warning_once ImportError)selfr#r$A __class__s f/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/mamba/modeling_mamba.pyr2zMambaMixer.__init__Ks{  !--$// & 2 2!'!9!9!&"7"78"#11ii..//%%**))&&*   !++&++,!--yy!1!143I3IA3MTZTcTcd ii 6 68K8KdNaNadeNe8elqr yy!4!4d6L6LSWX  LLD//!35== I$PQ' R HHT++R 0 ; ; =\\%))A,/ ejj)?)?@A $"8"8$:J:JQWQ`Q`a  %')''F &Z##J& hidden_states cache_paramscache_positionattention_maskc |j|jdd}|jr%|"t||jj |j r|jjnd|jj |jj |jj |jr$|jjjndtj|jj dd|j j|jjjd }|S|j#dd\}}|||j%dz}|jj j'|jj j)d|jj j)d}|m|ddkDret+|j-d|j.|j0||jj|j2}|j%d}n|Yt4j6j9||j:|j<dz df} |j?|j0| |tA|||jj|j2}|||j%dz}|j|jdd} tjB| |jD|jF|jFgd\} } } |jj | jddz}tj|jj }tI|jd r$|jjjnd}|e|ddkDr]tK|jL|j0|d |d || dddf| dddf|j |d |d j%d}nptO|||| jdd| jdd|j j||dd \}}|||jQ|j0||j|jdd}|S) Nrr,T) delta_biasdelta_softplusdimrr0)r?r().r) dt_softplus)rareturn_last_state))rD transposetrainingrr=weightr;r(rErFrQrCfloatrGexprNrPchunk unsqueezeviewsizer squeeze conv_statesr$r?r functionalpadr7shapeupdate_conv_statersplitr:r5hasattrr ssm_statesrupdate_ssm_state)rVr[r\r]r^projected_statescontextualized_statesgate conv_weightsrpssm_parameters time_stepBCdiscrete_time_steprWtime_proj_bias scan_outputs ssm_states rYcuda_kernels_forwardzMambaMixer.cuda_kernels_forwards7 << 6@@AF ==\1$2  ""$($6$6   D "" ## $$.2mm ""((*4::++-.. <<,,224#% !p%$O#3"8"8"8"B M4) -0H0H0K K  ;;--224;;3E3E3J3J13Mt{{OaOaOfOfghOijL'N1,=,A 4!))"- ,,T^^< KK$$OO ! !. 7 7 ; +"$--"3"3%(=(= @S@STV@W(WYZ'[#K!224>>;P^_ 0!<1A1Adoo! ) -0H0H0K K "[[)@)@A)FGN#kk!4!4d6I6I4K^K^ _egOIq!"&!4!4y7J7J1a7P!P 4::++-..A:A$,,PV:WT\\..446]aN'N1,=,A5 ++DNN;!&)&v.adGadGFFL" $ )B-+<!&KK1%KK1%FFLLN"#'&* +' i(\-E 11$..)L%)MM,2H2HA2N$O !$$rZc X |j\}}}|j}|j|jdd} | j dd\} } || |j dz} ||j |jj} | j| j} |jd|jk(rtjj| |j| jdz df} |j|j| ||j!|j#| dd|f} n9|j|j| |} | j|j"j$j} t'j(| |j"j$dddddfzd} |j*r| |j"j,z } |j!| j|j d} n`t'j.||j0|j2f| j|} |j!|j#| dd|f} || |j dz} |j5| jdd}t'j6||j8|j2|j2gd\}}}|j;|}tjj=|jdd}t'j>|j@jC }t'j>|ddddddf|dddddddfz}|dddddddf|dddddddfjCz}|| dddddddfjCz}|jDr|jFr|tI|jdd|jdd}||j dzjKdjdd}|| |jLddddfzz}||j!| z}ng}tO|D]}}|dddd|ddf| z|dddd|ddfz} t'jP| j||dd|ddfj d}|jS|dddddft'jT|d}|| |jLddddfzz}||j!| z}|(|j |jjW| |jY|jdd}|S) Nrr,rbrr0.devicer/r )-rsr/rDrfrkrlrwr$clonetorr7r rqrrrtr@r=rhrGsumr;r(zerosr8r5rErur:rFsoftplusrjrNrirArgrrorPrangematmulappendstackcopy_rQ)rV input_statesr\r]r^ batch_sizeseq_len_r/ryr[r{r conv_stater}r~rrrrW discrete_A discrete_BdeltaB_uhs scan_outputrirzs rY slow_forwardzMambaMixer.slow_forwards!-!3!3 GQ""<< 5??1E.44QA4> t  %)N,D,DQ,GGM  #$//?EEGI! ]%9%9:I##A&$*?*??]]..!**]-@-@-DDaH ..t~~z>Z $])CC'M)R S );;DNNM[ij ']]4;;+=+=+D+DE % *t{{7I7I!QPQ'7R*RXZ [ %%!T[[%5%55M $ 7 : :5 A K KB O  T33T5H5HI$++5I!HHT[[%?XgX %NOM  %)N,D,DQ,GGM]%<%!QPQST*AUU #ll9<<+>!Q' @T@TUW@XY ##K1a$89 : ++l;K%a 9N)NOK&$7K'''7==iH!% k.C.CAq.I J$$rZctr^d|jjjjvr2t j js|j||||S|j||||S)Ncuda) rRrErhrtyperG_dynamo is_compilingrr)rVr[r\r]r^s rYforwardzMambaMixer.forward?se "f 0B0B0I0I0N0N&NW\WdWdWqWqWs,,]L.Zhi i   nn]]rZr)__name__ __module__ __qualname____doc__rr9r2rGTensorrr LongTensorrrr __classcell__rXs@rYr"r"Cs:{:s:~.25959 c%||c%z*c%!!1!12 c% !!1!12 c%LO%x 7KO%aijojzjza{O%S[\a\l\lSmO%j.25959 ^z* ^!!1!12 ^ !!1!12 ^rZr"c,eZdZdfd ZdZdZxZS) MambaRMSNormct|tjt j ||_||_y)zL MambaRMSNorm is equivalent to T5LayerNorm and LlamaRMSNorm N)r1r2r rLrGrOrhvariance_epsilon)rVr3epsrXs rYr2zMambaRMSNorm.__init__Ls1 ll5::k#:; #rZc"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)Nr,r0T)keepdim) r/rrGrIpowmeanrsqrtrrh)rVr[ input_dtypevariances rYrzMambaRMSNorm.forwardTsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::rZcR|jjdd|jS)Nrz, eps=)rhrsrrVs rY extra_reprzMambaRMSNorm.extra_repr[s*++##A&'vd.C.C-DEErZ)gư>)rrrr2rrrrs@rYrrKs$;FrZrcteZdZfdZ ddeedeejdeejfdZxZ S) MambaBlockct|||_||_|j|_t |j |j|_t|||_ y)Nrr$) r1r2r#r$residual_in_fp32rr3layer_norm_epsilonnormr"mixer)rVr#r$rXs rYr2zMambaBlock.__init__`sR  " & 7 7 !3!39R9RS )< rZr\r]r^c|}|j|j|jjj}|jr|jt j }|j||||}||z}|S)Nr.r\r]r^)rrrhr/rrGrIr)rVr[r\r]r^residuals rYrzMambaBlock.forwardhs! -"2"29I9I9O9O"2"PQ  {{5==1H  ^dr# !=0 rZr) rrrr2rr rGrrrrs@rYrr_sQ=.25959 z*!!1!12  !!1!12 rZrc.eZdZdZeZdZddgZdZdZ dZ y)MambaPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. backbonerr"Tct|trWd|j_d|j_|j j dz|j jz}|j jdk(r5tjj|jj|nO|j jdk(r6tjj|jj| |tj tj"|j j$t'j(|j j*t'j(|j j,z zt'j(|j j,zj/|j j0}|tj(tj2|  z}tj45|jj6j9|dddd|jj6_t|tj<rM|j6t?|j6ddstjjA|j6nYt|tjBr?tjjE|j|j jF |j jHr|jKD]\}}|d vs tjjM|t'jNd  tj45|t'jN|j jPz}dddyy#1swYxYw#1swYxYw) zInitialize the weights.Tgconstantrandom)minN _no_reinitF)std)zout_proj.weight)a)) isinstancer"rN_no_weight_decayrPr#r:time_step_scaletime_step_init_schemer init constant_rFrhuniform_rGrjrandr8mathrM time_step_max time_step_minclamptime_step_floorexpm1no_gradr(rrrBgetattrzeros_ Embeddingnormal_initializer_rangerescale_prenorm_residualnamed_parameterskaiming_uniform_sqrtnum_hidden_layers)rVmodule dt_init_stddtinv_dtnameps rY _init_weightsz"MambaPreTrainedModel._init_weightss fj ),0FLL )(,FHH %++44d:T[[=X=XXK{{00J>!!&.."7"7E22h>  !6!6 kR 4;;88988DKK556$++B[B[9\\^((4;;4456e 33e4  %))U[["%5$566F 2##))&1 2-1FNN   * fbii ({{&v{{L%@GGNN6;;/  - GGOOFMMt{{/L/LO M ;; / /"224 Fa.. GG,,Q$))A,,?FTYYt{{'D'DEEFF F 0 2 22FFs &O"-O(O%(O1 N) rrrrr config_classbase_model_prefix_no_split_modulessupports_gradient_checkpointing _is_statefulrrZrYrr{s0 L"%|4&*#L-FrZrc|eZdZUdZdZeejed<dZ ee ed<dZ ee ejed<y) MambaOutputa# Class for the MAMBA model outputs. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. cache_params (`MambaCache`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. Includes both the State space model state matrices after the selective scan, and the Convolutional states hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. Nlast_hidden_stater\r[) rrrrrrrG FloatTensor__annotations__r\r r[rrrZrYrrsH$6:x 1 129)-L(:&-80`, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. cache_params (`MambaCache`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). use_cache (`bool`, *optional*): If set to `True`, the `cache_params` is returned and can be used to quickly generate the next logits. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. z_The bare MAMBA Model transformer outputting raw hidden-states without any specific head on top.c*eZdZfdZdZdZdZeee e e e  dde ejde ejde ed e ed e ed e ed e ejd e ejdeee ffdZxZS) MambaModelc t||tj|j|j |_tjt|jDcgc]}t||c}|_ d|_ t|j |j|_|j!|j"|j%ycc}w)NrFr)r1r2r r vocab_sizer3 embeddings ModuleListrrrlayersgradient_checkpointingrrnorm_f"_register_load_state_dict_pre_hook load_hook post_init)rVr#idxrXs rYr2zMambaModel.__init__"s  ,,v'8'8&:L:LMmmRWX^XpXpRq$r3Z#%F$rs &+#"6#5#56;T;TU  //?  %ss&Ccf|D],}d|vs|j|||jdd<yy)Nz embedding.z embeddings.)popreplace)rV state_dictprefixargsks rYr zMambaModel.load_hook.s; Aq EO^^TUEV 199\=AB rZc|jSNrrs rYget_input_embeddingszMambaModel.get_input_embeddings4s rZc||_yrrrVnew_embeddingss rYset_input_embeddingszMambaModel.set_input_embeddings7s (rZ checkpoint output_typer input_ids inputs_embedsr\ use_cacheoutput_hidden_states return_dictr]r^returnc ||n|jj}||n#|js|jjnd}||n|jj}|du|duz r t d||j |}|jr|jr|rd}|r|st|j|jd|j|j}tjd|jj|j}n| t dd}|} |rdnd} |jD]O} |jr,|jr |j!| j"| |||} n | | |||} |sJ| | fz} Q|j%| } |r| | fz} |st'd | || fDSt)| |r|| Sd| S) NFz:You must specify exactly one of input_ids or inputs_embedsrrrzYou have to specify the `cache_position` manually when `use_cache=True` and `cache_params` is passed, you don't have to pass a `cache_params` if you are in prefilling stage because in that case it will be initialized for you automaticallyrrc3&K|] }|| ywrr).0vs rY z%MambaModel.forward..sfqXYXefs)rr\r[)r#r$rgr#use_return_dict ValueErrorrr r rnrr/rGrHr6r_gradient_checkpointing_func__call__r tupler) rVr!r"r\r#r$r%r]r^r[all_hidden_states mixer_blocks rYrzMambaModel.forward:s$%9$D $++JjJj "+!6IZ^ZgZgT[[=R=Rmr %0% J 'tn J d^J !!1!12J !!1!12J  uk! "J C J rZrz The MAMBA Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ceZdZdgZfdZdZdZdZdZ dde de e e fd e d e e e ffd Z dd eed eej$deej$fdZeeeeee ddeej$deej$deej4d eedeej$deedeedeed eej8d eeeffdZxZ S)MambaForCausalLMzlm_head.weightct||t||_t j |j |jd|_|jy)NFr-) r1r2rrr rBr3rlm_headr )rVr#rXs rYr2zMambaForCausalLM.__init__sF  "6* yy!3!3V5F5FUS  rZc|jSrr;rs rYget_output_embeddingsz&MambaForCausalLM.get_output_embeddingss ||rZc||_yrr=rs rYset_output_embeddingsz&MambaForCausalLM.set_output_embeddingss % rZc6|jjSr)rrrs rYrz%MambaForCausalLM.get_input_embeddingss}}1133rZc8|jj|Sr)rrrs rYrz%MambaForCausalLM.set_input_embeddingss}}11.AArZoutputs model_kwargsnum_new_tokensr&c |jdd|d<|jddrd|vr|d|ddd|z|d<d|vr?|d}tj||j|jddfgd |d<|S) Nr\r#Tr]r0r^rrrb)getrGcatnew_onesrs)rVrCrDrEkwargsr^s rY#_update_model_kwargs_for_generationz4MambaForCausalLM._update_model_kwargs_for_generations(/{{>4'H ^$   [$ / L0-.:-9:J-KBC-PSa-aL) * | +)*:;N-2YY!8!8.:N:Nq:QST9U!VW]_.L) *rZr\r]r^c <|rh| td|ddkDr|dddfjd}|9d}n6tjd|jj |j }||d|i}nd|ji}|j||||d|S)Nz`cache_position` should not be None as it should have been initialized in `model.generate`, you are responsible for passing in a valid `cache_position` if you are calling `prepare_inputs_for_generation` directly with `use_cache=True`rr0r(r"r!)r\r#r]r^) r.rlrGrHr#r6rrKupdate) rVr!r"r#r\r]r^rJ model_inputss rYprepare_inputs_for_generationz.MambaForCausalLM.prepare_inputs_for_generations % e a 1$%ae,66r: !-%)N"'a1H1HQZQaQa!b  $)=+];L')=)=)?@L ,&"0"0   rZrr!r"labelsr$r%r#c ||n|jj}|j||||||| |} | d} |j| j |jj j j} d}||j | j}| dddddfj}|dddfj}t}||jd|jd|jd}|s| f| ddz}||f|zS|St|| | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N)r\r"r$r%r#r]r^r.r0r)rrr\r[)r#r-rr;rrhr/rirrKr rmrnrr\r[)rVr!r^r"r\rPr$r%r#r]rJ mamba_outputsr[rr shift_logits shift_labelsloss_fctoutputs rYrzMambaForCausalLM.forwards_2&1%r@rrrrstrrr9rKrr rGrrOrr4rr5rr6rr7rrrrrrs@rYr9r9s++&4BYZ"26sCx.RU c3h,-15959. z* . !!1!12 .!!1!12.`++AB&'$155959-1-1/3&*$(157 E,,-7 !!1!127   1 12 7 z* 7 ))* 7 'tn7 d^7 D>7 !.7  u)) *7 C 7 rZr9)r9rr)@rr dataclassesrtypingrrrrrrGtorch.utils.checkpointr torch.nnr activationsr cache_utilsr generationrmodeling_utilsrutilsrrrrrutils.import_utilsrrrconfiguration_mambar get_loggerrrS mambapy.pscanr&mamba_ssm.ops.selective_scan_interfacerr+mamba_ssm.ops.triton.selective_state_updater causal_conv1drr allrRr5r6Moduler"rrrrrMAMBA_START_DOCSTRINGr4rr9__all__rrZrYrms !44 %!%)-kj,   H %# EXR@P=-~DD-7**.0@BVXfg3E^E^PF299F(89F?9Fx =+= =0 =+= =6 Bei %i i X V +_V V r ErZ