[^(h>ddlZddlmZddlZddlZddlmcmZddl m Z m Z m Z m Z mZmZmZmZddlmZddlmZej,eZ d,dej2dej2d ej2d eej2fd Zd-d ed efdZd-d ed efdZd-d ed efdZ d-dej2ded efdZ d-d efdZ!d-d ed efdZ"d-d ed efdZ#d-d ed efdZ$d-d ed efdZ%dZ&dej2d e'ej2e(e(ffdZ)dej2fdZ*dej2de(de(d e(d ej2f d!Z+d"Z,dej2d#e(d$ed ej2fd%Z-d&Z.d'ej2fd(Z/d)Z0dej2dej2d ej2d eej2d e'ej2ej2ej2eej2ff d*Z1 d.dej2dej2d ej2d eej2fd+Z2y)/N)Optional)can_use_cudnn_attentioncan_use_efficient_attentioncan_use_flash_attentioncudnn_sdp_enabledflash_sdp_enabledmath_sdp_enabledmem_efficient_sdp_enabled SDPAParams) SDPBackend) NestedTensorquerykeyvalue attn_maskc 2t|tr t|trt|ts3td|jd|jd|jd|j|jk7s|j|jk7r3td|jd|jd|jd|j |j k7s|j |j k7r3td|j d |j d |j d|j d ks&|j d ks|j d kr?td |j d |j d|j d|j|jk7s|j|jk7r3td|jd|jd|jd| tdy)NzNExpected query, key, and value to be nested tensors, but got query.is_nested: z, key.is_nested: z, and value.is_nested: z instead.zLExpected query, key, and value to have the same dtype, but got query.dtype: z , key.dtype: z, and value.dtype: zSExpected query, key, and value to have the same device type, but got query.device: z, key.device: z, and value.device: zUExpected query, key, and value to all be at least 3 dimensional, but got query.dim: z , key.dim: z and value.dim: z[Expected query, key, and value to all be ragged on the same dimension, but got ragged dims z, z, and z, respectively.zMasks are not yet supported!) isinstancer ValueError is_nesteddtypedevicedim _ragged_idxtorchbool)rrrr dropout_p is_causalscales Y/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/torch/nested/_internal/sdpa.py_validate_sdpa_inputr"s ul +#|,%.((-'88I#--Y$$)OO#4I ?   {{cii5;;%++#=$$)KK= cii[I % }I 7   ||szz!U\\U\\%A%%*\\N. M!!&i 9   yy{Q#'')a-599;?cyy{m;swwyk1A%))+i Y   COO+u/@/@EDUDU/U%%&b(9@Q@Q?RRa c  788paramsreturnc|jjd}|jjd}|jjd}||k(xr||k(S)Nr)rsizerr)r$debug q_batch_size k_batch_size v_batch_sizes r!_check_batch_size_nestedr,MsU<<$$Q'L::??1%L<<$$Q'L < ' HLL,HHr#cd}|jjd}|jjd}|jjd}||k(xr||k(}|r |dzdk(r||ks|rtj d|||yy)NrzFor NestedTensor inputs, Flash attention requires q,k,v to have the same last dimension and to be a multiple of 8 and less than or equal to 256. Got Query.size(-1): %d, Key.size(-1): %d, Value.size(-1): %d instead.FTrr'rrlogwarningr$r(max_sizequery_size_last key_size_lastvalue_size_lastsame_head_dim_sizes r!!_check_head_dim_size_flash_nestedr:ZHll''+OJJOOB'Mll''+O=(O_-O  q A %  (  KKX    r#cd}|jjd}|jjd}|jjd}||k(xr||k(}|r |dzdk(r||ks|rtj d|||yy)Nr/r0rzFor NestedTensor inputs, cuDNN attention requires q,k,v to have the same last dimension and to be a multiple of 8 and less than or equal to 128. Got Query.size(-1): %d, Key.size(-1): %d, Value.size(-1): %d instead.FTr1r4s r!!_check_head_dim_size_cudnn_nestedr>tr;r#param param_namect|tsJd|jdk(r|rtj d|y|j dk(r|rtj d|yy)Nzparam should be a jagged NTr zMFused kernels do not support ragged num_head_dims, %s has a ragged num_heads.FrzAFused kernels do not support seq_len == 0, %s has a seq len of 0.T)rrrr2r3_get_min_seqlen)r?r@r(s r!:_check_for_seq_len_0_and_consistent_head_dim_nested_helperrCsr e\ *I,II * A  KK_  !#  KKS  r#c t|||}||k7r|dk7s||k7r|dk7s ||k7r$|dk7r|rtjd|||||||yy)Nr zzBoth fused kernels require query, key and value to have broadcastable %s, got Query %s %d, Key %s %d, Value %s %d instead.FT)maxr2r3)q_sizek_sizev_sizer@r(r5s r!_try_broadcast_param_sizerIsl666*H 8 ! h 6Q; h 6Q;  KKC  r#c|jjrt|jd|nd}|sy|jjrt|jd|nd}|sy|jjrt|jd|nd}|sy|jj d}|jj d}|jj d}||k(xr||k(}|si|jj s,|jj s|jj r|rtjdyt|||d|Sy) NrTFrrr zFBoth fused kernels do not support training with broadcasted NT inputs.z num heads) rrrCrrr' requires_gradr2r3rI) r$r( q_is_safe k_is_safe v_is_safe q_num_heads k_num_heads v_num_headssame_num_headss r!_check_for_seq_len_0_nestedrSsU << ! ! C LL'5    ::   C JJu    << ! ! C LL'5   ,,##A&K**//!$K,,##A&K K/NK;4NN  LL & &zz''||)) \( k;   r#cJtttf}|D] }|||r yyNFT)r,r:rSr$r( constraints constraints r!_can_use_flash_sdpa_jaggedrYs5 )#K " &%( r#c@ttf}|D] }|||r yyrU)r,rSrVs r!_can_use_efficient_sdpa_jaggedr[s2 #K" &%( r#cx|jjddjrT|jjddjr*|jjddjs|rt j dy|jr|rt j dyy)Nr zGIf inputs are nested tensors they must be contiguous after transposing.FzENested tensors for query / key are not supported when is_causal=True.T)r transpose is_contiguousrrr2r3r)r$r(s r!_can_use_math_sdpa_jaggedr`s LL " "1a ( 6 6 8zz##Aq)779||%%a+99;  KKY    KKW  r#c Fts.ts$tststj Stj tjtjtjf}t|||||||}|D]} | tjk(rt|rtjcS| tj k(r(t|rt|rtj cS| tjk(r(t|rt|rtjcS| tjk(stst!|stjcSt"j%dt|dt|dt"j%dt|dt|dt"j%dt!|dt"j%dt|dtj S)Nz)Memory efficient kernel not used because:T)r(z(Flash attention kernel not used because:z'Math attention kernel not used because:z(cuDNN attention kernel not used because:)rr r rr ERRORFLASH_ATTENTIONEFFICIENT_ATTENTIONMATHCUDNN_ATTENTIONr rrrYrr[r`r2r3) rrrrdropoutr enable_gqaorderingr$backends r!_select_sdp_backendrk%s  )+ "!# ""&&"" HsE9gy* UF' j00 0&v.!111 j00 0&v.3Mf3U!111 j44 4*627U8"555 joo %!&?&G!&' KK;<d3"66KK:;F$/vT2KK9:fD1KK:;F$/   r#qkvct|ts td|jg|j j t j|j}|j}|jjd}nt|jjdj t j|j}|j}t|dj}|||fS)Nzz+_is_computing_meta_flops..s1  Gu{{//@@ @ s:<F) rjit is_scriptingrrr_python_dispatch _get_current_dispatch_mode_stackany)rtorch_dispatch_mode_stacks r!_is_computing_meta_flopsrs_ 99 ! ! # (? KK ( ( I I K " .    r#c|jjt|stjs||||fSfd}||||||||fS)a* [Autocasting SDPA for NJT] Normal autocasting doesn't work for NJT+SDPA right now: * NJT intercepts the __torch_function__ call for scaled_dot_product_attention, which happens before we get to any aten ops or dispatcher logic; then the torch_function logic calls into efficient attention or flash attention. So, autocasting on the scaled_dot_product_attention op won't work because we never see that aten op. * If we put autocasting on `_flash_attention_forward`, then we'll get autocasting to run, but the kernel selection logic in torch_function handling (ie. jagged_scaled_dot_product_attention) won't work correctly: the kernel selection logic will run before autocasting, and choose a kernel based on the un-autocasted dtypes; but then autocasting will run and the actual attention computation will happen in a different dtype. An alternative is to just change the backend selection logic for SDPA+NJT to be autocast-aware and rely on autocasting to do the actual conversions for flash attention / efficient attention. However, by manually doing the actual autocast before the backend selection, we ensure that the autocast handling for backend selection doesn't diverge from the autocast handling for the actual dtype conversions. c||Stj}|jjr,|j|k(s|jtjk(r|S|j |Sr)rget_autocast_dtyperis_floating_pointfloat64rp)r target_dtype device_types r!cvtz_autocast..cvtsZ 9H// < **ww,&ww%--'HttL!!r#)rrrris_autocast_enabled)rrrrrrs @r! _autocastrs^4,,##K&e.G.G .Tc5)++ " u:s3xUS^ ;;r#cv t||||\}}}}t|||||||t|tr t|trt|tsJddlm}|j dkDr|j dkDr|j dkDr|jdk(rtj|j|j|jt|tr|jn||||} || |j|j|j|jS|jxs|jxs |j} t!|||||||} t#|rt$j&} | t$j&k(r|j)d} t+|dd } t+|dd }t+|dd }t-||}t/| ||\}}}}}}}}t0j2j4j7|||||||||d | \}}}}}||fi|j9dd }t;|| S| t$j<k(rt/|||\}}} }}}}}t0j2j4j?|jAd|jAd| jAdd|||||tC|| | \}}!}"}#}$}||jEdfi|j9dd S| t$jFk(rot/|||\}}} }}}}}t0j2j4jI||| |||||| ||d | \ }}%}&}'}$}(}"}#})||fi|j9dd S| t$jJk(r|j}*|j}+|j},|j}-|jLd}.|jLd}/d }0|0|}|0|}|0|}t1jN||||||| d}1|1j9dd jQj}1|1jSd|.|/}1||1|*|+|,|- j9dd }1|1StUd)Nr)'nested_view_from_values_offsets_lengthsrr )rrrr )rryr/r0F)r r]cV|jdd|jddz }tj|dd}|jj t |d}tj jt |}|jddj}|S)Nr r/r]r)r) _offsetsrr^rssplitlistnestedas_nested_tensorr)jagged_layout_ntrnr^ tensor_list strided_nts r! get_strided_layout_nested_tensorzMjagged_scaled_dot_product_attention..get_strided_layout_nested_tensors&//36F6O6OPSQS6TTG(8!Q?I#**,224=a2HK66tK7HIJ#--a3>>@J r#)rnrryz=No viable backend for scaled_dot_product_attention was found.)+rr"rr$torch.nested._internal.nested_tensorrrrFscaled_dot_product_attentionrsrorn_maybe_min_seqlen_maybe_max_seqlenrKrkrr rcr'rrrropsaten_flash_attention_forwardr^rrd_efficient_attention_forward unsqueezervsqueezerf_cudnn_attention_forwardre_size"_scaled_dot_product_attention_mathrrr)2rrrrrrr rhroutputcompute_logsumexpbackend_choicer query_padded key_padded value_paddedog_scalerrrrrrrr attention _logsumexp _philox_seed_philox_offset_debug_attn_maskquery_reshaped key_reshapedvalue_reshaped log_sumexpseedoffset max_seqlen_q logsumexp cum_seqlen_q cum_seqlen_kv max_seqlen_kv_ro q_lengthsrryd1d2rattn_outs2 r!#jagged_scaled_dot_product_attentionrs6$-UC #J E3yUIy)US 5,' sL ) ul + , yy{Q3779q=UYY[1_ARARVWAW// LLN JJL LLN&0L&I   "y  7  MMO MMO....   ++Ws/@/@WEDWDW( sE9iJN & $33333**R.$UAu5 "351 $UAu5 #E51 '|Z N !  ! ( )    IINN 3 3 !  ! ( )      4      <    )Aq/ *)W== :99 9 'uc5 9    ( )    IINN 7 7  $ $Q '  " "1 %  $ $Q '  ( )     N  8        7   a    )Aq/  :55 5 'uc5 9    ( )    IINN 3 3     ( )       4          7    )Aq/  :?? *--/MMO ,, ,, [[^ [[_ 17.s307;; 3y)Ye   %%a+668??A==R,:  !!  )Aq/  K  r#)NFN)F)NrFNF)3loggingtypingrrtorch.nntorch.nn.functionalrrrtorch.backends.cudarrrrrr r r torch.nn.attentionr nested_tensorr getLogger__name__r2Tensorr"rr,r:r>strrCrIrSrYr[r`rktuplervr{rrrrrrrrrr#r!rs    *'g!)- 0 <<0 0 <<0 % 0f IZ I Ij$4j$616 <<%( 4RV.8 8D8v z 4 :tj$(,^11% cSV@V:W16ell>1 LL1"1/21>A1 \\1lGT LL*-6: \\& ELL +< <<+< +< <<+< % +<  5<<u||Xell5K KL +