S^(h+I&NddlZddlZddlmZmZddlZddlmcmZ ddl m Z m Z m Z mZmZej eZdZe rddlmZmZmZddlmZmZddlmZerddlmZddlmZmZmZdd lmZdd lm Zer$d e!ejDejFvZ$d Z%d Z&dejNde(ejNejNe)ffdZ*dejNdejNdejNdejNde)f dZ+dZ, d4dejNdejNdejNdeejZfdZ.e dZ/ej`jcdddk(Z2 d5d!ejNd"ejNd#ejNdeejNde)d$e3d%e4d&eejNd'ee4d(ee)d)e3d*ee4d+ee3d,eejjd-eejjd.ee)d/ee)deejZf$d0Z6Gd1d2ed 3Z7y)6N)Optional TypedDict)is_flash_attn_2_availableis_flash_attn_greater_or_equal#is_flash_attn_greater_or_equal_2_10is_torch_npu_availablelogging)index_first_axis pad_input unpad_input)flash_attn_funcflash_attn_varlen_func)apply_rotary_emb)npu_rotary_mul)npu_flash_attn_func)npu_flash_attn_varlen_func window_sizec0trytryy)z5Determine whether flash-attention can be used or not.TF)rr i/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/modeling_flash_attention_utils.pyis_flash_attn_availabler6s!" rc\tr t Str ddlm}|Sy)zBDetermine whether flash-attention uses top-left or down-right maskr'is_npu_fa2_top_left_aligned_causal_maskF)rrr integrations.npu_flash_attentionrrs r!flash_attn_supports_top_left_maskrDs+!"6888]688 rattention_maskreturncd|jdtj}tj|j dj }|j j }tjtj|dtjd}|||fS)aq Retrieves indexing data required to repad unpadded (ragged) tensors. Arguments: attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: indices (`torch.Tensor`): The indices of non-masked tokens from the flattened input sequence. cu_seqlens (`torch.Tensor`): The cumulative sequence lengths, used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). max_seqlen_in_batch (`int`): Maximum sequence length in batch. )dimdtypeF)as_tupler)rr) sumtorchint32nonzeroflattenmaxitemFpadcumsum)rseqlens_in_batchindicesmax_seqlen_in_batch cu_seqlenss r_get_unpad_datar4Ts &))b )DmmN224uEMMOG*..0557u||$4!5;;OQWXJ r query_layer key_layer value_layer query_lengthct|\}}}|j\}} } } t|j|| z| | |}t|j|| z| | |}|| k(r't|j|| zd| |}|} |} |}nn|dk(rLd} t j |dztj |j} | dd}|jd}n|dd| df}t||^}}} } }||||| |f| |ffS)a Unpads query, key, and values tensors, using a single dimension for all tokens even though they belong to different batches. This function is used instead of `flash_attn.bert_padding.unpad_input` in order to avoid the recomputation of the same intermediary tensors for query, key, value tensors. Arguments: query_layer (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. query_length (`int`): Target length. Return: query_layer (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`Tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`Tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). r"r)r$deviceN) r4shaper reshaper'aranger(r:squeezer )r5r6r7rr8 indices_k cu_seqlens_kmax_seqlen_in_batch_k batch_size kv_seq_lennum_key_value_headshead_dim cu_seqlens_qmax_seqlen_in_batch_q indices_q_s r _upad_inputrJosWN6E^5T2I|2(9:JUVacqJrG Y .Ca  |$  56  rc|jd|jd|jd}|jjd|jd|jd}|jjd|jd|jd}|j}t j |jd|j tj}t j||dk(t j|j|j tjf}|jdz}||||||f||ffS)aI This function returns necessary arguments to call `flash_attn_varlen_func`. All three query, key, value states will be flattened. Cumulative lengths of each examples in the batch will be extracted from position_ids. NOTE: ideally cumulative lengths should be prepared at the data collator stage Arguments: query (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). position_ids (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: query (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`Tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`Tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). r"r)r:r$r) viewsize contiguousr*r'r=r:r(cattensorr+)querykeyvalue position_idsrH cu_seq_lens max_lengths rprepare_fa2_from_position_idsrXs0@ JJr5::b>5::b> :E ..   CHHRL#((2, ?C     # #B 2 2 GE'')L \..q1,:M:MUZU`U`aI)) la' ( LL**,\5H5HPUP[P[ \ K!!#a'J 3y; *DzS]F^ __rrRrSrT target_dtypec||||fS|j}|tjk(rLtj d|d|j |}|j |}|j |}|||fS)aG PEFT usually casts the layer norms in float32 for training stability reasons therefore the input hidden states gets silently casted in float32. Hence, we need cast them back in float16 / bfloat16 just to be sure everything works as expected. This might slowdown training & inference so it is recommended to not cast the LayerNorms! Args: query (`torch.Tensor`): Input query states to be passed to Flash Attention API key (`torch.Tensor`): Input key states to be passed to Flash Attention API value (`torch.Tensor`): Input value states to be passed to Flash Attention API target_dtype (`torch.dtype`, *optional*): The dtype to convert the attention tensors to. Conversion can be ignored by not providing the target dtype. zThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)r$r'float32logger warning_onceto)rRrSrTrY input_dtypes rfa_peft_integration_checkras.c5  ++Kemm#~Q  &ff\"& #u rz2.4.1FLASH_ATTENTION_DETERMINISTIC01F query_states key_states value_states is_causaldropoutrU softmax_scalesliding_windowuse_top_left_masksoftcap deterministic cu_seq_lens_q cu_seq_lens_k max_length_q max_length_kc ,| s|}n |xr|dk7}txr| duxr|jd| kD}|rd| | fini}tr | t} | |d<| | |d<t ||||\}}}|U|jd}t |||||\}}}}}}|\}}|\}}t |||f|||||||d|}t||||}|S|C|/|dk7r;tj|d dk\js|jd}| | t||||\}}}}}}|\} }|\}}n|jd|jd |jd}|jd|jd |jd}|jd|jd |jd}t |||f| ||||||d|}|j|d|jd |jd}|St||||f||d |}|S) a Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, then computes the attention scores and pad the final attention scores. Args: query_states (`torch.Tensor`): Input query states to be passed to Flash Attention API key_states (`torch.Tensor`): Input key states to be passed to Flash Attention API value_states (`torch.Tensor`): Input value states to be passed to Flash Attention API attention_mask (`torch.Tensor`, *optional*): The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the position of padding tokens and 1 for the position of non-padding tokens. dropout (`float`): Attention dropout softmax_scale (`float`, *optional*): The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim) use_top_left_mask (`bool`, defaults to `False`): flash_attn<2.1 generates top-left aligned causal mask, while what is needed here is bottom-right alignment, that was made default for flash_attn>=2.1. This attribute is used to handle this difference. softcap (`float`, *optional*): Softcap for the attention logits, used e.g. in gemma2. deterministic (`bool`, *optional*): Determines if the deterministic option introduced in flash_attn>=2.4.1 is enabled. rNrrnrmr)rFr@ max_seqlen_q max_seqlen_k dropout_prjcausalr")r#rL)rjrw)_flash_supports_window_sizer; flash_241deterministic_grarJrr r'diffallrNrXr<rMr) rerfrgrr8rhrirUrjrkrlrmrnrorprqrrrYkwargsrwuse_sliding_windows flash_kwargsrBrHrV max_seq_lensrFr@rGrAattn_output_unpad attn_outputs r_flash_attention_forwardrs\ 0|q0 $kd(BkzGWGWXYGZ]kGkI\MNN#CDacL  +M(5 _%") Y.Gj, .*L*l !!''* Wb *lNLX T j, ; &1" l7C442     &%..'     19j,W X O  ! \Q%6 <]_@`de@e?j?j?l!&&q)  M$9-lJ Vbc YL*lI{L,7 (M=)5 &L,(//L4E4Eb4Irs &    H %PPB9<ZZXf"/48I8I8I/8Z8e8e3f"f   ELLU5<<WZ;Z5[6FF||FFLL F  FR/`l+/ & <<& & <<&5;;' &R +7 3 **..!@#F#M+/%)$(##$(0404"&"&*.%I,,I I,,IU\\* I  I  II5<<(IE?ISMIIe_ID>IE,,-IE,,-I 3-!I"3-#I$5;;'%IX 9E r