T^(hrdZddlmZmZmZddlZddlZddlmZddl m Z ddl m Z ddl mZdd lmZdd lmZmZd d lmZd d lmZmZmZmZmZmZmZmZmZd dl m!Z!ddl"m#Z#ejHe%Z&dZ'GddeZ(GddeZ)GddeZ*GddeZ+Gdde!Z,Gdde eZ-GddeZ.Gd d!eZ/Gd"d#eZ0Gd$d%eZ1gd&Z2y)'zPyTorch Qwen3 model.)CallableOptionalTupleN)Cache)FlashAttentionKwargs)CausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONS)Unpack) LossKwargslogging)GemmaMLP) LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaForQuestionAnsweringLlamaForSequenceClassificationLlamaForTokenClassification LlamaRMSNormapply_rotary_pos_embeager_attention_forward) MistralModel) Qwen3Configz Qwen/Qwen3-8Bc eZdZy) Qwen3RMSNormN__name__ __module__ __qualname__e/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/qwen3/modular_qwen3.pyrr4r#rc eZdZy)Qwen3MLPNrr"r#r$r'r'8r%r#r'c2eZdZdedeffd Z d dejdeejejfde ejde e de ejd e e d eeje eje eejffd ZxZS) Qwen3Attentionconfig layer_idxct|||t|j|j|_t|j|j|_|j|_|jjr:t|jdd#|j|jjk\sd|_yy)N)epssliding_window) super__init__rhead_dim rms_norm_epsq_normk_normr.r*use_sliding_windowgetattrr+max_window_layersselfr*r+ __class__s r$r0zQwen3Attention.__init__=s +"4==f6I6IJ "4==f6I6IJ $33 KK * * %5t<H$++"?"??"&D @r# hidden_statesposition_embeddingsattention_maskpast_key_valuecache_positionkwargsreturnc |jdd}g|d|j}|j|j|j |j dd} |j |j|j |j dd} |j|j |j dd} |\} } t| | | | \} } |'| | |d}|j| | |j|\} } t}|jjdk7r^|jjdk(r(|jddrt j#d nt$|jj}||| | | |f|j&sd n |j(|j*|j,d |\}}|j.g|dj1}|j3|}||fS) Nrr)sincosr?eagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.g)dropoutscalingr.)shaper1r3q_projview transposer4k_projv_projrupdater+rr*_attn_implementationgetlogger warning_oncer trainingattention_dropoutrJr.reshape contiguouso_proj)r9r;r<r=r>r?r@ input_shape hidden_shape query_states key_states value_statesrErD cache_kwargsattention_interface attn_output attn_weightss r$forwardzQwen3Attention.forwardIs$))#2. 88b8$--8 {{4;;}#=#B#B<#PQ[[\]_`a [[]!;!@!@!NOYYZ[]^_ {{=166|DNNqRST &S#7 jRUWZ#[ j  %#&snUL'5'<'d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL.. %  % ! \*k));;;;FFH kk+. L((r#)NN)rr r!rintr0torchTensorrrr LongTensorr rrd __classcell__r:s@r$r)r)<s '{ 's '"+/59 0)||0)#5<<#=>0)!. 0) ! 0) !!1!12 0)-.0) u||Xell3XeELL>Q5RR S0)r#r)c(eZdZdedeffd ZxZS)Qwen3DecoderLayerr*r+ct|t|||_t ||_|j r4|jdk7r$tjd|jdyyy)N)r*r+flash_attention_2z=Sliding Window Attention is enabled but not implemented for `z)`; unexpected results may be encountered.) r/r0r) self_attnr'mlpr.rRrTrUr8s r$r0zQwen3DecoderLayer.__init__}sp 'vKF#  ! !f&A&AEX&X   OPVPkPkOlm99 'Y !r#)rr r!rrer0rirjs@r$rlrl|s { s  r#rlc eZdZy) Qwen3ModelNrr"r#r$rrrrr%r#rrc eZdZy)KwargsForCausalLMNrr"r#r$rtrtsr#rtc.eZdZdeedeffd ZxZS)Qwen3ForCausalLM super_kwargsrAc "t|di|S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. logits_to_keep (`int` or `torch.Tensor`, *optional*): If an `int`, compute logits for the last `logits_to_keep` tokens. If `0`, calculate logits for all `input_ids` (special case). Only last token logits are needed for generation, and calculating them only for that token can save memory, which becomes pretty significant for long sequences or large vocabulary size. If a `torch.Tensor`, must be 1D corresponding to the indices to keep in the sequence length dimension. This is useful when using packed tensor format (single dimension for batch and sequence length). Returns: Example: ```python >>> from transformers import AutoTokenizer, Qwen3ForCausalLM >>> model = Qwen3ForCausalLM.from_pretrained("Qwen/Qwen3-8B") >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```r")r/rd)r9rwr:s r$rdzQwen3ForCausalLM.forwardsFw...r#)rr r!r rtr rdrirjs@r$rvrvs%#/01#/ #/#/r#rvc eZdZy)Qwen3ForSequenceClassificationNrr"r#r$rzrzr%r#rzc eZdZy)Qwen3ForTokenClassificationNrr"r#r$r|r|r%r#r|c eZdZy)Qwen3ForQuestionAnsweringNrr"r#r$r~r~r%r#r~)rvr~rrQwen3PreTrainedModelrzr|)3__doc__typingrrrrftorch.utils.checkpoint cache_utilsrmodeling_flash_attention_utilsrmodeling_outputsr modeling_utilsr processing_utilsr utilsr r gemma.modeling_gemmarllama.modeling_llamarrrrrrrrrmistral.modeling_mistralrconfiguration_qwen3r get_loggerrrT_CHECKPOINT_FOR_DOCrr'r)rlrrrtrvrzr|r~__all__r"r#r$rs,,  B65&,   4,   H %% <  x =)^=)@ )   ?,j>$/'$/N %C  "=  9  r#