T^(h#VddlmZmZmZddlZddlmZddlmcmZ ddl Zddl m Z ddl mZddlmZddlmZmZmZmZmZmZmZd d lmZej6eZGd d ej<ZGd deZ GddeZ!GddeZ"GddeZ#GddeZ$y))CallableOptionalTupleN)Cache)ALL_ATTENTION_FUNCTIONS)logging)LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModelapply_rotary_pos_embeager_attention_forward) OlmoConfigcdeZdZdZdeddffd ZdejdejfdZxZ S) OlmoLayerNormz/LayerNorm but with no learnable weight or bias. hidden_sizereturnNc2t||f|_yN)super__init__normalized_shape)selfr __class__s c/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/olmo/modular_olmo.pyrzOlmoLayerNorm.__init__s !, hidden_statesc|j}tj|jtj |j dddj|S)N)dtypegh㈵>)eps)r#F layer_normtotorchfloat32r)rr! orig_dtypes rforwardzOlmoLayerNorm.forward!sO"(( ||M,,5==,A4CXCXZ^`djnorr   r ) __name__ __module__ __qualname____doc__intrr(Tensorr+ __classcell__rs@rrrs49/C/D/ U\\ ell r rceZdZfdZxZS)OlmoMLPcJt||tj|j|j d|_tj|j|j d|_tj|j |jd|_y)NF)bias) rrnnLinearrintermediate_size gate_projup_proj down_proj)rconfigrs rrzOlmoMLP.__init__)ss  4#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXr )r,r-r.rr2r3s@rr5r5(sYYr r5c eZdZ d dejdeejejfdeejdeedeejdeejeejeeejff dZ y) OlmoAttentionNr!position_embeddingsattention_maskpast_key_valuecache_positionrc V|jdd}g|d|j}|j|} |j|} |j |} |j j | j|j j |j j | j|j j |j j | j|j j |j j | j|jdd} | j|jdd} | j|jdd} |\} } t| | | | \} } |'| | |d}|j| | |j|\} } t}|j jdk7r^|j jdk(r(|jdd rt j#d nt$|j j}||| | | |f|j&sd n |j(|j*d |\}}|j,g|dj/}|j1|}||fS) N)minmaxrr )sincosrDeagersdpaoutput_attentionsFz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.g)dropoutscaling)shapehead_dimq_projk_projv_projr>clip_qkvclamp_view transposerupdate layer_idxr_attn_implementationgetlogger warning_oncertrainingattention_dropoutrOreshape contiguouso_proj)rr!rArBrCrDkwargs input_shape hidden_shape query_states key_states value_statesrJrI cache_kwargsattention_interface attn_output attn_weightss rr+zOlmoAttention.forward1sy$))#2. 88b8$--8 {{=1 [[/ {{=1 ;;   +   T[[%9%9$9t{{?S?S  T   4;;#7#7"7T[[=Q=Q  R   T[[%9%9$9t{{?S?S  T#((6@@AF __\2<d##L '>dkk>^>^&_#$7      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r )NN) r,r-r.r(r1rrr LongTensorr+r rr@r@0s +/59 8)||8)#5<<#=>8)!. 8) ! 8) !!1!12 8) u||Xell3XeELL>Q5RR S8)r r@c(eZdZdedeffd ZxZS)OlmoDecoderLayerr>rZct|||t|j|_t|j|_t |||_y)N)r>rZ)rrrrinput_layernormpost_attention_layernormr@ self_attnrr>rZrs rrzOlmoDecoderLayer.__init__msF +,V-?-?@(5f6H6H(I%&f Jr )r,r-r.rr0rr2r3s@rrqrqlsKzKcKKr rqc$eZdZdeffd ZxZS) OlmoModelr>c t||tjt |j Dcgc]}t ||c}|_t|j|_ ycc}wr) rrr8 ModuleListrangenum_hidden_layersrqlayersrrnormrvs rrzOlmoModel.__init__us[  mmBGH`H`Ba bY fi 0 b  "&"4"45  csA1)r,r-r.rrr2r3s@rrxrxts6z66r rxc eZdZy)OlmoForCausalLMN)r,r-r.ror rrr}sr r)%typingrrrr(torch.nnr8torch.nn.functional functionalr%torch.utils.checkpoint cache_utilsrmodeling_utilsrutilsr llama.modeling_llamar r r rrrrconfiguration_olmor get_loggerr,r]Modulerr5r@rqrxrror rrs,,  5+   H %  BII  YhY9)N9)xK(K6 6 & r