g^(h"*dZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z m Z m Z mZddlmZddlmZddlZddlmZmZdd lmZmZmZdd lmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&dd l'm(Z(m)Z)dd l*m+Z+m,Z,m-Z-m.Z.dd l/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:m;Z;mZ> ddl?m@Z@e rddlBmCZCddlDmEZEmFZFede ZGdQdRdZHdRdZIdeJeKed?eQZbGd@dAe<ZcGdBdCe<ZdGdDdEe<ZeGdFdGe<ZfddHlDmgZgmhZhGdIdJehZieiZjGdKdLe<ZkGdMdNe<ZlGdOdPe<Zmy#eA$rdZ@YewxYw)Sa Function-related variable tracking classes for Dynamo's symbolic execution. This module contains classes that track different types of functions during graph compilation, including: - User-defined functions and methods - Built-in functions and methods - Wrapped functions (e.g. from decorators) - Special function types (e.g. functools.partial) - Triton kernels and related function types These classes are responsible for: - Tracking function calls and their arguments - Managing function closures and cell variables - Handling function attributes and special methods - Maintaining guards for function identity and closure contents - Supporting function inlining and specialization - Enabling proper symbolic execution of different function types The variable trackers here work together with the rest of Dynamo to enable accurate graph capture while handling Python's various function-related behaviors. N)Sequence)AnyCallableOptional TYPE_CHECKINGTypeVar)Never)patch) polyfills variables)create_call_function create_rot_n is_generator) get_dynamo_observed_exceptionhandle_observed_exceptionInfiniteGeneratorErrorObservedExceptionObservedGeneratorExitObservedUserStopIterationraise_observed_exception SkipFrame unimplementedunimplemented_v2 Unsupported) GuardBuilder install_guard) AttrSourceConstantSourceDefaultsSource GetItemSource) check_constant_argscheck_unspec_or_constant_argscmp_name_to_op_mappingcountersidentity is_functionis_wrapper_or_member_descriptoristype make_cell)typestrValueMutationNewVariableTrackerConstantVariable)_fsdp_param_group)InstructionTranslator)TritonGridTypeTritonKernelType_F)boundct|tr|S|stj||Stjj ||SN) isinstancer.buildr LazyVariableTrackercreate)txvalsources _/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/torch/_dynamo/variables/functions.pywrap_bound_argrAXsB#' $$R--,,33C@@ct|jD]+\}}t|ttfst ||||<-yr8)listitemsr9tupledictrA)r=resultkvs r@wrap_args_kwargsrKds>V\\^$.1 a% '&r1-F1I.rBrHc|jj}|jD]<}|j}||vr!|j ||j ||||<>y)z Update `result` to add mapping from local name to new cells created directly by `code`, or update SideEffects in `parent` if the a local cell is already in `result` (cell argument). N)output side_effects co_cellvarstrack_cell_new store_cellpop)parentrHcoderNnamenew_cells r@ init_cellvarsrWksa ==--L   ..0 6>  # #Hfjj.> ?t  rBcddlm}||||||}||_t|trddlm} t| |}|t|tsJ||_|S)Nr) FunctionTypepairwise) typesrY__kwdefaults__r9rF itertoolsr[rG__annotations__) rT f_globalsrUdefaultsclosure kwdefaults annotationsrYfuncr[s r@_create_nested_fnrf|sa# ix AD$D+u%&8K01   *[$"?? ?&D KrB>__doc____code__ __closure__ __globals__ __defaults__r]r_c&|xr t||} tj||}|t vr t ||}|r tjj|Stj|S#t$rt t|YgwxYwr8) rinspectgetattr_staticAttributeErrorrfn_known_dunder_attrsgetattrr r;r<r.r:)r=fnr?rUsubobjs r@fn_var_getattrrts  0 640F5''D1 $$T" ,,33FFCC  V ,, 5 !45sA44BBcPeZdZdZdZ d dZdddedefdZd Z d Z y ) BaseUserFunctionVariablec6|jjSr8get_code co_filenameselfs r@ get_filenamez%BaseUserFunctionVariable.get_filename}}***rBc6|jjSr8ryco_namer{s r@get_namez!BaseUserFunctionVariable.get_name}}&&&rBr=r2returncL|j|g|j||Sr8)inline_user_function_return self_argsr|r=argskwargss r@ call_functionz&BaseUserFunctionVariable.call_functions, --d4Ndnn6F4N4NPVWWrBrUcd} t|j|}t j j|S#t$r|dk(rt|trd}YAwxYw)NF__name__T)hasattr get_functionNotImplementedErrorr9NestedUserFunctionVariabler r0r<r|r=rUrHs r@call_obj_hasattrz)BaseUserFunctionVariable.call_obj_hasattrse T..0$7F))0088# z!j7Q&R s= A A cnttj|jjSr8)rDrm signaturer parametersr{s r@inspect_parameter_namesz0BaseUserFunctionVariable.inspect_parameter_namess'G%%d&7&7&9:EEFFrBciSr8r|r=s r@ closure_varsz%BaseUserFunctionVariable.closure_vars rBNr=r2rlist[VariableTracker]rdict[str, VariableTracker]rr.) r __module__ __qualname__r}rrstrr.rrrrrBr@rvrvsd+'X #X&X- X  X 9) 914 9  9GrBrvceZdZdZddhej ZedZddfd ZfdZ dZ d Z d Z d Z d Zd ZdeeeffdZdddefdZdddedefdZ dfd ZxZS)UserFunctionVariablez-Some unsupported user-defined global functionrr is_constantcft|jtj|||SNr?)r make_guardr CLOSURE_MATCHclsvaluer?s r@create_with_sourcez'UserFunctionVariable.create_with_sources(f'' (B(BCD5((rBrc ,t|di|t|ddrd|_nd|_t |t j tjjfsJdt|d|tj|d|}||_ y)N_dynamo_marked_constantFTzexpected FunctionType found  _torchdynamo_inliner)super__init__rqrr9r\rYtorchjitScriptFunctionr,rmrnrrr|rrrr __class__s r@rzUserFunctionVariable.__init__s "6" 20% 8#D $D "u115993K3KLM *72;-q = M  # #B(=r B&(rBcXt|tr |jSt|Sr8)r)rrrras_python_constantr|rs r@rz'UserFunctionVariable.as_python_constants% $, -77Nw)++rBcgSr8rr{s r@rzUserFunctionVariable.self_argsrrBc|jSr8rrr{s r@rz!UserFunctionVariable.get_function wwrBc.|jjSr8)rrrhr{s r@ryzUserFunctionVariable.get_codeswwrBc"tjSr8r\rYr{s r@ python_typez UserFunctionVariable.python_type!!!rBc4t|jddduSN__self__)rqrrr{s r@has_selfzUserFunctionVariable.has_selfstww D1==rBc.|jjSr8)rrrjr{s r@ get_globalsz UserFunctionVariable.get_globalssww"""rBc|jrJ|jj}tjt |}|j }|jxsg}t|D cgc])\}} |jdnt|j|+} }} tj|j|j|jt!t#|| D cgc]\} } || | c} } |j$} |j&r||j&Dcic])}||jdnt|j|d+}}|j&j)Dcic]\}}|||||c}}| _t+j,| j.|i|}|j1t3|j4j)}t7||t9|||j|j j$xsd}t;|t;|j jj<k(sJt#t?j@|j jj<|D]\}}}|jjB}||vr||}n|jrbtEtG|jd|}tG|d} tIjJ||jL|}|jU|||}n4 tIjJ||jL}|jUd||}|||<|Scc} }wcc} } wcc}wcc}}w#tN$rtQjR}YwxYw#tN$rtQjR}YuwxYw) z Assume `args` and `kwargs` are VariableTracker arguments for a call to this function, create new bindings for initial locals. )r=N)r>r?T)is_kwrri cell_contents)+rrMroot_tx functoolspartialrArrrk enumerater?r r\rYrhrjrrFziprir]rErmrbindapply_defaultsrG argumentsrKrWlen co_freevarsr^countrNr!rr.r:r ValueErrorr DeletedVariabletrack_cell_existing)r|rSrrrwraprrraidx_defaults_sourcesargr? fake_funcrIkwdefaults_sourcesrJr6rHrbrUcellrNcell_var closure_cellclosure_cell_contents contents_vars r@ bind_argszUserFunctionVariable.bind_argss ####--''  G<!%??(b$H- QKK'D^DKK-M M  && KK NN KK (+85E'F#VS0  NN     ** "  {{*' QdCD" "--335(Aq4A&8&;<<(I $ 2!!),114B6B eoo++-.&)ffbkk2''%%+7|s477#3#3#?#?@@@@" OO tww//;;W " $OCt"==55L|#'-,t{{M:C  )3<(Q%?#2#8#8 2 24I$L (;; $ ?#2#8#8ASAS#TL(;;D$ U#F4LE" $H U "(<"?#,#<#<#>L?"?#,#<#<#>L?s<*.M-M3$.M91M>;!N1 N'N$#N$'OOr=r2rUc|tvrtj||St||j|j |Sr8)r$r GetAttrVariablertrrr?r|r=rUs r@ var_getattrz UserFunctionVariable.var_getattr\s7 ) ),,T48 8b$''4;;==rBclt|j|}tjj |Sr8)rrrr r0r<rs r@rz%UserFunctionVariable.call_obj_hasattras+$'))0088rBc|jtjjurt j |jj |i|}|jd}t|ts)|j}td|jdt|ts|j}td|d|j}tj |dS|j"r't%||j|j||S|j&j(j*r|j&j(j,sq ddlm} | h|j| j4| j6fvrDtjj8j-|5t: |y|||cdddSt: |y|||S#t2$rd} YwxYw#1swY+xYw) Nrz=`nonstrict_trace` expects a callable, but got value of type <>z) Applying `nonstrict_trace` to function ; however, `nonstrict_trace` currently requires the function to be defined outside `torch.compile` region. Tnonstrict_traceable) FSDPState)rrr_dynamononstrict_tracermrrrr9rvrrrrrr TorchInGraphFunctionVariablerinvoke_and_store_as_constantrMcurrent_tracerunder_activation_checkpoint#allow_side_effects_under_checkpoint/torch.distributed.fsdp._fully_shard._fsdp_stater Exception _pre_forward _post_forwardrNrr) r|r=rrr6fn_vartypfn_namerrrrs r@rz"UserFunctionVariable.call_functiongs 77emm33 33G%%dgg.33TDVDEZZ]Ff&>?((*STWT`T`Saabcf&:; //+))0 2 B99"RVW W   /DGGT]]_dF  II $ $ @ @II,,PP !U$&&''5*]]//SSTVWC 70T6BCCw$Rv66 !  ! CCs%G&:G7& G43G47HFrNr)rrrrgrv_nonvar_fields classmethodrrrrrryrrrrGrr.rrrr __classcell__rs@r@rrs7  " 0 0N )))",  ">#Uc?6J1KUn>5>S> 9)9149 9 -7 #-7&-7- -7  -7-7rBrcJeZdZddfd ZedZ ddZxZS)BuiltinMethodVariablec jt|di|t|tjsJ||_yNr)rrr9r\BuiltinMethodTyperrrs r@rzBuiltinMethodVariable.__init__s0 "6""e55666rBc|j}|j}|tuxr|dk(xst|tuxr|dk(S)N__new__ __contains__)rrrFtype frozenset)obj method_self method_names r@is_supported_builtin_methodz1BuiltinMethodVariable.is_supported_builtin_methodsIll ll u$A )A   * L{n/L rBc|jj}|jj}|jxrt |jd}t j |||}|j||||Sr)rrrrr?rr.r: call_method)r|r=rrrrU obj_sourceobj_vts r@rz#BuiltinMethodVariable.call_functionsf gg&& ww[[HZ Z%H  &&r; C!!"dD&99rBrrr)rrrr staticmethodrrrrs@r@r r sG    : # :& :- :  :rBr c eZdZdejdedffd ZdZdZdZ dZ d Z d Zd Z e Z d Zd ZdZdZdZdZdZdej,fdZdeefdZdZdZdZdddeddddddf fd ZxZ S)!LocalGeneratorObjectVariablerT inline_tracerr2c Nt|di|||_||_||_yr )rrrTr`r)r|rTr`rrrs r@rz%LocalGeneratorObjectVariable.__init__s+ "6" "*rBc|jSr8)rTr{s r@ryz%LocalGeneratorObjectVariable.get_codes yyrBc6|jjSr8rxr{s r@r}z)LocalGeneratorObjectVariable.get_filenamer~rBc6|jjSr8rr{s r@rz%LocalGeneratorObjectVariable.get_namerrBctr8)rr{s r@rz)LocalGeneratorObjectVariable.get_functions!!rBcyNFrr{s r@rz%LocalGeneratorObjectVariable.has_selfrBc"|jSr8)rr{s r@rz%LocalGeneratorObjectVariable.__name__s}}rBcV|jjd|jdS)N())rrrr{s r@__str__z$LocalGeneratorObjectVariable.__str__s'..))*!DMMO+>rBcddlm}ddlm}m}m}|j }||}||}||} |5|5| 5|j|} | js|j||_ tj|jj|dddddddddy#1swYxYw#1swYxYw#1swYyxYw)Nr)"disallow_side_effects_in_generator)r2 save_and_restart_speculation_log(temporarely_allow_writes_to_output_graph)torch._dynamo.side_effectsr,torch._dynamo.symbolic_convertr2r-r. current_tx_get_inline_tracergenerator_exhaustedforce_unpack_var_sequenceremaining_itemsr ListIteratorVariable reconstruct) r|codegenr,r2r-r.r=savedisallowtemptracers r@r7z(LocalGeneratorObjectVariable.reconstructsQ  . " - - //35b97;  V8 VT V,,R0F--'+'E'Eb'I$  * *4+?+? @ L LW U  V V V V V V V V Vs;CC A"B=$C ,C=C C  C CCc<|jj|||Sr8)rrrrs r@rz&LocalGeneratorObjectVariable.bind_argssww  T622rBc|jSr8r`r{s r@rz(LocalGeneratorObjectVariable.get_globals ~~rBc"tjSr8)r\ GeneratorTyper{s r@rz(LocalGeneratorObjectVariable.python_types"""rBcpddlm}|j|j||gi|_|jSNr)InliningInstructionTranslator)r0rErbuild_inline_tracer)r|r=rEs r@r2z/LocalGeneratorObjectVariable._get_inline_tracers;P    %!>!R!RD"b"D !!!rBcd|j|}|jrtt| t j t dt di5|jcdddt dxxt dzcc<S#1swYnxYwnf#t$r}|d}~wt$rt$rC}tjjj|jt |d}~wwxYw t dxxt dzcc<y#t dxxt dzcc<wxYw)Nr inline_call)r2_is_generator_exhaustedr StopIterationr rGr% inline_call_rrrrr eval_frame skip_coderyr)r|r=r<es r@ next_variablez*LocalGeneratorObjectVariable.next_variables ((,  ' ' ) $]B 7 AH8O&PQ -**, - - _ %-)@ @ % - - -  G%   # MM $ $ . .t}} ? " # - _ %-)@ @ %H_ %-)@ @ %sM#BB& BB BD C8B  C85>C33C88DD/cyr$rrs r@has_unpack_var_sequencez4LocalGeneratorObjectVariable.has_unpack_var_sequencer%rBrcyNTrrs r@has_force_unpack_var_sequencez:LocalGeneratorObjectVariable.has_force_unpack_var_sequencesrBcg} |j|j|"#t$rt|Y|SwxYwr8)appendrOrr)r|r=rHs r@r4z6LocalGeneratorObjectVariable.force_unpack_var_sequencesL  d0045- )"-  s &>>c|j|} |j|y#t$r}|j|Yd}~yd}~wwxYwr8)r2_raise_exception_variablerexception_handler)r|r=excr<rNs r@_setup_exceptionz-LocalGeneratorObjectVariable._setup_exception%sH((, (  , ,S 1  (  $ $Q ' ' (s% A AA cT|jduxs|jjdk(S)Nr)rinstruction_pointerr{s r@_is_generator_just_startedz7LocalGeneratorObjectVariable._is_generator_just_started.s*!!T)XT-?-?-S-SWX-XXrBc0t|jddS)Nr3F)rqrr{s r@rIz4LocalGeneratorObjectVariable._is_generator_exhausted1st))+@%HHrBr=rUrrrrr.c|dk(r|j|S|dk(r|S|dk(rp|jr-t|r"td|Dst t ||j |}|j||j|S|dk(r|j |}|js|jrd|_ tjdS |j|tjtdtj dk\r5|j"j$d k(rd|_ tjdS |j|rt t(|n|d k(r|j |} |j|t|d k(r|d n|d |j|}t1dt2fi} |j|tj|d|j|t t(||St6|q||||y#t&$rd|_ tjdcYSwxYw#t&$rd|_ tjdcYSt*$r|j,J|j,cYSwxYw#t.$rwxYw#t5|$rY|SwxYw)N__next____iter__sendc3\K|]$}t|txr|jdu&ywr8)r9r0r).0rs r@ z;LocalGeneratorObjectVariable.call_method..Gs1s$45K#))t:KKs*,closeTr) CALL_INTRINSIC_1throwrhr+r__InternalThrowException)rOr^rallr TypeErrorr2 push_manyrIr3r r0r[ExceptionVariable GeneratorExitsys version_infonext_instructionopnamer RuntimeErrorrsymbolic_resultrrrrrr) r|r=rUrrr<retvalexc_typers r@rz(LocalGeneratorObjectVariable.call_method4s : %%b) ) Z K V^..0SY#-Y;,,R0F   T "%%b) ) W_,,R0F..0D4P4P4R-1* 11$77 8%% 33M2F($$///66:LL15F.$55d;; .%%b),\2>W_,,R0F  %%bSY!^$q'aQ ''+F|6 bIH ?%%b)*E*EhPR*ST""2& )v>M BdF3a) 8-1* 11$77 8) 8-1* 11$77, . --999---  .,%  N1: M  sI'A1H;!I&(J57K;%I#"I#&%J2 "J21J25 KKK)!rrrr\CodeTyperrryr}rrrr*__repr__r7rrrr2rOrQbuiltinsboolrTrDr.r4r[r^rIrrrrs@r@rrs +nn + 78 ++'"?HV&3#"A.8==tO/D(YIz4 #z4z4& z4 - z4  z4z4rBrceZdZdZy)4ContextlibContextManagerLocalGeneratorObjectVariablez .. note:: This is only used when the function is annotated with @contextlib.contextmanager It is a special case of a generator function as we do not allow return a context manager from a torch.compile function. N)rrrrgrrBr@rrsrBrcReZdZdZeddeffdZdZdZ ddZ xZ S) LocalGeneratorFunctionVariablezxfunctions that behaves like iterators .. note:: This is a wrapper around (Nested)UserFunctionVariable ) generator_clsvtc @t|di|||_||_yr )rrrr)r|rrrrs r@rz'LocalGeneratorFunctionVariable.__init__s$ "6"*rBc||jjjvr t||St|j|Sr8)r__dict__keysrqr)r|rUs r@ __getattr__z*LocalGeneratorFunctionVariable.__getattr__s; 4>>**//1 14& &tww%%rBc6ddlm}|j||||SrD)r0rErF)r|r=rrrEs r@_build_inline_tracerz3LocalGeneratorFunctionVariable._build_inline_tracers$P,@@       rBct|jjsJ|j|||}|jj}|jj }|j ||||j Sr)rrryrrrr?)r|r=rrrrTr`s r@rz,LocalGeneratorFunctionVariable.call_function s DGG,,.///11"dFC ww!GG'') !!   ;; "  rBr) rrrrgrr.rrrrrrs@r@rrsO3 +  +&   # & -    rBrc,eZdZdZfdZfdZxZS)3FunctionDecoratedByContextlibContextManagerVariablezm .. note:: This is only used when the function is annotated with @contextlib.contextmanager c 2t||fdti|y)Nr)rrr)r|rrrs r@rzs%   N  rBct||||}t|tjj j sJd|_|SrS)rrr9rrsymbolic_convert&InliningGeneratorInstructionTranslatoris_generator_from_ctx_manager)r|r=rrr<rs r@rzHFunctionDecoratedByContextlibContextManagerVariable._build_inline_tracerEsO -b$?  MM * * Q Q   04, rB)rrrrgrrrrs@r@rr5s    rBrcveZdZdZd fd ZdefdZdZdZ dfd Z fd Z ddd effd Z xZ S)UserMethodVariablez$Some unsupported user-defined methodrc 6t|dd|i|||_y)Nrrr)rrr)r|rrrrrs r@rzUserMethodVariable.__init__Vs )B)&)rBch|jjd|jd|jdS)Nr(z, r))rrrrrr{s r@r{zUserMethodVariable.__repr__Zs...))*!DGG9BtxxjBBrBc|jgSr8)rr{s r@rzUserMethodVariable.self_args]szrBc"tjSr8)r\ MethodTyper{s r@rzUserMethodVariable.python_type`srBr=r2cddlm}||jrHg|j|}t j |jd}|j |||S|jjrt|jtjryt|jdd}||jdr|dk7s |jr|jj||jj |||j St"d|jt"j$j&ur>t j(|jj ||jg||S|jrQt|jj*|jj }t-|||j/||St0 ||||S) Nr )is_nonstrict_trace_callableTrrz torch.nn.ztorch.nn.utils.parametrize)constant) trace_rulesrrrrr rrrMis_root_tracerr9rNNModuleVariablerq startswithrrrr1FSDPParamGroupuse_training_stateTorchCtxManagerClassVariablerrrr) r|r=rrr call_argsvar module_attrrrrs r@rz UserMethodVariable.call_functioncs > &tww /2$..*2T2I88TC$$RF; ; 99 # # %* HHi00+ "$''<DKK!> : 88O : "((TWWf= =w"2t,,rBrr) rrrrgrrr{rrrrrrrs@r@rrSsm.C#C ;7 #;7&;7- ;7  ;7z5-5-S--rBrc<eZdZdfd Z dfd ZxZS)WrappedUserMethodVariablec |jdd|jddt||j|jfi|||_||_yNrrr)rRrrrrrwrappedcontextr|rrrrs r@rz"WrappedUserMethodVariable.__init__sI 4 5$ W[[;F;  rBc|jj|t| |||}|jj ||Sr8renterrrexitr|r=rrrHrs r@rz'WrappedUserMethodVariable.call_functionA 2&r48 " rBrrrrrrrrrs@r@rr8  # & -    rBrc<eZdZdfd Z dfd ZxZS)WrappedUserFunctionVariablec |jdd|jddt||jfi|||_||_yr)rRrrrrrrrs r@rz$WrappedUserFunctionVariable.__init__sC 4 5$ .v.  rBc|jj|t| |||}|jj ||Sr8rrs r@rz)WrappedUserFunctionVariable.call_functionrrBrrrrs@r@rrrrBrcd}|Dcgc] }|| }}|jDcic]\}}|||}}}||i|} |jj| |t|Scc}wcc}}w)Ncvt|tjr|jS|j Sr8)r9r TensorVariableget_real_valuer)xs r@convertz-invoke_and_store_as_constant..converts0 a11 2##% %##%%rBr)rErMregister_attr_or_moduler) r=rrrUrrrrrIrJress r@rrs& !% %1GAJ %D %(.  71am 7F 7 d f C 99 , ,  d# -  & 7s A1A6czeZdZdhejZ d d fd ZdZdZdZdZ dZ dZ d Z d Z d ZxZS)rr`c  Lt |di| t|jtsJt|jt j sJt|tsJ||_||_ ||_ ||_ ||_ ||_ ||_||_yr )rrr9rrr\rzrGrrTr`rarcrdrb wrapped_fn) r|rrTr`rarcrdrbrrrs r@rz#NestedUserFunctionVariable.__init__s "6"'446<<<$113U^^DDD)T***  "  $& 5?rBcgSr8rr{s r@rz$NestedUserFunctionVariable.self_argsrrBc6|jjSr8)rTrr{s r@ryz#NestedUserFunctionVariable.get_codesyy++--rBc"tjSr8rr{s r@rz&NestedUserFunctionVariable.python_type rrBcD|jrttj|jj |j |jj }|jr|jj |_ |jr|jj |_ |jrZ|jj }t|trddlm}t#||}t|t"sJ||_|S)NrrZ)rbrr\rYrTrr`rrarkrcr]rdr9rFr^r[rGr_)r|rerdr[s r@rz'NestedUserFunctionVariable.get_functions <<% %!! II ( ( * NN LL + + -  == $ @ @ BD  ??"&//"D"D"FD    **==?K+u-."8K#89 k40 00#.D  rBc|jduSr8)rbr{s r@ has_closurez&NestedUserFunctionVariable.has_closure&s||4''rBcyr$rr{s r@rz#NestedUserFunctionVariable.has_self)r%rBc|jSr8r?r{s r@rz&NestedUserFunctionVariable.get_globals,r@rBcH|j}tj||j|jj |j rt|j jndtdtt|jjD}|jr|jj|_tj |j"|i|}|j%t'|j(j}t+|j,j.|t1|||t3|jD])\}} | |vsJ|j4j|} | || <+|S)Nc32K|]}tdywr8)r*)rers r@rfz7NestedUserFunctionVariable.bind_args..6sSa)D/Ss)ryr\rYr`rrrarFrErangerrrckeys_as_python_constantr]rmrrrrGrrKrMrrWrrb) r|rSrrrTrer6rHrrUrs r@rz$NestedUserFunctionVariable.bind_args/sH}}!!  NN LL + + -*.--E$--%% &T S5T]]_5P5P1Q+RS S   ??"&//"I"I"KD ,!!$',,d=f= eoo++-...7ffd+"4#3#34 ICv% %%<<%%c*DF4L  rBcjfd|jjj|jgt j |jjj|jr|jn!jjdg|jr|jn!jjdg|jr|jn!jjdg|jr= |jj}jj|gn!jjdgjt!dd|j"rwjfd|j"jt!ddjt%djt!ddyy#t$r|jYwxYw)Nc0jtdS)Nrf)load_import_fromrr8sr@z8NestedUserFunctionVariable.reconstruct..IsG,,X7JKrBFc(jddS)Nrwrapsrrsr@rz8NestedUserFunctionVariable.reconstruct..ms00gFrBr+r T) add_push_nullrT extend_outputcreate_load_const_uncheckedr`r0r<rrracreate_load_constrbrcrdrrrrr)r|r8rds ` r@r7z&NestedUserFunctionVariable.reconstructGs K   wBB4>>RST '' (?(?@A == DMM "  ! !7#<#z?Dynamo developers have intentionally marked that the function `z ` in file `z` should not be traced.zAvoid calling the function `z`.rzRemove the function `z` or the file `zv` from torch/_dynamo/trace_rules.py. More graph breaks may occur as a result of attempting to trace into the function.z Please file an issue to PyTorch._abc _warningsz6Dynamo does not know how to trace the Python builtin `.zIf you are attempting to call a logging function (e.g. `_warnings.warn`), you can try adding it to `torch._dynamo.config.reorderable_logging_functions`.zKPlease file an issue on GitHub so the PyTorch team can add support for it. optreez*Dynamo cannot trace optree C/C++ function zi Consider using torch.utils._pytree - https://github.com/pytorch/pytorch/blob/main/torch/utils/_pytree.py z/Dynamo does not know how to trace the builtin `z.` This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind).zIf it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround.aIf it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use `torch.compiler.allow_in_graph`.allow_in_graphzFound an allow_in_graph decorator to a function which is created inside the parent function that is getting compiled. This is not supported for now.zz,Attempted to call function marked as skippedzmodule: z , qualname: z, skip reason: )rmrnrrrrr graph_breakgetrr9r\WrapperDescriptorTypeutils warn_oncerrqgetfilernrrjoinr) r|r=rrgraph_break_msgrqualnamepathrrknown_python_builtin_modulesrs r@rz"SkipFunctionVariable.call_functionsn  ! !$**.De L KDJJ5djj\Auv>  ZZ5==44 4$jj5O"1"D"D"F ?J4&P[\b[ccdeB?BSTD   E$?$? @DTZZLQ?@  MM   ) )# . # tzz>;OPH< Ytzz2UV^U_` $v%<> 38*B? D(/zOAA; E`++? $(KKT[[5GF F"4::#8#8"9hZ_e^fg'  c( Y06 /D,::((,HH JJ112!H:RA iGEJJ))5 --88B$NtzzOdOdNeefgofppq"rK^E MM''11+2DtyyQVGW2WXJ$**J_J_I``abjaklab m;EMM''11+2DtyyQVGW2WXQ( Ys:AG ELLchtjjt|j|Sr8)r r0r<rrrs r@rz%SkipFunctionVariable.call_obj_hasattrs$))00T1JKKrBrUc|tvrtj||St||j|j |Sr8)r$r rrtrr?rs r@rz SkipFunctionVariable.var_getattrs7 ) ),,T48 8b$**dkk4@@rBr8rrr=r2)rrrr.rrrrrrrrrrrs@r@rrus  ' 'N  ))o #o&o- o  obLA5ASArBrcHeZdZdZdfd Zdfd Z ddZxZS)WrapperUserFunctionVariablea3 Used to represent a wrapper object that contains the actual callable as an attribute. For example, torch.jit.script/trace have the original function at their _torchdynamo_inline attribute. Similarly, functions with __script_if_tracing_wrapper have the original attr at "__original_fn". c @t|di|||_||_yr )rr wrapper_obj attr_to_trace)r|r r rrs r@rz$WrapperUserFunctionVariable.__init__s# "6"&*rBc||jk(r[t|j|j}|jxrt |j|}t j |||St|!||Sr8) r rqr r?rr.r:rr)r|r=rUr>r?rs r@rz'WrapperUserFunctionVariable.var_getattrsi 4%% %$**D,>,>?C[[BZ T%BF"((S&9 9w"2t,,rBctjtjj ||tj |j g||Sr8)r rr getattr_and_tracerr0r rs r@rz)WrapperUserFunctionVariable.call_functionsL --  ' ' - y11$2D2DEMMv  rBrrr)rrrrgrrrrrs@r@rrs=+ -  #  &  -    rBrcRtjjrddlm}|SiS)Nrtraceable_collective_remaps)r distributed is_available)torch.distributed._functional_collectivesrrs r@_traceable_collective_remapsr*s& %%' +* IrBctjjsJd|tj vsJ|j }|j d}t||S)NzIllegal invocation.z)torch.distributed._functional_collectives)rrrrvaluesr import_sourcer)r=rr inner_name path_sources r@_traceable_collectives_sourcer5sb    ) ) +B-BB + -/668 88 8J""#NOK k: ..rBcpeZdZdZdfd ZeddZedZeddZ d dZ xZ S) !CollectiveFunctionRewriteVariablea Some of the torch.distributed.* collective APIs are possible to rewrite to 'traceable' collectives. This class provides both a way to check if a function is remappable, and perform the remapping. In the case that a function is 'remappable' but only for some combinations of call-time arguments, we check the args at `call_function` time and fall back to graph-breaking if needed. This is no worse than status-quo as we currently graph-break on all distributed.* collectives. c Xt||fi|t|tsJ||_yr8)rrr9rreplacement_var)r|rrrrrs r@rz*CollectiveFunctionRewriteVariable.__init__Is. &v&/+?@@@.rBc jtj||\}}t|ft|fd|i||d|S)Nr?)rr?)rrewriter)r=old_fnr?optionsnew_fn new_sources r@r<z(CollectiveFunctionRewriteVariable.createNsM>FFr6R 0  0V VgV    rBcHtj|xr |tvSr8)rm isfunctionr)variables r@ can_rewritez-CollectiveFunctionRewriteVariable.can_rewriteXs%   x ( WX9U9W-W rBc8t|}|t||fSr8)rr)r=rrr#s r@r z)CollectiveFunctionRewriteVariable.rewrite^s"-/34R@@@rBczddlm}ddlm}t j |j }t|j|i|j}d}d|vr+|djrtd|j |j |j|j|jfvro|jd}| |j n|j"dj$}||vrt'd|t(j*j-|||d<|j.j1|||S)Nr)REDUCE_OP_TO_STRrasync_opzBCollectiveFunctionRewriteVariable can't support async_op=True for opzUnsupported all_reduce op: )torch.distributedrrr+rmrrrrGrrrr all_reducereduce_scatter_tensor_reduce_scatter_baserrrdefaultrr r0r<rr) r|r=rrdistr+r reduce_op_var reduce_ops r@rz/CollectiveFunctionRewriteVariable.call_functioncs< )N%%dgg. ninnd5f5??@  F:$6$I$I$K TUYU\U\T]^  77 OO  & &  % %  #JJt,M!,##))$/77   00 #>yk!JKK$55<< +F4L##11"dFCCrBrrr) rrrrgrrr<r(r rrrs@r@rr>s{/     AA'D #'D&'D- 'D  'DrBrc0eZdZ dfd ZxZS)FunctoolsWrapsVariablec||s(tdk(rfd}tj|St|||S)Nr+ct|tjr|jdSt d|dy)Nr)rzfunctools.wraps(r))r9r rcloner)rrrs r@rz3FunctoolsWrapsVariable.call_function..wrapss:b)"F"FG88tAw877 0A67rB)rr LambdaVariablerr)r|r=rrrrs ` r@rz$FunctoolsWrapsVariable.call_functions> #d)q. 8 ++E2 2w$Rv66rBr)rrrrrrs@r@r7r7s37 #7&7- 7  77rBr7c*eZdZdZ ddZy)CollectionsNamedTupleFunctionc|jSr8rr{s r@rz0CollectionsNamedTupleFunction.as_python_constantrrBc Ft||}|r}|j|Dcgc]}|jc}i|jDcic]\}}||jc}}}t j |t Stdycc}wcc}}w)N) mutation_typez!namedtuple with non constant args)r"rrrrEr UserDefinedClassVariabler-r) r|r=rr constant_argsrrIrJrs r@rz+CollectionsNamedTupleFunction.call_functions ,D&9 DGG267Q!&&(79?HA1a**,,HE55%5%7  9: 8Hs B BNr)rrrrrrrBr@r=r=s2; #;&;- ;  ;rBr=ceZdZdeddffd ZdZdZdZ dd Zdd d e defd Z dd d e fd Z dZ dZ xZS)FunctoolsPartialVariablererNc t|di|||_t|tsJ||_t|t sJ||_tjt|_ yr ) rrrer9rDrrGkeywordsrrr& fake_value)r|rerrFrrs r@rz!FunctoolsPartialVariable.__init__s\ "6" $%%% (D)))  $++H5rBc"tjSr8)rrr{s r@rz$FunctoolsPartialVariable.python_types   rBc<jfd|j|jrj|j|js2j t t|jdzdyj|jjt|jj}j jt|t|jzdz|dy)Nc(jddS)Nrrrrsr@rz6FunctoolsPartialVariable.reconstruct..sg&>&>{I&VrBr+F) rrerforeachrFrrrrrFrcreate_call_function_kw)r|r8rs ` r@r7z$FunctoolsPartialVariable.reconstructsVW  99 OODII &}}  ! !"6s499~7I5"Q R  ,,./T]]'')*  + +CIDII,F,JDRW X rBc"|jSr8rr{s r@rz%FunctoolsPartialVariable.get_function&&((rBr=r2cz|j|z}i|j|}|jj|||Sr8)rrFrer)r|r=rr merged_args merged_kwargss r@rz&FunctoolsPartialVariable.call_functions> ii$& 34==3F3 yy&&r; FFrBrUctjjtt j t |Sr8)r r0r<rrrr&rs r@rz)FunctoolsPartialVariable.call_obj_hasattrs2))00 I%%h/ 6  rBc|jxrt|j|}|dk(r |jS|dk(r!tj|j |S|dk(rW|j jDcic]\}}tj||}}}tj||Stt|ycc}}w)NrerrrF) r?rrer ListVariablerrFrEr0r<ConstDictVariablerro)r|r=rUr?rIrJrEs r@rz$FunctoolsPartialVariable.var_getattrs>DKK!> 6>99  6>))$))FC C : ?C}}?R?R?TUtq!%,,Q/2UEU..uVD D 4Vs> C c :tj|jjg|jDcgc]}|jc}i|j j Dcic]\}}||jc}}Scc}wcc}}wr8)rrrerrrFrE)r|rrIrJs r@rz+FunctoolsPartialVariable.as_python_constants  II ( ( * 26)) <3c$$& < 6:]]5H5H5JKTQq!&&((K  <K B 1Bc :tj|jjg|jDcgc]}|jc}i|j j Dcic]\}}||jc}}Scc}wcc}}w)zcSimilar to as_python_constant(), but add ID_MATCH guards to try to force things to become constants)rrreguard_as_python_constantrrFrE)r|rJrIs r@rZz1FunctoolsPartialVariable.guard_as_python_constants  II . . 0 48II >qa((* > <@==;N;N;PQ41aq!,,..Q  >QrXr)rrrr.rrr7rrrrrrrZrrs@r@rDrDs 6_ 64 6!  )G #G&G- G  G ) 14   55 5S 5  rBrDceZdZdddhejZeejddee de fe jffdZ edZdeddffd Zedefd Zd Zd Z dd Z ddZdZxZS)PolyfilledFunctionVariablerrr traceable_fnNr.ciSr8r)rs r@_get_polyfill_handlersz1PolyfilledFunctionVariable._get_polyfill_handlers s  rBcft|jtj|||Sr)rrrrrs r@rz-PolyfilledFunctionVariable.create_with_sources(f'' (C(CDE5((rBc *t|di|||_|jj ||}t |s Jd|d|dD]"}t ||d}|st |sJ|}ntd|d||_||_ y)NzPolyfill handler z is not callable for )__torch_dynamo_polyfill____python_implementation__z# does not have a traceable functionr) rrrrr_rcallablerqrvrr])r|rrrhandlercandidate_attr candidater]rs r@rz#PolyfilledFunctionVariable.__init__s "6"--/33B; X$5gY>STVSW"XX  N >I ***(  #G9,OP & ,rBc|jSr8)r]r{s r@ polyfill_fnz&PolyfilledFunctionVariable.polyfill_fn/s   rBc0t|jddS)N*__torch_dynamo_can_constant_fold_through__F)rqrr{s r@can_constant_fold_throughz4PolyfilledFunctionVariable.can_constant_fold_through3s OOI5  rBc"|jSr8rNr{s r@rz'PolyfilledFunctionVariable.get_function8rOrBc .|jrt||rt|j|Dcgc]}|jc}i|j Dcic]\}}||jc}}}t j ||S|jtjur&t|dk(r|st|dtjtjfrtd|djDrtjj!||j"j%dt&j(t+d|djDfit'j(|djDcgc]4}t|tj,r |j.n |j06c}St j ||j2}|j5|||Scc}wcc}}wcc}w)Nr+rc3K|]n}t|tjxrt|jtxs2t|tj xr|j tupywr8)r9r r0rintSymNodeVariabler)rers r@rfz;PolyfilledFunctionVariable.call_function..RseAy99:Wz!''SV?WYq)";";<WTWAWYsA4A6rc3<K|]}|jywr8)as_proxy)reas r@rfz;PolyfilledFunctionVariable.call_function..]s?A1::<?s)sym_num)rlr#rrrrEr.r:r|sumrr9r rU TupleVariablermrqr<rM create_proxyrsym_sumrFr0rrur]r) r|r=rrrrIrJrHtraceable_function_variables r@rz(PolyfilledFunctionVariable.call_function;s  ) ) +0M &1 6:;a**,;=C\\^LTQq!..00L  #((V4 4 GGx|| #D Q47Y%;%;Y=T=T$UVa ,,33 &&#MM?a??A   "&a   *!Y-G-GHGG!"+ 4 ('6&;&;B@Q@Q&R#*88T6JJO<L4sHH 9Hcb|dk(r|j|||St|j|d}|Jd|d|jt|sJd|d|ji}|jrt |j||d<t |fi|}|j|||S)N__call__zMember z not found in z is not callable in r?)rrqrrr'r?rr\)r|r=rUrrmethodr"polyfilled_method_variables r@rz&PolyfilledFunctionVariable.call_methodos : %%b$7 7$-!JWTF. #JJ!6"QgdV3Gy$QQ" ;; *4;; =GH %?%R'%R")77D&IIrBc|jSr8rr{s r@rz-PolyfilledFunctionVariable.as_python_constantrrBrrrrrrr.)rrrr.rrr lru_cacherGrrr\rYr_rr5rpropertyrirlrrrrrrs@r@r\r\s   ' ' NYtHS#X,>@R@R,R'S)) -2-D-.!R!! )2K #2K&2K- 2K  2KhJ& J - J  J&rBr\ceZdZdZy)TracebackVariablecyr8rrs r@rzTracebackVariable.call_functionsrBN)rrrrrrBr@rrs2rBrc0eZdZfdZdZdZdZxZS)SysFunctionVariablec 2t|di|||_yr )rrr)r|rrrs r@rzSysFunctionVariable.__init__s "6" rBcht|jrK|jd}|j}d}tj|||tj||g}n>t j dt j dt j dg}t j|S)N)r exn_vt_stackryr.r:r r0rw)r|r=exnrtbrEs r@exc_infozSysFunctionVariable.exc_infos r //"%C,,CB%%b#.%%b"-E**40**40**40E &&u--rBc>|j|jdSr)rrErs r@ exceptionzSysFunctionVariable.exceptions}}R &&q))rBc|jtjur|j|S|jtjur|j|St d|jj y)Nzsys.)rrrrrrrrs r@rz!SysFunctionVariable.call_functionsZ :: %==$ $ ZZ3== (>>"% % D!4!4 56 7rB)rrrrrrrrrs@r@rrs.$*8rBr)TMADescriptorMetadataTritonHOPifierceZdZdedefdZdedefdZdedefdZ de e jjjdffd Zd Zd Zd Zd ZdedefdZdddeeddfdZdefdZy)DynamoTritonHOPifierrrct|r8)r)r|rs r@raise_unsupportedz&DynamoTritonHOPifier.raise_unsupporteds #rBmaybe_callablec.t|ttfSr8)r9rr)r|rs r@ is_callablez DynamoTritonHOPifier.is_callables 79MN  rBr>c|jSr8r)r|r>s r@ get_valuezDynamoTritonHOPifier.get_values yyrB.cvddlm}t||r|jSt dt |y)Nr+)BaseListVariablezgrid for the triton kernel is )listsrr9rsrr)r|gridrs r@ check_gridzDynamoTritonHOPifier.check_grids0+ d, -==? " :4:,G HrBc|jDcic]%\}}tjj||'}}}|j ||gi}|Scc}}wr8)rEr r0r<r)r|rmetar=rIrJs r@ call_gridzDynamoTritonHOPifier.call_gridsVDHJJLQDAq **11!4a7QQ!!"tfb1 Rs*Ac\ddlm}|j||}|j|||}|S)Nr+)SourcelessBuilder)builderrr<r) r|user_fnrrr=r'rwrapped_user_functionrHs r@call_user_defined_fnz)DynamoTritonHOPifier.call_user_defined_fns1. 1 8 8W E&44RvF rBclddlm}||t|j|j |}|S)Nr+)VariableBuilder)rrr kernel_source_wrap)r|user_objr=r'rUrwrapped_user_objs r@wrap_user_defined_objz*DynamoTritonHOPifier.wrap_user_defined_objs7,*  811dV= %/  rBcl|j|}|Dcgc]}|j}}|Scc}wr8)unpack_var_sequencerZ)r|configsr=configs r@maybe_unpack_configsz)DynamoTritonHOPifier.maybe_unpack_configss=--b1DKK6224KKLs1rHcd|js|jd|jS)Nz`@triton.heuristics must return constant values because configs can only contain constant values.)is_python_constantrrZ)r|rHs r@maybe_unpack_heuristic_resultz2DynamoTritonHOPifier.maybe_unpack_heuristic_results0((*  " "r ..00rBr'TritonKernelVariablerc|jt|dk7r|jdt||j|j |d|j S)Nr+z7Triton kernels should be called with only a single gridr)kernel kernel_idxrr)rrrrrrr?)r|r'rs r@ call_getitemz!DynamoTritonHOPifier.call_getitems\ == $D Q  " "I tH~??**a"//   rBc $ddlm}ddlm}i}t |j D]D}||} t | ts| j||<| jj||<F|jD cic]%\}} tjj|| '} }} ddlm} m} |jD cic]"\}} t | |r|| j#$} }} | jD cic]\}} t | |s|| }}} |j%D][} | j'} t | tj(tj*fr>|j-dt/| d]| j1| }||t2}|j4j7d| d |j8||||j;d tjdScc} }wcc} }wcc} }w) Nr+r/)rVr)kernel_side_tabletriton_kernel_wrapper_mutationz.Unexpected argument type for a Triton kernel: rrr)rconstant_args_idxrtma_descriptor_metadatar)rr0dictsrVrDrr9TMADescriptorVariable to_metadatadata_ptr from_tensorrEr r<*torch._higher_order_ops.triton_kernel_wraprrrrrealizerrqrrepradd_constant_argsrGrMrxrrs)r|r'gridscombined_args_rawr=r0rVrrIrJ combined_argsrrrBnon_constant_argsrrs r@call_HOPzDynamoTritonHOPifier.call_HOPs.,:<',,./ >A!!$A!23-.]]_'*'(zz'='=!!$  >*//1 1  & & - -a 0! 3   *//1 1!-. q##% %  &++- 1a!12 qD  #))+ A Aa)":":I "6"(66tVZQUVrBc2tj||||Sr8)rcall_triton_kernelrs r@rz"TritonKernelVariable.call_functionSs 0BB $  rBc|dk(rtj||S|dk(rtj||||St|||||S)N __getitem__run)rrcall_runrr)r|r=rUrrrs r@rz TritonKernelVariable.call_method]sW = 3@@tL L U]3<? ? rBrrr) rrrr_rrprrrrrrrs@r@rrHs   W  # & -     ;& ; - ;  ;ssrBrc<eZdZ dfd ZdZdZxZS)rc t|tjsJt|di|||_||_||_||_yr ) r9r DataPtrVariablerrrdims block_dims element_size)r|rrrrrrs r@rzTMADescriptorVariable.__init__wsH(I$=$=>>> "6"   $(rBc|jDcgc]}|jc}|jDcgc]}|jc}|jjfScc}wcc}wr8)rrsrr)r|dims r@rz!TMADescriptorVariable.to_metadatasT'+yy 1S\\^ 1'+ 7S\\^ 7    & & (  1 7s A*A/cjfdjjgjjj }j |jt|dzdy)NcVjddtjdS)Nz$triton.tools.experimental_descriptorcreate_d_tma_descriptor)rrr)r8r|sr@rz3TMADescriptorVariable.reconstruct..s+G,,6#dii.))9:rBr+F) rrr7rrrrKrr)r|r8rs`` r@r7z!TMADescriptorVariable.reconstructsp  !!'*@@T__@d.?.?@c$i!mU3rB)rzvariables.DataPtrVariablerlist[ConstantVariable]rrrr0)rrrrrr7rrs@r@rrvs8 )- )' )- ) ) )  4rBrc@eZdZdeddffd Z ddZxZS)CreateTMADescriptorVariablerankrNc >|dvsJt|di|||_y)N)r+r r)rrr)r|rrrs r@rz$CreateTMADescriptorVariable.__init__s( v~~ "6" rBcd|vr|dn|d}t|tjstd|jd|jdk(r;t |t |zdk(sJd|vr|dn|dg}d|vr|dn|d g}nTt |t |zd k(sJd |vr|d n|dd |vr|d n|d g}d |vr|d n|dd|vr|dn|dg}d|vr|dn|d}t ||||S)Nptrrz8Please ensure there were no graph breaks between create_z3d_tma_descriptor and the upstream .data_ptr() call.r+r block_dimr dim1dim0 block_dim1rh block_dim0rr)rrrr)r9r rrrrr)r|r=rrrrrrs r@rz)CreateTMADescriptorVariable.call_functionsa %fUmDG#y889))%$$  99>t9s6{*a/ //!&&u d1gD(3f'<{#$q'Jt9s6{*a/ //"(F"2vQ"(F"2vQD )5(>|$DG(4(>|$DGJ2@61Ivn-tTVx $!%   rBr)rrrrprrrrs@r@rrsE  ( #( &( - (  ( rBrr8r)nrgr|rrmr^rrr\collections.abcrtypingrrrrrtyping_extensionsr unittest.mockr rrr r bytecode_transformationrrrrZrrrrrrrrrrrguardsrrr?rrr r!rr"r#r$r%r&r'r(r)r*baser,r-r.rr0#torch.distributed.fsdp._fully_shardr1ModuleNotFoundErrorr0r2rr3r4r5rArKrGrrWrfrprtrvrr rrrrrrrrrrrrrrr7r=rDr\rrrrrrrrrrrBr@rsR. $BB# #VV    1NN   =<&E D  T" A. $sO';"< "(-"DF73F7R:4:N b$8#9 +?+\!4O!4H2 /2 c:s&HHH