[^(h#ddlZddlZddlZddlZddlZddlmZddlm Z m Z m Z m Z ddl mZddlZddlmZddlmZddlmZmZmZmZmZmZddlmZdd lmZed e fZ!ed e fZ"ejFe$Z%Gd d ejLZ'e'jPde'jRde'jTde'jVde'jXde'jZde'j\dddiZ/e0e/Dcic]\}}|| c}}Z1GddejLZ2e2Dcic]}||jfc}Z4ejjdddGdd Z6ejjGddZ7ejjdddGdd e6Z8ded eee e8e e8ffd!Z9ded ee8fd"Z:ded eee e8e8ffd#Z;d$e ed eed%ffd&Z<Gd'd(Z=Gd)d*Z>Gd+d,Z?ejjGd-d.Z@Gd/d0ZAGd1d2ZBejjGd3d4ZCejjGd5d6ZDGd7d8ZEGd9d:ZFycc}}wcc}w);N)Iterator)AnycastOptionalUnion)Literal)FunctionSchema)_ProfilerResult) _EventType_ExtraFields_Allocation_ExtraFields_TorchOp_ProfilerEvent_TensorMetadata RecordScope) _element_size)_utilsKey TensorKeyceZdZejZejZejZejZejZ ejZ ejZ y)CategoryN) __name__ __module__ __qualname__enumautoINPUT TEMPORARY ACTIVATIONGRADIENTAUTOGRAD_DETAIL PARAMETEROPTIMIZER_STATE]/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/torch/profiler/_memory_profiler.pyrr s] DIIKE IJtyy{HdiikO IdiikOr$r darkgreen goldenrodblack mediumpurplered mediumblue royalbluegreyceZdZejZejZejZejZy)ActionN) rrrrr PREEXISTINGCREATEINCREMENT_VERSIONDESTROYr#r$r%r/r/8s7$))+K TYY[F! diikGr$r/TF)eq unsafe_hashfrozenc,eZdZUejed<y)rdeviceN)rrrtorchr8__annotations__r#r$r%rrBs LLr$cNeZdZUdZeed<eed<defdZdede fdZ defdZ y ) _StoragezBundle storage pointer and id. All profiling logic should use `allocation_id`, however it is useful to print storage pointers for debugging and unit tests sometimes look up values using the storage data pointer of a live Tensor.ptr allocation_idreturncNt|jdd|jdS)Nz>18 ())hexr=r>selfs r%__repr__z_Storage.__repr__Rs'dhh-$Bt'9'9&:!<rErGs r%__eq__z_Storage.__eq__Us%%*Xt/A/AUEXEX/XXr$c,t|jSrI)hashr>rDs r%__hash__z_Storage.__hash__XsD&&''r$N) rrr__doc__intr:strrFobjectboolrLrOr#r$r%r<r<GsB? H=#=YFYtY(#(r$r<c eZdZUdZeed<eed<defdZddde fdZ e de ed e ed e ed e jde df d Zed ede dfdZede ede dfdZedeeeeeffdZy)ra~Hashable identifier for a storage which has been asigned an ID. A detailed description of Tensor IDs and why they are needed is given in `torch/csrc/profiler/collection.h` when `TensorID` is declared. To summarize, multiple Storage buffers can map to the same logical Tensor. This dataclass is used to refer to a concrete in-memory StorageImpl of a Tensor. idstorager?cjd|jdt|jdd|jdS)Nzid=z: z<24rArB)rVreprrWr8rDs r%rFzTensorKey.__repr__js1TWWIRT\\ 237r$++aHHr$rGc4|j|jkSrI) _as_sortablerKs r%__lt__zTensorKey.__lt__ms  5#5#555r$ tensor_id storage_ptrr>r8c>|||t||t||SyrI)rr<)r]r^r>r8s r%_makezTensorKey._makeps/  !')VYm0TU Ur$alloccz|j|j|j|j|jSrI)r`rVr=r>r8)clsras r%from_allocationzTensorKey.from_allocations)yy599e.A.A5<<PPr$tc|<|j|j|j|j|jSyrI)r`rVstorage_data_ptrr>r8)rcres r% from_tensorzTensorKey.from_tensors1 =99QTT1#5#5qQ Qr$c|j|jj|jj|jj fSrI)rVrWr>r8typeindexrDs r%r[zTensorKey._as_sortables3ww 22DKK4D4DdkkFWFWWWr$N)rrrrPrQr:r<rRrFrTr\ staticmethodrr9r8r` classmethodr rdrrhpropertytupler[r#r$r%rr\s G I#I6K6D6 C= c]  }   +   Q$;Q@UQQH_5(;:O XeCc3$67XXr$noder?c#4K|j}|jdtjk(r|jdjt j k(r|jdk(r|r|djdtjk(r|djdvr|djdjrdt|djdjdtr7dtj|djdjdfy|jdtjk(r|jd}|j|jJ|jK|jj D]2\}}}tj|tj|f4|jL|jj D]2\}}}tj|tj|f4yyyw)Nrztorch::autograd::AccumulateGrad)z aten::detachz aten::add_)childrentypedr TorchOpscoperBACKWARD_FUNCTIONnameinputsrJrrrhPyCallmodule optimizer parameters)rprs typed_fields_pp_grads r%!_extract_parameters_and_gradientsrs}}H 1 +++ JJqM  ;#@#@ @ II: :  QK  a J$6$6 6 QK   > > QK  a ' ' x{((+2215 GI))(1+*;*;A*>*E*Ea*HIII A*++ +zz!} ""*l.D.D.LLL    * , 3 3 > > N 1f++A. 0E0Ef0MMM N  ! ! - , 6 6 A A N 61++A. 0E0Ef0MMM N . ,sHHc#@Kt|D] \}}| |ywrIr)rpr_p_grads r%extract_parametersrs(7= 7 =Gsc#DKt|D]\}}| ||fywrIr)rprrs r%extract_gradientsrs/7t< 6  V)Os  event.cg}|rW|jdtjk(r(|j|jdj|j }|rWt |SNrrr)rtr ruappendrvparentro)rscopess r% get_scopesrsT F  ;;q>Z// / MM%++a... /   =r$ceZdZdZededeeedffdZ ededee dffdZ edefdZ e dedeee dffd Zy ) SchemaMatcheraLookup operator schema based on profiled name. When profiling we record the operator's name but not the schema. However some analysis requires that information. Fortunately we can look up registered schema from the recorded name. We do not, however, record the overload and so we must compare the profiled arguments with all overloads to determine viable matches. Note: Once https://github.com/pytorch/pytorch/issues/78871 is completed this code will be obsolete. rer?.c 6d}|j|D]_}|xs|jDcgc]}dc}}t|jD]'\}}||xxt|jddzcc<)at |xsd|j DScc}w)aDetermine which inputs may have mutated based on function schema. Note that we don't need to resolve down to a single schema to perform this analysis. An input is mutable if it is mutable in any overload. In practice, however, it is overwhelmingly common to match a single overload. If we cannot find any valid schema then we must be conservative and assume all inputs are mutable. NFis_writec3 K|]}dywrIr#).0rs r% z3SchemaMatcher.inputs_are_mutable..s 8! 8s ) match_schemas arguments enumerategetattr alias_inforory)rcremutableschemariargs r%inputs_are_mutablez SchemaMatcher.inputs_are_mutables)-''* IFB1A1A!BA%!BG#F$4$45 I3 gcnnj%HH  I I W8 8qxx 899 "Cs Bctd|jDdtffd tfdj|jxsdDS)Nc3K|]^}t|trtj|n4t|tr#|Dcgc]}tj|c}n|`ycc}wwrI)rJrrrhlistrrjs r%rz.SchemaMatcher.match_schemas..sb  )31o(FI ! !! $;EQ:MA6q)''*6    7s>A+A&A+r?ct|jtk(xr(tfdt|jDS)Nc3\K|]#\}}j||j%ywrI) _types_matchrj)robserved schema_argrcs r%rz?SchemaMatcher.match_schemas..matches..s0C(Hj  :??;Cs),)lenrallzip)rrc signatures r%matchesz,SchemaMatcher.match_schemas..matchessHv''(C N:sC,/ 6;K;K,LC@ r$c34K|]}|s |ywrIr#)rsrs r%rz.SchemaMatcher.match_schemas..sO1GAJQOs r#)roryrTlookup_schemasrx)rcrerrs` @@r%rzSchemaMatcher.match_schemassS  XX    t  O 2 2166 : @bOOOr$c t|tjjr(|j }|duxs|j ||St|tjj ry|jtjjjr$t|txrtd|DStjjtftjjtdftjj t"ftjj$t&ftjj(t*ftjj,t.ftjj0t"t&t*t.fff}|D]\}}t||st||cS|duS)NTc3<K|]}t|tywrI)rJrrrs r%rz-SchemaMatcher._types_match..s6-. 1i(6s)rJr9_C OptionalTypegetElementTyperAnyType isSubtypeOfListType ofTensorsrr TensorTyperNoneTyperjBoolTyperTIntTyperQ FloatTypefloat ComplexTypecomplex NumberType)rcr schema_typetype_mapjit_typepy_typess r%rzSchemaMatcher._types_matchsp k588#8#8 9%446Kt#Ns'7'7+'N N k588#3#3 4  " "588#4#4#>#>#@ Ah-#62:63  XX ) , XX  T + XX   % XX  s # XX   ' XX ! !7 + XX 4eW"= >K #+ 6 Hh+x0!(H55 64r$rxc| d|vryttjj|S#t$rYywxYw)Nz::)ror9r_jit_get_schemas_for_operator RuntimeError)rxs r%rzSchemaMatcher.lookup_schemas&s@ 4??EF F  s/'/ ;;N)rrrrPrmr rorrTrr rrrlrRrr#r$r%rrs :#7:E(4.RUBUz!OpTree.__init__..?s AOOr$key)experimental_event_tree _root_nodesrosorteddfs _sorted_nodesrErs r%__init__zOpTree.__init__=s.!99;"6$((*:S#TUr$c/fKtj|jg|i|Ed{y7wrI)r traverse_dfsrrEargskwargss r%rz OpTree.dfsAs*&&t'7'7I$I&IIIs '1/1.c|jSrI)rrDs r% sorted_nodeszOpTree.sorted_nodesDs!!!r$) rrrr rrrrrnrorr#r$r%rr<sPVV4VJh~&>J"eNC$78""r$rc^eZdZdeddfdZdeeddfdZede de efdZ d e fd Z y) SizeMapop_treer?Ncbi|_|jD]X}|jdtjk(r5|j |jdD]}|j |Y|jdtjk(sz|jd}|j|jJ|jA|jjD](\}}}|j ||j |*|j|jjD]C\}}}|j ||j ||D]\}}|j |E[i} |jD]}|jdtjk(s$|jd} tj| } | sKt| j} | j!| | } | | k7sx| d| }t"j%d||jj'| y)Nrrrz vs. z(Mismatch between allocation and free: %s)_valuesrrtr ru_flat_tensor_inputs_update_valuesrzr{r|r} Allocationrrdabs alloc_size setdefaultlogwarningupdate)rErrprer~rrrstate allocations alloc_fieldsrnew_size prior_sizedeltas r%rzSizeMap.__init__Js-/ (( 3Dzz!} 2 2211$**Q-@+A''*+A*"3"33#zz!} #**2l6L6L6TTT&&2(4(;(;(F(F4 1f++A.++F34 ))5,8,B,B,M,M3(65++A.++F3$)3DAq //233 3(-/ (( WDzz!} 5 55#zz!} // ="<#:#:;H!,!7!7X!FJ"X-#-,eH: > $NPUV! W$ K(r$rectj|}|||jtjk(rt dt |jxsdg|jxsdgD}|t|jz}|dk\sJ|t |jj|d||j|<yyyy)Nc32K|]}|d|dzywrrrNr#rs r%rz)SizeMap._update_values..zsOAAaD1Q4KOsrrr) rrhlayoutr9stridedmaxrsizesstridesrdtyperget)rErern num_bytess r%rzSizeMap._update_valuesvs##A& ?q}U]]1JOQWW^QYY=M1#)NOOAM!''22I> 1i[ 1> #DLL$4$4S!$ A!_-At$  s5AAAArc |j|SrI)rrErs r% __getitem__zSizeMap.__getitem__s||C  r$)rrrrrrrrrlr rrrr r#r$r%rrIse*)*)4*)XI 9IdI 4/9R!y!r$rcbeZdZUdZeeed<dZeeed<e defdZ e defdZ y) DataFlowEdgeN input_versionFmutatedr?c|jduSrI)rrDs r% is_allocationzDataFlowEdge.is_allocations!!T))r$c|jduSrI)rrDs r% is_deletionzDataFlowEdge.is_deletions||t##r$) rrrrrrQr:rrTrnrrr#r$r%r r sR#'M8C='#GXd^# *t**$T$$r$r ceZdZdeddddfdZdeeeffdZe deee e e fffdZ e deee ffd Ze de ed ffd Ze de fd Zy) DataFlowNodergraph DataFlowGraphr?Nc ||_||_|j|_|jj D]:\}}|j s|j r |jj|<|jj Dcic]#\}}|||jj|f%}}}td|jDsJ|d|jycc}}w)Nc3,K|] \}}||k(ywrIr#rs r%rz(DataFlowNode.__init__..s8da168sz, ) _event_graph_determine_edges_edgesitemsrrbumpoutputslookuprvalues)rErrredgekvversionss r%rzDataFlowNode.__init__s  595J5J5L **, &IC||D$6$6   % & ?Cll>P>P>RSdaA4;;--a011SS8hoo&788VXJb :VV8Ts(C:cttj|jg}i}d|DD]}t |j t j|D]\}}t|tr?tj|}|j|tj|Ut|tsf|D]@}tj|}|j|tj|Bt!j"t$}|j'D]M\}} | |r|j(j+|nd||_d| vxst| dk(} | ||_O|D]} | j0dt2j4k(s$| j0dj6dksAtj9| j0d}||} || j. Jd|d| _|r|j(j+|nd| _|D]j} | j0dt2j4k(s$| j0dj6dkDsAd|tj9| j0d_lt;t=d|j'DS) Nc3|K|]4}|jdtjk(s$|jd6ywr)rtr rurs r%rz0DataFlowNode._determine_edges..s-S!!''!* @R@R2R1771:Ss%<<TrIrrrzDouble delete: c30K|]\}}| ||fywrIr#rr$r%s r%rz0DataFlowNode._determine_edges..sMdaq}Aq6Ms  )rorrrrryrrrJrrrhrsetaddr collections defaultdictr rrr!rrrtr rrrddictr) rEsubtreemutable_by_keyrop_inputrr op_input_iedges mutable_setrrr#s r%rzDataFlowNode._determine_edgess++T[[M:;JLSwS KB%( =;;B?& K!'h8#//9C"--c359==gF $/&.K '33J?&11#su=AA'JK K K '' 5 . 4 4 6 - CFI4;;+=+=c+Brc (  ;.RE+4F'4Q%,c " - LAwwqzZ222qwwqz7L7Lq7P// ;Sz{dll&>W/RUQV@WW># @CT[[%7%7%<"  L RAwwqzZ222qwwqz7L7Lq7PMQi// ;<J R FMekkmMMNNr$c |jjDcic]A\}}|js0|t|jt t |jfCc}}Scc}}wrI)rrrrTrrrQrrEr$r%s r%ryzDataFlowNode.inputss[  ))+  1?? QYYc1??!;< <   sAA(c|jjDcic]G\}}|jr |jr |jr||j dn|j dzIc}}Scc}}wr)rrrrrrr8s r%r zDataFlowNode.outputssa ))+ 1 !)) AOO+q11D D   sA A..cVtd|jjDS)Nc3ZK|]#\}}|js|js |%ywrI)rrr+s r%rz-DataFlowNode.intermediates..s% !QQ]]A s + ++)rorrrDs r% intermediateszDataFlowNode.intermediatess) ++++-   r$c.|jjSrI)rrrDs r% start_timezDataFlowNode.start_times{{(((r$)rrrrrr0rr rrnrorTrQryr r<r>r#r$r%rrs Wn W_ W W1O$y,'>"?1Of YdCi(889   in-   uY^4  )C))r$rceZdZdeddfdZedeedffdZdZ edee dffdZ e dedee dffd Z d edefd Zd eddfd Zd eddfd Zy)rrr?Nc||_|j||_i|_|jDcgc]}t ||c}|_|j jd|jycc}w)Nc|jSrI)r>rs r%rz(DataFlowGraph.__init__..s ALLr$r) _op_tree_extract_leaf_events _leaf_events_active_version leaf_eventsr _flow_nodessortvalidate)rEres r%rzDataFlowGraph.__init__sk  55g>?A;?;K;KLaLD1L "89 MsA<.c,t|jSrI)rorGrDs r% flow_nodeszDataFlowGraph.flow_nodessT%%&&r$c0t}|jD]^}t|jj}||z}|r-J|jj d|j d|||z}`i}|jD]}|jjD](\}\}}|j|d} | |k(r!J| |f|jjD]*\}}|j||} || k\s J|| f|||<,y)N r) r,rLr rrrxrryr) rEr rp node_outputs duplicatestensor_versionsrrversionexpected prior_versions r%rIzDataFlowGraph.validates5.1eOO $Dt||1134L </J! Sdkk&6&6%7q Qzl#S S> | #G  $13OO /D%)[[%6%6%8 @!\a*..sA67*?Xw,??* @!% 2 2 4 / W / 3 3C A -/I'=1II/'.$ /  /r$c|jSrI)rDrDs r%rFzDataFlowGraph.leaf_eventss   r$cgdtdtfddtffd }|j|D]}tt dS)aPartially traverse the op tree and extract top level ops. Consider the following code: ``` with record_function("My annotation"): x.zero_() y.zero_() ``` The op tree (assuming no Autograd) will look like: TorchOp: "My annotation" TorchOp: zero_ TorchOp: fill_ TorchOp: zero_ TorchOp: fill_ The recursive structure of operator calls makes data flow unwieldy. In order to simplify analysis we would like to select the highest level ops to represent in the graph. In this case those are the `zero_` ops; the fact that `fill_` is called is an implementation detail. We also do not want to group everything under "My annotation" as this could create overly coarse bundles and lose critical semantics. To address this issue we walk over the graph and select the topmost torch ops ** which match at least one operator schema **. These form the leaves of the first pass through the op tree. (As well as any allocations or frees which do are not part of a kernel.) These events form the logical nodes in our data flow graph. rJr?c|jdtjk(xrW|jdjtj k(xs+t tj|jdSr) rtr rurvrrwrTrr)rJs r%leaf_opz3DataFlowGraph._extract_leaf_events..leaf_opAs_771:!3!33   K$A$AAA 33AGGAJ?@ r$c|s|jtjk(rj|gS|jSrI)tagr rrrs)rJrFrXs r% children_fnz7DataFlowGraph._extract_leaf_events..children_fnGs8qzQUUj&;&;;""1% :: r$)r[c|jSrIrrs r%rz4DataFlowGraph._extract_leaf_events..Qs qr$r)rrTrror)rr[rrFrXs @@r%rCz"DataFlowGraph._extract_leaf_eventss^B-/  ~ $   > 5 A  VK-FGHHr$rcF|jj|d}|J|SNrrEr)rErrRs r%r!zDataFlowGraph.lookupSs+&&11#q9"""r$ch|jj|d}|J|dz|j|<y)Nrr)rEr)rErrTs r%rzDataFlowGraph.bumpXs;,,00d; ((($1A$5S!r$c^|jj|dJd|j|<yr^r_r s r%deletezDataFlowGraph.delete]s1##..sA6BBB$(S!r$)rrrrrrnrorrLrIrrFrlrCrrQr!rrbr#r$r%rrs4'E,"34''/*!U>3#67!!2If2I~s7J1K2I2Ih) 6 6d6 ))))r$rceZdZUdZeeed<eje Z e e efed<eje Z e e efed<ejeZee ed<y)CategoryElementNby_iddefault_factoryby_key by_version _by_id_keyset)rrrrerrr: dataclassesfieldr0rhrri TensorAndIDr,rjr#r$r%rdrdbsq $E8H $(9 (9(9$(OFDH$ %O.?k.?.?PT.UJ[(*+U%6K$5$5c$JM3y>Jr$rdceZdZUejdZejee fe d<de de ddfdZ de de ddfd Zde d ede ddfd Zde d ede ddfd Zded edee fd Zy) CategoryDictc4tjtSrI)r.r/rdr#r$r%rzCategoryDict.ps 7 7 Hr$rfrrcategoryr?Nc||j|j_|j|jjj |yrI)rrVrerjr-rErrqs r% set_by_idzCategoryDict.set_by_idss8%- SVV" SVV**..s3r$cP||j|jj|<yrI)rrVrhrss r% set_by_keyzCategoryDict.set_by_keyws+3 SVV##C(r$rRcT||j|jj||f<yrI)rrVrirErrRrqs r%set_by_versionzCategoryDict.set_by_versionzs#:B SVV''g7r$cn|j|jjj||f|yrI)rrVrirrxs r%setdefault_by_versionz"CategoryDict.setdefault_by_version}s+ SVV''22C>8Lr$c t|trt|tsy|j|j}|j xs<|j j|dxs|jj||fdSrI) rJrrrrVrerhrri)rErrRelements r%rzCategoryDict.getsn c3  3 (B,,svv& MM <~~!!#t, <!!%%sGnd; r$)rrrrkrlrr.r/rQrdr:rrrtrvryr{rrrr#r$r%roroms=N[=N=NH>G[ $ $S/%9 :4Y4(4t44i4844C)CcCXCRVCMM'*M6>M M  s S Xh-? r$roceZdZdeddfdZedeeeee efdffdZ de fdZ de eeeffdZdeefd Zdd Zdd Zdd Zdd ZddZddZdZy) MemoryProfilerr?Nct||_t|j|_t |j|_t |_|j|j|j|j|j|j|jyrI)rrBr_data_flow_graphr _size_mapro _categories_set_gradients_and_temporaries#_set_parameters_using_python_tracer _set_inputs_set_parameters_using_data_flow_set_activations_set_optimizer_state_set_autograd_detailrs r%rzMemoryProfile.__init__sv -dmm < /'> ++- 002  ,,.  !!# !!#r$.c ^g}i}i}jjD]S}|jdtjk(s%|jd}|j }|dkD}|j }tj|} | ||| |f<pt|j} |j| jf} |rU| |vr&|j|tj| df|fd|| <|j|tj| df|f|j|tj | df| f|j#| dr.|jdtj$| df| fVj'} t)t+| j-} | j-D cgc]%\} }| df|vr|dk(rdtj$| |ff'}} }j.j0D]}|j2j5D]\} }|j6r,|| df}|j|tj| dffnV|j8rJ|j:j }|j<}|J|j|tj| |ff|j>s|| df}|j|tj | | | ff|jAfd|D|jCdtE|Scc}} w) NrrrTFr)c3VK|] \}}\}}||||fj|f"ywrI)r)rtimeactionrrRrEs r%rz)MemoryProfile.timeline..s9 ,fnsG6C>4>>#+> ? s&)c*|d|djfSr)valuers r%rz(MemoryProfile.timeline..s1Q41"4r$r)#rBrrtr rrrrrdrr8r=rr/r2r1r3popr0_category_snapshotr0rkeysrrLrrrrrrrextendrHro)rEoutputallocation_times live_unknownrrrrretkeyrptr_and_devicesnapshot last_versionrReventsrpr#s` r%timelinezMemoryProfile.timelines :<>@FH ]]&&( E{{1~!6!66${{1~ )44 *Q '' 00>#>?$dM%:;l112C&2&6&6 %CN$)\9"MM!"F$<$?\\ 11A"00G"...MM1f&>&>g"OP##(#u6AMM1fnnsL*L)c\|jj|i|tjk(SrI)rrrrrs r% _is_gradientzMemoryProfile._is_gradients+#t##T4V48I8IIIr$c Zt}|jjD]}}|jd|jj D|jd|j D|j|jj |jjjD]$}|jd|jD&t|Dcic]$\}}||f|jj||&c}}Scc}}w)Nc30K|]\}\}}||fywrIr#)rr$rr%s r%rz3MemoryProfile._category_snapshot..s'T91fq!A'Tc3$K|]}|df ywrNr#rrs r%rz3MemoryProfile._category_snapshot..s&NCQx&Nc3$K|]}|df ywrr#rs r%rz3MemoryProfile._category_snapshot..s&KCQx&Kr)r,rrLrryrr<r rrr"rjrr)rEall_tensor_versionsrprrrRs r%rz MemoryProfile._category_snapshots03))44 =D  & &'T @Q@Q@S'T U  & &&N4;M;M&N N  & &t||'9'9'; < = !!))002 LA  & &&K1??&K K L !'': ; W'ND,,00g> >   s:)D'cPt t}jjD]e}t fd|j j D}|s3j|jd|jDgt|k(rS)a\Extract IDs of Tensors which depend or will depend on a gradient. Note that this weakened definition of "depends" requires us to loop over the data flow graph multiple times because it allows dependency information to flow backward through edges and removes the guarantee that nodes are topologically sorted. (Or indeed, even that a valid topological order exists.) Put another way, we have converted an acyclic data flow graph into a cyclic graph and we are attempting to partition cycles involving a gradient from the rest of the graph. c3K|]`\}\}}jj||tjtjfvs|j vr|j bywrI)rrrrr!rV)rrrrRdepends_on_gradientrEs r%rzAMemoryProfile._any_version_depends_on_gradient..sa)\a''++C9 ))8+=+=>?vv!44 FFsA&A)c34K|]}|jywrI)rVrs r%rzAMemoryProfile._any_version_depends_on_gradient.. s.N#svv.Ns) r,rrrLroryrrr )rE start_sizerpidsrs` @r% _any_version_depends_on_gradientz.MemoryProfile._any_version_depends_on_gradients),01J--88 O-1[[->->-@'..s3'...N.NN O$&':5**+r$ch|jjD]?}t|D]/\}}|jj |t j 1A|jjD]=}|jD],}|jj|t j.?y)z>Mark Tensors which are unambiguous and simple to reason about.N) rBrrrrtrrrrLr<rvr)rErrrrprs r%rz,MemoryProfile._set_gradients_and_temporariess ]]&&( FE.u5 F 6  **683D3DE F F))44 CD'' C  ++Ax/A/AB C Cr$c|jjD]?}t|D]/}||jj |t j 1AyrI)rBrrrrtrr!)rErrs r%rz1MemoryProfile._set_parameters_using_python_tracer&sW]]&&( FE'. F=$$..q(2D2DE F Fr$cj}ttjjD]m}|j j Dchc] \}\}}||f}}}}||jj z}tfd|Dsi|zoj}jjD]N}tjt|jvs)|t|jj z}P|D]?\}}|j|vsjj!||t"j$Aycc}}}w)aMark inputs based on which Tensors are updated using gradients. The process for differentiating between inputs and activations is more involved. Most Tensors in a training loop depend on at least one gradient: parameters depend on them through updates, and activations and optimizer state depend on them transitively through parameters. Critically, we do not need to know which Tensors are parameters to apply this method; we can simply walk the data flow graph to build the set of all values which depend on a gradient and then obtain the set of inputs from the conjugate set. There is, however, one hiccup. The first time we see a parameter is generally on the forward pass of the first step. We know from inspection of the data flow graph that v1 of that Tensor depends on a gradient (provided we profile an optimizer step), but not v0. To address this problem we weaken the definition of "depends on a gradient" to "any version of this Tensor depends on a gradient", which in turn strengthens the criteria for the input set enough to filter the activations in the forward pass of the first step.c3K|]C}jj|tjtjfvxs|vEywrI)rrrrr!)rrproduces_gradientrEs r%rz,MemoryProfile._set_inputs..MsW%  $$a(X->->@R@R,SS*))*sA A N)rr,reversedrrLryrr anycopyrrwrrrVrr{rr) rErrprrrRtensorsinput_candidatesrs ` @r%rzMemoryProfile._set_inputs,sO2#CCE/2eT22==> -D?C{{?P?P?RSS*;#|7W~SGS t||))+ +G! "W,! --113))44 >D,, 4;;0GG C (:(:(<$==  >- ULCvv00  66sGX^^T U!TsE( c j}t}|jDchc]\}}|tjk(s|}}}j j D]}|jjD chc] \}\}} || f} }}} tjt|jvs\tfd| Drqtfd|jjDr|j| s||jjz}|| j|z}t t!j j D]`}t fd|jjDs1 j#d|jjDb|j% |Dchc]\}}|j&} }}| j)z} |j+D]>\}}|j&| vsj,j/|tj0@ycc}}wcc} }}wcc}}w)aDeduce which Tensors are parameters. Consider the following code for the step of SGD with momentum (nesterov=False), where `d_p` is the gradient of `param` and `buf` is the momentum buffer. ``` buf.mul_(momentum).add_(d_p, alpha=1 - dampening) d_p = buf param.add_(d_p, alpha=-lr) ``` Both `param` and `buf` take a gradient and perform an in-place update. The python tracer will inspect calls to `nn.Module.forward` and `optim.Optimizer.step` to extract parameter and optimizer state respectively (including parameters), so this is generally a non-issue. However as a fallback we can also exploit several properties of parameters to distinguish them from other model state. First, they are directly used in the forward pass. (At this point we haven't established which parts of the graph correspond to the forward pass but we can deduce enough to suffice.) Some mutable state such as batch norm moving averages also contribute to the forward pass, but optimizer state does not. Second, a parameter is by definition used to compute at least one gradient and depends on at least one gradient. c3<K|]}j|ywrIrrrrEs r%rz@MemoryProfile._set_parameters_using_data_flow..sBa-D--q1Bc3<K|]}j|ywrIrrs r%rz@MemoryProfile._set_parameters_using_data_flow..sPa-D--q1Prc3HK|]}j|xs|vywrIr)rrrEused_for_gradients r%rz@MemoryProfile._set_parameters_using_data_flow..s4"!!1%?.?)??s"c30K|]\}\}}||fywrIr#)rrrrRs r%rz@MemoryProfile._set_parameters_using_data_flow..s!)'8sLQS'N)rN)rr,rrrrrLryrrwrrrr  intersection differencerrintersection_updaterVrrrrtr!) rErcandidate_parametersrrqcandidate_fwd_tensorsrprrrryparameter_keysrs ` @r%rz-MemoryProfile._set_parameters_using_data_flow_s*:**, 25!)!13 !XX5OA3 3 ))44 QD:>++:K:K:MNNsJQsElNFN--Z 5LLB6BBP4<<;M;M;OPP*66v>%););)==%$(9(9:O(PP$ Q/2eT22==> D++-"(()<@KK,>? ))44 TD:>++:K:K:MNNsJQsElNFNBHIQ 4 0 0 4 4a 8I I"H,)X -DE 11DKK9PP++-TA:D$$::SASx?R?RST TNIs 8D6  D=c|jjD]}|jdtjk(s$|jdj s>|jdj j }tjjd|DD]G\}}tj|}||jj|tjIy)Nrrrc3(K|] \}}}| ywrIr#)rrrs r%rz5MemoryProfile._set_optimizer_state..s9{q!UU9s)rBrrtr rzr|r}itchain from_iterablerrhrrtrr")rErr}rrers r%rz"MemoryProfile._set_optimizer_states]]&&( RE{{1~!2!22u{{1~7O7O"[[^55@@ HH229j9RDAq$//2C((2238P8PQ R Rr$cdtjh}|jjD]}tj t |jvs)|jjD]W\}}|dk(s"|jj||dz |vs-|jj||tjYyr) rr rrLrrwrrr rrrr{)rEpriorrprrRs r%rz"MemoryProfile._set_autograd_detailsx//0))44 D,, 4;;0GG$(LL$6$6$8LC!|t'7'7';';C1'MQV'V((>>(*B*B r$r?N)rrrr rrnrorQr/KeyAndIDrrTrr0rmrrrr,rrrrrrrrr#r$r%rrs $ $4 $@%c68S&@ A3 FG@@DJtJ Dhx6H)H$I !+#c(!+FC"F 1UfHDTT& Rr$rc6eZdZdZdZddZddZ d ddZy) MemoryProfileTimelinecH|j|_|j|_y)aThe minimum representation of the memory profile timeline includes the memory timeline and categories. The timeline consists of [timestamp, action, (TensorKey, version), numbytes] elements, to denote any actions (pre-existing, create, destroy, or increment_version) that occurred to a specific Tensor for a chunk of memory. The categories help map each (TensorKey, version) pair into a category.N)rr categories)rEmemory_profiles r%rzMemoryProfileTimeline.__init__s'// (44r$c> tj|}g}g fd}d}jD]G\}}\}} } |j|k7r|dk7rt|dz }|dk(s ||kr|dkDr|}t |dk(r9|j | j dgt D cgc]} dc} zn;||dk7r3|j | j dj|tjtjfvr ||| | |tjk(r||| | ||| dz| |tjk(r ||| | .updatesVc9-##C1  'x014E "Ie E * r$r)irrrUnknown action: )r9r8rrQrrrrr/r0r1r2r3 ValueError) rE device_strr8timesrt_minrerrrRnumbytesrrs ` @r%_coalesce_timelinez(MemoryProfileTimeline._coalesce_timelinesj)!# +37==! > /Av~WxzzV#BwDM{q5yQU5zQ Q aS/A#B!A#BBCeBi Q U2Y^^-.&,,fmm<<sGX.6333sGhY/sGaK26>>)sGhY/!#3F8!<==C! >F1661!a%Q&66e|+$C(7s $ F ?FNc|j|\}}ddl}t|d5}|j||g|dddy#1swYyxYw)zSaves the memory timeline as [times, sizes by category] as a JSON formatted file to the given path for the given device.rNw)rjsonopendump)rEpathrrrrfs r%export_memory_timelinez,MemoryProfileTimeline.export_memory_timelinesM..z: u $_ ) IIuena ( ) ) )s AA c tj|}g}fd}jD]\}}\}} } |j|k7r|tjtj fvr$|j |t|| ||| fb|tjk(rK|j |t|| ||| f|j |t|| ||| dzf|tjk(r%|j |t|| ||| ftd|ddl } t|d5} | j|| dddy#1swYyxYw)zSaves the memory timeline as raw memory event tuples in the form of (timestamp, action, numbytes, category) as a JSON formatted file to the given path for the given device.crt|trjj||nd}t|SrI)rJrrrr)rrRrqrEs r%get_category_indexzLMemoryProfileTimeline.export_memory_timeline_raw..get_category_index-s:c9-##C1  &h/ /r$rrrrNr)r9r8rr/r0r1r_ACTION_TO_INDEXr2r3rrrr) rErrr8 raw_eventsrrerrrRrrrs ` r%export_memory_timeline_rawz0MemoryProfileTimeline.export_memory_timeline_raw%s j)68  048==+ > /Av~WxzzV#&,,fmm<<!!(0 *38 6333!!(0! *38 !!(0 *3! < 6>>)!!(0! *38 !#3F8!<==W+ >Z  $_ % IIj! $ % % %s EE&c ddl}|jjd}| tdyddlm}ddlm}ddlm } ddl m } ddl } |j|} | j| d| j| d}} t| }| |z} | j!|d d z }t#j$|}t"j&j)|}t"j&j+|}| j-|d }|j/}t0j3D]8\}}t4|}|j7| d z |dd|f|dd|dzf|d:|j9t0Dcgc]}|dn |j:c}|j=d|j?ddjA|r|gngd|d z dd|d z ddgz}|jC|| ddd}|jE|jG|j:dtI|j:d5}||jKjMd}d |d!}tI|d"5}|jO|dddddd||j:ycc}w#1swY)xYw#1swY-xYw)#zsExports the memory timeline as an HTML file which contains the memory timeline plot embedded as a PNG file.rN matplotlibzDexport_memory_timeline_html failed because matplotlib was not found.) b64encode)remove)NamedTemporaryFilerr)axisi@P)figsizedpig@@gffffff?)coloralphaUnknownz Time (ms)z Memory (GB)z zMax memory allocated: z.2fz GiB Max memory reserved: z GiBwbz.pngF)suffixrbpng)formatrbzutf-8z} GPU Memory Timeline HTML r)(importlib.utilutil find_specprintbase64rosrtempfilermatplotlib.pyplotpyplotnumpyrarraymincumsumr9r8cudamax_memory_allocatedmax_memory_reservedfiguregca_CATEGORY_TO_COLORSrr fill_betweenlegendrx set_xlabel set_ylabeljoin set_titleclosesavefigrreaddecodewrite)rErrrtitle importlibmatplotlib_specrrrpltnpmtrrrstackedr8rrfigaxesrqrrtmpfiletmpencodedhtmlrs r%export_memory_timeline_htmlz1MemoryProfileTimeline.export_memory_timeline_htmlgs #..22<@  " V  $/'  $ $Z 0xx1AuE  ))E)*W4j)$zz>>vF#jj<r@s $--% #.'!  K$% g!"tyy"   k NNG   | k& (11D'EFtq!adFTYY)//1AqwwJ/$E$?@ (((($D>.X.X?.Xb'N 'N eHY'))<<=>'NT^0C   eHY'234h~.5c9I3JjjZ " "@!@!F $ $ $[)[)|g)g)T KKK    >BBJ ZZ{G0s 1 I+I1