g^(hU nddlZddlZddlZddlZddlmZddlmZddlm Z m Z ddl m Z m Z ddlmZmZmZmZmZddlmZddlZddlZddlmZmZdd lmZdd lmZdd lmZdd l m!Z!dd l"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)ddl*m+Z+m,Z,erddl-m.Z.e/ej`jcdddk(rdndZ2dZ3dgZ4Gdde5e Z6Gdde5e Z7Gdde6Z8Gdde7Z9Gdd Z:Gd!d"Z;d#e5d$e/fd%Zd*e?eje?e5e/ffd#e5d$dfd+ZAd*e?eje?e5e/ffd#e5d$dfd,ZBd-e?e7eCe?eje?e5e/fffd#e5d$dfd.ZDd-e?e7eCe?eje?e5e/fffd#e5d$dfd/ZEGd0d1e ZFGd2de&ZGy)3N) nullcontext)deepcopy)autoEnum)partialwraps)AnyCallableOptional TYPE_CHECKINGUnion)Self)nnoptim)active_fake_mode)get_untyped_storages) ModTracker)DTensor)!register_optimizer_step_post_hook register_optimizer_step_pre_hook)TorchDispatchMode) tree_flatten tree_map_only)WeakIdKeyDictionaryweakref)RemovableHandlePYTORCH_NO_CUDA_MEMORY_CACHINGiTotal MemTrackerceZdZdZy)_RefTypeziBase Class for defining memory reference types, categorizing tensors based on their usage within a model.N__name__ __module__ __qualname____doc__b/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/torch/distributed/_tools/mem_tracker.pyr"r")ssr)r"ceZdZdZy)_Statez;Base Class for defining module state to capture snapshots .Nr#r(r)r*r,r,-sEr)r,c,eZdZdZdZdZdZdZdZdZ dZ y ) _MemRefTypea An enum to define memory reference types, categorizing tensors based on their usage within a model. - PARAM: Tensors registered as nn.Parameter within modules. - BUFFER: Tensors registered as nn.Buffer within modules. - GRAD: Gradients associated with parameters. - ACT: Tensors produced during the forward pass and recomputation in activation checkpointing. - TMP: Temporary memory used during the backward pass, including gradients of activations. - OPT: Tensors holding optimizer states. - OTH: Tensors registered via `track_external` that do not fit the above categories. ParameterBufferGradient ActivationTempOptstateOtherN) r$r%r&r'PARAMBUFFERGRADACTTEMPOPTOTHr(r)r*r.r.1s,  E F D C D C Cr)r.c0eZdZdZdZdZdZdZdZdZ dZ d Z y ) _ModStatea An enum to define the state of a module. - PRE_FW: The module is about to run the forward pass. - POST_FW: The module has finished running the forward pass. - PEAK_FW: The module has reached the peak memory usage during the forward pass. - PRE_BW: The module is about to run the backward pass. - PRE_FW_AC: The module is about to run the forward pass with activation checkpointing. - POST_FW_AC: The module has finished running the forward pass with activation checkpointing. - POST_BW: The module has finished running the backward pass. - PEAK_BW: The module has reached the peak memory usage during the backward pass. z Pre-Forwardz Post-Forwardz Peak-Forwardz Pre-BackwardzPre-Forward-ACzPost-Forward-ACz Post-Backwardz Peak-BackwardN) r$r%r&r'PRE_FWPOST_FWPEAK_FWPRE_BW PRE_FW_AC POST_FW_ACPOST_BWPEAK_BWr(r)r*r>r>Gs1 FGG F I"JGGr)r>ceZdZdZdefdZy) _ModMemStatsa A class to store the memory statistics of a module. Args: mod_fqn (str): The fully qualified name of the module. Attributes: mod_fqn (str): The fully qualified name of the module. parameter_mem (int): The memory usage of the parameters of the module. buffer_mem (int): The memory usage of the buffers of the module. input_mem (int): The memory usage of the inputs to the module. output_mem (int): The memory usage of the outputs from the module. snapshots (Dict[_ModState, Dict[torch.device, Dict[str, int]]]): A dictionary of memory snapshots of the module at different states defined by ``_ModState``. Note: The memory snapshot is stored as a dictionary - Dict[torch.device, Dict[str, int]], where each key is a device, and each value is another dictionary with keys as memory reference types defined by `_MemRefType` and values as the memory consumed in bytes. mod_fqnc>||_||||i|_i|_yN)rI local_peak snapshots)selfrIs r*__init__z_ModMemStats.__init__ss#     35TVr)N)r$r%r&r'strrOr(r)r*rHrH_s&WWr)rHceZdZdZdededej deddf dZdefd Z d ejdefd Z e dd ejdej ded e eeej"gefdeeej"ff d Zy) _WeakRefInfozN Manages memory statistics and device attributes for tensor storages. size element_sizedevicereftypereturnNcf||_||_||_||_|j |_y)a Initializes the ``_WeakRefInfo`` object with tensor storage properties. Args: size (int): The number of elements in the tensor storage. element_size (int): The size of each element in the tensor storage. device (torch.device): The device on which the tensor is allocated. reftype (_RefType): The reference type of the tensor. N)rSrTrVrU_calculate_mem_consumed mem_consumed)rNrSrTrUrVs r*rOz_WeakRefInfo.__init__s2 (   88:r)c|j|jz}|jjdk(r#t j |t z t zS|S)z Calculates the memory consumed by the tensor storage, considering device-specific allocation rules. Returns: int: The memory consumed in bytes. cuda)rSrTrUtypemathceil_PYTORCH_MIN_ALLOCATE)rNmems r*rYz$_WeakRefInfo._calculate_mem_consumedsJii$+++ ;;  v %99c%::;>SS S r)stc|j|jk7r*|j|_|j|_|jS)a Updates and returns the memory consumed if the storage size has changed. Args: st (torch.UntypedStorage): The tensor storage to check for size updates. Returns: int: The updated memory consumed in bytes. )rSrYrZ)rNrbs r*update_mem_consumedz _WeakRefInfo.update_mem_consumeds@ 779 ! DI $ < < >D    r)callbackc||j|j||}tj||r t ||nd}||fS)a Creates a new ``_WeakRefInfo`` instance and a weak reference to a ``torch.UntypedStorage`` object, optionally attaching a callback to the weak reference. Args: st (torch.UntypedStorage): The storage object for which to create the weak reference info. device (torch.device): The device associated with the storage object. reftype (_RefType): The type of reference, used to categorize the storage. callback (Optional[Callable[[Self, weakref.ref]]]): A callback function that is called when the storage object is about to be finalized (garbage collected). The callback function should accept two arguments: the ``_WeakRefInfo`` instance and the weak reference to the storage. Returns: Tuple[Self, weakref.ref]: A tuple containing the newly created ``_WeakRefInfo`` instance and the weak reference to the storage object. The weak reference may have an attached callback if provided. N)rSrTrrefr)clsrbrUrVrewinfow_sts r* create_winfoz_WeakRefInfo.create_winfosF0BGGIr0&'B{{28wx7Nd{r)rK)r$r%r&r'inttorchrUr"rOrYUntypedStoragerd classmethodr r rrrgr tuplerkr(r)r*rRrR}s;;'*;49LL;KS; ;$   !e&:&: !s ! BF      8T7;;$7$<=>  tW[[ ! r)rRunitsrWc ~ddddd}||vr||Std|ddj|j) Nriii@)BKiBMiBGiBzUnsupported unit: z. Supported units are: z, ) ValueErrorjoinkeys)rq unit_dicts r*_get_mem_divisorr{sReEBI  '>tyyIY?Z>[ \  r)valuedivisor precisionc.|dk(r|St||z |S)Nr)round)r|r}r~s r* _rounding_fnrsqL5GeEGOY&GGr)snapshotct|dk(r tdyt|jD]G\}}t |t ddkrtd|gfd|jDddiIy)NrNo memory tracked.zDevice: c 3K|]M\}}t|tr d|jdt|ddnd|dt|ddOyw) z: r N) isinstancer"r|r).0kvr}rqs r* z"_print_snapshot..srAqa*QWWIR Q ;  se  %NN,     r)c t|dk(r tdy ddlm}t  g}t t t|jj}dg|Dcgc]"}t|tr |jn|$c}z}|jD]c\}} t| t ddkrt!|g} | j# fd| jD|j%| et|||d y#t$r}td|d}~wwxYwcc}w) Nrrtabulate3Please install tabulate to use the tabulate option.Devicerc3BK|]}t|ddyw)rrN)r)rrr}rqs r*rz*_print_snapshot_tabular..s&Wl1gq12!E7;Wsrstheaderstablefmt)rrr ImportErrorr{listnextitervaluesryrr"r|rrrrPextendappend) rrqrerr table_datakey_listkeyrrrrowr}s ` @r*_print_snapshot_tabularrs2 8} "#% u%GJDhoo/016689HjMUFI*S(3399+C5AG") X ,gq 9Q > 3xj WX__EVWW#   (:w ?@#  A  sD'0'E' E0 D<<ErMc|jD]L\}}t|jt|D]#\}}td|dzdt ||%Nty)Nz# r:)rrr| enumerater)rMrqstate snapshot_listirs r*_print_state_snapshotsr sj!* 1-}  $]3 -KAx Bq1ugQ- He , --  Gr)c V ddlm}g}d}t|}|jD]\}}t |D]\} } |j d| dz} | jD]\} } t | t|ddkr| |k7r| ndt| d}| }| jD]8\}}t ||dd ||t|tr |j n|<:|j|t||d d y#t$r}td|d}~wwxYw) Nrrrz # rr)z State & Callrrryrr) rrr{rrr|rrrPrr"rr)rMrqrrrlast_state_callr}rrrr state_callrrrrrs r*_print_state_snapshots_tabularrsX% JOu%G ) 1'}$]3 'KAx!KK=AE73J!)!1 ' X 4gqAQF'1O&C !#h  #-$NN,DAq'7A67q@ 1h(?177)sL!!#& ' ''$ (:v >?5  A  sD D( D##D(cDeZdZeZeZeZeZy) _UpdateTypeN)r$r%r&rADDDELREFSIZEr(r)r*rr7s  &C &C &C 6Dr)rc "eZdZdZd3dZ d4dededeedee ddf d Z d5d e jd e d e deefd Zdedej"ddfdZd3dZd3dZdeddfdZd e d e jddfdZ d6dedee j4eeefffdZ d7dej:de deeeffdZde defdZ!dej:de ddfdZ"dej:de de ddfdZ#dej:de ddfd Z$dej:de ddfd!Z%d e d"e&jNddfd#Z(d3d$Z)d3d%Z*d&e+ej:e&jNe jfddfd'Z, d8ded(ed)e ddfd*Z- d9d+ed(ed)e ddfd,Z.d3d-Z/d3d.Z0d3d/Z1d:fd0 Z2de ddffd1 Z3d;d2Z4xZ5S)+@(  $*+ &L*5" ! .2 !0088&-&<&<&E&E# r)u_typeriold_mem_consumed old_reftypecd}|jj|jtj |j d}|jt d|tjk(r>||jxx|jz cc<|t xx|jz cc<n|tjk(r?||jxx|jzcc<|t xx|jzcc<d}n|tjk(r=|J||xx|jzcc<||jxx|jz cc<n_|tjk(r>|J|j|z }||jxx|z cc<|t xx|z cc<d}ntd||r<|j|jt dk(r|j|j=yyy)NFrTzInvalid update type: )r setdefaultrUdictfromkeysrrrrrVrZrrrrw)rNrrirr maybe_zerorchanges r* _update_snapzMemTracker._update_snaps &&11 LL$--;  J* [__ $ U]] #u'9'9 9 # Z E$6$6 6 { & U]] #u'9'9 9 # Z E$6$6 6 J { &* ** [ !U%7%7 7 ! U]] #u'9'9 9 # {'' '#/ //''*::F U]] #v - # Z F * J4VH=> > ""5<<0<A'' 5B r)trVupdate_existingc2t|}t}|D]}|jj|d\}}|L|j} | |k7r)||_|j t j|| |j|p|r tdtj||j||j\}} || f|j|<|jdkDr |j t j||j||S)NNN)rzNo existing winfo foundr)rsetrgetrVrrraddKeyErrorrRrkrU_delete_callbackrZr) rNrrVrstswinfosrbri_rrjs r*_update_and_maybe_create_winfosz*MemTracker._update_and_maybe_create_winfoss #1% "B{{r<8HE1 #mm ')$+EM%%koou+%V 5! 899+77'4+@+@ t$)$- B%%)%%koou= 5!1 "2 r)rjcd|jdkDr!|jtj|yyNr)rZrrr)rNrirjs r*rzMemTracker._delete_callbacks+    !   koou 5 "r)ctjdtjdtddffd }|tj_y)NrbrSrWc(j||jj|d\}}|^|j|jk7r@|j}|j |j tj||yyy)Nrr) rrrrSrZrdrrr)rbrSrirrrNs r*rz)MemTracker._track_resize..resize_s   b$ '{{r<8HE1 UZZ2779%<#(#5#5 ))"-!!$$e>N"&= r))rrrmrnrlr)rNrs` r* _track_resizezMemTracker._track_resizesM t  ! ,, C D  " (/$r)cB|jtj_yrK)rrmrnrrs r*_restore_resizezMemTracker._restore_resizes'+'8'8$r) peak_statec~|j}|jjD]}|j|jj vs&||j vs5|jD]`\}}|jj|d|tks-|t|j|<t||j |d|<b|jD]Z\}}|jj|d|tks-|t|j|<t||j|<\y)Nr)rrrrIrparentsrMrrLrrrrr)rNr curr_snap mod_statsrrs r*_update_peak_statszMemTracker._update_peak_statss) '' --446 I  D$5$5$=$==!4!44)2): X$//33C;hz>RR8@8LI005GO (HI// ;B?D '__. >MC~~!!#q)HZ,@@&.z&:s#+3H+=##C( >r)ct|}|D]}|jj|d\}}|^|j|jk7r?|j}|j ||j tj||ytj||j||j\}}||f|j|<|jdkDs|j tj|y)Nrrr)rrrrSrZrdrrrrRrkrUrr) rNrVrrrbrirrrjs r*_trackzMemTracker._track s#1% >B{{r<8HE1 ::*','9'9$--b1%%#((%BR&*77'4+@+@ t$)$- B%%)%%koou=# >r)r]c|dk(rt|jS|dk(rt|jStd|)aE Capture a snapshot of the memory usage breakdown per device, based on the specified type. Args: type (str): The type of snapshot to capture. Can be "current" for the current memory usage or "peak" for the peak memory usage. Defaults to "current". Returns: Dict[torch.device, Dict[str, int]]: A dictionary where each key is a torch.device, and each value is another dictionary. This inner dictionary has keys representing memory reference types as defined in ``_MemRefType`` and values representing the amount of memory consumed in bytes. Raises: ValueError: If an invalid type is specified. currentpeakz Invalid type )rrrrw)rNr]s r*get_tracker_snapshotzMemTracker.get_tracker_snapshot#sG" 9 D//0 0 V^D//0 0}TF34 4r)moduleinstall_grad_hooksc dtjddffd d}|jD]}j|tj }|t d|Dz }|j*j|jtjjj|d|s|j }|j fd}||fj|<d}|jD]7} j| tj}|t d|Dz }9||fS)NgradrWcFj|tjyrK)rr.r8)rrNs r* _grad_hookz?MemTracker._track_module_params_and_buffers.._grad_hookCs  0 0   r)rc34K|]}|jywrKrZrris r*rz>MemTracker._track_module_params_and_buffers..OsGu 2 2Gc(|jSrK)r)prs r*z=MemTracker._track_module_params_and_buffers..[sz!&&1r)c34K|]}|jywrKrr s r*rz>MemTracker._track_module_params_and_buffers..gs H!3!3 Hr )rmTensor parametersrr.r6sumrr8rr register_hook"register_post_accumulate_grad_hookbuffersr7) rNrr param_memoryparamrgrad_hook_handlepost_acc_grad_hook_handle buffer_memorybufferrs ` @r* _track_module_params_and_buffersz+MemTracker._track_module_params_and_buffers;sP U\\ d   &&( E99!!F CGGG GLzz%44JJ$$ 0044UDAI&#(#6#6z#B ,1,T,T2-)%-;007' . nn& IF99""F S H HH HM  I m,,r)argscvddtjddffd }ttj||S)NrrrWct|}|D]3}jj|d\}}|%|jz 5y)Nr)rrrrZ)rrrbririnput_or_output_memoryrNs r*add_inps_or_outsz=MemTracker._track_inputs_or_outputs..add_inps_or_outsnsM&q)C A;;??2|<q$*e.@.@@* Ar))rmrr)rNrr rs` @r*_track_inputs_or_outputsz#MemTracker._track_inputs_or_outputsjs<!" A  A A ell$4d;%%r)inputscz|jj|}|J||jvrgt|}|j |d\}}|j |}||_||_||_||j|<tj}n|jjrM|j|}tj}|jtj||_ d|_n~t#|jj$|hz } t'| dk(rd| vr t)d|j|}tj}|j |}||_||_|j-} |tjk(ri| j/D cic]\} } | | t0c} } |_|j4j7tj8gj;| |j4j7|gj;t=| ycc} } w)NTrrGlobalzMemTracker does not support memory tracking for multiple iterative calls. Either use ``reset_mod_stats`` to clear module memory stats for the previous iteration or file a github issue if you need this feature.)r get_known_fqnrrHrr! parameter_mem buffer_mem input_memr>r?is_bwrCrrrgrrrrNotImplementedErrorrIrrrrLrMrrArr) rNrr"mod_namer param_memr(r)rr mem_snapshotrrs r* _pre_fw_hookzMemTracker._pre_fw_hookys $$226:### -- -$X.I$($I$I4%J% !Iz55f=I&/I ##-I "+I +4D  ($$E    $ $,,V4I''E||#&{{62 " $++334zAG7|q X%8)H ,,V4I$$E55f=I (I "+I 002 I$$ $?K?Q?Q?S$.;c8Xj))$I     * *9+<+rDrrr@r! output_memrMrrr)rNrr"r0rrr3s r* _post_fw_hookzMemTracker._post_fw_hooks((0    " "((E||'DLLNf,D# # %%E66w?J#-I &&ub1889R9R9TUr)c|tjddy|j|}|j}|j Dcic]\}}||t c}}|_|jjtjgj||jjtjgjt|ycc}}w)Nz%Module is None. Skipping PRE_BW hook.r stacklevel)warningswarnrrrrrLrMrr>rFrrBr)rNrrrr.rrs r* _pre_bw_hookzMemTracker._pre_bw_hooks > MMAa P ((0 002 ;G;M;M;O *7#xC*% %  &&y'8'8"=DD\R&&y'7'7<CC \ "  s C&c|tjddy|j|}|jj t j gj|jy)Nz&Module is None. Skipping POST_BW hook.rr6) r8r9rrMrr>rErr)rNrrrs r* _post_bw_hookzMemTracker._post_bw_hooks^ > MMBq Q ((0 &&y'8'8"=DD  % % ' r) optimizerc|jjD]D}|jD]/}t|tjs|j ||1FyrK)rrrrmrr)rNrVr=statesvals r*_track_optimizer_statesz"MemTracker._track_optimizer_statessW oo,,. F}} c5<<088  r)cdtjdtdtddffd }dtjdtdtddffd }t|t |f_y)Nr=rkwargsrWcd_y)NT)rr=rrCrNs r*_opt_step_pre_hookzFMemTracker._register_global_optimizer_hook.._opt_step_pre_hooks  DLr)cTjtj|d_yr2)rAr.r;rrEs r*_opt_step_post_hookzGMemTracker._register_global_optimizer_hook.._opt_step_post_hooks   ( () D DLr))r Optimizerr rrr)rNrFrHs` r*_register_global_optimizer_hookz*MemTracker._register_global_optimizer_hookss  .1 ;>   ! !.1 !;> !  ! --? @ -.A B( $r)c&|jjD]%\}}|j|j'|jj|j)|jD]}|jd|_yyrK)rrremoveclearr)rNrrhandles r*%_deregister_param_and_optimizer_hooksz0MemTracker._deregister_param_and_optimizer_hookss - - 4 4 6 /  %  # # % % , , .  / ((..0  ' ' 366   +/D ( 4r)externalct|\}}|D]}t|tjr!|j |t j >t|tjjr|j|dvt|tjr!|jt j||tdt|dy)a Track tensors and stateful objects like modules, optimizers etc. that are created outside the MemTracker. This method should be called before the ``MemTracker`` is used. Any tensors that are not module parameters, buffers, gradients activations, or optimizer states will be categorized as ``Other``. If you want them categorized with a custom name, please file a GitHub issue. Any tensors created outside the MemTracker and not supplied to this method will not be be tracked by ``MemTracker``. Args: *external (Union[nn.Module, optim.Optimizer, torch.Tensor]): The external modules, optimizers, and tensors to be tracked. Fr$NzObject of type zj is not supported for tracking. Only stateful objects like modules, optimizers, and tensors are supported.)rrrmrrr.r<rModulerrrIrAr; TypeErrorr])rNrP flat_externalrobjs r*track_externalzMemTracker.track_externals(1 q  C#u||,44OOC155ce5TC1,,[__cB%d3i[1ab r)rqrc\|j|}|r t||yt||y)a' Display the memory usage breakdown snapshot of the tracker based on the specified type and units. Keyword args: type (str): The type of snapshot to display. Can be "current" for the current memory usage or "peak" for the peak memory usage. Defaults to "current". units (str): The units to use for displaying memory usage. Defaults to "B". Supports ["B", "KiB", "MiB", "GiB"]. tabulate (bool): Whether to display the snapshot in a tabular format. Defaults to False. N)rrr)rNr]rqrrs r*display_snapshotzMemTracker.display_snapshot3s+,,T2  #He 4 He ,r)depthcndtdttttffdt |j j fdD]e}|j}|jddz}||kDr)td||rt|j|Pt|j|gy ) aZ Print per device memory breakdown snapshot for each module called within MemTracker. Snapshots are displayed for the states defined by ``_ModState``. The module hierarchy is displayed up to the specified depth. Keyword Args: depth (int, optional): The depth of the module hierarchy to display. Defaults to 2. units (str, optional): The units to use for memory tracking. Defaults to "B". Supports ["B", "KiB", "MiB", "GiB"]. tabulate (bool, optional): Whether to display the snapshot in a tabular format. Defaults to False. srWctjd|Dcgc]-}|jr t|n|j /c}Scc}w)Nz([0-9]+))resplitisdigitrllower)r[texts r*natural_sort_keyzAMemTracker.display_modulewise_snapshots..natural_sort_keyTsFHHZ3"\\^D = s2Ac(|jSrK)rI)m_statsrbs r*r z9MemTracker.display_modulewise_snapshots..\s 0 Ar))r.rz Module: N) rPrr rlsortedrrrIcountrrrMr)rNrYrqrrrI mod_depthrbs @r*display_modulewise_snapshotsz'MemTracker.display_modulewise_snapshotsEs  U38_(=    ' ' )A CI ''G c*Q.I5  IgY' (.y/B/BEJ&y':':EB Cr)c8|jjy)z[ Reset all the module memory stats. Clears ``memory_tracking`` dictionary. N)rrMrs r*reset_mod_statszMemTracker.reset_mod_statshs ""$r)cdtjjdttdfdt t tfdtffd }|tj_ y)Nop_callr.rCrWc|tjjvrn t5j |||cdddS#1swYyxYwrK)rr_custom_op_handlersrr)rmrrCrNs r*track_dtensor_dispatchzBMemTracker._track_dtensor_dispatch..track_dtensor_dispatchosPg44HHH ] J 227D&I  J J Js AA) rm_ops OpOverloadrpobjectrrPrrr)rNrps` r*_track_dtensor_dispatchz"MemTracker._track_dtensor_dispatchns^ JZZ** J $ Jf% J J+A'r)cB|jtj_yrK)rrrrrs r*_restore_dtensor_dispatchz$MemTracker._restore_dtensor_dispatch}s*.*E*E'r)c6|jdk(r|j|jj|j|j |j |j|j|j|j|_ |jjDcic]\}}||tc}}|_|jjt |=|xjdz c_|Scc}}w)Nrr)rrJrregister_user_hooksr/r4r:r<rrtrrrrr __enter__super)rNrr __class__s r*ryzMemTracker.__enter__s ;;!   0 0 2    1 1!!""!!""       ( ( *"&";";"=D &*%8%8%>%>%@!CXj))DN    ' ' )  q  s9Dc,|xjdzc_|jdk(rc|j|jj|j |j |jj |t||y)Nrr)rrOrclear_user_hooksrrv__exit__rz)rNrr{s r*r~zMemTracker.__exit__sv q ;;!   6 6 8    . . 0  "  * * , &D   & & - $r)cL|tjjjjk(rt r|d}n ||i|xsi}|j rtj}nC|jjr|jstj}ntj}ttjt!|j"|||jjrt$j&nt$j(}|j+||Sr)rmops_c10d_functional wait_tensordefaultrrr.r;rr*rr:r9rrrrr>rFrAr)rNfunctypesrrCresrVrs r*__torch_dispatch__zMemTracker.__torch_dispatch__s EII..::BB B "q'C- "-C <<!ooG    $ $T[[!&&G!ooGellGDKK$A3G*.*;*;*A*AY&&yGXGX   + r))rWNr)F)r)T)rrsF)rrsF)rWr )r(N)6r$r%r&r'rOrrRr rlr"rrmrboolrrrrgrrrr,rrrPrrUrrrRrprr r!r/r4r:r<rrIrArJrOr rVrXrirkrtrvryr~r __classcell__)r{s@r*r r Ds.`6+/*. *6*6*6#3- *6 h' *6  *6`!& ! <<!! ! \  !F6l6'++6$6 / 9>V>>.>h>5<<>D>2$55 ellDcN* +52=A--ii--59-- sCx--^ &S &S &6Q2996Qc6Qd6QpVBIIVsVSVTV& 299 C D &  BII  S  T    ,1OO    ( 0 ryy%//5<<GH   FIN--,/-AE- -&BG!C!C%(!C:>!C !CF% AF* c d r))Hr^osr]r8 contextlibrcopyrenumrr functoolsrrtypingr r r r r typing_extensionsrrm)torch.distributed._tools.fake_collectivesrr torch._guardsr%torch.distributed._tools.common_utilsr$torch.distributed._tools.mod_trackerrtorch.distributed.tensorrtorch.optim.optimizerrrtorch.utils._python_dispatchrtorch.utils._pytreerrtorch.utils.weakrrtorch.utils.hooksrrlenvironrr`r__all__rPr"r,r.r>rHrRr{floatrrrUrrrrrrr r(r)r*rs( "$@@" 0*F;,;;91  ?C D IDq .tsDtFS$F(,0WWOH d5<<c3h#?@  QU (A5<<c3h/0A9<A A8 FDellDcN&B!CDDE NQ   @FDellDcN&B!CDDE@NQ@ @D $ q "q r)