g^(h STddlZddlZddlZddlZddlmZmZmZddlm Z ddl m Z m Z ddl mZddlmZmZmZmZmZmZmZddlZddlmZddlmcmcmZddlm Z ddl!m"Z"ddl#m$Z$dd l%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,dd l-m.Z.dd l/m0Z0m1Z1dd l2m3Z3m4Z4dd l5m6Z6m7Z7ddl8m9Z9m:Z:m;Z;ddlZ>ddl?m@Z@erddlAmBZBejeDZEe GddZFdeGeHefdeeIeHeffdZJe GddZKGddeZLGddeZMdeFdeGeHefdeNd eNd!eNdeOeGeHeff d"ZPd#edeNfd$ZQdeFdeGeHefdeKfd%ZRdeFd&eKd eNdeOeGeHeffd'ZSd(e'd)eGeHefd*eejdeGeHeffd+ZUd(e'd&ed*eejdefd,ZVdeFd-eHd)eGeHefdeGeHeffd.ZW dhd/eGeHefd0e jd1eNd2eejjd3eNd*eejdeGeHeffd4Z[deFd5eGeHeGeHeffd6eOeHdeGeHeffd7Z\d8eHd9eOejd6eOeHd:eejd;e1dejf d<Z_d8eHd=eOejd6eOeHdejfd>Z`d8eHd?eOed6eOeHdefd@Za didAeGeHefd0e jd2ejjdBeeeOeGeHefee jfdCeNdDeNdeGeHeffdEZc djd0e jdBeeeOeGeHefee jfdeGede jffdFZed0ej@jdeGe0eHffdGZf dkd2ejjd0ee jdDeNdHeeGe jeOeHfdIeeGe0eHfdeGeedeHfe jff dJZg dkd2ejjd0ee jdDeNdHeeGe jeOeHfdIeeGe0eHfdeGe jeedeHfff dKZh djd0e jdBeeeOeGeHefee jfdeGe jedffdLZidMeOeMdNeGeMeeHedffdOeGeeHedfe jfd*eejddf dPZj did/eGeHefd*eejdOeGeedeHfe jfdHeGe jeOeHfdQeGeHeFfdReNdeIeOeMeGeMeedeHffffdSZkdTeGeHefdOeGeedeHfe jfdHeGe jeOeHfdeOeGeHeffdUZld/eGeHefdeNfdVZme GdWdXZnd(e'dYeGeHefdeOeGeHenffdZZodeFd[eOeGeHenfdYeGeHefd\eGeHeGeHeffdeIeejeGeHeOeejfff d]ZqdeFd\eGeHeGeHeffd8eHd eNdeNd!eNddfd^ZrdeFd[eOeGeHenfdYeGeHefd eNdeNd!eNdeGeHeGeHefffd_ZsdeFdYeGeHefd eNdeNd!eNdeGeHeff d`Zt dldaeOeMdNeGeMeedeHffdQeGeHeFfd/eGeeHedfefdeNd eNd!eNdeGeHeffdbZu dldaeOeMdNeGeMeedeHffdQeGeHeFfd/eGeeHedfefdeNd eNd!eNdeGeHeffdcZvej dmd0e jd2ejjd/eGeHefdBeeeOeGeHefee jfd3eNd eNd*eejdCeNd1eNd!eNdeGeHeffddZxd0e jdeGeHeFffdeZyed(e'dfe:ddfdgZzy)nN)IterableIteratorSequence) ExitStack) dataclassfield)chain)Anycast NamedTuple no_type_checkOptional TYPE_CHECKINGUnion)_gather_state_dict)_get_pg_default_device)_apply_to_modules _FSDPState._get_module_fsdp_state_if_fully_sharded_module_get_param_to_fqns_module_handle!_named_parameters_with_duplicatesclean_tensor_name)SimpleProfiler) FlatParameterFlatParamHandle)_ext_chunk_dtensor_ext_chunk_tensor) _lazy_init%_reset_flat_param_grad_info_if_needed)ShardingStrategyStateDictSettings StateDictType)DTensor Replicate) tree_map_only) ShardedTensorcFeZdZUeed<eed<eeefed<e e ed<y) FSDPParamInfostatehandle param_indicesparam_requires_gradN) __name__ __module__ __qualname__r__annotations__rdictstrintlistboola/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/torch/distributed/fsdp/_optim_utils.pyr)r)5s&  S>!d#r8r) dictionaryreturnc#^Kt|j}|D] }|||f ywN)sortedkeys)r:r?ks r9 sorted_itemsrA=s5 *//# $D As+-ceZdZUdZeeZeeejfe d<eeZ eeejfe d<eeZ eee fe d<y)_ConsolidatedOptimStateap This holds the consolidated optimizer state on the target rank. Positive- dimension tensor state is communicated across ranks, while zero-dimension tensor state and non-tensor state is taken directly from the target rank. PyTorch version 1.12 moved to using zero-dimension tensors for scalar values, but user implemented optimizers may still use float (i.e. a non-tensor). Thus, we support both and handle them identically. Attributes: tensor_state (Dict[str, torch.Tensor]): Mapping from positive-dimension tensor state name to the unsharded flat tensor representing the state. zero_dim_tensor_state (Dict[str, torch.Tensor]): Mapping from zero- dimension tensor state name to its value. non_tensor_state (Dict[str, Any]): Mapping from non-tensor state name to its value. )default_factory tensor_statezero_dim_tensor_statenon_tensor_stateN)r.r/r0__doc__rr2rEr3torchTensorr1rFrGr r7r8r9rCrCCs\&-2$,GL$sELL()G5:45P4U\\ 12P',T'Bd38nBr8rCcNeZdZUdZej ed<ejed<y)_PosDimTensorInfoa Metadata for positive-dimension tensors used internally for :meth:`scatter_full_optim_state_dict`. Attributes: shape (torch.Size): Sharded tensor shape (which is equal to the unsharded tensor shape if the tensor is optimizer state for a non-FSDP parameter and is hence not sharded). dtype (torch.dtype): Data type of the tensor. shapedtypeN)r.r/r0rHrISizer1rNr7r8r9rLrL]s  :: ;;r8rLc0eZdZUdZeedfed<eed<y)_OptimStateKeyz This represents an optimizer state key that may be used commonly across ranks. It is based on the unflattened parameter names rather than parameter IDs to make it independent of each rank's own optimizer construction. .unflat_param_namesis_fsdp_managedN)r.r/r0rHtupler3r1r6r7r8r9rQrQms c3h'r8rQfsdp_param_infoflat_param_stateto_save shard_state cpu_offloadc |r |sJdt||}|rjt|||}|D]V}|st|jD]5}||} t | t j s#| j||<7X|SgS)aO Unflattens the optimizer state, consisting of the "state" part and the "param_groups" part. Unflattening the "state" part involves consolidating the state on the target rank and remapping from flattened to unflattened parameter IDs, and the "param_groups" part only involves remapping from flattened to unflattened parameter IDs. Args: fsdp_param_info (FSDPParamInfo): The FSDP state, the handle, and a mapping from FQN to original parameter index. flat_param_state (Dict[str, Any]): Entry for the flat parameter in the "state" part of the optimizer state dict. to_save (bool): Whether to save the state on this rank. Returns: List[Dict[str, Any]]: A :class:`list` holding the entries in the "state" part of the optimizer state dict corresponding to the unflattened parameters comprising the flat parameter if on the target rank or an empty :class:`list` otherwise. The final optimizer state dict will need to map these entries using the proper unflattened parameter IDs. z7If ``shard_state`` is True, ``to_save`` has to be True.)_communicate_optim_state#_unflatten_communicated_optim_stater5r? isinstancerIrJcpu) rUrVrWrXrYconsolidated_stateunflat_param_state optim_statekeyr*s r9_unflatten_optim_statercxs:gA %2@     . 3K 0 0 233C',E%eU\\: ',yy{K$ 3 3"! r8xcVtj|xr|jdk(SNr)rI is_tensordim)rds r9_is_zero_dim_tensorris ??1  .!%%'Q,.r8c|j}|jj}t}|j|j |j }}}t|D]\}} tj| rB| jdkDr.|jdk(s|jtjk(r| ||<c|jJd| j j"|jj"k7r| j%|j} |j&j)} | j*| } t-j.| | |j0|j2j5t7t8j:|j<j?} | d| ||<_tA| r#| jCjE||<| ||<|S)a Communicates the optimizer state for a flat parameter across ranks. All ranks will hold the entire non-sharded optimizer state on GPU. If ``N`` is the number of tensor optimizer states in the optimizer state dict, then the communication complexity is 0 if ``N = 0`` and ``N + 1`` otherwise (where the plus 1 comes from all-gathering the padding per rank). Args: fsdp_param_info (FSDPParamInfo): The FSDP state, the handle, and a mapping from FQN to original parameter index. flat_param_state (Dict[str, Any]): The entry in the "state" part of the optimizer state dict corresponding to the flat parameter. Returns: ConsolidatedOptimState: Consolidated optimizer state for the target flat parameter. rNz'compute_device has not been initializedgroup)#r*r+ flat_paramrCrErFrGrArIrgrh world_sizesharding_strategyr!NO_SHARDcompute_devicedevicetypeto_full_param_paddedsize new_zerosdistall_gather_into_tensor process_group_device_handle synchronizer nn Parameter_unpadded_unsharded_sizenumelridetachclone) rUrV fsdp_staternr*rErFrG state_namevalue buffer_size tensor_bufferunpadded_numels r9r[r[s,!&&J ''22J # %E  ## *:'L **:;#5 E ??5 !eiikAo %%*//3C3L3LL+0 Z(,,8 9 8||  J$=$=$B$BB!:!:;%77<<>K+EOO[9M  ' 'uJ,D,D   % % 1 1 3! jAAeg (5_n'EL $#5)49LLN4H4H4J%j1/4 ,G#5H Lr8r*c R|j}|j}|j}g}i}|j}|j|j |j } } } t|D]?} i} t| D]\}}||v}|s|j|}|||<n||}t|}|r|j}t|ddr;|jJt||j|j|j }n^|j"Jt%||j|j&|j(j+|j"|j }|| |<t| D] \}}|| |< t| D] \}}|| |< |j-| B|S)a Unflattens the communicated optimizer state (given by ``tensor_state``, ``non_tensor_state``, and ``zero_dim_tensor_state``) for a single flat parameter. This should only be called on the target rank. Args: fsdp_param_info (FSDPParamInfo): The FSDP state, the handle, and a mapping from FQN to original parameter index. state (_ConsolidatedOptimState): Consolidated optimizer state. Returns: List[Dict[str, Any]]: A :class:`list` holding the entries in the "state" part of the optimizer state dict corresponding to the unflattened parameters comprising the flat parameter. The final optimizer state dict will need to map these entries using the proper unflattened parameter IDs. _use_dtensorF)r*r+rn _num_paramsrErFrGrangerA_get_unflat_viewsnext_optim_state_dict_configgetattr _device_meshrrank_fsdp_extensionr{rror| device_countappend)rUr*rXrr+rnr`flat_param_viewsnum_unflat_paramsrErFrG_unflat_state_paramr flat_tensorviews_generatedviewsra osd_configzero_dim_tensor non_tensors r9r\r\s,!&&J  # #F""J/1,."..  ## *:'L $ %'6'3L'A 9 #J (,<>>"4#""//"22 #K&33???"3#""--"11>>@"00"22 #K.9 z *9 9>,88M+N = 'J-< z * ='33C&D 8 "J -7 z * 8!!"45O'6P r8rrarmcdg}tj|dk(rttjd||d<tj |d|tj|dk(r|S|dS)Nrc|jdk(r|jSt|j|jSrf)rhr^rLrMrN)vs r9z,_broadcast_processed_state..Os,Aaeeg3DQWWagg3Vr8srcrm)ryget_rankr&rIrJbroadcast_object_list)rrarmobjectss r9_broadcast_processed_staterFsl G }}Uq " LL V    wAU; }}Uq qzr8ctj|dk(rKt|tjr|j dk(r|S|j |j}n~t|tjr|j dk(sJd|St|ts|Stj|j|j|j}tj|d||S)NrzlFor non-zero ranks, a tensor state should have zero dimension, but got the state with shape {state.shape()}.rNrsr) ryrr]rIrJrhrurrrLzerosrMrN broadcast)rr*rmtensors r9_broadcast_staterYs }}Uq %.%))+2BL*334 eU\\ *99;!# @ #LE#45L KKu{{:3L3L  NN6q. Mr8fqnc.|siS|j}|jj}|j|}|j|}t ||j |j}|jsiSi}|j}|j} |jD]s\} } tj| rT| jdkDrA|jt j"k7r$| j%|| dzj'} | || <u|S)z Shard the optimizer state for the original parameter with the name ``fqn``. This API should only be used when ``use_orig_params`` is True. pgrsrrk)r*r+rnr,_shard_param_infosrr{rrin_shardintra_param_start_idxintra_param_end_idxitemsrIrgrhrpr!rqflattenr) rUrrarrn param_idxshard_param_infonew_optim_staterrrrs r9_shard_orig_param_staterps   &&J ''22J--c2I!44Y?$ 009R9RK  $ $ &(O,BB*>>(..0 , E OOE " a,,0@0I0IIMMO%(;)eg ', # , r8optim_state_dictmodeluse_orig_paramsoptim rank0_onlyctj|}d|vr |s tdt|}t |}t t |jj} |rt| ||}i} |d} t| j} |jD]\} }|d}|| vr| j||rD|D]:}| |s | |jD]}t| | |||| ||<<|d}||vr||}|rGtjtj j"5t%||| |}dddn t'|| |}t)t+|d}r|| |<n|ryt-|dk(s Jd|d ||jj/| d}|t1j2|| |<nvt5j6d |d | j8d nOt;d |d t-|dk(sJt)t+|d}t1j0| || |<|s|D]X}| |s t=| |jD]1\}}| j8dkDr| ||=|j?| ||<3Z| D]K}| |}tA|tBjDr|r|rt| ||}t1j0|| |<MtjFdd|vrt1j2|d}| |dSd| iS#1swYxYw)a Flattens the full optimizer state dict, still keying by unflattened parameter names. If ``use_orig_params`` is True, each rank will have all FSDP-managed parameters but some of these parameters may be empty due to the sharding. For a regular optim.Optimizer, states for those empty parameters will not be initialized. So, when aggregating the FQNs across ranks, no assert will be raised on a rank even if it does not have all the states -- it is valid and FSDP know how to aggregate them. However, FSDP has to ignore handling those parameters that are not managed by FSDP and do not exist on the local rank -- it is managed by other parallelism and FSDP does not know ho to handle/aggregate them. Note that ``_flatten_tensor_optim_state`` does not need ``optim`` to flatten/shard the state. However, NamedOptimizer and KeyedOptimizer require all the states even if the corresponding parameters are empty. To this end, ``optim`` will be used to to get the initial state of the empty parameters. ``optim`` should only be non-None if the ``optim` is KeyedOptimizer or NamedOptimizer. Returns: Dict[str, Any]: The flattened optimizer state dict. r*zO`optim_state_dict` must have the keys "state"to be a valid optimizer state dictrlrNTrkz5use_orig_params is True but there are multiple FQNs, .z optim_state[z] is not on rankz The state of z8 is empty. This should happen when use_orig_params=True.Fz,FSDP _flatten_optim_state_dict() profiling: param_groupsr*r)$rreset ValueErrorr_get_fqn_to_fsdp_param_inforitervaluesr*rsetr?rdifference_updaterprofileType RESHARDINGr_flatten_optim_staterQrTlengetcopydeepcopywarningswarnr RuntimeErrorr5r^r]rIrJdump_and_reset)rrrrrrm unflat_osd param_to_fqnsfqn_to_fsdp_param_inforflat_osd_stateunflat_osd_stateall_state_keysparamfqnsrrrU flat_staterbr* param_state user_stateflat_osd_param_groupss r9_flatten_optim_state_dictrs@!Jj  1  'u-M8?d188:;<BBJ/ JeT =?N!'*)..01N$**,FN t1g & & ((.  ',"23"7"<"<">J8H"$4S$9*$EU9$S)*5 q'C ( (4S9O#++N,?,?,J,JK!8'(-"J2#$ !td3C&0s# 4yA~KD6QRS~$!KKOOE48E(.2mmE.Bs+ *3%/? ?PPQR ##C5),, t9> !> te4C"&)), "4    0 0  . /'',,     K1   %   05578Kc"4"9"9";<< ##5"68   "" "57J!: ': "/A.L z *RV V  -9 JqAM J J$%)Jz " GKKK2_& 6A 5??1#5#E!%%'A+ E  $7$: : 5??1#55 5O 6#88Qa88 u:? #7? @  f E_.AB  >c%jAoUV V  FLLE$9$ll^1UG5  e$J#&o7J"K  K  "  knnZ01 ]] KK!     (();V=R=RSK!::    0 0 {0011BCSBTU 0 UE 0. sFFF!.AF&zero_dim_tensorsc |Dcgc]}|| }}|Dchc]}||jnd}}|Dchc]}| |jnd}}t|t|k7st|dk7st|dk7rtd|d|d|d|t t |}t t |}t j||t jdScc}wcc}wcc}w) a Flattens the zero-dimension tensor optimizer state given by the values ``zero_dim_tensors`` for the state ``state_name`` for a single flat parameter corresponding to the unflattened parameter names ``unflat_param_names`` by enforcing that all tensors are the same and using that common value. NOTE: The requirement that the tensors are the same across all unflattened parameters comprising the flat parameter is needed to maintain the invariant that FSDP performs the same computation as its non-sharded equivalent. This means that none of the unflattened parameters can be missing this state since imposing a value may differ from having no value. For example, for Adam's "step", no value means maximum bias correction, while having some positive value means less bias correction. Args: state_name (str): Optimizer state name. zero_dim_tensors (List[torch.Tensor]): Zero-dimension optimizer state for the unflattened parameters corresponding to the single flat parameter. unflat_param_names (List[str]): A :class:`list` of unflattened parameter names corresponding to the single flat parameter. Returns: torch.Tensor: A zero-dimensional tensor giving the value of the state ``state_name`` for all unflattened parameters corresponding to the names ``unflat_param_names``. NrkAll unflattened parameters comprising a single flat parameter must have scalar state with the same value and dtype but got values z and dtypes rrr^r) itemrNrrrrrIrrs) rr rRrr values_setrrrNs r9rrsB$4Eaq}EE?OP!am!&&(5PJP:J KQaggD0 KF K %5!66 z?a  v;!  (\fX[l;!"  $   j! "E f E <<U5<<3F GG%FP KsC.C.C3C8 non_tensorsc|Dcgc]}|| }}t|}t|t|k7st|dk7rtd|d|d|tt |}|Scc}w)a Flattens the non-tensor optimizer state given by the values ``non_tensors`` for the state ``state_name`` for a single flat parameter corresponding to the unflattened parameter names ``unflat_param_names`` by enforcing that all values are the same and using that common value. See the note in :func:`_flatten_zero_dim_tensor_optim_state`. Args: state_name (str): Optimizer state name. non_tensors (List[Any]): Non-tensor optimizer state for the unflattened parameters corresponding to the single flat parameter. unflat_param_names (List[str]): A :class:`list` of unflattened parameter names corresponding to the single flat parameter. Returns: Any: A non-tensor giving the value of the state ``state_name`` for all unflattened parameters corresponding to the names ``unflat_param_names``. rkr rz" and unflattened parameter names )rrrrr)rrrRntnon_none_non_tensorsnon_tensor_setrs r9rr8s2*5G2BGG%N  C $44N8Kq8P ,-[ E++=*> @  d>*+J Hs A-A- sharded_osd optim_inputusing_optim_inputis_named_optimizerc t|}t|}tttj t ttff|r t||nt|||||}t|t|ksJi} i} |jD])\} } | |vr || } | | t| <| D]}| | |< +|d}i}|jD]F\}}t|tr|||<| j|j |j } ||| <Hd|vr[g}|dD]L}t#j$|}t'|dDchc]}| | c}}||d<|j)|N||dSd|iScc}w)a Rekeys the optimizer state dict from unflattened parameter names to flat parameter IDs according to the calling rank's ``optim``, which may be different across ranks. In particular, the unflattened parameter names are represented as :class:`_OptimStateKey` s. r*rparamsr)r_get_flat_param_to_fqnr r2r~rrr4r3'_get_param_to_param_id_from_optim_input_get_param_to_param_keyrrrTr]rrRrrr>r)rrrrrrrflat_param_to_fqnparam_to_param_key$unflat_param_names_to_flat_param_key#unflat_param_name_to_flat_param_keyrrRflat_param_keyrsharded_osd_staterekeyed_osd_staterbrrekeyed_osd_param_groupsunflat_param_groupflat_param_groupflat_param_keyss r9_rekey_sharded_optim_state_dictr)_s &'u-M.u5>B R\\5c? *+! 4E; G(u0-AR ? ! "c-&8 88 8 ) (&3%8%8%:T!! * * +E2JX,U3E-FG!3 T ES /0A B T T$G,46-3358[ c3 %0 c " =AA  " "C$:$: -8.)8$9; "-n"= > #}}-?@ $.@-I)88IJO *9 X & $ + +,< = >+??KI-z%66 Zt,, - ySTTIf%&& 9,.8 ![0  & ~   X!678  +, --;  "m %  s*D D(D##D(c td}d}i}t|||t|Dcgc]\}}| c}}|Scc}}w)a Constructs a mapping from ``FlatParameter`` to a cleaned (devoid of prefixes from wrappers) fully qualified name (FQN). Note that this FQN is "non-canonical" because ``FlatParameter`` s do not come from the original module but are registered only after FSDP has been applied. This function returns the FSDP-given name for the ``FlatParameter`` (usually module._flat_param) as opposed to the canonical FQNs returned for ``FlatParameter`` s in ``_common_utils._get_param_to_fqns(...)``). Consequently, this function will only return a non-empty mapping if FSDP was applied with ``use_orig_params=False`` as, otherwise, the original parameters are used within the module and there would be no ``FlatParameter`` s in the module. cvt|dD])\}}t|tst||z}|||<+y)NF)recurse)rr]rr)moduleprefix tree_levelr param_namerrs r9 module_fnz)_get_flat_param_to_fqn..module_fn sI!B E"  + Je]3#FZ$78C'* e $  +r8c|Sr=r7)rs r9 return_fnz)_get_flat_param_to_fqn..return_fn  r8rr)rr=r?flat_param_to_fqn_retrrs r9rrsJ+!79  -J R J   7> @FCi#6H2I&I = I $E *F,,L 12%++fUKOOKu-A"'(;(;E(B"CDQDDD xUC /  $H- JD$^,d3D4y1}dV#J:>?3..?@B  9%%D@s  E":E'r merge_keysctj|}i}g}|jD]\} } | |dvr|| } t| t} | r*| d|vs#J| dt |j f| d|v} tt| | } |dk(s|r|j| | || <|rrttj|Dcgc]}g}}tj|||gtj|}tt!|}||fS|dk(r|gndg}tj"|d||dJ|d}t%||||||fScc}w)a@ Construct the local mapping between the ``_OptimStateKey`` and parameter keys and all the ``_OptimStateKey`` across ranks. If ``merge_keys`` is False, rank0 must contain all the ``_OptimStateKey``, an exception will be raised otherwise. Note that ``merge_keys`` should equal to ``use_orig_params``. r*r)rRrSrlNr)ryrrr]rr5r?rQrTrrrRrSr from_iterabler>rrrZ)rrmrGrrr[rrNall_optim_state_keysrVrrrSoptim_state_keyrall_keysmerge_all_optim_state_keys key_obj_lists r9_map_param_key_to_optim_keysrcs == DJL 13.446B 5 ,W5 5 U#$UM: 744 Q+00237 4q'%;;($T{+  19 ' ' 88A$_5'B*d11%890 B0 0  x)=UK%Eu':':8'D%E"%c*D&EF != =='+ai ! "dV  "":K U[[;O++J7;@++J7 @%4S!4 && 17s D rs output_statesc i}|jD]`\}}|Dcgc]}|| }}t|D chc]#} | jjD]} | %c} } } t } d} | D]%}g}t }t |D]\}}|j d|jj|d}|=|jj|d<| s |j} n| |jk(sJ|ddk(s{|j|| r| |k(sJ|} ||vr|jDcgc]}dc}||<||j|d}|%|j|jj}||||j|<(t |D]\}}|| vr |j jD]:\}}|j|d}|||k(sJd|d|d|dd|z|||<<|j"jD]K\}}|j|d}|-t%j&||sJd|d|d|dd|z|||<Mc |fScc}wcc} } wcc}w) a1 Given the ``gathered_state_info`` and ``input_states``, the API converted the StateInfo into the original state if the state is not a non-scalar tensor. For a multi-dimensional tensor, the local state will be stored in ``state_buffer`` in a correct order for later allgather purpose. NrzRank z has different values for z: rz Other ranks: )rr>rnr?rr+rrrMrrNaddr,rur*rrrrorIequal)rUrsrprv state_buffersrgathered_states state_infor*nall_tensor_states empty_ranksrNrnumels _empty_ranksr object_stateinfor local_statenamenon_tensor_valuecurr_non_tensor_valuescalar_tensor_valuecurr_scalar_tensor_values r9_convert_all_state_infor1s1>@M,224?;^&9:af: :"& E5 0B0B0D E1Q EQ E !$ '+, XJF%(UL&/ &; +"l a #++// DA#!%!1!1!3F2J $ $ 222":? $$T* +#k\&A AA&K."1"?"?-D- j)'s+// DAK&)nn_-B-B-Q-QR LWM* %o&C&CC&H I7 X@#,J"7 ; D,{"*6*B*B*H*H*J 8&&(6(:(:4(F%)1,0@@D6!;D6DTCUUVW&'<&=>? A (8t$ 8.:-H-H-N-N-P ;))+9+=+=dD+I(/75;;')A<D6!;D6DWCXXYZ&'?&@AB (;t$ ; ;S?;B - A; E0-s I;(J : Jc D|sy|jj}|j}|jD]O\}} | |} |j|} t | t r| jd} | tk7r| j} | jtft|j| }|| xx| jjdzcc<tj |}| j#|} n=| j#|j| } n| j#|j| } |r|j$}t'|ddr;|j(Jt+| |j,|j(|j.} n|j0Jt3| |j,|j4|j6j9|j0|j.} n>|s59!..:::*OO++.. "//;;;)OO))--::<,,.. ''0 / ,,. / ''(;(;(?(?@ $  $%*z"e2+X / / $ $sJ (J J J c |j}|jdk(r]tjtjj k(r.t jd|jj|jDcic]}|i}}t||||\} } t| dk(r|S|jjD cgc] \}} ||vrdnd} }} |jj } t#j$t&j(| |j*}|| j,}|jj/| jD]\}}g}|j| j0j3z}|| j0j3zdz }d\}}t5| j6| j8D]\}}| xr|j:| xr| | }|s|r|||zdz }}||cxkr|krnn||k\r||z dzn||z dz}n=||cxkr|krnn||kr||z dzn||z dz}n||cxkr |cxkr|krnn|}nd}|r|j=|||s6||,|j=t?t&j@|||dz }||z }| j0j3tCd |Dz }| jD|k(sFJd | jDd |d | j0j3d | j6d|d|d |dkDr|j=||t'jF|}|j3|jHz|j3k(sJd|jj/tKjLtJjNjP5tjR|||jT|jj/ddd|d| jVj3}|j} | jY|}!t|!t|jk(sJd|jjD]$\}} |j:| s||vs|!| |||<&t[||||||~|Scc}wcc} }w#1swYxYw)z Given the ``gathered_state_info`` and ``input_states``, the API allgathers all tensor states and restore non-tensor states from ``gathered_state_info``. rz@Memory Summary before calling to _allgather_orig_param_states %sTFrrk)rrNc3<K|]}|jywr=)r).0rs r9 z/_allgather_orig_param_states..>s1a 1szLManually calculated _sharded_numel_padded is incorrect. _shard_numel_padded=z, shard_numel_padded=z, _sharded_size.numel=z, _numels_with_padding=z, begin=z, end=,zThe size of local shard times the world size should equal to the gathered tensor size. The inconsistency may be from a bug of FlatParameter's metadata or the reconstruction logic in optimizer state dict.rlzThe number of parameters from FlatParameter is not consistent to the number of states used by optimizer state dict reconstruction logic.).r*rryget_debug_level DebugLevelDETAILloggerrr|memory_summaryr?rrr,rr+rn functoolspartialrIemptyrr_padded_unsharded_sizer} _sharded_sizerr_numels_with_padding_is_padding_maskr-rr rJsum_shard_numel_paddedcatrorrr ALLGATHERrzr{r_get_unflat_views_alignedr)"rUrsrprXrWrYrrrvrNr{idxhas_state_paramsrn empty_funcgathered_tensorrbuffers local_buffersbeginend mem_offsetrr is_paddingfrozen_and_no_state padding_begin padding_end padding_lenshard_numel_padded local_shardunpadded_tensorflat_param_handle orig_statess" r9_allgather_orig_param_statesrso!&&J! 4 4 6$//:P:P P N  % % 4 4 6 DPCTCTCV/WCR/WM/W2,lME= =Q(55;;=$ C}$%/$$!''22J"" 5)B)BJ!!B!BCO))+,224r  G,. *":":"@"@"BBj..4466: $ I!$  + +Z-H-H" 8 E:'1.##77 BB4(33 0 .8e9Ka9O{  E8[8 +-$e+a/ 5[1_ #c8[8 !M1m+a/ 5[1_ ]?k?C?#(K"#K!((K)@A9%1!((ellGI!? @ii .   "Z%:%::o>S>S>UU   U !!--/  # #N$7$7$A$A B 4  ' 'J4L4L   % % 1 1 3  4**WJ,O,O,U,U,WX+22'AA/R ;3'D'D#EE   E (55;;= BHC22373-;O1r?) rUrprXrWrYrrsrvrbrs r9_gather_all_orig_param_statertsl!&&J"  ' '+;+D+D D&|.B.    3 3 > > ?   # #N$7$7$E$E F R"7 L"Q  R4          '55;;= HCm#"66s;%8/77<<>?@A8-,,./0 3   5 R R   s$,.E E'EE EE$r^c i}i}|D]P} |j| d} | | js%| jrZ| jd} |j| d} | U| in|| } t| |vri|t| <| |t| | <|st | jdk(sJ| jd}t j d5ttttf| } tj|| ||<|rBt||D]1\}}tj|s|j|||<3dddS|j!D]}t#t%|j'} || } t | j(dkDsJd| j*j-D]e\}}||vr | j(|st/|dt1| j*j'dt1|j'd|j3t5| |||||S#1swY[xYw)Nrrknone_fsdp_managed_copyzgWith use_orig_params, FSDPParamInfo should have requires_grad information. However, the length is zero.zE is not in the optimizer state. The FSDPParamInfo has the param keys z( while the optimizer has the param keys r)rrSrRidrrrr rr3r4rrArIrgr^rrrr?r-r,rrr>updater)r^rNrrrWrXrYfsdp_osd_state all_statesr_rVrrUr*rrr _all_statesrbrs r9_convert_state_with_orig_paramsrs&(N -/J0&T+G+K+K T,   _%D%D   * *!44Q7C488dCO&#+B1A)1LE/"*424 2o./38Jr/* +C 0 99:a? ?? / B B1 E ''(@A T sCx)< 48II$Y/501-9&'89.T) E %u5$HM '89*E T T T7&TT"((* 4 ((*+,05?667!;  8 ;(55;;= HCk!"66s;%8/77<<>?@A4+**,-.a 1    (   ' : Y T Ts A=II" cni}|D],}|j|d} | Jd|d| |jrp|jd} || } t| || |||} |sZt | t |jk(sJt |j| D] \} }||| < |st |jdk(sJ|jd} t j || || <|st|| D]1\}}tj|s|j|| |<3/|S)NzQIf use_orig_params is False, we must be able to find the corresponding param id. rrrk) rrSrRrcrrrrArIrgr^)r^rNrrrWrXrYrr_rVrrU unflat_staterr`rrs r9_convert_state_with_flat_paramsrs&(N0(P+G+K+K T, $  ''6&7q  E $  * *"44Q7C4S9O1 + L<(C0R0R,SSSS=@#66 >K9%'99KN#45 K 99:a? ?? / B B1 E 04 :J9:U0VN, -)5"#45*P%J!??51 DIIIKN#45jA PG(PT r8c tjt} | jtjtj j ttj|| xstj|dk(xs|} tjd5t|} t|} t|}tt t"t$t&ft(j*f|r t-||nt/|||| | }t1|}dddtjd5t3|| |\}}dddtjd5|rt4nt6}||d| || }ddd| siSdi}t9 j;}|dj=D]2\}}||vr ||vr|vrt?j@d|d |||<4d |vrtC| |d <| jEtjFd |S#1swYxYw#1swYxYw#1swYxYw) a Consolidates the optimizer state and returns it as a :class:`dict` following the convention of :meth:`torch.optim.Optimizer.state_dict`, i.e. with keys ``"state"`` and ``"param_groups"``. The flat parameters in ``FSDP`` modules contained in ``model`` are mapped back to their unflattened parameters. Parameter keys are not well-defined. For a regular optimizer, the optimizer state_dict contains a mapping from parameter IDs to parameter states. Parameter IDs are the order of parameters in ``optim.param_groups()`` across all the groups. This API also allows user to pass ``optim_input`` for the mapping between parameters and parameter IDs. Using ``optim_input`` is being deprecated. If the optimizer is a ``NamedOptimizer``, the optimizer state_dict does not contain parameter IDs mapping but a mapping from parameter FQNs to parameter states. This API finds the mapping from FQNs to parameters if the optimizer is a ``NamedOptimizer``. If ``use_orig_params`` is True, each rank will have all FSDP-managed parameters but some of these parameters may be empty due to the sharding. For a regular optim.Optimizer, states for those empty parameters will not be initialized. So, when aggregating the FQNs across ranks, no assert will be raised on a rank even if it does not have all the states -- it is valid and FSDP knows how to aggregate them. However, FSDP has to ignore handling those parameters that are not managed by FSDP and do not exist on the local rank -- those are managed by other parallelisms and FSDP does not know how to handle/aggregate them. Args: model (nn.Module): Root module (which may or may not be a :class:`FullyShardedDataParallel` instance) whose parameters were passed into the optimizer ``optim``. optim (torch.optim.Optimizer): Optimizer for ``model`` 's parameters. rank0_only (bool): If ``True``, saves the populated :class:`dict` only on rank 0; if ``False``, saves it on all ranks. (Default: ``True``) shard_state (bool): If ``True``, shard and distribute all non-zero-dimension states. Returns: Dict[str, Any]: A :class:`dict` containing the optimizer state for ``model`` 's original unflattened parameters and including keys "state" and "param_groups" following the convention of :meth:`torch.optim.Optimizer.state_dict`. If ``rank0_only=False``, then nonzero ranks return an empty :class:`dict`. r preprocessingNpreprocessing_with_comm)r[state_convertingr*zFound a optim state, aK, that FSDP cannot process. FSDP will directly copy everything to the returned state_dict. In most cases, this is a user-defined state that is not associated with any particular parameter. Another possible case is this state is managed by TorchRec. Otherwise, there may be a mismatched assumption of optim_state_dict of this mode.rz$FSDP _optim_state_dict() profiling: )$rrr enter_contextrrALLr traversal_utils_get_fsdp_handlesryrrrrkr r2rr4r3r~rr5rIrrcrrrrrrrrhcloser)rrrrrrXrmrrrYcmrWrrrrGrr^rN convert_fnrfsdp_osdflat_param_fqnsrbrs r9_optim_state_dictr:sD B^++N,?,?,C,CDE)/*K*KE*RSnH e 4 9H[G    0D*51 25901AB! sCx",,. /%8{K,5"4mEV  "=U!CD"    9 :   )     "&   (      2 3  ,0  $ ( " W %      $  '8H+2245O&w/557$ U .  / !  $ $   #C5)L L $s%$())#: 0-$  HHJ!!"HI OeDD"     s%/A.module_fnsCFK   :v& F3 && ' FBC( (8(89 NC#FY$67C''(-44??:MRsRM%4 c "14O ) )# .!!-33::&&s+99 r8c|Sr=r7)rs r9r?z._get_fqn_to_fsdp_param_info..return_fnr@r8rA)rr=r?rrrs r9rrsL.!35  |j}|tjk(r t dddd|j _yy)Nrz'Found state_dict_type LOCAL_STATE_DICT.z3DeviceMesh is not compatible with LOCAL_STATE_DICT.zKPlease set state_dict_type to SHARDED_STATE_DICT to get DTensor state_dict.T)rstate_dict_typer#LOCAL_STATE_DICTroptim_state_dict_configr)rrrs r9_set_optim_use_dtensorrsWz>40-== m<< <9E]  HL  7 7 D1r8)FNFN)Fr=)NFNN)T)FT){rrloggingrcollections.abcrrr contextlibr dataclassesrr itertoolsr typingr r r r rrrrItorch.distributed distributedry'torch.distributed.fsdp._traversal_utilsfsdp_traversal_utilsrtorch.nnr~#torch.distributed._state_dict_utilsr"torch.distributed.distributed_c10dr$torch.distributed.fsdp._common_utilsrrrrrrr#torch.distributed.fsdp._debug_utilsr"torch.distributed.fsdp._flat_paramrr'torch.distributed.fsdp._fsdp_extensionsrr%torch.distributed.fsdp._runtime_utilsrr torch.distributed.fsdp.apir!r"r#torch.distributed.tensorr$r%torch.utils._pytreer&'torch.distributed._shard.sharded_tensorr' getLoggerr.rr)r2r3rTrArCrLrQr6r5rcrir[r\ ProcessGrouprrrModuler OptimizerrrrJrOrrrrr)r4r5rrIrrrZrcrhrkrmrurNrrrrrrno_gradrrrr7r8r9rs  88 (WWW AABE?M 8-E   8 $ $$ $T#s(^sCx0I  CC C2   Z4"438n44 4  4  $sCx. 4n/3/4/C"C38nCCLJ"J "JJ $sCx. JZc3h D%% & #s(^ &#&/78I8I/J.#"# #c3h# #s(^ #R"-1)- L)38nL) 99L)L) EKK)) * L)  L) D%% & L) #s(^L)^F"F3S#X./FS F #s(^ FRQQ%,,'QS Q"%**- Q  Q  \\ Qh3H3H5<<(3HS 3H \\ 3Hl$$c$S $ $d %M,c3hM, 99M, ;; M, c3h R\\ " $  M,M,M, #s(^M,n H. 99H. c3h R\\ " $ H. #r|| H.V"%((//"d=#;M6N"N"&$=A<@ 0 ;; 0 BII 00DtCy!89: 0  ]C%7 89 0  %S/2<< '( 0j"&$=A<@ N ;; N BII NNDtCy!89: N  ]C%7 89 N  ",,c3h '( N2  N 99 N c3h R\\ " $  N ",,  N'&n-'&"&~uS#X'F"G'&U38_bll:;'& D%% & '&  '&` <>38n<> D%% &<>U38_bll:;<> d3i/0 <> !m!34 <>  <> 4 neCHo&E!F FG<>~S#XU38_bll:; d3i/0 $sCx. , $sCx. T &     ""sCx." $sI~ "JP "P d3 >23P sCx.P T#s(^+, P  8EKK $sD%,,1G,H'H"I IJ P fE+"E+T#s(^+,E+E+ E+  E+  E+ E+Pe"ed3 >23esCx.e e  e  e #tCH~ eP/"/sCx.// /  /  #s(^ /rX~.X"&~uS#X'F"GX!m!34X5c?C/0 X  X  XX #s(^XD6~.6"&~uS#X'F"G6!m!3465c?C/0 6  6  66 #s(^6r"Y 99Y ;; Y38nY c3h R\\ " $  YYY D%% &YYYY  #s(^!YYx.ryy.T#}:L5M.bLL*L LLr8