g^(h9,ddlZddlZddlmZddlmZddlmZmZddl m Z  ddl m Z m Z mZmZmZmZmZmZmZmZej0eZej7ej8 dded ed ed eee d eee d e!e"e efeeff dZ#GddeZ$de d ed eefdZ%y#e$rZededZ[wwxYw)N)IntEnum)Optional)Graph is_submodule)SACStats) lpDot LpInteger LpMaximize LpMinimize LpProblemLpStatuslpSum LpVariable PULP_CBC_CMDvaluezBPlease install pulp package. See: https://github.com/coin-or/pulp.graph memory_budget world_sizeac_units fsdp_unitsreturnc %tj}d}d}tdt}t j dt t|ddt} t j dt t|dd} t j dt t|d} t j d t t|d} t j d t t|d} t j d t t|d}t j d t t|d}t d d}|r;t|}t|D]"%j%d|vs|| %dk(z }$|r1t|D]#%t%fd|Ds|| %dk(z }%t|D]=%t%dz|D])}j%|dk(s|| %| |zdkz }+?t|D] %j%ds|| %dk(z }"t|D]?%j%dsj%dj%dkrtjdj%dtjdj%dtjdj%dtjdtjdj%dj%d<j%d|z }j%d|z }|| %|| %z||z | %zz k(z }Bt|D]d%|| %| %k\z }j%dr$j%d|z }j%d|z }|| %||z |z | %zk\z }ft|D]%j%d|z }j%d|z }j%d}dg|z}t|D]&}jj|d}d||<(|| %||zt!|| z k(z }jdd|z }t|D].%j%d|z }|| %| %||z|z zk(z }0t|D] %||| %k\z }t|D]`%tj%dD]@}j%d|}j%d |}||%|| %z|zk\z }Bbt|D]S%j%d!}||%|| %zkz }||%||%zkz }||%||%z|d| %z zz k\z }U|||kz }|t#|z }t%d"d#d$} |j'| }!|!dk7r"tj)d%t*|!idd&fSi}"t|D]J%t-| %j.dk(st-| %j.d'|"j%d<Lt-t1|j2d(}#t-|j.|z}$|"|#|$fS))a6 MILP to decide which modules to AC and how much memory to discard. The objective is to minimize recomputation time. The constraint is to ensure peak memory is under budget. Args: graph: graph representation of the model as a module submodule tree where each node is a submodule with memory & runtime stats memory_budget: memory budget in GiB world_size: number of GPUs. In the case of FSDP, world_size will be used to compute the amount of parameter and gradient memory on each rank ac_units: a list of user-specified AC units. fsdp_units: a list of FSDP units. AC units cannot be supermodules of FSDP units. Returns: Dict[str, float]: the optimal SAC solution, mapping from module fqn to the percentage of activation memory to **discard** float: the recomputation time of the optimal SAC solution int: upper bound on the peak memory of the optimal SAC solution. note that value of -1 means that the ILP solver failed to find a solution. di@SACyrrdamrcprctmax_mfqnc3VK|] }t|jd"yw)r$N)rnodes).0 fsdp_unitris ^/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/torch/distributed/_tools/sac_ilp.py zsac_milp..ds-Y Au(=>s&)is_leaf sac_memoryact_fw_per_modulezFor module {%s}: z.activation memory from memory tracker is {%d},z-activation memory from SAC estimator is {%d}.z!Something is wrong. Please check!z&Overriding the latter with the former.act_grad_per_module act_totalpos_fw_post_orderindexparam_per_module grad_total n_segmentsslopes intercepts sac_runtime皙?gapRel timeLimitmsg$Solver failed to find a solution: %s)lenr&r r rmatrixlistranger setany ad_matrixloggerwarning name2node fw_post_orderrrrsolveerrorr roundvarValuer objective)&rrrrr num_nodesMMEM_MULTIPLIERprobrrrrr r!r"r# ac_units_setjACM_iIA_iAG_iTA_iposcoeffpP_1TG_isslope interceptACT_isolverstatus ac_decisionsrecomputation_timepeak_memr)s&` @r*sac_milprk#sj:EKK I AN UJ 'D #tE)$45q!YGA#tE)$45q!) $ AaDAI D 9 =  Ay)u{{1~ 0 KKN. /00 NN. Au0E F NN@ A23  NN? A|,  NN> ? NNC D+0;;q>:M+NEKKN< ( A|,~={{1~12^C !!  !'<<<</=<9 6 !!  ;;q>) $  A|,~={{1~12^C !.1555 69  6{{1~34~E{{1~k*^;kk!n01is A 3 3A 67@AE!H  !t eE1o555 6 ++a.+ ,~ =C 9 9{{1~l+n< !!d j88889 9  1 9 7u{{1~l34 7AKKN8,Q/E A|4Q7I CFeadlY66 6D 779 : A}- A!ad("" A%#a&.(( A%#a&.1AaD>999 : E] ""D E#JD! t'j(|t+|j,z} |t/||j,| kz }|t/||j0z }t3d d d } |j5| } | dk7rt6j9d t:| gSt|Dcgc]}t=||j>c}Scc}w) aB This is adapted from -- https://github.com/facebookresearch/xformers/blob/c6c0ac31f1b08542a0bc27278c6ed10f825f6963/xformers/checkpoint.py#L375 Given the SACStats of a module, including list of operators, their memory, runtimes, and metadata, decide via MILP an optimal set of operators to checkpoint under a given ``memory_budget``. Args: sac_stats: the SACStats object of the module memory_budget: a float between zero and one Returns: List[int]: the decision whether each operator should be saved (1) or recomptued (0). rrz5`memory_budget` must be a float between 0 and 1. Got .zSAC-per-modulexNr@r9 r;r?) ValueErrorrC func_namesr r rrDrErFr force_store_randomrand_opsrnrsrzip view_like_opsrr inplace_opsmathceilsummemoryrruntimesrrNrJrOr rPrQ) rurnum_opsrVrxr)i1i2op op_parent max_memoryrfrgs r*+get_optimal_checkpointing_policy_per_modulersD"  #! #CM?RS T  )&&'G %z 2D #tE'N3Q9EA #### 3A AaDK,,222 2D 3),,Sb193E3Eab3IJ #FB AbEQrUN "D # $ $4 ! --33334#..4 I ? AbEQy\) )D AbE[--333 3D 4=3y/?/?+@@AJE!Y%% &* 44D  E!Y'' ((D ;F ZZ F{ ;Xf=MN +0. 9QE!A$-- 99 9sH?)rNN)&loggingrenumrtypingr"torch.distributed._tools.ilp_utilsrr&torch.distributed._tools.sac_estimatorrpulprr r r r r rrrr ImportErrorerr getLoggerrorJsetLevelINFOfloatintrEstrtupledictrkrnrrtrlr*rs B;   $   8 $  $(&* v6 v6v6v6tCy! v6 c# v6  4U UC '( v6r ' F:F:(-F: #YF:U L  sB--C2 B;;C