g^(h^ ddlZddlZddlmZddlZddlZddlmZddlm Z m Z ddl m Z GddeZ Gd d eZejd efd Zd ej"d e fdZd ej"d efdZd ej"d efdZGddeZGddeZGddeZdgdggZdgdggdgdggdgdgggZgdgdgdgZd ej"d efdZy) N)IntEnum)ir)get_dtype_size sympy_product)VceZdZdZdZdZy) NCCL_COLLrrN)__name__ __module__ __qualname__ ALL_REDUCE ALL_GATHERREDUCE_SCATTER[/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/torch/_inductor/comm_analysis.pyr r sJJNrr ceZdZdZdZdZy)NVIDIA_GPU_TYPErrr N)r r rVOLTAAMPEREHOPPERrrrrrs E F Frrreturnc8tjjjtjjjxsd}d|vrt j Sd|vrt jSd|vrt jSt jS)NV100A100H100) torchutils collect_env get_gpu_inforunrrrr)gpu_infos r get_gpu_typer&s|{{&&33EKK4K4K4O4OPVTVH $$$ 8 %%% 8 %%%%%%rnodect|tjstd||j}|Jd|vrt j Sd|vrt jSd|vrt jStd|)Nz!node is not a collective kernel: all_reduce all_gatherreduce_scatterzUnsupported collective kernel: ) isinstancer_CollectiveKernel ValueErrorpython_kernel_namer rrr)r' kernel_names rget_collective_typer1(s dB00 1>7v>??rceZdZdZdZdZy)NCCL_HWrrr N)r r rNVLINKPCINETrrrrKrKSs F C CrrKceZdZdZdZy) NCCL_ALGOrrN)r r rTREERINGrrrrPrPYs D DrrPceZdZdZy) NCCL_PROTOrN)r r rLLrrrrTrT^s  BrrTg333333@gffffff@g333333??g@g@)C@rWgffffff4@)gU@g6@g3@cjt|}|dz dz dz }d}t|}tj||z }|}|dkrytj }t j}t|} tjjj} tjjj} t} |dkr|dz nd} |dk(r| nd}t|| }|dk(r| n| }d}||z}t!|||dkDs| t"j$k(rdndz}| t"j$k(r d|dz z}n'| t"j&t"j(fvr|dz }d|zz }||z}|d z }t*j,}| t"j$k(r|dkDrd|z}n*d}n'| t"j&t"j(fvr|dz }t.||}t0|||}t0t*j2||}d }|dkDrd}t5||}||z |z||zzz }|d z}||z }||zS) a9 Returns estimated NCCL collective runtime in nanoseconds (ns). The following heuristics are copied from https://github.com/NVIDIA/nccl/blob/master/src/graph/tuning.cc. We aim to estimate the runtime as accurately as possible. Assumptions: - only ring algorithm (NCCL_ALGO_RING) is used - only Low-Latency protocol (NCCL_PROTO_LL) is used, i.e. Simple or LL128 is not used - 8 gpus per node # TODO: Need to find a way to get accurate "gpus per node" and "# nodes" info. - collective is one of: allreduce, reducescatter, allgather irrr g?gUUUUUU?rVgeAgg@@)rArImathceilrPrRrTrUr1r _inductorconfig intra_node_bw inter_node_bwr&llMaxBwsminr rrrrKrLbaseLathwLatrNmax)r'tensor_storage_size_bytestensor_storage_size_GBnum_gpus_per_node group_sizenNodesnRanks nccl_algo nccl_protocollbwIntrabwInter compCapIndexindex2index1llMaxBwbw nChannelsbusBwnstepsratio bandwidthbandwidth_GB_per_nsintraHw nInterStepslatencyintraLatinterLat netOverhead latency_ns transport_nss r estimate_nccl_collective_runtimersr!@ E6=DtK*40J YYz$55 6F F {IJ t $D oo$$22Goo$$22G>L!Q;VaZAF#q[\aFvv&GaKWBI NE  !ty/C/C'C9) U E  y###fqj! )**I,@,@A A!6\V #E I#c/nnG y### A:f*KK )**I,@,@A Aqj i ,GW~i(4HW[[!),Z8HK z 8[)H $0;3IIIG3J*,??L * $$r) functoolsrZenumrr7r rrr!rr virtualizedrr r lru_cacher&IRNoder1r9rArIrKrPrTrbrcr`floatrrrrrsJ  0 g   &o & & Jbii JI J "))  @BII@#@ g               ,   ,b%299b%b%r