f^(h ddlZddlmZddlmZddlZddlmZmZm Z m Z m Z m Z m Z mZddlmZddlmZmZddlmZddlmZGd d ZGd d ZGd deZ dZde de dee ddfdZdee ddfdZdee ddfdZdee ddfdZ dee de!ee"ffdZ#deede!e"effdZ$dee deede%e!eee fe!ee"fee ffdZ&dee deefdZ'dZ(GddZ)y) N)deque) NamedTuple)Deviceget_extra_size_of get_latency_of_partitioned_graph get_partition_to_latency_mapping NodeLatency PartitionPartitionerConfig PartitionMode) GraphModule)map_argNode)get_size_of_all_nodes) split_modulec NeZdZdZdedeedeedeededdf d Zdefd Z y) DAGNodezDAGNode class maintains useful information for a partition (submodule), and its input submodules and output submodules. submodule_node input_nodes output_nodeslogical_device_ids size_bytesreturnNcJ||_||_||_||_||_yN)rrrrr)selfrrrrrs k/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/torch/fx/experimental/accelerator_partitioner.py__init__zDAGNode.__init__s+%3'2(4-?$c,t|jSr)strrrs r__str__zDAGNode.__str__*s4&&''r) __name__ __module__ __qualname____doc__rlistintrr!r#rrrrs^ % %$Z %4j % !I %  %  %((rrc JeZdZdZd dZdedeedeedeed eddf d Zy) DAGz$DAG class contains all the DAG nodesrNcg|_yr)nodesr"s rrz DAG.__init__1s $& rrrrlogical_devicesrcXt|||||}|jj|yr)rr.append)rrrrr/rnodes r create_nodezDAG.create_node4s- K   $rrN) r$r%r&r'rrr(r)r3r*rrr,r,.sU.'    $Z  4j  c       rr,c&eZdZUdZeed<eed<y)PartitionResultz4NameTuple used for returning DAG and a new fx moduledagmodule_with_submodulesN)r$r%r&r'r,__annotations__r r*rrr6r6Bs> H''rr6c |D] }g|_ yr)r) partitions partitions rreset_partition_devicer=Ls* ') $*r partition_0 partition_1r;rctt|}|jj|j|_|j |j ||j ||j |t|y)zGiven a list of partitions and its two partitions, combine these two partitions into a new one appending to the partitions and remove the previous two partitions from the list of partitions N)r lenr.unionrecalculate_mem_sizer1removereorganize_partitions)r>r?r;r<s rcombine_two_partitionsrFQsq#j/*I!''--k.?.?@IO ""$i k"k"*% rcf|D] }t|_t|_"|D]}|jD]v}|j}|D]c}|D]\}||k7s ||jvs||jvs'|jj ||jj |^exy)zHGiven a list of partitions, mark parents and children for each partitionN)setchildrenparentsr.usersadd)r;r<r2rKnps rset_parents_and_childrenrObs  "  U E " 1 OO 1DJJE 1$1AI~!qww,4qww;N!**..q1 i01  1 1 1 rcNt|D] \}}||_t|y)zmGiven a list of partitions, reorganize partition id, its parents and its children for each partition N) enumerate partition_idrO)r;ir<s rrErEzs/ "*-# 9!" #Z( rct}t}|D],}t|jdk(s|j|.t}d}|ru|j }||_|j||j }|D]}||vs|j||s|j}t}|dz }|ruy)zJGiven a list of partitions, mark the bfs level for each partition rN)rHrArJrLpop bfs_levelrIcopy)r; current_levelvisitedr< next_levellevelrIchilds rget_bfs_level_partitionr^s%(EM!eG) y  !Q &   i ()"%J E !%%' #  I%% &EJ&u% &&OO-MJ QJE  rcXi}|D]"}|jD]}|j||<$|S)z;Given a list of partitions,return node to partition mapping)r.rR)r;node_to_partitionr<r2s rget_node_to_partition_mappingrasC)+= OO =D&/&<&< d # == rdevicesc6i}|D]}|||j<|S)z6Get a mapping from device logical ID to Device object.) logical_id)rblogical_id_to_deviceds rget_logical_id_to_devicergs,.0 /-.Q\\*/ rc6t|}i}i}|D]}g||<|j||<g}|D]d}|jgk7rB|jD]2}||} || j||| xx|jzcc<4T|j|f|||fS)zGiven a list of partitions and a list of devices, returns: 1. A mapping from device to partitions on it; 2. A mapping from device to its remaining memory size; 3. A list of partitions that do not have a device. )rgavailable_mem_bytesrr1used_mem_bytes) r;rbredevice_to_partitionsdevice_to_left_mem_bytesrfno_device_partitionsr<rddevices rget_device_partition_statsros4G<:<24 <"$Q&'&;&; #< 3  ' '2 -':: M -j9$V,33I>(0I4L4LL0 M ! ' ' 23   rc dtdttfddtffd }t||\}d}|D]F}tt j t jd||}|rE|S|S)z\Given a list of partitions and a list of devices, map each partition into a device. r<r;c t}|D]}|j|j}t|dk(r |jS|j|j}d}|jD]}|t ||z }|S)Nr)rHrBr.rArjr)r<r; all_nodesrNextra_size_neededr2s r$calculate_extra_mem_bytes_needed_forzNget_device_to_partitions_mapping..calculate_extra_mem_bytes_needed_fors #u  1A!0I 1 y>Q ++ +OOIOO4 OO DD !24!C C  D  rcD]^}||}||ks|j||jj|j|xx|zcc<yy)a3Given a partition, find a logical device for the partition The algorithm is to put the partition on the device that has just enough mem left for that partition. device_to_left_mem_bytes is a dictionary between device and its left mem size sorted by its left mem size TF)r1rrd)r<rfrsrtrlrks rfind_device_forz9get_device_to_partitions_mapping..find_device_fors~* A D/2! !#;A#>>$Q'..y9,,33ALLA(+/@@+ rTrUkey)r r(rodictsorteditemsoperator itemgetter) r;rbrvrm found_devicer<rtrlrks @@@r get_device_to_partitions_mappingrs ! !*.y/ !9, #:w7  L) #' +1139L9LQ9O P$  'y1    rc|h}t|g}|rR|j}|jD]0}||k(ry||vs|j||j |2|rRy)z^Given a partition,check if there is a circular dependency on this partition using bfs TF)rpopleftrIrLr1)r<rZqueuerNr]s rcheck_dependencyrsm )kG#YK0E  MMOZZ (E !'KK&LL'  (  rceZdZdZddZdedejjde de fdZ dd e ddfd Z dd Zdd Zdefd ZdedefdZdefdZdZde ddfdZdedeeefddfdZdedeeefddfdZdZy) PartitioneraA fx module may not fit into one device. Partitioner class helps partition one fx module into submodules (partitions), so that the submodules can be executed crossing different accelerators. The main function of this class is self.partition_graph. It partitions the fx module based on the scheme specified in partition_config A DAG structure is returned along with a new fx module with submodule nodes. rNc.g|_i|_g|_yr)r;r`rbr"s rrzPartitioner.__init__,s+-24%' r fx_module torch_modulepartitioner_configc ||_||_|j|_t|jdk(r t dt |j|jj j}td|Dr t dd}|D],}|jdk(rn||jjz }.t|jd}|jtjk(r(|j!|j"|j$na||j&kr|j)||j*n3|t-d |jDkDr t d |jtj.k(rT|jdj& t fd |jDs t d |j1 n|jtj2k(r'|j5|j6|j8nT|jtj:k(r'|j=|j6|j8n|j?|j@r|jA|jC}|jE|} tG| |} | S) zGiven the fx module, torch module and partitioner_config, find the partitions, do the partitions, and then return a DAG and a new fx module with submodule nodes (partitions) rz No devicesc38K|]}|jdvyw)>outputget_attr placeholderN)op).0r2s r z.Partitioner.partition_graph..DsRDtww??Rsz.No Partition since no operations in the modulerc|jSrrirfs rz-Partitioner.partition_graph..Ms a>S>Srrw)logical_device_idc34K|]}|jywrr)rrfs rrz.Partitioner.partition_graph..Ys&Sq'<'<&Ssz,Devices have no enough memory for the modulec3<K|]}|jk(ywrr)rrnris rrz.Partitioner.partition_graph.._s%..2EEsz'All devices must have same memory size!)$ graph_modulerrbrA RuntimeErrorrgraphr.allrr total_sizemaxmoder aot_basedaot_based_partitionnode_to_partition_mapping#partition_to_logical_device_mappingrifind_single_partitionrdsum sparse_nnsparse_nn_partition cost_awarecost_aware_partitiontransfer_rate_bytes_per_secnode_to_latency_mappingkl_basedkl_based_partitionsize_based_partition saturate_host do_partitiondump_dagr6) rrrrr.total_size_of_graphr2device_with_max_memr8r7retris @rpartition_graphzPartitioner.partition_graph1sx&()11 t||  !|, ,d//0!!''-- RER ROP P >Dww(" 4??#=#= =  > "$,,4ST  " "m&=&= =  $ $"<<"FF  !$7$K$K K  & &#7J7U7U ' !3&Sdll&S#S SMN N"&&-*A*AA&*ll1o&I&I#"&,,''PQQ(()<=#((M,D,DD))&BB&>> $((M,B,BB''&BB&>> ))+  + +    "&!2!2!4mm23c#9: rrc|j}|jjjD]-}|jdk(r|jj |/||_|g|_t|j|_ y)z'Fit the whole fx module into one devicerN) create_partitionrrr.rrLrjrrar;r`)rrrr>r2s rrz!Partitioner.find_single_partitions++- %%++11 (Dww("    ! !$ '  ( &9 "*;)< &!>t!Orc\dtffd }i}gj}jjjD]}|j dvst jt jkr:t||j}|jdk(rN||}j||j||<|jj|jn|||krt jt jk(rj|||}j}t||j}|j||<|jj|j|j!|||xx|zcc<yj|t#jt%j_t)jj}|s t+dy)aThis method is to partition the fx module based on memory size. It uses greedy approach. The result may not be the best. The basic idea is: Step 1: Find a device which has enough memory to fit the current node, create a empty partition with the size of that device. Then keep adding the following nodes into the partition until the partition is full. Step 2: Repeat Step 1 until no device left Step 3: If some nodes are left, create a partition for each left node (single node partition). and then try to map those partitions into logical devices with enough mem left. rc t|t}tddd}jD]}|vs|j|k\s|}n|jdkrt t |dzj||S)ziGiven a node, this function is to find a logical device that could fit the node. rzis too large to fit any device)rrHrrbrirr!r1)r2mem_size_neededrnrfoccupied_devicesrs rfind_device_based_on_sizezCPartitioner.size_based_partition..find_device_based_on_sizes0ce call_method call_module call_functionrz6Cannot Get a Valid Partition to Logical Device MappingN)rrrrr.rrAr;rbrrjr1rirrdcreate_single_node_partitionadd_noderErar`rr) rrpartition_to_left_mem_bytesr<r2total_size_of_input_nodesrn!found_partition_to_device_mappingrs ` @rrz Partitioner.size_based_partitions v &=?#)+))+ %%++11, WW:55d;Y, t!O,L OOT\\- )1WX Xrct|j|j\}}}t|dk(sJdt||jDcgc]}t||dkDs|}}i}t|dzt|zt|jkrd}|jDcgc] }||vr||vr|}}i} |D]f} |Dcgc]#}|j| j|| z k\r|%} }t| dk(rd}n&t | d} |j | | | | <h|snB|j| t|dzt|zt|jkr|jD]6\} }| j}||D]}|jj|8|jD]}t|jycc}wcc}wcc}w) aSaturate host by assigning replicates to unused devices with enough memory. It uses a greedy approach to find a next available set of devices to place all split partitions: For each used device, it searches for an idle device with minimal memory size that can hold all the partition located on that device; If the search is successful for all used devices, it then assigns the new devices' logical ID to the corresponding partition. rz2Expect no_device_partitions has 0 device, but get TFc|jSrrrs rrz+Partitioner.saturate_host..$s !BWBWrrwN) ror;rbrAriminrDupdater{rdrr1print)rrkrlrmrf used_devices replicated_device_to_used_devicesuccess idle_devicestemp_replicate_mapping used_deviceavailable_devices new_devicereplicate_deviceoriginal_devicerdr<rNs rrzPartitioner.saturate_hosts_ 't E  $  $ % * \ ?DX@Y?Z [ \ *$(<<Ta37KA7N3ORS3ST TAC(,!#c*J&KKs LLP  GL(Q6V-VL &( " , A *%,,"66.{;<<%!%()Q.#G !28WX ##J/5@&z2 A  , 3 34J KC,!#c*J&KKs LLP  N. 3 3 5 @  )44J1/B @ ,,33J? @  @ (A !&& ' (_U%sG(+G(3G-(G2cPtjjfd}|S)z9Return a new fx module with submodule nodes (partitions).c"j|Sr)r`)r2rs rrz*Partitioner.do_partition..<s//5r)rrr)rr8s` rrzPartitioner.do_partition7s+!-       5"  &%rr8ct}|jjD]-}|jdk(r|S|jdvr%|jt j k(rCi}t|j|jt|j|jt|jdkDrt|j}n|g}t|jj!ddd}|j"|j$}|j"|j&}|j)|t||||0|S)z?Return the dag structure and the new fx module with submodules.r>rrrU_r)r,rr.rtargetr| __getitem__rargs setdefaultkwargsrArKr(r)namersplitr;rrjr3) rr8r7r2rrrR device_idsrs rrzPartitioner.dump_dag@s e*0066 Dww(", +ww55{{h222,.K DII{55 6 DKK!7!7 8 4::"#DJJ/ $v tyy//Q7;.combine_partitions_based_on_size..s QEUEUrrwN)rzr^r;)r;rifind_combinationsorted_partitions'find_partition_to_combine_based_on_sizers r combine_partitions_based_on_sizezIPartitioner.sparse_nn_partition..combine_partitions_based_on_size|sJ" $ "$*:;U$V!'8/V%':J0, * # rc|jj|j}d}|D]}|t||z }|S)zuGiven two partitions, calculate how many mem bytes are needed if two partitions are combined r)r.rBr)p1p2r.mem_bytes_neededr2s rcalculate_mem_bytes_neededzCPartitioner.sparse_nn_partition..calculate_mem_bytes_neededsJHHNN288,E  C $5dE$BB  C# #rrcpd}|jd}|dddD]}t|j|jz dks)||}||ks8t||j|j ||j ||j jdd}||fS||fS)z+step 1 in combine_partition_based_on_size()FrNrrUT)rVabsrWrFr;rDr1) rrir;rsmallest_partitionrNrrrs rrzPPartitioner.sparse_nn_partition..find_partition_to_combine_based_on_sizes % !2!6!6q!9 &tt, )33akkABaG'A!EW'X$'+>>.q2DdooV"))*<="))!,"))$//"*=>+/(#Z/ / $Z/ /rcrj|nj||rj}|_|Sy)zyIf crossing the boundary between non-embedding nodes and embedding nodes, create a new partition N)r1rleft_mem_bytes)r< new_partitionriembedding_partitionsin_embedding_regionnon_embedding_partitionsrs rreset_partition_in_sparse_nnzEPartitioner.sparse_nn_partition..reset_partition_in_sparse_nnsF#$++I6(// : 113 +> (  rr2c|jdk(rmj}t|jj dD]:}t ||st d|d|t||}dt|vs:yy)z$Check if a node is an embedding noder.zModule z has no attribute EmbeddingTF)rrr!rsplithasattrrgetattr)r2 submoduleatomrs ris_embedding_nodez:Partitioner.sparse_nn_partition..is_embedding_nodesww-' --  ,2237$D"9d3*%i[0B4&I!( 4 8I"c)n4#$rF>rrrrz!is too large to fit into a device)rzNeed z devices, but only z provided partition_zN(embedding partition) and non embedding partitions can not fit into one device)T)r(r r)tupleboolrrrrr.rrjrrrrrOr;rArbr!rQrRrdrr1rar`)rrirrrr<r2r&total_size_of_non_embedding_partitionsmsgrrSrrrrrs`` @@@@@rrzPartitioner.sparse_nn_partitionjs $ Y >A   8 $ 0#I 0!$ 0Y 04i( )  0*   D T 1346 $)))+ %%++11 )DwwII$T*.AA!//14$@$K .A*A',=dIOO,T)- 0H0HH)*!=Y GI0A$ 0X-03FF* KK*MM""4(+ ), %YeD 1()ACVW()=?RS12.1 OI 2i6N6N N 2 O # $s4<<'8 8c./01'(c$,,'()  s# #%&:; DLAy79Q9QQ%&# )0012fg15 Q0J0J/K , '' Q(B(BC D 2 .try_combining_partitions/sH%BH%B R\\BLL01Q6"**$"++&&r2z:#JrN3 <'&z2?   $ <'/O 70,80/  < rc t j|}t j||}t jdk(ryg}t t jdz D]`}t |dzt jD]9} || jdd}||kr||g}|}t j;bt|dk7r; j|d} j|d} t || jt jt jt j jt|dk7S)aGiven transfer rate between partitions and each node's latency, find two partitions to combine so the cost of the partitions can be reduced. The algorithm is : 1. Go through all the partition pairs and see if any pair of partitions can be combined. 2. Calculate the cost after the combination. 3. Select the minimum cost and combine its corresponding partition pair. rUFNr) rr;rrArangerErFr^r=rrb) rrrrpartition_pairrSjnew_costrrrrs rsearch_combinationz.search_combinationUsN,L!8, (4,+D 4??#q((*N3t/!34 ;q1uc$//&:;;A 81dooa>PQH4'*+Q')$//:; ;>"a'__^A%67__^A%67&r2t? #DOO 4 "4?? 3 ,T__dll K~&!+ +r>rrrTN) r rrrr.rrrOr;r^rErar`)rrrrr2rrs``` @rrz Partitioner.cost_aware_partitions*$ $ L( , ( ,T%%++11 8DwwCC11$7 8 !10 2+-D   doo.!>t!Orc 4dfdfd}jtj|}tj|}g}g}jj j Dcgc]}|jdvr|} }| D]} j| } j| } tjD]7\} }| | k7s j| }|| | ||\}}||ks0|}|}| |g}9t|dk7s|d|d|d|dtjtjjtjtjjycc}w)aThis function is a cost aware partition based on Kernighan-Lin algorithm. First, the graph is partitioned using size_based_partition. Then, each node is swapped with any other node in a different partition, and at the same time, the cost is estimated after the swapping. For example, we have nodes n0, n1, n2, n3 and n4. Using size_based_partition, n0 and n1 are in Partition p0. n2, n3 and n4 in Partition p1. The current cost is estimated. We first tried using n0 to swap with n2 from the other partition. Then we see that swapping n0 and n2 shows a lower cost than the current cost and it is the minimum among other pairs like (n0, None)(This means moving n0 to Partition without swapping other nodes), (n0, n3) and (n0, n4). We swap n0 and n2 and set the new cost as the current cost. Then We repeat this process for all the other nodes until all swapping pairs are tried. c|"|j||j||#|j||j|yyr) remove_noder)n0n1rrs r swap_nodesz2Partitioner.kl_based_partition..swap_nodessA~r" B~ Br"rctd} ||||t jt|s{t|spt jt j|}t j j}|s td}nt j| } ||||t jt jt j j|S)Nr ) r rEr;rr=rrrbr) rrrrrtransfer_rate_per_secrrr~rrrs rtry_swap_nodesz6Partitioner.kl_based_partition..try_swap_nodessrr)r(r.r r) r2rrrr!p1_nodesmin_cost node_pairrrr"s rswap_node_to_partitionz>Partitioner.kl_based_partition..swap_node_to_partitions BHH~.HU|H$&I $>bee/J&J%"b"&=?T(?!%r I#H $? "r>rrrrrUN)rrr;rrrr.rr`rQrArErrb)rrrr'rrr&rrMop_nodesr2r rrrrr new_node_pairrr"s`` @@rrzPartitioner.kl_based_partitions0 # @ #. !!#'G OO4( $0 OO9;V !# *,&&,,22 tt@@    PD--d3H*B )9 2 !x'2B.D/3 /+Hm $'$1 *,b 2"9~"aL)A,q0A>RSCT&doo60$,,O9 P: doo.($,,GK s7Fci}||_|jD]n}|j|}||vr6t|}|jj||||<|||_n||j|}|j |py)zqThis function helps to rebuild the partitions given the nodes and its corresponding partition id N)r`r r;r1rr)rrr!partition_id_to_partition_mappingr2rRr<s rrzPartitioner.aot_based_partition!s CE)!:** %D11$7L#DD%l3 &&y1BK1,?/R 0 ,>**40    t $ %rr4)r)r$r%r&r'rr torchnnModuler r6rr)rrrrr,rr rrrr ryrr rrrr*rrrr"s( PPhhooP. P  Pf=>69 "\|C(J&k&{s:) nsntn`w%*w"&dK&7!8w  wrL%*L"&dK&7!8L  L\%rr)*r| collectionsrtypingrr,'torch.fx.experimental.partitioner_utilsrrrrr r r r torch.fx.graph_moduler torch.fx.noderr"torch.fx.passes.graph_manipulationrtorch.fx.passes.split_modulerrr,r6r=r(rFrOrEr^ryr)rargrrorrrr*rrr6s    .'D5((.  ((j(F*   )2 @DY   " i T 0 d9o $  Y D 8d9o$tSy/ d6l tCK7H "Y"*.v," 4Y' ($vs{*;T)_ LM"J6Y6*.v,6r$V %V %r