[^(hH0ddlmZddlmZddlZddlmcmZddlm Z ddl m Z m Z ddl mZddlmZd gZed d d gZGd d eZy)) namedtuple)SequenceN)Tensor) ModuleList Sequential)Linear)ModuleAdaptiveLogSoftmaxWithLoss _ASMoutputoutputlossc eZdZUdZeed<eed<eeed<eed<eed<e ed<e ed< ddedede ededed d f fd Z dd Z d eded efdZdZded efdZded efdZxZS)r uEfficient softmax approximation. As described in `Efficient softmax approximation for GPUs by Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou `__. Adaptive softmax is an approximate strategy for training models with large output spaces. It is most effective when the label distribution is highly imbalanced, for example in natural language modelling, where the word frequency distribution approximately follows the `Zipf's law`_. Adaptive softmax partitions the labels into several clusters, according to their frequency. These clusters may contain different number of targets each. Additionally, clusters containing less frequent labels assign lower dimensional embeddings to those labels, which speeds up the computation. For each minibatch, only clusters for which at least one target is present are evaluated. The idea is that the clusters which are accessed frequently (like the first one, containing most frequent labels), should also be cheap to compute -- that is, contain a small number of assigned labels. We highly recommend taking a look at the original paper for more details. * :attr:`cutoffs` should be an ordered Sequence of integers sorted in the increasing order. It controls number of clusters and the partitioning of targets into clusters. For example setting ``cutoffs = [10, 100, 1000]`` means that first `10` targets will be assigned to the 'head' of the adaptive softmax, targets `11, 12, ..., 100` will be assigned to the first cluster, and targets `101, 102, ..., 1000` will be assigned to the second cluster, while targets `1001, 1002, ..., n_classes - 1` will be assigned to the last, third cluster. * :attr:`div_value` is used to compute the size of each additional cluster, which is given as :math:`\left\lfloor\frac{\texttt{in\_features}}{\texttt{div\_value}^{idx}}\right\rfloor`, where :math:`idx` is the cluster index (with clusters for less frequent words having larger indices, and indices starting from :math:`1`). * :attr:`head_bias` if set to True, adds a bias term to the 'head' of the adaptive softmax. See paper for details. Set to False in the official implementation. .. warning:: Labels passed as inputs to this module should be sorted according to their frequency. This means that the most frequent label should be represented by the index `0`, and the least frequent label should be represented by the index `n_classes - 1`. .. note:: This module returns a ``NamedTuple`` with ``output`` and ``loss`` fields. See further documentation for details. .. note:: To compute log-probabilities for all classes, the ``log_prob`` method can be used. Args: in_features (int): Number of features in the input tensor n_classes (int): Number of classes in the dataset cutoffs (Sequence): Cutoffs used to assign targets to their buckets div_value (float, optional): value used as an exponent to compute sizes of the clusters. Default: 4.0 head_bias (bool, optional): If ``True``, adds a bias term to the 'head' of the adaptive softmax. Default: ``False`` Returns: ``NamedTuple`` with ``output`` and ``loss`` fields: * **output** is a Tensor of size ``N`` containing computed target log probabilities for each example * **loss** is a Scalar representing the computed negative log likelihood loss Shape: - input: :math:`(N, \texttt{in\_features})` or :math:`(\texttt{in\_features})` - target: :math:`(N)` or :math:`()` where each value satisfies :math:`0 <= \texttt{target[i]} <= \texttt{n\_classes}` - output1: :math:`(N)` or :math:`()` - output2: ``Scalar`` .. _Zipf's law: https://en.wikipedia.org/wiki/Zipf%27s_law in_features n_classescutoffs div_value head_biasheadtailNreturnc ||d}t |t|}t|dk(r t d|t |k7sQt |dksCt||dz kDs2tt|t|k7std|Dr t d||_ ||_ ||gz|_ ||_ ||_|jd|_t|jdz |_|j|j z|_t%|j|j"fd|ji||_t)|_t-|j D]} t/|j|j| dzzz} |j| dz|j| z } t1t%|j| fddi|t%| | fddi|} |j*j3| y) N)devicedtyperz4cutoffs should be a sequence of length larger than 0rc38K|]}t||k7ywN)int).0cs W/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/torch/nn/modules/adaptive.py z6AdaptiveLogSoftmaxWithLoss.__init__..s013q6Q;0szcutoffs should be a sequence of unique, positive integers sorted in an increasing order, where each value is between 1 and n_classes-1biasF)super__init__listlen ValueErrorsortedminmaxsetanyrrrrrshortlist_size n_clusters head_sizer rrrrangerrappend)selfrrrrrrrfactory_kwargsihszosz projection __class__s r r$z#AdaptiveLogSoftmaxWithLoss.__init__ts%+U; w- w<1 ST Tw 'G !G A .CL!S\1000:  '"), """ll1odll+a/,,t>   dnn 37>> ES  L t' )Ad&&4>>a!e+DEFC,,q1u% Q7C#t''K5KNKsC>e>~>J II  Z ( )c|jj|jD]%\}}|j|j'yr)rreset_parametersr)r2i2hh2os r r;z+AdaptiveLogSoftmaxWithLoss.reset_parameterssA ""$  #HC  "  " #r9input_target_c |j}|dk(r[|jd|jdk7r td|jdk7rWtd|j|dk(r-|jdk7r%td|jtd|dkD}|r|n|jd}|r|n|jd}d}|jd}|j |} |j |} dg|j z} tt| dz D]-} | | } | | dz}|| k\||kz}|jj}|jdk(rN| dk(r| jd|||n||| z }|jd|}|j| dz |}|j| zdz }| j!d||t#j$|d}|j'd|jd}| jd||jd||jz }0||k7rZtd |j(dz d |j+j-d |j/j-d |j1|}t#j$|d}| |j'd| jdjz } | j3}|s| jd} t5| |S) NrrzBInput and target should have the same size in the batch dimension.zE1D target tensor expects 2D input tensors, but found inputs with sizezE0D target tensor expects 1D input tensors, but found inputs with sizez;0D or 1D target tensor expected, multi-target not supporteddimzTarget values should be in [0, z], but values in range [z, z] were found. )rCsize RuntimeError unsqueeze new_zeros new_emptyrr0r&nonzerosqueezenumel index_copy_ index_selectrr- index_fill_F log_softmaxgatherrr)itemr*rmeanr )r2r>r?targ_dim is_batchedinputtarget used_rows batch_sizer gather_inds cutoff_valuesr4low_idxhigh_idx target_mask row_indicesrelative_target input_subsetcluster_output cluster_indexcluster_logprob local_logprob head_output head_logprobrs r forwardz"AdaptiveLogSoftmaxWithLoss.forwardsO;;= q={{1~a0".zz|q "1KKM ]zz|q "1KKM M \ $&*:*:1*=&G,=,=a,@ [[^ ,&&z2 dll* s=)A-. -A#A&G$QU+H!W,(1BCK%--/779K  "a'Av'';{8KL#)"5"?$11![A !11q5!1,!? $ 3 3a 7! ; ''; F"#--A"F / 6 6q/:S:STU:V W ""1k=3H3H3KL **, ,I3 -6  "1$..12D1EF((. (9(9(;'ST  ii& }}[a8 ,%%a)>)>q)ABJJLL~~^^A&F&$''r9c|j|jd|jf}tj|d}|ddd|j f|ddd|j f<t t|j|jddD]g\}\}}|j||}tj|d} | |dd|j |zfjdz} | |dd||f<i|S)zZGiven input tensor, and output of ``self.head``, compute the log of the full distribution.rrrBN) rHrDrrOrPr- enumerateziprrrF) r2rVrfoutrgr4 start_idxstop_idxrbrdoutput_logprobs r _get_full_log_probz-AdaptiveLogSoftmaxWithLoss._get_full_log_probs oo{//2DNNCD}}[a8 (4Q8M$:M:M8M5M(NA$$$$ $%(1#dllDLLQRQSDT2U(V 8 $A$ 8)TYYq\%0NmmNBO,|4&&**0ilN*8C9X%% & 8 r9rVcH|j|}|j||S)aCompute log probabilities for all :math:`\texttt{n\_classes}`. Args: input (Tensor): a minibatch of examples Returns: log-probabilities of for each class :math:`c` in range :math:`0 <= c <= \texttt{n\_classes}`, where :math:`\texttt{n\_classes}` is a parameter passed to ``AdaptiveLogSoftmaxWithLoss`` constructor. Shape: - Input: :math:`(N, \texttt{in\_features})` - Output: :math:`(N, \texttt{n\_classes})` )rrp)r2rVrfs r log_probz#AdaptiveLogSoftmaxWithLoss.log_probs% ii& &&uk::r9ct|j|}tj|d}||jk\}|j }|r|S|j r)|j ||}tj|dS|j ||||}tj|d||<|S)aReturn the class with the highest probability for each example in the input minibatch. This is equivalent to ``self.log_prob(input).argmax(dim=1)``, but is more efficient in some cases. Args: input (Tensor): a minibatch of examples Returns: output (Tensor): a class with the highest probability for each example Shape: - Input: :math:`(N, \texttt{in\_features})` - Output: :math:`(N)` rrB)rtorchargmaxr-r,allrp)r2rVrfr not_in_shortlistall_in_shortlistrrs r predictz"AdaptiveLogSoftmaxWithLoss.predict*sii& kq1!T%8%88 0 4 4 67 M  ! ! #..ukBH<<a0 0..&'5E)FH(-||H!'DF# $Mr9)g@FNN)rN)__name__ __module__ __qualname____doc__r__annotations__r%floatboolr rrr$r;rr rhrprrry __classcell__)r8s@r r r sUnN #YO L 7)7)7)# 7)  7)  7) 7)r# P(fP(vP(*P(d$;f;;& V  r9) collectionsrcollections.abcrrttorch.nn.functionalnn functionalrOr containerrrlinearr moduler __all__r r r9r rsK#$ - ( (  x&8 9 vvr9