l^(hMddlmZddlmZmZddlmZddlmZddl m Z m Z ddl Z ddl Z ddl mZmZddlmZmZdd lmZdd lmZmZGd d Z ddZGdd ej4Zy)) annotations)IterableIterator) nullcontext)partial)AnyLiteralN)Tensornn)get_device_statesset_device_states)SentenceTransformer)StaticEmbedding Transformerc(eZdZdZddZddZddZy) RandContexta Random-state context manager class. Reference: https://github.com/luyug/GradCache. This class will back up the pytorch's random state during initialization. Then when the context is activated, the class will set up the random state with the backed-up one. c`tj|_t|\|_|_yN)torch get_rng_state fwd_cpu_stater fwd_gpu_devicesfwd_gpu_states)selftensorss n/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/sentence_transformers/losses/CachedGISTEmbedLoss.py__init__zRandContext.__init__s("0024Ew4O1d1ctjj|jd|_|jj tj |jt|j|jy)NT)devicesenabled) rrandomfork_rngr_fork __enter__ set_rng_staterr rrs rr%zRandContext.__enter__s^\\**43G3GQU*V   D../$..0C0CDrcL|jj|||d|_yr)r$__exit__)rexc_typeexc_valexc_tbs rr)zRandContext.__exit__#s Hgv6 rN)returnNone)__name__ __module__ __qualname____doc__rr%r)rrrrsPE rrCachedGISTEmbedLossc |jJ|jJtj5t ||j|jD]v\}}}t |j |dd||D]N\\}}}}tj |j|j|z} | jPx dddy#1swYyxYw)zOA backward hook to backpropagate the cached gradients mini-batch by mini-batch.NTF)sentence_feature with_gradcopy_random_state random_states) cacher9r enable_gradzipembed_minibatch_iterdotflattenbackward) grad_outputsentence_featuresloss_objr6gradr9reps_mb_grad_mb surrogates r_backward_hookrI(s >> %% %  ! ! -- -     %589JHNN\d\r\r5s % 1 dM,/--%5"&+"/ . - %(!Q"IIgoo&79JKkY ""$ % % % % %s BCCceZdZ d d fd Zd dZ d d dZ d ddZddZ d ddZddZ ddZ xZ S)r4ct|t|dtr t d||_||_||_tjd|_ t|dtrt|dts t dtj|_ ||_d|_d|_||_|j$j&|j$j&k7xs|j(|j(k|_|j*r|j$|_|dvr t d||_||_y) a This loss is a combination of :class:`GISTEmbedLoss` and :class:`CachedMultipleNegativesRankingLoss`. Typically, :class:`MultipleNegativesRankingLoss` requires a larger batch size for better performance. :class:`GISTEmbedLoss` yields stronger training signals than :class:`MultipleNegativesRankingLoss` due to the use of a guide model for in-batch negative sample selection. Meanwhile, :class:`CachedMultipleNegativesRankingLoss` allows for scaling of the batch size by dividing the computation into two stages of embedding and loss calculation, which both can be scaled by mini-batches (https://arxiv.org/pdf/2101.06983.pdf). By combining the guided selection from :class:`GISTEmbedLoss` and Gradient Cache from :class:`CachedMultipleNegativesRankingLoss`, it is possible to reduce memory usage while maintaining performance levels comparable to those of :class:`GISTEmbedLoss`. You can apply different false-negative filtering strategies to discard hard negatives that are too similar to the positive. Two strategies are supported: - "absolute": Discards negatives whose similarity score is greater than or equal to ``positive_score - margin``. - "relative": Discards negatives whose similarity score is greater than or equal to ``positive_score * (1 - margin)``. Args: model: SentenceTransformer model guide: SentenceTransformer model to guide the in-batch negative sample selection. temperature: Temperature parameter to scale the cosine similarities. mini_batch_size: Mini-batch size for the forward pass, this denotes how much memory is actually used during training and evaluation. The larger the mini-batch size, the more memory efficient the training is, but the slower the training will be. It's recommended to set it as high as your GPU memory allows. The default value is 32. show_progress_bar: If True, a progress bar for the mini-batches is shown during training. The default is False. margin_strategy: Strategy used for false negative filtering. One of {"absolute", "relative"}. margin: The margin value for filtering negatives. Defaults to 0.0, together with the "absolute" strategy, this only removes negatives that are more similar to the query than the positive is to the query. References: - Efficient Natural Language Response Suggestion for Smart Reply, Section 4.4: https://arxiv.org/pdf/1705.00652.pdf - Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup: https://arxiv.org/pdf/2101.06983.pdf - GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning https://arxiv.org/abs/2402.16829 Requirements: 1. (anchor, positive) pairs or (anchor, positive, negative pairs) 2. Should be used with large batch sizes for superior performance, but has slower training time than :class:`MultipleNegativesRankingLoss` Inputs: +-------------------------------------------------+--------+ | Texts | Labels | +=================================================+========+ | (anchor, positive) pairs | none | +-------------------------------------------------+--------+ | (anchor, positive, negative) triplets | none | +-------------------------------------------------+--------+ | (anchor, positive, negative_1, ..., negative_n) | none | +-------------------------------------------------+--------+ Recommendations: - Use ``BatchSamplers.NO_DUPLICATES`` (:class:`docs `) to ensure that no in-batch negatives are duplicates of the anchor or positive samples. Relations: - Equivalent to :class:`GISTEmbedLoss`, but with caching that allows for much higher batch sizes Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") guide = SentenceTransformer("all-MiniLM-L6-v2") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = losses.CachedGISTEmbedLoss( model, guide, mini_batch_size=64, margin_strategy="absolute", # or "relative" (e.g., margin=0.05 for max. 95% of positive similarity) margin=0.1 ) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() rzCachedGISTEmbedLoss is not compatible with a SentenceTransformer model based on a StaticEmbedding. Consider using GISTEmbedLoss instead.dimz_Both the training model and the guiding model must be based on the `transformers` architecture.N)absoluterelativez1margin_strategy must be 'absolute' or 'relative'.)superr isinstancer ValueErrormodelguide temperaturer CosineSimilaritysimilarity_fctrCrossEntropyLosscross_entropy_lossmini_batch_sizer:r9show_progress_bar tokenizervocabmax_seq_lengthmust_retokenizemargin_strategymargin) rrTrUrVr[r\rarb __class__s rrzCachedGISTEmbedLoss.__init__@s.~  eAh 08   & 11b9%(K0 58[8Yq #%"5"5"7.04 =A!2 OO ! !U__%:%: : ie>R>RUZUiUi>i    "__DN ": :PQ Q. rcb|j|jd|jdS)Nr)rX unsqueeze)rembed1embed2s r sim_matrixzCachedGISTEmbedLoss.sim_matrixs+""6#3#3A#68H8H8KLLrc |rtntj}| tn|}|jD cic] \} } | | || } } } |5|5|rt | j nd}|j | d} dddtj5|jr|jj| dd} |jj| } | jDcic]+\}}||j|jj-} }}|j| d}dddddd |fScc} } w#1swYxYwcc}}w#1swY.xYw#1swY2xYw)zYDo forward pass on a minibatch of the input features and return corresponding embeddings.Nsentence_embedding input_idsT)skip_special_tokens)rrno_graditemsrvaluesrTr`r] batch_decoderUtokenizetodevice)rr6beginendr7r8 random_state grad_contextrandom_state_contextkvsentence_feature_minibatchrepsdecodedkeyvalue guide_repss rembed_minibatchz#CachedGISTEmbedLoss.embed_minibatchs'0{U]] 0<0D{},BRBXBXBZ%[$!Qa5o%["%[ ! Z TTe{,F,M,M,OPko zz"<=>RS T Z''"nn992;?UY:G261D1DW1M.KeKkKkKm2=GS%UXXdjj&7&7882.2"ZZ(BCDXY  Z ZZ--!&\ T T2 Z Z Z ZsTEE=/EE="AE1=0E+ -E1E=E( $E=+E11E: 6E==Fc #$K|d}|j\}}ttjd||jd|j D]=\}} | |jz} |j || | |||dn||\} } } | | | f?yw)z`Do forward pass on all the minibatches of the input features and yield corresponding embeddings.rlrzEmbed mini-batchesdescdisableN)r6rurvr7r8rw)shape enumeratetqdmtranger[r\r)rr6r7r8r9rlbszrFiber}rrws rr=z(CachedGISTEmbedLoss.embed_minibatch_iters-[9 Q KK$$) 222    1DAqD(((A-1-A-A!1#"3%2%:T a@P .B. *D*l L0 0% 1sBBc|j||d}|jj}|Dcgc]}|Dcgc]}|jc}c}}|_|Scc}wcc}}w)zeGeneralized function to calculate the cross-entropy loss and cache the gradients wrt. the embeddings.T) with_backward)calculate_lossdetachrequires_grad_rDr:)rr} reps_guidedlossrsrs r"calculate_loss_and_cache_gradientsz6CachedGISTEmbedLoss.calculate_loss_and_cache_gradientss[""4D"I{{}++-59:rr*!qvv*:  +:s A)A$ A)$A)ct|t|k7r td|Dcgc]}tj|d}}|Dcgc]}tj|d}}tj|dj dj j|dj}|djd} g} tjd| jdj D]D} | jz} j|d| | |d} j|d| | |d}j|d| | |d}| j| j!ddj|d| | |d}j|d| | |d}j|d| | |d}d dfd }tj"| jtj$| jd }|j'| }|| || }|||}|||}tj|||gd}t|d kDrrt)d t|D]Z}j|d| | ||}j|d| | ||}|||}tj||gd}\|j*z }j-||| | t|z| z }|r |j/|j1}| j3|Gt5| }|Scc}wcc}w)zSGeneralized function to calculate the cross-entropy loss without caching gradients.z.reps and reps_guided must have the same lengthrrMzCalculating lossrre)offsetrLcjdk(r|jz kD}n$jdk(r|djz zkD}||z}tj |<|S)NrOrPre)rarbrinf)guided_sim_matsim_mat positive_maskmask guided_simrs rmask_false_negativesz@CachedGISTEmbedLoss.calculate_loss..mask_false_negatives*so'':5)Z$++-EFD))Z7)Z1t{{?-KLD ,=.0D!&  r)dtypert)rr)rz Tensor | None)lenrSrcatarangesizelongrsrtrrrr[r\ridiagonalvieweyeboolrollrangerVrZr@rappendsum)rr}rrrepconcatenated_reps rep_guideconcatenated_guided_repslabels batch_sizelossesrr guided_ap_sim guided_aa_sim guided_pp_simap_simaa_simpp_simrrscoresrguided_neg_simneg_sim loss_mbatchrrs` @rrz"CachedGISTEmbedLoss.calculate_losss t9K( (MN N?CCsUYYs2CCQ\#]IEIIiQ$?#] #]/277:;@@BEEFWXYFZFaFab&q)//2 %'   #...  ? 'AD(((A!OO,DQ,G!,LNfghNijM OO,DQ,G!,LNfghNijM OO,DQ,G!,LNfghNijM'//q/9>>r1EJ__%6q%9!A%>@QRS@TUF__%6q%9!A%>@QRS@TUF__%6q%9!A%>@QRS@TUF "II}':':%**UbUiUijM)..q1M*-}]F)-@F)-@FYY7Q?F$%)q#&7"89AA%)__5Ma5PQRST5UWopqWr%sN"oo.?.B1Q.GIZ[\I]^G2>7KG"YY'8a@F Ad...F(,(?(?qQR (TWZ[aWb(beo(oK$$&)002 MM+ &? 'B6{ SD#]s M McZg}g}g|_|D]}g}g}g}|j|ddD]c\} } } |j| jj |j| j|j| e|j||j||jj|t j r5|j||} | jtt||| S|j||} | S)NFT)r6r7r8)rBrC) r9r=rrrris_grad_enabledr register_hookrrIr) rrBrr}rr6reps_mbsreps_guided_mbsrandom_state_mbsrEreps_guided_mbrwrs rforwardzCachedGISTEmbedLoss.forwardWs3  1 8 H O! 9=9R9R!1"&:S: 65  0 ? ? AB&&~'<'<'>? '' 5 6 KK !    /    % %&6 7 8  "::4MD   w~IZeij k &&t[9D rcv|j|j|j|j|jdS)NrUrVr[rarbrr's rget_config_dictz#CachedGISTEmbedLoss.get_config_dictws5ZZ++#33#33kk   r)g{Gz? FrOg)rTrrUrrVfloatr[intr\rrazLiteral['absolute', 'relative']rbrr-r.)rgr rhr r-r r)r6dict[str, Tensor]rurrvrr7rr8rrwzRandContext | Noner-z)tuple[Tensor, Tensor, RandContext | None]) r6rr7rr8rr9zlist[RandContext] | Noner-z3Iterator[tuple[Tensor, Tensor, RandContext | None]])r}list[list[Tensor]]rrr-r )F)r}rrrrrr-r )rBIterable[dict[str, Tensor]]rr r-r )r-zdict[str, Any]) r/r0r1rrirr=rrrr __classcell__)rcs@rr4r4?sB "!"';Ez"z#z z  z  z9zz zxM,0.+.. .  .  .). 3.F37 1+11 1 0 1 = 1<`eQ&Q5GQX\Q Qf@ r)rAr rBrrCr4r-r.) __future__rcollections.abcrr contextlibr functoolsrtypingrr rrr r torch.utils.checkpointr r sentence_transformersrsentence_transformers.modelsrrrrIModuler4r3rrrsk"." G5E.%%2%"% %. ")) r