l^(h9ddlmZddlZddlZddlZddlmZddlmZm Z ddl Z ddl Z ddl Z ddlmZmZddlmZddlmZerddlmZej.eZGd d eZy) ) annotationsN) nullcontext) TYPE_CHECKINGCallable)average_precision_score ndcg_score)SentenceEvaluator)cos_sim)SentenceTransformerc eZdZdZdddedddddf dfd Z d dd Zd Zd Zd Z d Z xZ S)RerankingEvaluatoraC This class evaluates a SentenceTransformer model for the task of re-ranking. Given a query and a list of documents, it computes the score [query, doc_i] for all possible documents and sorts them in decreasing order. Then, MRR@10, NDCG@10 and MAP is compute to measure the quality of the ranking. Args: samples (list): A list of dictionaries, where each dictionary represents a sample and has the following keys: - 'query': The search query. - 'positive': A list of positive (relevant) documents. - 'negative': A list of negative (irrelevant) documents. at_k (int, optional): Only consider the top k most similar documents to each query for the evaluation. Defaults to 10. name (str, optional): Name of the evaluator. Defaults to "". write_csv (bool, optional): Write results to CSV file. Defaults to True. similarity_fct (Callable[[torch.Tensor, torch.Tensor], torch.Tensor], optional): Similarity function between sentence embeddings. By default, cosine similarity. Defaults to cos_sim. batch_size (int, optional): Batch size to compute sentence embeddings. Defaults to 64. show_progress_bar (bool, optional): Show progress bar when computing embeddings. Defaults to False. use_batched_encoding (bool, optional): Whether or not to encode queries and documents in batches for greater speed, or 1-by-1 to save memory. Defaults to True. truncate_dim (Optional[int], optional): The dimension to truncate sentence embeddings to. `None` uses the model's current truncation dimension. Defaults to None. mrr_at_k (Optional[int], optional): Deprecated parameter. Please use `at_k` instead. Defaults to None. Example: :: from sentence_transformers import SentenceTransformer from sentence_transformers.evaluation import RerankingEvaluator from datasets import load_dataset # Load a model model = SentenceTransformer("all-MiniLM-L6-v2") # Load a dataset with queries, positives, and negatives eval_dataset = load_dataset("microsoft/ms_marco", "v1.1", split="validation") samples = [ { "query": sample["query"], "positive": [text for is_selected, text in zip(sample["passages"]["is_selected"], sample["passages"]["passage_text"]) if is_selected], "negative": [text for is_selected, text in zip(sample["passages"]["is_selected"], sample["passages"]["passage_text"]) if not is_selected], } for sample in eval_dataset ] # Initialize the evaluator reranking_evaluator = RerankingEvaluator( samples=samples, name="ms-marco-dev", ) results = reranking_evaluator(model) ''' RerankingEvaluator: Evaluating the model on the ms-marco-dev dataset: Queries: 9706 Positives: Min 1.0, Mean 1.1, Max 5.0 Negatives: Min 1.0, Mean 7.1, Max 9.0 MAP: 56.07 MRR@10: 56.70 NDCG@10: 67.08 ''' print(reranking_evaluator.primary_metric) # => ms-marco-dev_ndcg@10 print(results[reranking_evaluator.primary_metric]) # => 0.6708042171399308 T@FNc t |||_||_| !tj d| d| |_n||_||_||_||_ ||_ | |_ t|jtr(t|jj|_|jD cgc](} t!| ddkDst!| ddkDs'| *c} |_d|rd|zndzd |j d z|_d d d d|j d|j g|_||_d|j |_ycc} w)Nz?The `mrr_at_k` parameter has been deprecated; please use `at_k=z ` instead.positivernegativer _rz _results_@z.csvepochstepsMAPMRR@NDCG@ndcg@)super__init__samplesnameloggerwarningat_ksimilarity_fct batch_sizeshow_progress_baruse_batched_encoding truncate_dim isinstancedictlistvalueslencsv_file csv_headers write_csvprimary_metric) selfrr!rr.r"r#r$r%r&mrr_at_ksample __class__s q/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/sentence_transformers/evaluation/RerankingEvaluator.pyrzRerankingEvaluator.__init__VsZ     NN\]e\ffpq r DIDI,$!2$8!( dllD ) 3 3 56DL"& VJ5G1H11LQTU[\fUgQhklQlF  -dd KPZ[_[d[dZeeiNjj   499+ DII;   # %dii[1 s;EE$Ec(|dk7r|dk(rd|}n d|d|d}nd}|j|d|jd z }tjd |jd |d |j |}|d }|d}|d} |j D cgc]} t | d} } |j D cgc]} t | d} } tjdt |j dtj| ddtj| ddtj| ddtj| ddtj| ddtj| dtjd|dzdtjd|jd|dzdtjd|jd| dzd||jrtjj||j } tjj#| }t%| d|rdndd !5}t'j(|}|s|j+|j,|j+||||| gdddd |d"|j|d#|j| i}|j/||j}|j1|||||Scc} wcc} w#1swYgxYw)$a Evaluates the model on the dataset and returns the evaluation metrics. Args: model (SentenceTransformer): The SentenceTransformer model to evaluate. output_path (str, optional): The output path to write the results. Defaults to None. epoch (int, optional): The current epoch number. Defaults to -1. steps (int, optional): The current step number. Defaults to -1. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. z after epoch z in epoch z after z stepsrNz (truncated to )z0RerankingEvaluator: Evaluating the model on the z dataset:mapmrrndcgrrz Queries: z Positives: Min z.1fz, Mean z, Max z Negatives: Min zMAP: dz.2frz: rawzutf-8)newlinemodeencodingzmrr@r)r&rinforcompute_metricesrr+npminmeanmaxr!r.ospathjoinr,isfileopencsvwriterwriterowr-prefix_name_to_metrics store_metrics_in_model_card_data)r0model output_pathrrout_txtscoresmean_apmean_mrr mean_ndcgr2 num_positives num_negativescsv_pathoutput_file_existsfrNmetricss r4__call__zRerankingEvaluator.__call__s B;{)%1&ugWUG6BG    ( ):):(;1= =G FtyykQYZaYbbcde&&u--%=6N @D||LVVJ/0L L?C||LVVJ/0L L DLL)**=bff]>STW=XX_`b`g`ghu`vwz_{|BCECICIJWCXY\B]]pqsqwqwxEqFGJpKKRSUSZSZ[hSijmRnntuwu{u{|IuJKNtO P   eGcM#./0 d499+R3s';<= eDII;bS(=>?  "t~~ww||K?H!#!9 h8JPS^ef NjkA)OOD$4$45w) LM  N 7499+ DII;   --gtyyA --eWeUK9ML N NsK>5LA LLc^|jr|j|S|j|S)a Computes the evaluation metrics for the given model. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. )r%compute_metrices_batchedcompute_metrices_individual)r0rRs r4rCz#RerankingEvaluator.compute_metricess7((  ) )% 0 11%8 c$g}g}g}|j tn|j|j5|j|jDcgc]}|d c}d|j |j }g}|jD]*}|j|d|j|d,|j|d|j |j }dddd\} } |jD]=} | } | dz } t| d} t| d}| | | z|z}| | |zz } | d k(s|d k(rH|j| |}t|jdkDr|d }tj| }|jj}dg| zd g|zz}d }t|d |j D]\}}||s d|dzz }n|j#||j#t%|g|g|j |j#t'||@t)j*|}t)j*|}t)j*|}|||d Scc}w#1swYxYw) aE Computes the evaluation metrics in a batched way, by batching all queries and all documents together. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. NqueryTconvert_to_tensorr#r$rr)rrrkr9r:r;)r&rtruncate_sentence_embeddingsencoderr#r$extendr+r"shapetorchargsortcputolist enumerater!appendrrrDrF)r0rRall_mrr_scoresall_ndcg_scores all_ap_scoresr2all_query_embsall_docs all_docs_embs query_idxdocs_idxinstance query_embnum_posnum_negdocs_emb pred_scorespred_scores_argsort is_relevant mrr_scorerankindexrVrWrXs r4raz+RerankingEvaluator.compute_metrices_batcheds "//7[]U=_=_`d`q`q=r "\\/3||*GGO, xCCq=  sJ J BJJJcg}g}g}tj|j|j dD]}|d}t|d}t|d}t |dk(st |dk(rB||z} dgt |zdgt |zz} |j t n|j|j 5|j|gd |jd } |j| d |jd } ddd|j  } t | jdkDr| d} tj| }| jj} d}t!|d|j"D]\}}| |s d|dzz }n|j%||j%t'| g| g|j" |j%t)| | t+j,|}t+j,|}t+j,|}|||d S#1swY;xYw)aO Computes the evaluation metrics individually by embedding every (query, positive, negative) tuple individually. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. Samples)disabledescrerrrrhNTFrfrirk)tqdmrr$r)r+r&rrlrmr#r"rorprqrrrsrtr!rurrrDrF)r0rRrvrwrxr~rerrdocsrrrrrrrrrVrWrXs r4rbz.RerankingEvaluator.compute_metrices_individualsH  $,,D*GGO, xCCC  s AII cXd|ji}|j|j|d<|S)Nr!r&)r!r&)r0 config_dicts r4get_config_dictz"RerankingEvaluator.get_config_dict[s2tyy)    (*.*;*;K 'rc)rz list[dict[str, str | list[str]]]r!intrstrr.boolr"z4Callable[[torch.Tensor, torch.Tensor], torch.Tensor]r#rr$rr%rr& int | Noner1r)Nr6r6) rRr rSrrrrrreturnzdict[str, float]) __name__ __module__ __qualname____doc__r rr_rCrarbr __classcell__)r3s@r4r r s<BOV"'%)#'#.21.2.2 .2  .2 M .2.2 .2#.2!.2.2bbd=(=7:=JM=[^= =~ HDT:Dxrcr ) __future__rrMloggingrH contextlibrtypingrrnumpyrDrprsklearn.metricsrr2sentence_transformers.evaluation.SentenceEvaluatorr sentence_transformers.utilr )sentence_transformers.SentenceTransformerr getLoggerrrr rcr4rsP"  "* ?P.M   8 $H*Hrc