l^(h>ddlmZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z mZddlmZmZmZmZmZddlZddlZddlZddlmZmZddlmZmZddlmZdd l mZdd l!m"Z"ejFe$Z%erdd l&m'Z'dd l(m)Z)dd l*m+Z+d=dZ,d>dZ-d=dZ.d?dZ/d@dZ0d?dZ1d@dZ2d?dZ3d@dZ4dAdZ5d@dZ6dAdZ7dBdZ8dCdZ9edDdZ:edEdZ:dFdZ:dd d!d"d#d$e0f dGd%Z;d!d"d#d$e0f dHd&Z dK dLd*Z?dMd+Z@dNd,ZAdOd-ZBdPd.ZC dQ dRd/ZDGd0d1eZEe ejfd2ZG dS dTd3ZH dS dUd4ZI dS dVd5ZJd6ZKdOd7ZLdWd8ZMdXd9ZNdXd:ZOdXd;ZPe d<ZQy)Y) annotationsN)contextmanager)PackageNotFoundErrormetadata) TYPE_CHECKINGAnyCallableLiteraloverload)hf_hub_downloadsnapshot_download)Tensordevice)trange)tqdm)is_torch_npu_availableDataset) CrossEncoder)SentenceTransformercPt|tstj|}|S)z Converts the input `a` to a PyTorch tensor if it is not already a tensor. Args: a (Union[list, np.ndarray, Tensor]): The input array or tensor. Returns: Tensor: The converted tensor. ) isinstancertorchtensoras X/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/sentence_transformers/util.py_convert_to_tensorr!s  a LLO HcN|jdk(r|jd}|S)z If the tensor `a` is 1-dimensional, it is unsqueezed to add a batch dimension. Args: a (Tensor): The input tensor. Returns: Tensor: The tensor with a batch dimension. r)dim unsqueezers r_convert_to_batchr$0s# uuw!| KKN Hrc2t|}t|}|S)z Converts the input data to a tensor with a batch dimension. Args: a (Union[list, np.ndarray, Tensor]): The input data to be converted. Returns: Tensor: The converted tensor with a batch dimension. )rr$rs r_convert_to_batch_tensorr&?s 1A!A Hrct||S) Computes the cosine similarity between two tensors. Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Matrix with res[i][j] = cos_sim(a[i], b[j]) )cos_simrbs rpytorch_cos_simr,Ns 1a=rct|}t|}t|}t|}tj||j ddS)r(rr!)r&normalize_embeddingsrmm transpose)rr+a_normb_norms rr)r)\sJ !#A #A !! $F !! $F 88FF,,Q2 33rcjt|}t|}tt|t|S)a Computes the pairwise cosine similarity cos_sim(a[i], b[i]). Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Vector with res[i] = cos_sim(a[i], b[i]) )rpairwise_dot_scorer.r*s rpairwise_cos_simr5os1 1A1A 2157KA7N OOrczt|}t|}tj||jddS)a Computes the dot-product dot_prod(a[i], b[j]) for all i and j. Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Matrix with res[i][j] = dot_prod(a[i], b[j]) rr!)r&rr/r0r*s r dot_scorer7s4 !#A #A 88Aq{{1a( ))rcXt|}t|}||zjdS)a Computes the pairwise dot-product dot_prod(a[i], b[i]). Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Vector with res[i] = dot_prod(a[i], b[i]) r")rsumr*s rr4r4s- 1A1A E;;2; rc`t|}t|}tj||d S)a: Computes the manhattan similarity (i.e., negative distance) between two tensors. Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Matrix with res[i][j] = -manhattan_distance(a[i], b[j]) g?pr&rcdistr*s r manhattan_simrA. !#A #A KK1 $ $$rct|}t|}tjtj||z d S)a< Computes the manhattan similarity (i.e., negative distance) between pairs of tensors. Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Vector with res[i] = -manhattan_distance(a[i], b[i]) r9r:)rrr;absr*s rpairwise_manhattan_simrEs9 1A1A IIeiiA&B / //rc`t|}t|}tj||d S)a: Computes the euclidean similarity (i.e., negative distance) between two tensors. Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Matrix with res[i][j] = -euclidean_distance(a[i], b[j]) g@r=r?r*s r euclidean_simrGrBrct|}t|}tjtj||z dzd S)a: Computes the euclidean distance (i.e., negative distance) between pairs of tensors. Args: a (Union[list, np.ndarray, Tensor]): The first tensor. b (Union[list, np.ndarray, Tensor]): The second tensor. Returns: Tensor: Vector with res[i] = -euclidean_distance(a[i], b[i]) r9r:)rrsqrtr;r*s rpairwise_euclidean_simrKs= 1A1A JJuyy!a%A26 7 77rcJt|}t|}tj|dd\}}tj|dd\}}tj|dz|dzzdd}||z||zz|z }||z||zz |z }tj|dz|dzzdddz} tj|dz|dzzdddz} || | z z}|| | z z}tjtj||fdd} tj | S)aP Computes the absolute normalized angle distance. See :class:`~sentence_transformers.losses.AnglELoss` or https://arxiv.org/abs/2309.12871v1 for more information. Args: x (Tensor): The first tensor. y (Tensor): The second tensor. Returns: Tensor: Vector with res[i] = angle_sim(a[i], b[i]) rIr!r:T)r"keepdim?)rrchunkr;concatrD) xyrr+cdzreimdzdw norm_angles rpairwise_angle_simr[s 1A1A ;;q! #DAq ;;q! #DAq !Q$A+1d3A a%!a%-1 B a%!a%-1 B 1a4!Q$;At 4 ;B 1a4!Q$;At 4 ;B"r'MB"r'MB5<<Ra8a@J 99Z  rcZtjjj|ddS)z Normalizes the embeddings matrix, so that each sentence embedding has unit length. Args: embeddings (Tensor): The input embeddings matrix. Returns: Tensor: The normalized embeddings matrix. rIr!)r>r")rnn functional normalize) embeddingss rr.r.s& 88   ( (qa ( @@rcyNr` truncate_dims rtruncate_embeddingsrfsY\rcyrbrcrds rrfrfs]`rc|dd|fS)a Truncates the embeddings matrix. Args: embeddings (Union[np.ndarray, torch.Tensor]): Embeddings to truncate. truncate_dim (Optional[int]): The dimension to truncate sentence embeddings to. `None` does no truncation. Example: >>> from sentence_transformers import SentenceTransformer >>> from sentence_transformers.util import truncate_embeddings >>> model = SentenceTransformer("tomaarsen/mpnet-base-nli-matryoshka") >>> embeddings = model.encode(["It's so nice outside!", "Today is a beautiful day.", "He drove to work earlier"]) >>> embeddings.shape (3, 768) >>> model.similarity(embeddings, embeddings) tensor([[1.0000, 0.8100, 0.1426], [0.8100, 1.0000, 0.2121], [0.1426, 0.2121, 1.0000]]) >>> truncated_embeddings = truncate_embeddings(embeddings, 128) >>> truncated_embeddings.shape >>> model.similarity(truncated_embeddings, truncated_embeddings) tensor([[1.0000, 0.8092, 0.1987], [0.8092, 1.0000, 0.2716], [0.1987, 0.2716, 1.0000]]) Returns: Union[np.ndarray, torch.Tensor]: Truncated embeddings. .Nrcrds rrfrfs: c=L=( ))rF iii dc N|j|||d} t| |||||S)a Given a list of sentences / texts, this function performs paraphrase mining. It compares all sentences against all other sentences and returns a list with the pairs that have the highest cosine similarity score. Args: model (SentenceTransformer): SentenceTransformer model for embedding computation sentences (List[str]): A list of strings (texts or sentences) show_progress_bar (bool, optional): Plotting of a progress bar. Defaults to False. batch_size (int, optional): Number of texts that are encoded simultaneously by the model. Defaults to 32. query_chunk_size (int, optional): Search for most similar pairs for #query_chunk_size at the same time. Decrease, to lower memory footprint (increases run-time). Defaults to 5000. corpus_chunk_size (int, optional): Compare a sentence simultaneously against #corpus_chunk_size other sentences. Decrease, to lower memory footprint (increases run-time). Defaults to 100000. max_pairs (int, optional): Maximal number of text pairs returned. Defaults to 500000. top_k (int, optional): For each sentence, we retrieve up to top_k other sentences. Defaults to 100. score_function (Callable[[Tensor, Tensor], Tensor], optional): Function for computing scores. By default, cosine similarity. Defaults to cos_sim. Returns: List[List[Union[float, int]]]: Returns a list of triplets with the format [score, id1, id2] T)show_progress_bar batch_sizeconvert_to_tensor)query_chunk_sizecorpus_chunk_size max_pairstop_kscore_function)encodeparaphrase_mining_embeddings) model sentencesrlrmrorprqrrrsr`s rparaphrase_miningrx=sC>%6:aeJ ()+%  rc |dz }tj}d}d}tdt||D])} tdt||D] } ||| | |z|| | |z} t j | t |t| dddd\} } | jj} | jj} tt| D]n}t| |D][\}}| |z}| |z}||k7s| |||kDs"|j| ||||f|dz }||k\sG|j}|d}]p,t}g}|jsg|j\}}}t||g\}}||k7r-||f|vr'|j||f|j!|||g|jsgt|dd}|S) a Given a list of sentences / texts, this function performs paraphrase mining. It compares all sentences against all other sentences and returns a list with the pairs that have the highest cosine similarity score. Args: embeddings (Tensor): A tensor with the embeddings query_chunk_size (int): Search for most similar pairs for #query_chunk_size at the same time. Decrease, to lower memory footprint (increases run-time). corpus_chunk_size (int): Compare a sentence simultaneously against #corpus_chunk_size other sentences. Decrease, to lower memory footprint (increases run-time). max_pairs (int): Maximal number of text pairs returned. top_k (int): For each sentence, we retrieve up to top_k other sentences score_function (Callable[[Tensor, Tensor], Tensor]): Function for computing scores. By default, cosine similarity. Returns: List[List[Union[float, int]]]: Returns a list of triplets with the format [score, id1, id2] r!r9rTFr"largestsortedc |dS)NrrcrQs rz.paraphrase_mining_embeddings..s !A$rkeyreverse)queue PriorityQueuerangelenrtopkmincputolist enumerateputgetsetemptyr|addappend)r`rorprqrrrspairs min_score num_addedcorpus_start_idxquery_start_idxscoresscores_top_k_valuesscores_top_k_idx query_itr top_k_idx corpus_itrijentry added_pairs pairs_listscoresorted_isorted_js rrurujs/0 QJE    !EII!!S_6GH1$QJ9IJ 1O#?_?O-OP+.>AR.RSF 5:JJE3vay>24PU5 1 !1#6"9"9";"B"B"D /335<<> "3v;/ 1 -67G 7R-S 1)Iz')3A(:5AAv"5i"@"Ki"W #6y#A)#LaQR"ST!Q $ 1$)IIKE(-aI 1 1 116%KJkkmiik q!#QF^( x Xx$8 $K OOXx0 1   uh9 : kkm EJ rct|i|S)z8This function is deprecated. Use semantic_search instead)semantic_search)argskwargss rinformation_retrievalrs D +F ++r c t|tjtjfrt j |}n%t|t rt j|}t|jdk(r|jd}t|tjtjfrt j |}n%t|t rt j|}|j|jk7r|j|j}tt|Dcgc]}g}}tdt||D])}tdt||D] } |||||z|| | |z} t j| t|t| dddd\} } | j!j#} | j!j#} tt| D]n} t%| | | | D]W\}}| |z}|| z}t|||krt'j(||||f=t'j*||||fYp,tt|D]I}tt||D]}|||\}}||d|||<t-||dd||<K|Scc}w) a This function performs a cosine similarity search between a list of query embeddings and a list of corpus embeddings. It can be used for Information Retrieval / Semantic Search for corpora up to about 1 Million entries. Args: query_embeddings (:class:`~torch.Tensor`): A 2 dimensional tensor with the query embeddings. corpus_embeddings (:class:`~torch.Tensor`): A 2 dimensional tensor with the corpus embeddings. query_chunk_size (int, optional): Process 100 queries simultaneously. Increasing that value increases the speed, but requires more memory. Defaults to 100. corpus_chunk_size (int, optional): Scans the corpus 100k entries at a time. Increasing that value increases the speed, but requires more memory. Defaults to 500000. top_k (int, optional): Retrieve top k matching entries. Defaults to 10. score_function (Callable[[:class:`~torch.Tensor`, :class:`~torch.Tensor`], :class:`~torch.Tensor`], optional): Function for computing scores. By default, cosine similarity. Returns: List[List[Dict[str, Union[int, float]]]]: A list with one entry for each query. Each entry is a list of dictionaries with the keys 'corpus_id' and 'score', sorted by decreasing cosine similarity scores. r!rTFrz) corpus_idrc |dS)Nrrcr~s rrz!semantic_search..s \]^e\frr)rnpndarraygenericr from_numpyliststackrshaper#rtorrrrrzipheapqheappush heappushpopr|)query_embeddingscorpus_embeddingsrorprrrs_queries_result_listrr cos_scorescos_scores_top_k_valuescos_scores_top_k_idxr sub_corpus_idrrquery_iddoc_itrs rrrs0"RZZ$<= ++,<= $d + ;;'78  ! !"a'+55a8#bjj"**%=>!,,->? %t ,!KK(9:#3#:#::+../@/G/GH',S1A-B'CD!2DD C(8$9;KL] %a->)?AR S ] ' ?EU3UV!"25EHY5YZJ =BJJCs:a='9:4X]= 9 #%9'>&A&A&C&J&J&L ##7#;#;#=#D#D#F "3z?3 ] ,/0DY0OQhirQs,t](M5 0= @I.:H.x89EA/9E9;M))*=h*G%QZI[\] ] ]]8#123vS!4X!>?@ ^G28 K.c#ts tdddlm}|j}|r||vr|d}|r||vr|d}|s|st |dk7r t d||rdnd }tjd |d |rA|dk7s | ||| d k7rtjd |d k7rtjdd }||} tjd| d t|jj|jj|j}t|}|A| | | || dzz|z}n|| z|z}|dkDr|rd}|r t!d|rt!d|di}||}||}|du}|s|}tt#|t#|z}t%|D cic]\}} | | }!}} |j'}"tt#|}t%|D#cic]\}}#|#| }$}}#t |}%t)j*|%j-d}&|j.}'|%t |"k7r|rt!d|%dt |"d|dkDr%t1j2|}(t!d|(dd|rb|j5t7|t8rdn|})|j;||)|dd}*|j;||)|dd}+|j=|)n,|j?||ddd}*|j?||ddd}+|r=ddl },|,jC|jE}- |,jG}.d|._$d|._%|,jM|-|. }-|-jQ|*g}/g}0tSdt |+|d!"D]E}1|+|1|1|z}2|-jU|2|dz#\}3}4|/jW|3|0jW|4Gt)jXt1jZ|/d$j]|'}3t)jXt1jZ|0d$j]|'}4n?|j_|+|*j]|'}3t)j`|3||zd%\}3}4tc|%D5cgc]}5g}6}5te|"|D]!\}#}7|$|#}8|6|8jW|!|7#|6D9cgc] }9t |9}:}9g}g}"tc|%D]B}|jg|6|D;cgc]};||; c};|"jg||g|:|zD|6D9cgc]}9t)jh|9|'&}6}9|+tc|%D5cgc]}tc|:|D]}5|c}5}}+|*t)jj|6jm}<|jo|+|<j]|'}=~+~<~*|| | |tqt%|4d't |4(D]O\}}>||}#|>D?cgc]}?||? }@}?|jstte|#g|dzz|@|d)}A|A|3|<Q|jstte|"||d)}=|sUt)jttc|%DBcgc]}Bt)jv|4|B|6|B c}B}Ctyd* |3|C<|3j{}D| | Qt)j||%|=j.|=j~+}Ed}Ftc|%D].}Bt)j|=F|F|:|BzE|B<|F|:|Bdz z }F0| p|3| zEj|3jddjkD}Gtyd* |3|G<|Gjj}H|HrH|HDz d,|d-<|D|Hz}D| s|3Ej|3jddjd| z zkD}Gtyd* |3|G<|Gjj}H|HrH|HDz d,|d.<|D|Hz}D|?|3|kD}Gtyd* |3|G<|Gjj}H|Hr H|HDz d,|d/<| ?|3| k}Gtyd* |3|G<|Gjj}H|Hr H|HDz d,|d0<t)j`|3|d%\}I}J|4|&|Jf}4|r|4dd|df}4Idd|df}I| d k(r|4ddd| f}4Iddd| f}In| d1k(r|4jdIjjdz }K|Kj| 2}K|KDLcgc]"}Ltjtc|L| #$}M}L|4|&|Mf}4I|&|Mf}I|Ijdd3\}I}J|4|&|Jf}4t)jjtc|%Dcgc]}|4|j|:|dc}}4t)jjtc|%Dcgc]}I|j|:|dc}}I|dk(rGItyd* k7}Nt)j|4}Ot)j|4}P|4|N}4|I|N}Id}Ftc|%D]c}Bt)jh|Bj|:|B| OF|F|:|Bz|6|Bj| djP|F|F|:|Bz|F|:|Bz }FeON}OP|N}6|g|gd4gi}Qte|O|6|4D]K\}R}S}TQ|jW||R|Q|jW||S|Qd4jW||TM|=j| djNIz }Un|d5k(rItyd* k7}N|g|gd6gi}Qtc|%D]}8|6|8D]D}SQ|jW||8|Q|jW||S|Qd6jWdFte|4|8I|8D]W\}T}V|Vtyd* k(rQ|jW||8|Q|jW|T|Qd6jWdYIN}I|=j| dj|N|Iz }Un|d k(rItyd* k7jd7}N|I|N}I|4|N}4|t%|NDWcgc] \}}W|Ws |"|c}W}|t%NDWcgc] \}}W|Ws ||c}W}it%|4jd8D1XYcic]\}1}Xd9|1|XDYcgc]}Y||Y c}Yc}Y}X}1}QIj}I|=j| djNj|Iz }Un |d:k(rItyd* k7}N|t%|NDZcgc]\}}Z|Zjs|"|c}Z}|t%teN|4DZ[W-cgc]D\}\}Z}[|Zjr,||gteZ[DW-cgc] \}W}-|Ws ||-c}-}WzFc}-}W}[}Z}d;NDZcgc]&}Z|ZjsdgdgtZzz(c}Zi}QIN}I|=j| dj|N|Iz }Ut Qdk(r t d<|jQ}\|rd=}]d>}^t!|]jd?d@dAdBt!|]jdC|^t |=|^t IdDdEt(j2fdFt(jfdGt(jfdHdIfdJdKfdLdMfdNdOfdPdQffD]T\}_}`t!]j|_j^|`|=|^|`I|^|`UVd-| fd.| fd/|fd0| ffD]4\}a}b|a|vs |adR}c||adS}dt!dT|cdUdV|ddWdX|adYbdZ 6| t |zt Iz }e|edkDrd[g}f|dkDrfjWd\| fjWd-| fjWd.|fjWd/d]jfdd}gt |fdkDr gd^fdzz }ge| t |zz }ht!d_|ed`|hdWdagdbt fdkDrdcndDdd \Scc} }wcc}#}w#tN$rY wxYwcc}5wcc}9wcc};wcc}9wcc}5}wcc}?wcc}Bwcc}Lwcc}wcc}wcc}W}wcc}W}wcc}Ywcc}Y}X}1wcc}Z}wcc}-}Wwcc}-}W}[}Z}wcc}Zw)eu4 Add hard negatives to a dataset of (anchor, positive) pairs to create (anchor, positive, negative) triplets or (anchor, positive, negative_1, ..., negative_n) tuples. Hard negative mining is a technique to improve the quality of a dataset by adding hard negatives, which are texts that may appear similar to the anchor, but are not. Using hard negatives can improve the performance of models trained on the dataset. This function uses a SentenceTransformer model to embed the sentences in the dataset, and then finds the closest matches to each anchor sentence in the dataset. It then samples negatives from the closest matches, optionally using a CrossEncoder model to rescore the candidates. You can influence the candidate negative selection in various ways: - **range_min**: Minimum rank of the closest matches to consider as negatives: useful to skip the most similar texts to avoid marking texts as negative that are actually positives. - **range_max**: Maximum rank of the closest matches to consider as negatives: useful to limit the number of candidates to sample negatives from. A lower value makes processing faster, but may result in less candidate negatives that satisfy the margin or max_score conditions. - **max_score**: Maximum score to consider as a negative: useful to skip candidates that are too similar to the anchor. - **min_score**: Minimum score to consider as a negative: useful to skip candidates that are too dissimilar to the anchor. - **absolute_margin**: Absolute margin for hard negative mining: useful to skip candidate negatives whose similarity to the anchor is within a certain margin of the positive pair. A value of 0 can be used to enforce that the negative is always further away from the anchor than the positive. - **relative_margin**: Relative margin for hard negative mining: useful to skip candidate negatives whose similarity to the anchor is within a certain margin of the positive pair. A value of 0.05 means that the negative is at most 95% as similar to the anchor as the positive. - **sampling_strategy**: Sampling strategy for negatives: "top" or "random". "top" will always sample the top n candidates as negatives, while "random" will sample n negatives randomly from the candidates that satisfy the margin or max_score conditions. .. tip:: The excellent `NV-Retriever paper `_ is a great resource for understanding the details of hard negative mining and how to use it effectively. Notably, it reaches the strongest performance using these settings:: dataset = mine_hard_negatives( dataset=dataset, model=model, relative_margin=0.05, # 0.05 means that the negative is at most 95% as similar to the anchor as the positive num_negatives=num_negatives, # 10 or less is recommended sampling_strategy="top", # "top" means that we sample the top candidates as negatives batch_size=batch_size, # Adjust as needed use_faiss=True, # Optional: Use faiss/faiss-gpu for faster similarity search ) This corresponds with the `TopK-PercPos (95%)` mining method. Example: >>> from sentence_transformers.util import mine_hard_negatives >>> from sentence_transformers import SentenceTransformer >>> from datasets import load_dataset >>> # Load a Sentence Transformer model >>> model = SentenceTransformer("all-MiniLM-L6-v2") >>> >>> # Load a dataset to mine hard negatives from >>> dataset = load_dataset("sentence-transformers/natural-questions", split="train") >>> dataset Dataset({ features: ['query', 'answer'], num_rows: 100231 }) >>> dataset = mine_hard_negatives( ... dataset=dataset, ... model=model, ... range_min=10, ... range_max=50, ... max_score=0.8, ... relative_margin=0.05, ... num_negatives=5, ... sampling_strategy="random", ... batch_size=128, ... use_faiss=True, ... ) Batches: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 588/588 [00:32<00:00, 18.07it/s] Batches: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 784/784 [00:08<00:00, 96.41it/s] Querying FAISS index: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:06<00:00, 1.06it/s] Metric Positive Negative Difference Count 100,231 487,865 Mean 0.6866 0.4194 0.2752 Median 0.7010 0.4102 0.2760 Std 0.1125 0.0719 0.1136 Min 0.0303 0.1702 0.0209 25% 0.6221 0.3672 0.1899 50% 0.7010 0.4102 0.2760 75% 0.7667 0.4647 0.3590 Max 0.9584 0.7621 0.7073 Skipped 427,503 potential negatives (8.36%) due to the relative_margin of 0.05. Skipped 978 potential negatives (0.02%) due to the max_score of 0.8. Could not find enough negatives for 13290 samples (2.65%). Consider adjusting the range_max, range_min, relative_margin and max_score parameters if you'd like to find more valid negatives. >>> dataset Dataset({ features: ['query', 'answer', 'negative'], num_rows: 487865 }) >>> dataset[0] { 'query': 'when did richmond last play in a preliminary final', 'answer': "Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieved since 1995. A series of close losses hampered the Tigers throughout the middle of the season, including a 5-point loss to the Western Bulldogs, 2-point loss to Fremantle, and a 3-point loss to the Giants. Richmond ended the season strongly with convincing victories over Fremantle and St Kilda in the final two rounds, elevating the club to 3rd on the ladder. Richmond's first final of the season against the Cats at the MCG attracted a record qualifying final crowd of 95,028; the Tigers won by 51 points. Having advanced to the first preliminary finals for the first time since 2001, Richmond defeated Greater Western Sydney by 36 points in front of a crowd of 94,258 to progress to the Grand Final against Adelaide, their first Grand Final appearance since 1982. The attendance was 100,021, the largest crowd to a grand final since 1986. The Crows led at quarter time and led by as many as 13, but the Tigers took over the game as it progressed and scored seven straight goals at one point. They eventually would win by 48 points – 16.12 (108) to Adelaide's 8.12 (60) – to end their 37-year flag drought.[22] Dustin Martin also became the first player to win a Premiership medal, the Brownlow Medal and the Norm Smith Medal in the same season, while Damien Hardwick was named AFL Coaches Association Coach of the Year. Richmond's jump from 13th to premiers also marked the biggest jump from one AFL season to the next.", 'negative': "2018 NRL Grand Final The 2018 NRL Grand Final was the conclusive and premiership-deciding game of the 2018 National Rugby League season and was played on Sunday September 30 at Sydney's ANZ Stadium.[1] The match was contested between minor premiers the Sydney Roosters and defending premiers the Melbourne Storm. In front of a crowd of 82,688, Sydney won the match 21–6 to claim their 14th premiership title and their first since 2013. Roosters five-eighth Luke Keary was awarded the Clive Churchill Medal as the game's official man of the match." } >>> dataset.push_to_hub("natural-questions-hard-negatives", "triplet-all") Args: dataset (Dataset): A dataset containing (anchor, positive) pairs. model (SentenceTransformer): A SentenceTransformer model to use for embedding the sentences. anchor_column_name (str, optional): The column name in `dataset` that contains the anchor/query. Defaults to None, in which case the first column in `dataset` will be used. positive_column_name (str, optional): The column name in `dataset` that contains the positive candidates. Defaults to None, in which case the second column in `dataset` will be used. corpus (List[str], optional): A list containing documents as strings that will be used as candidate negatives in addition to the second column in `dataset`. Defaults to None, in which case the second column in `dataset` will exclusively be used as the negative candidate corpus. cross_encoder (CrossEncoder, optional): A CrossEncoder model to use for rescoring the candidates. Defaults to None. range_min (int): Minimum rank of the closest matches to consider as negatives. Defaults to 0. range_max (int, optional): Maximum rank of the closest matches to consider as negatives. Defaults to None. max_score (float, optional): Maximum score to consider as a negative. Defaults to None. min_score (float, optional): Minimum score to consider as a negative. Defaults to None. absolute_margin (float, optional): Absolute margin for hard negative mining, i.e. the minimum distance between the positive similarity and the negative similarity. Defaults to None. relative_margin (float, optional): Relative margin for hard negative mining, i.e. the maximum ratio between the positive similarity and the negative similarity. A value of 0.05 means that the negative is at most 95% as similar to the anchor as the positive. Defaults to None. num_negatives (int): Number of negatives to sample. Defaults to 3. sampling_strategy (Literal["random", "top"]): Sampling strategy for negatives: "top" or "random". Defaults to "top". include_positives (bool): Whether to include the positives in the negative candidates. Setting this to True is primarily useful for creating Reranking evaluation datasets for CrossEncoder models, where it can be useful to get a full ranking (including the positives) from a first-stage retrieval model. Defaults to False. output_format (Literal["triplet", "n-tuple", "labeled-pair", "labeled-list"]): Output format for the `datasets.Dataset`. Options are: - "triplet": (anchor, positive, negative) triplets, i.e. 3 columns. Useful for e.g. :class:`~sentence_transformers.cross_encoder.losses.CachedMultipleNegativesRankingLoss`. - "n-tuple": (anchor, positive, negative_1, ..., negative_n) tuples, i.e. 2 + num_negatives columns. Useful for e.g. :class:`~sentence_transformers.cross_encoder.losses.CachedMultipleNegativesRankingLoss`. - "labeled-pair": (anchor, passage, label) text tuples with a label of 0 for negative and 1 for positive, i.e. 3 columns. Useful for e.g. :class:`~sentence_transformers.cross_encoder.losses.BinaryCrossEntropyLoss`. - "labeled-list": (anchor, [doc1, doc2, ..., docN], [label1, label2, ..., labelN]) triplets with labels of 0 for negative and 1 for positive, i.e. 3 columns. Useful for e.g. :class:`~sentence_transformers.cross_encoder.losses.LambdaLoss`. Defaults to "triplet". batch_size (int): Batch size for encoding the dataset. Defaults to 32. faiss_batch_size (int): Batch size for FAISS top-k search. Defaults to 16384. use_faiss (bool): Whether to use FAISS for similarity search. May be recommended for large datasets. Defaults to False. use_multi_process (bool | List[str], optional): Whether to use multi-GPU/CPU processing. If True, uses all GPUs if CUDA is available, and 4 CPU processes if it's not available. You can also pass a list of PyTorch devices like ["cuda:0", "cuda:1", ...] or ["cpu", "cpu", "cpu", "cpu"]. verbose (bool): Whether to print statistics and logging. Defaults to True. as_triplets (bool, optional): Deprecated. Use `output_format` instead. Defaults to None. margin (float, optional): Deprecated. Use `absolute_margin` or `relative_margin` instead. Defaults to None. Returns: Dataset: A dataset containing (anchor, positive, negative) triplets, (anchor, passage, label) text tuples with a label, or (anchor, positive, negative_1, ..., negative_n) tuples. zGPlease install `datasets` to use this function: `pip install datasets`.rrr!rIz)Dataset must contain exactly two columns.Ntripletzn-tuplezrThe `as_triplets` parameter is deprecated. Use the `output_format` parameter instead. Setting `output_format` to `z`.topzWhen using `include_positives=True`, updating `range_min`, `range_max`, `max_score`, `margin`, or `sampling_strategy` from the default values may still discard the positive values.z~When using `include_positives=True`, `output_format` will be set to `"n-tuple"` to ensure that the ranking order is preserved.zThe `margin` parameter is deprecated. Use the `absolute_margin` and/or `relative_margin` parameter instead. Setting `absolute_margin` to `riz\Using FAISS, we can only retrieve up to 2048 documents per query. Setting range_max to 2048.zSetting range_max to z" based on the provided parameters.r9zFound z unique queries out of z total queries.zFound an average of z.3fz positives per query.)target_devicesT)rmr.rl)rmr.convert_to_numpyrl)cozQuerying FAISS index)desc)k)axis)rr"rzRescoring with CrossEncoder)rtotal)rmrninf)rdtype)skippedratioabsolute_marginrelative_margin max_scorerrandom)r)r" descendingnegativez labeled-pairlabelr:)start negative_z labeled-listlabelszHNo triplets could be generated. Please check the parameters and dataset.z{:<6} {:>14} {:>14} {:>14}cbt|tjr|jdS|dS)Nz.4f,)rrritem)values rrz%mine_hard_negatives..Fs)Juell<[ejjl3%7dijkclrMetricPositiveNegative DifferenceCountmeanmedianstdrch|jdkDrtj|StdS)Nrr)numelrrfloatrs rrz%mine_hard_negatives..Ts' 8J599V#4PUV[P\rz25%c|jdkDr%tj|jdStdS)Nrg?qrrrquantilerrs rrz%mine_hard_negatives..U0V\\^^_M_5>>&,,.D#Iejkpeqrz50%c|jdkDr%tj|jdStdS)NrrNrrrrs rrz%mine_hard_negatives..Vs0FLLN]^L^5>>&,,.C#Hdijodprz75%c|jdkDr%tj|jdStdS)Nr?rrrrs rrz%mine_hard_negatives..Wrrmaxch|jdkDrtj|StdS)Nrz-inf)rrrrrs rrz%mine_hard_negatives..Xs' 8J599V#4PUV\P]rrrzSkipped rz potential negatives (z.2%z ) due to the z of . range_max range_minz, z and z$Could not find enough negatives for z samples (z). Consider adjusting the z parametersz, if you'd like to find more valid negatives.)Uis_datasets_available ImportErrordatasetsr column_namesr ValueErrorloggerwarningr to_pandasgroupbycountto_dictvaluesrprintrrcopyraranger#rrrstart_multi_process_poolrboolencode_multi_processstop_multi_process_poolrtfaiss IndexFlatIP get_sentence_embedding_dimensionGpuMultipleClonerOptionsshard useFloat16index_cpu_to_all_gpus Exceptionrrsearchrr concatenater similarityrrrextendrcatrsimilarity_pairwiserpredictrisinrrrrrrepeatsizeTr;risinfclamprsamplesort empty_likeallflattenany from_dictformatrr capitalizejoin)idatasetrvanchor_column_namepositive_column_namecorpus cross_encoderrrrrrr num_negativessampling_strategyinclude_positives output_formatrmfaiss_batch_size use_faissuse_multi_processverbose as_tripletsmarginrcolumnspositives_per_query max_positives log_countersqueries positivesseparate_corpusidxtext corpus_idx all_queriesquery queries_idx n_queries batch_idxravg_positives_per_querypoolrrrindexr scores_list indices_listr query_chunkrindicesrpositive_indicespositive query_idxr> n_positivesdoc_idxpositive_embeddingspositive_scores candidate_idx_idxcandidate_passages pred_scoresq_idx positive_masknum_candidatesmax_positive_scores start_idxremoved_indices num_skippednegative_scores local_indices num_optionsoptions sampled_idxindices_to_keepanchor_indices pos_indices dataset_data anchor_idx positive_idx negative_idxdifference_scoresnegative_scorekeep neg_indicesneg_idxkeep_row indices_rowoutput_dataset row_format formattermetricfunction param_name param_valuerrmissing_negatives solutionsconsiderationsmissing_negatives_ratiosi rmine_hard_negativesrs` ! "cdd ""G !37!B$QZ #7w#F&qz &:s7|q?PDEE%0 i  ++8/ =  N$$! E) NNe  I % NNQ &M   --<,=R A ##$67==?GGIJ^_ffh+,M  &/*EI^!]R%78=HI"M1MAI t  Itu  ))4VW XL()G,-ID(O #f+I. /F.7v->? T$)?J?,,.K3w< G09'0BC*#u5#:CKCG I Y'11"5I \\FC $$ yk!8[9I8J/Z[q"$''*="> $%>,Q#GHKKFS!!"24EFII&Q **Vy=/HaP %*)$45q55{I6Ax& #**:h+?@A$44a3q64K4IKY>9I#9NOg&/OPGCL>K,<<=>AQQ1 Qv6QQ(i8H(jRWXcdgXhRi(jQ(j(jk+EII6F,G,N,N,PQ//0@BUVYYZ`aO #'BiF["&y'9@]ehipeq"r & CCLE;H!I4&,!I !I'//S%IM24FGH%"&0K &F3K &(// [), -!"0   NST]N^ _UUZZ(8(? @ _ "'u }\\^N"o&A$kk)OIT_bpTp2q ./+-  &$':'A'A&++a.RS'T'V'VZ[^mZm'nnO',U|mF? #)--/446K>IT_bpTp2q ./+- 9,#(<-%))+002 &$~5)L %  9,#(<-%))+002 &$~5)L % &+ZZ)%K"O]i./G!YZ-()!YZ-8E!!^m^+,)!^m^*;< h &ll1o(=(=(?(C(CA(FF !''M': U`a'v}}U7^}Ea a)[01))[*@A)8)=)=!PT)=)U&)]23iiUS\M]^c,,[-=qA^_Gii]bcl]m nVY!5!"f '0 &C"A{A3!Mg&/!MM *113+22=!DFFW__adss . ()eEl]: yQ`Ga tmc8emeqeqesS!1 t 4=c/SZ>[4\##0C0(K<<>3 SS^E_#hkdEcgF5M#hh# cXT\T`T`TbsaS3x=00c */:+22=!DFFWZii <Acdd&W&&|4N1 n  j*j,OP   #o./#o./   UZZ u|| $ EII  \ ] q r p q q r ] ^ !   FH !!%%'h78h78h'89:   ( 0  0 ) $ ) $ (  #J \)&z29=$Z09wqk)?c{-XbWccghsgttuv +S\9S=QQ q $ I1}  -*  !23*  !23$  -!YYy"~6N9~!'IbM"99&7=3w<;W&X # 67H6ITkloSpq++9*:*CPYN]^L^SdfDghTU  s @ D^   46 5 PR)k"J ``b_ nH!g"f!M!u#h# ds AD  AD1AD? AD(>AD-6 AD2 *AD7AD<< AE#AE'AE 7AE9AE7 AEAE AE! &AE! AE,  AE'(AE, AE32AE3/AE? AE9AE9AE?,AFAFD AD%D$AD%E'AE, E9AE?ctjj|dk7r4tjtjj|dt j |d}|j dk7r6 eCj& )s}5E#J6 Lrc|jj}||tjjk(r|jjS|dz|jjzS)a Gives a full name (package_name.class_name) for a class / object in Python. Will be used to load the correct classes from JSON files Args: o: The object for which to get the full name. Returns: str: The full name of the object. Example: >>> from sentence_transformers.losses import MultipleNegativesRankingLoss >>> from sentence_transformers import SentenceTransformer >>> from sentence_transformers.util import fullname >>> model = SentenceTransformer('all-MiniLM-L6-v2') >>> loss = MultipleNegativesRankingLoss(model) >>> fullname(loss) 'sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss' r) __class__ __module__str__name__)omodules rfullnamersR*[[ # #F ~3==#;#;;{{###|akk2222rc8 |jdd\}} tj|} t ||S#t$r|d}t|wxYw#t $rtj|}YJwxYw#t$rd|d|d}t|wxYw)a: Import a dotted module path and return the attribute/class designated by the last name in the path. Raise ImportError if the import failed. Args: dotted_path (str): The dotted module path. Returns: Any: The attribute/class designated by the last name in the path. Raises: ImportError: If the import failed. Example: >>> import_from_string('sentence_transformers.losses.MultipleNegativesRankingLoss') rr!z doesn't look like a module pathzModule "z" does not define a "z" attribute/class)rsplitrr importlib import_modulergetattrAttributeError) dotted_path module_path class_namemsgrs rimport_from_stringrs$"-"4"4S!"< Z 6((5vz**  =># 6((56  %::,FWX#s%:A A;AA87A8;Bct|tjstj|}tj||j}t |}g}t |t|}t td|zdt|}ttdt||d| D]}||||z|jz}|jjdvr||k\} | jd} | |k\} | js]| | } || }| j} |j| d \} }t!| |D]'\}}|j#|d |j%)|j|d \}} tt|D]}||d |k\s||j|d \}}|d |kDrV|t|krHt d|zt|}||j|d \}}|d |kDr|t|krH|j#|||k\j%t'|d d }g}t)}t+|D]U\}}g}|D]}||vs|j#|t||k\s4|j#||j-|Wt'|dd }|S)a Function for Fast Community Detection. Finds in the embeddings all communities, i.e. embeddings that are close (closer than threshold). Returns only communities that are larger than min_community_size. The communities are returned in decreasing order. The first element in each list is the central point in the community. Args: embeddings (torch.Tensor or numpy.ndarray): The input embeddings. threshold (float): The threshold for determining if two embeddings are close. Defaults to 0.75. min_community_size (int): The minimum size of a community to be considered. Defaults to 10. batch_size (int): The batch size for computing cosine similarity scores. Defaults to 1024. show_progress_bar (bool): Whether to show a progress bar during computation. Defaults to False. Returns: List[List[int]]: A list of communities, where each community is represented as a list of indices. rrI2rzFinding clusters)rdisable)cudanpur!T)rr{Nr9ct|Srbrr~s rrz%community_detection..Ns Arrct|Srbrr~s rrz%community_detection..^s #a&r)rrrrrr.rrrrrr"typer;r*rrrrr|rrr)r` thresholdmin_community_sizermrlextracted_communities sort_max_sizercrthreshold_maskrow_wise_countlarge_enough_maskrr top_k_indicesrrS top_k_valuesr top_val_large top_idx_largeunique_communities extracted_ids cluster_id communitynon_overlapped_communityrEs rcommunity_detectionrs.0 j%,, /\\*-  Yz/@/@AI%j1J/ZAA 22B7ZIM aZ*-4FTePe*e  I ,BCjllR     ! !_ 4'94N+//2N!/2D D $((*+,=>N#$56J""$A)DA A}#&nm"D Gw%,,WVe_-C-C-EF G)oo0BDoQOL!3|,- e?2&)33=a=3E3E _c3E3d0M=(+i7MCPZO<[(+A ,=s:(O 7A!}7I7IMcg7I7h4 }(+i7MCPZO<[*00}PY?Y1Z1a1a1cd eA*eZ##8>NX\]EM!*+@!A; I#%  5C-'(//4 5 ' (,> >  % %&> ?  !9 :; 28HRVW rc.eZdZdZfdZdfd ZxZS) disabled_tqdmz Class to override `disable` argument in case progress bars are globally disabled. Taken from https://github.com/tqdm/tqdm/issues/619#issuecomment-619639324. c.d|d<t||i|y)NTr)super__init__)selfrrrs rrzdisabled_tqdm.__init__os y $)&)rcP t||y#t$r |dk7rYywxYw)zBFix for https://github.com/huggingface/huggingface_hub/issues/1603_lockN)r __delattr__r)rattrrs rrzdisabled_tqdm.__delattr__ss3  G  % w s %%)rrreturnNone)rr __qualname____doc__rr __classcell__)rs@rrrhs *rrc#Ktjjj}tj| dtj|y#tj|wxYww)z A context manager that will prevent any logging messages triggered during the body from being processed. Args: highest_level: the maximum logging level allowed. N)loggingrootmanagerr) highest_levelprevious_levels rdisable_loggingr|sI\\))11N OOM"( ''s:A1AA1A..A1c 6tt|d||||S)a} Checks if the given model name or path corresponds to a SentenceTransformer model. Args: model_name_or_path (str): The name or path of the model. token (Optional[Union[bool, str]]): The token to be used for authentication. Defaults to None. cache_folder (Optional[str]): The folder to cache the model files. Defaults to None. revision (Optional[str]): The revision of the model. Defaults to None. local_files_only (bool): Whether to only use local files for the model. Defaults to False. Returns: bool: True if the model is a SentenceTransformer model, False otherwise. z modules.json)token cache_folderrevisionlocal_files_only)r load_file_path)model_name_or_pathrrrrs ris_sentence_transformer_modelrs+(   %-    rc tjj||}tjj|r|S t |||d|||S#t $rYywxYw)a Loads a file from a local or remote location. Args: model_name_or_path (str): The model name or path. filename (str): The name of the file to load. token (Optional[Union[bool, str]]): The token to access the remote file (if applicable). cache_folder (Optional[str]): The folder to cache the downloaded file (if applicable). revision (Optional[str], optional): The revision of the file (if applicable). Defaults to None. local_files_only (bool, optional): Whether to only consider local files. Defaults to False. Returns: Optional[str]: The path to the loaded file, or None if the file could not be found or loaded. sentence-transformers)filenamer library_namer cache_dirrN)rrr.existsr r)rrrrrr file_paths rrrsj. /:I ww~~i   0"-   sA A! A!c Jtjj||}tjj|r|S|||dvr|dndd|||td} t di|}tjj||S#t $rd|d<t di|}Y;wxYw) af Loads the directory path for a given model name or path. Args: model_name_or_path (str): The name or path of the model. directory (str): The directory to load. token (Optional[Union[bool, str]]): The token for authentication. cache_folder (Optional[str]): The folder to cache the downloaded files. revision (Optional[str], optional): The revision of the model. Defaults to None. local_files_only (bool, optional): Whether to only use local files. Defaults to False. Returns: Optional[str]: The directory path if it exists, otherwise None. )rrz/**Nr)repo_idrallow_patternsrrrr tqdm_classTrrc)rrr.rrr r) r directoryrrrrdir_pathdownload_kwargs repo_paths r load_dir_pathrs.ww||. :H ww~~h&/8 /IYKs+t/!,# O9%88 77<< 9 -- 9.2*+%88 9s BB"!B"cBtjfd}|S)Nc|jdd}|rd|vrtjd||d<t|dk\rg|ddd|dd}|g|i|S)N repo_namerzfProviding a `repo_name` keyword argument to `save_to_hub` is deprecated, please use `repo_id` instead.rI)poprrr)rrrr funcs rwrapperz+save_to_hub_args_decorator..wrapper s|JJ{D1 &0 NNx !*F9  t9>/T"1X/t/d12h/DD*4*6**r) functoolswraps)r r s` rsave_to_hub_args_decoratorr s%__T + + Nrctjjrktjj rtjj }n)t tjjdd}d|Stjjjrytrytjjdddlmcm}|jryy ) aO Returns the name of the device where this module is running on. This function only supports single device or basic distributed training setups. In distributed mode for cuda device, it uses the rank to assign a specific CUDA device. Returns: str: Device name, like 'cuda:2', 'mps', 'npu', 'hpu', or 'cpu' LOCAL_RANKrzcuda:mpsrhabana_frameworksNhpur)rr is_available distributedis_initializedget_rankrrenvironrbackendsrrrutil find_spechabana_frameworks.torch.hpur) local_rankhthpus rget_device_namer!s zz    + + -**335JRZZ^^L!<=Jzl##    ( ( *  !  ! !"5 6 B33     rcZ t|}|d|k(xr||dvS#t$rYywxYw)zB Checks if a package is available from the correct owner. Namez Home-pageF)rr) package_nameownermetas rcheck_package_availabilityr'9sA %F||+Jk9J0JJ s  **ctddS)zJ Returns True if the Huggingface accelerate library is available. accelerate huggingfacer'rcrris_accelerate_availabler,Ds &lM BBrctddS)zH Returns True if the Huggingface datasets library is available. rr*r+rcrrrrKs &j- @@rc.txr tS)z Returns True if we have the required dependencies for training Sentence Transformers models, i.e. Huggingface datasets and Huggingface accelerate. )r,rrcrris_training_availabler/Rs # $ @)>)@@rc#|Kddlm}m}m}|} |r|d|r|yy#|r|wwxYww)zN A context manager that will disable caching in the datasets library. r)disable_cachingenable_cachingis_caching_enabledN)rr1r2r3)r1r2r3is_originally_enableds rdisable_datasets_cachingr5ZsF ML.0     !   !s< - < 9<)rlist | np.ndarray | Tensorrr)rrrr)rrr+rrr)rr6r+r6rr)rr6r+r6)rQrrRrrr)r`rrr)r` np.ndarrayre int | Nonerr7)r` torch.Tensorrer8rr9)r`np.ndarray | torch.Tensorrer8rr:)rwz list[str]rlr rmrrorrprrqrrrrrs"Callable[[Tensor, Tensor], Tensor]rlist[list[float | int]])r`rrorrprrqrrrrrsr;rr<)r"list[list[dict[str, int | float]]])rrrrrorrprrrrrsr;rr=)NNNNrNNNNNrFrrii@FFTNN)0r/rrvrr0 str | Noner1r?r2zlist[str] | Noner3zCrossEncoder | Nonerrrr8r float | Nonerr@rr@rr@r4rr5zLiteral['random', 'top']r6r r7z=Literal['triplet', 'n-tuple', 'labeled-pair', 'labeled-list']rmrr8rr9r r:zlist[str] | boolr;r r<z bool | Noner=r@rr)rrrrrr)rdict[str, Any]rrrrA)rr)rrrr)rrrF) r`ztorch.Tensor | np.ndarrayrrrrrmrrlr rzlist[list[int]])NNNF) rrrbool | str | Nonerr?rr?rr rr )rrrrrrBrr?rr?rr rr?)rrrrrrBrr?rr?rr rr?)r$rr%rrr )rr )R __future__rrrrrrrrr contextlibrimportlib.metadatarrtypingrrr r r numpyrrrhuggingface_hubr r rrrrtqdm.autonotebook transformersr getLoggerrrrr0sentence_transformers.cross_encoder.CrossEncoderr)sentence_transformers.SentenceTransformerrrr$r&r,r)r5r7r4rArErGrKr[r.rfrxrurrrrrrrrrCRITICALrrrrrr!r'r,rr/r5rcrrrOs"  %=BB > "/   8 $ MM       4&P"*""%"0"%"8"!D A \ \ ` `*F$ #9@*** *  *  ** *7**^!#9@ FFFF F  F 7 FFR, #9@ MMMM M  M 7 M(Mf&*'+#)- ""$($(27#S\!*/#/|  |  | #| % |   |  ' | | | | | "| "| | 0| |  Q!| "#| $%| &'| (()| *+| ,-| . /| 0 1| ~"J"38!L # c)ccc c  c  cVD(")"2"2((* $#"       F $#" ''' ' '  '  ''Z $#" ,.,.,. ,. ,.  ,.  ,.,.^(8CAAr