l^(ho&ddlmZddlZddlZddlmZddlZddlZddl m Z ddl m Z ddlmZddlmZddlmZdd lmZGd d ej*Zy) ) annotationsN)Path) load_file) save_file) Tokenizer)nn)PreTrainedTokenizerFast)get_device_nameceZdZ d d fd Zd dZddZddZeddZddZ dddZ ddZ e d dd Z e dd ZxZS)StaticEmbeddingc t|t|tr |j}nt|t s t d|Vt|tjrtj|}tjj|d|_n7|*tj|j||_n t d|jj |_|jj"|_||_|j$j'|j)dd|_y)a Initializes the StaticEmbedding model given a tokenizer. The model is a simple embedding bag model that takes the mean of trained per-token embeddings to compute text embeddings. Args: tokenizer (Tokenizer | PreTrainedTokenizerFast): The tokenizer to be used. Must be a fast tokenizer from ``transformers`` or ``tokenizers``. embedding_weights (np.ndarray | torch.Tensor | None, optional): Pre-trained embedding weights. Defaults to None. embedding_dim (int | None, optional): Dimension of the embeddings. Required if embedding_weights is not provided. Defaults to None. Example:: from sentence_transformers import SentenceTransformer from sentence_transformers.models import StaticEmbedding from tokenizers import Tokenizer # Pre-distilled embeddings: static_embedding = StaticEmbedding.from_model2vec("minishlab/potion-base-8M") # or distill your own embeddings: static_embedding = StaticEmbedding.from_distillation("BAAI/bge-base-en-v1.5", device="cuda") # or start with randomized embeddings: tokenizer = Tokenizer.from_pretrained("FacebookAI/xlm-roberta-base") static_embedding = StaticEmbedding(tokenizer, embedding_dim=512) model = SentenceTransformer(modules=[static_embedding]) embeddings = model.encode(["What are Pandas?", "The giant panda, also known as the panda bear or simply the panda, is a bear native to south central China."]) similarity = model.similarity(embeddings[0], embeddings[1]) # tensor([[0.8093]]) (If you use potion-base-8M) # tensor([[0.6234]]) (If you use the distillation method) # tensor([[-0.0693]]) (For example, if you use randomized embeddings) Raises: ValueError: If the tokenizer is not a fast tokenizer. ValueError: If neither `embedding_weights` nor `embedding_dim` is provided. zThe tokenizer must be fast (i.e. Rust-backed) to use this class. Use Tokenizer.from_pretrained() from `tokenizers` to load a fast tokenizer.NF)freezez?Either `embedding_weights` or `embedding_dim` must be provided. base_model)super__init__ isinstancer _tokenizerr ValueErrornpndarraytorch from_numpyr EmbeddingBagfrom_pretrained embeddingget_vocab_sizenum_embeddings embedding_dim tokenizer no_paddinggetr)selfrembedding_weightsrkwargs __class__s j/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/sentence_transformers/models/StaticEmbedding.pyrzStaticEmbedding.__init__sZ  i!8 9!,,IIy1^   (+RZZ8$)$4$45F$G!__<<=NW\<]DN  &__Y-E-E-GWDN^_ _"nn;;!^^99$- !!#!**\48c |jj|d}|Dcgc]}|j}}tjt j dg|ddDcgc] }t|c}z}tj|Dcgc] }|D]}| c}}tj} | |dScc}wcc}wcc}}w)NF)add_special_tokensr)dtype) input_idsoffsets) r encode_batchidsrrrcumsumlentensorlong) r"textsr$ encodingsencoding encodings_ids token_idsr-token_idr,s r&tokenizezStaticEmbedding.tokenize]sNN//%/P 6?@(@ @""299aSTabeceTf3gyC N3g-g#hiLLM!dyZc!dh(!d(!dlqlvlvw &7;; A3g!dsB>CC c <|j|d|d|d<|S)Nr,r-sentence_embedding)r)r"featuresr$s r&forwardzStaticEmbedding.forwardes()-8MxXaOb)c%&r'ciSNr"s r&get_config_dictzStaticEmbedding.get_config_dictis r'c"tjSr@)mathinfrBs r&max_seq_lengthzStaticEmbedding.max_seq_lengthls xxr'c|jSr@)rrBs r& get_sentence_embedding_dimensionz0StaticEmbedding.get_sentence_embedding_dimensionps!!!r'c ^|r9t|jtjj |dnBt j |jtjj |d|jj tt|dz y)Nmodel.safetensorspytorch_model.bintokenizer.json) save_safetensors_file state_dictospathjoinrsaverstrr)r"save_dirsafe_serializationr$s r&rSzStaticEmbedding.savessl  !$//"3RWW\\(L_5` a JJt("'',,xAT*U V CX1A ABCr'c tjtt|dz }tj j tj j|dr*ttj j|d}nItjtj j|dtjdd} |d}t|| S#t$r|d}YwxYw) NrMrKrLcpuT) map_location weights_onlyzembedding.weight embeddings)r#)r from_filerTrrPrQexistsrRload_safetensors_filerloaddeviceKeyErrorr )load_dirr$rweightss r&r_zStaticEmbedding.loadzs''DN=M,M(NO 77>>"'',,x1DE F+BGGLLCV,WXGjj X':;%,,W\J]lpG ,01GyGDD ,l+G ,sC..C?>C?cR ddlm}t}|||||||}t |j t jr tj|j } n|j j} |j} || | |S#t$r tdwxYw)a Creates a StaticEmbedding instance from a distillation process using the `model2vec` package. Args: model_name (str): The name of the model to distill. vocabulary (list[str] | None, optional): A list of vocabulary words to use. Defaults to None. device (str): The device to run the distillation on (e.g., 'cpu', 'cuda'). If not specified, the strongest device is automatically detected. Defaults to None. pca_dims (int | None, optional): The number of dimensions for PCA reduction. Defaults to 256. apply_zipf (bool): Whether to apply Zipf's law during distillation. Defaults to True. use_subword (bool): Whether to use subword tokenization. Defaults to True. Returns: StaticEmbedding: An instance of StaticEmbedding initialized with the distilled model's tokenizer and embedding weights. Raises: ImportError: If the `model2vec` package is not installed. r)distillz\To use this method, please install the `model2vec` package: `pip install model2vec[distill]`) vocabularyr`pca_dims apply_zipf use_subwordr#r) model2vec.distillre ImportErrorr rrrrrrweightr) cls model_namerfr`rgrhrire static_modelr#rs r&from_distillationz!StaticEmbedding.from_distillations<  1 !" !!#   l,,bjj 9 % 0 01G1G H  , 6 6 = = +55 90AjYY) n  s BB&cD ddlm}|j|}t |j t jr tj|j }n|j j}|j}||||S#t$r tdwxYw)aH Create a StaticEmbedding instance from a model2vec model. This method loads a pre-trained model2vec model and extracts the embedding weights and tokenizer to create a StaticEmbedding instance. Args: model_id_or_path (str): The identifier or path to the pre-trained model2vec model. Returns: StaticEmbedding: An instance of StaticEmbedding initialized with the tokenizer and embedding weights the model2vec model. Raises: ImportError: If the `model2vec` package is not installed. r) StaticModelzSTo use this method, please install the `model2vec` package: `pip install model2vec`rj) model2vecrsrlrrrrrrrrmr)rnmodel_id_or_pathrsrpr#rs r&from_model2veczStaticEmbedding.from_model2vecs" u -#223CD l,,bjj 9 % 0 01G1G H  , 6 6 = = +55 90AN^__ ust t us B B)NN)rz#Tokenizer | PreTrainedTokenizerFastr#z np.ndarray | torch.Tensor | Noner int | NonereturnNone)r4z list[str]rxdict[str, torch.Tensor])r=rzrxrz)rxzdict[str, float])rxint)T)rUrTrVboolrxry)rbrTrxr )NNTT)rorTrfzlist[str] | Noner`z str | Nonergrwrhr|rir|rxr )rurTrxr )__name__ __module__ __qualname__rr:r>rCpropertyrGrIrSr_ classmethodrqrv __classcell__)r%s@r&r r s?C$( H96H9<H9" H9  H9T<"D E(,!" 3Z3Z%3Z 3Z  3Z  3Z3Z 3Z3Zj``r'r ) __future__rrErPpathlibrnumpyrrsafetensors.torchrr^rrN tokenizersrr transformersr sentence_transformers.utilr Moduler rAr'r&rs="  @@ 06J`biiJ`r'