l^(hddlmZddlZddlmZmZddlmZmZddl Z eje Z eGddZ y)) annotationsN) dataclassfield)AnyCallablecjeZdZUdZded<edZded<eedd Zd ed <dd Z dd Z y)SentenceTransformerDataCollatoraCollator for a SentenceTransformers model. This encodes the text columns to {column}_input_ids and {column}_attention_mask columns. This works with the two text dataset that is used as the example in the training overview: https://www.sbert.net/docs/sentence_transformer/training_overview.html It is important that the columns are in the expected order. For example, if your dataset has columns "answer", "question" in that order, then the MultipleNegativesRankingLoss will consider "answer" as the anchor and "question" as the positive, and it will (unexpectedly) optimize for "given the answer, what is the question?". r tokenize_fnc ddgS)Nlabelscorera/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/sentence_transformers/data_collator.pyz(SentenceTransformerDataCollator.s GWCUr)default_factory list[str]valid_label_columnsF)rinitreprzset[tuple[str]]_warned_columnsct|dj}i}d|vr|jd|dd|d<t||jvr|j ||j D]B}||vstj|Dcgc]}|| c}|d<|j|n|D]}|jdrK|dtd |vr:tj|Dcgc]}|| c}tj||<_|j|Dcgc]}|| c}}|jD]\}} | ||d|<|Scc}wcc}wcc}w)Nr dataset_namer _prompt_length)dtype_)listkeysremovetuplermaybe_warn_about_column_orderrtorchtensorendswithlenintr items) selffeatures column_namesbatch label_columnrow column_name tokenizedkeyvalues r__call__z(SentenceTransformerDataCollator.__call__sHQK,,./  \ )    /$,QK$?E. !  d&:&: :  . .| <!44 L|+!&H.USsr?zColumn z is at index z?, whereas a column with this name is usually expected at index a?. Note that the column order can be important for some losses, e.g. MultipleNegativesRankingLoss will always consider the first column as the anchor and the second as the positive, regardless of the dataset column names. Consider renaming the columns to match the expected order, e.g.: dataset = dataset.select_columns()N)r'indexloggerwarningraddr )r(r*column_name_to_expected_idxr. expected_idxproposed_fix_columnss rr!z=SentenceTransformerDataCollator.maybe_warn_about_column_order=s ' #*E)J)J)L  %Kl*|/A/A+/NR^/^"DD+M( $::,6+A( $99,3Z+@( $QQ+X(k_M,:L:L[:Y9Z[LLX>Z89M7MQ P) ,   |!45rN)r)zlist[dict[str, Any]]returnzdict[str, torch.Tensor])r*rrHNone) __name__ __module__ __qualname____doc____annotations__rrsetrr2r!rrrr r s? %*;U%VV',SuSX'YO_Y@%6rr ) __future__rlogging dataclassesrrtypingrrr" getLoggerrJrBr rrrrUsB"(   8 $ U6U6 U6r