S^(h<dZddlZddlmZmZddlZddlmZddl m Z ddl m Z m Z dZe jeZd ed ed ej(fd ZGd dZGddZy)z%REALM Retriever model implementation.N)OptionalUnion)hf_hub_download) AutoTokenizer)logging strtoboolzblock_records.npyblock_records_pathnum_block_recordsreturncddlmcm}|jj |d}|j |d}t |jdj}|S)Nri ) buffer_sizeT)drop_remainder) tensorflow.compat.v1compatv1dataTFRecordDatasetbatchnexttakeas_numpy_iterator)r r tfblocks_dataset np_records r/var/www/skyplay_api_hub/venv/lib/python3.12/site-packages/transformers/models/deprecated/realm/retrieval_realm.pyconvert_tfrecord_to_npr!s]%%WW,,-?M^,_N#))*;D)QN^((+==?@I c&eZdZdZ ddZdZy) ScaNNSearcherztNote that ScaNNSearcher cannot currently be used within the model. In future versions, it might however be included.cddlm}|||d}|j|||}|j|}|j |_y)zBuild scann searcher.r)builder dot_product)db num_neighborsdistance_measure) num_leavesnum_leaves_to_searchtraining_sample_size)dimensions_per_blockN)#scann.scann_ops.py.scann_ops_pybindr#treescore_ahbuildsearcher) selfr%r&r+r(r)r*Builderr#s r__init__zScaNNSearcher.__init__.sU KR}}],,!8Lcw ""8L"M  rc|jj|jj\}}|j dS)Nint64)r0search_batcheddetachcpuastype)r1question_projectionretrieved_block_ids_s rr6zScaNNSearcher.search_batchedCs@!%!=!=>Q>X>X>Z>^>^>`!aQ"))'22rN)idi)__name__ __module__ __qualname____doc__r3r6rrr!r!+s~  #(*3rr!cpeZdZdZfdZddZedeee e jffdZ dZ dZxZS) RealmRetrieverahThe retriever of REALM outputting the retrieved evidence block and whether the block has answers as well as answer positions." Parameters: block_records (`np.ndarray`): A numpy array which cantains evidence texts. tokenizer ([`RealmTokenizer`]): The tokenizer to encode retrieved texts. c>t|||_||_yN)superr3 block_records tokenizer)r1rIrJ __class__s rr3zRealmRetriever.__init__Ss *"rctj|j|d}|jj |dd}g}g} |D]2} |j || j | j 4|j|| ddd|} | j |} ||j| || fzSddd| fS)Nr)indicesaxisT)skip_special_tokens)padding truncationreturn_special_tokens_mask max_length)nprrIrJdecodeappendconvert_to_tensorsblock_has_answer) r1r;question_input_ids answer_idsrSreturn_tensorsretrieved_blocksquestiontext text_pairretrieved_block concat_inputsconcat_inputs_tensorss r__call__zRealmRetriever.__call__Xs774#5#5?RYZ[>>(();A)>TX(Y / 7O KK !   _335 6 7 )TdW[hr' !. @ @ P  !(( CG\F^^ ^$&;< ?f$ F"  # Iw..r)Npt)r?r@rArBr3rc classmethodrrstrrkPathLikertrzrX __classcell__)rKs@rrErEHsM# =*-HU3PRP[P[K[E\<]--*7 #/rrE)rBrktypingrrnumpyrThuggingface_hubrrutilsrr ro get_loggerr?loggerrintndarrayrr!rErCrrrsl, "+(!4   H %ssrzz33:d/d/r