Q^(h dZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z ddl mZddlmZmZddlmZmZmZmZmZmZmZmZmZmZmZdd lmZd d l m!Z!d d l"m#Z#m$Z$m%Z%m&Z&d d l'm(Z(d dl)m*Z*m+Z+m,Z,d dl-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7d dl-m8Z9d dl:m;Z;erd dl>V #!..77rSexcs r?r@z CommitOperationAdd.__post_init__s243D3DE d**D 1#&t';';#>/2 #3O3DDl!mnnD0023D3De2LM#  d**B,=,= > $$))+$$))!R[[9 d**C 0)33D4H4HID  ,,e 4)44T5I5IJD )66t7K7KLD ^,  d s7AHH>- H99H> with_tqdmc#Kt|jtst|jtrI|r#t |j5}|dddyt |jd5}|dddyt|jt r"tj|jyt|jtjrS|jj}|j|jj|tjyy#1swYyxYw#1swYyxYww)u A context manager that yields a file-like object allowing to read the underlying data behind `path_or_fileobj`. Args: with_tqdm (`bool`, *optional*, defaults to `False`): If True, iterating over the file object will display a progress bar. Only works if the file-like object is a path to a file. Pure bytes and buffers are not supported. Example: ```python >>> operation = CommitOperationAdd( ... path_in_repo="remote/dir/weights.h5", ... path_or_fileobj="./local/weights.h5", ... ) CommitOperationAdd(path_in_repo='remote/dir/weights.h5', path_or_fileobj='./local/weights.h5') >>> with operation.as_file() as file: ... content = file.read() >>> with operation.as_file(with_tqdm=True) as file: ... while True: ... data = file.read(1024) ... if not data: ... break config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s] >>> with operation.as_file(with_tqdm=True) as file: ... requests.put(..., data=file) config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s] ``` Nrb) r:rSrFrr(openrfrdBytesIOrergrhSEEK_SET)r>rpfileprev_poss r?as_filezCommitOperationAdd.as_filesH d**C 0Jt?S?SUY4Z%d&:&:;tJ$..5J ,,e 4**T112 2 ,,b.?.? @++002H&& &  % %h <As7A E D4E 2E7B=E 4D=9E E E c|j5}tj|jcdddS#1swYyxYw)z[ The base64-encoded content of `path_or_fileobj` Returns: `bytes` N)rxbase64 b64encodereadr>rvs r? b64contentzCommitOperationAdd.b64contents8 \\^ 1t##DIIK0 1 1 1s #>Ac|jy|jdk(r$|jjjS|j 5}t j |jcdddS#1swYyxYw)aReturn the OID of the local file. This OID is then compared to `self._remote_oid` to check if the file has changed compared to the remote one. If the file did not change, we won't upload it again to prevent empty commits. For LFS files, the OID corresponds to the SHA256 of the file content (used a LFS ref). For regular files, the OID corresponds to the SHA1 of the file content. Note: this is slightly different to git OID computation since the oid of an LFS file is usually the git-SHA1 of the pointer file content (not the actual file content). However, using the SHA256 is enough to detect changes and more convenient client-side. Nr.)rXrVsha256hexrxr'git_hashr|r}s r? _local_oidzCommitOperationAdd._local_oid sp    $   % '##**..0 0 14||DIIK0 1 1 1s #A>>Br]N)F)rBrCrDrErFrGrrrfr rrVrrXr UploadModerYr;rZr[r\r@rrrxr~propertyrrHrAr?rRrR|s.3eX566#U;K; */Et)TL(:&T&+E4%PNHTNP"'Et!LK#LEuEL$E UFM4F!MF/=/=(82D/=/=b1E11HSM11rArRr3r]c4|jdr|dd}|dk(s|dk(s|jdrtd|d|jdr|d d}tD]7tfd |j dDs'td d |d |S)Nr7r.z..z../z,Invalid `path_in_repo` in CommitOperation: ''z./c3(K|] }|k( ywrPrH).0part forbiddens r? z)_validate_path_in_repo..+sETty EszHInvalid `path_in_repo` in CommitOperation: cannot update files under a 'z/' folder (path: 'z').) startswithr<r anysplit)r3rs @r?r8r8"ss##AB' sld2l6M6Me6TG ~UVWXXt$#AB' & E\-?-?-DE EZ[dZef!N#'  rA operationsctt}|D]}|j}t|tr^||dkDrt j d|d||xxdz cc<t|jD]}|t|xxdz cc<t|ts|tt|dkDs|jrt j d|dt j d|dy)a Warn user when a list of operations is expected to overwrite itself in a single commit. Rules: - If a filepath is updated by multiple `CommitOperationAdd` operations, a warning message is triggered. - If a filepath is updated at least once by a `CommitOperationAdd` and then deleted by a `CommitOperationDelete`, a warning is triggered. - If a `CommitOperationDelete` deletes a filepath that is then updated by a `CommitOperationAdd`, no warning is triggered. This is usually useless (no need to delete before upload) but can happen if a user deletes an entire folder and then add new files to it. rzBAbout to update multiple times the same file in the same commit: 'z9'. This can cause undesired inconsistencies in your repo.rz_About to delete a folder containing files that have just been updated within the same commit: 'zLAbout to delete a file that have just been updated within the same commit: 'N) rintr3r:rRwarningswarnr parentsrFr2r5)rnb_additions_per_path operationr3parents r?_warn_on_overwriting_operationsr6s-8,<  -- i!3 4$\2Q6 %'"" ", /1 4 /' 5== 8&c&k2a72 8 i!6 7$S|)D%EFJ&&MM==INKII MM**6899/rA)endpoint num_threadsrevision additions repo_typerepo_idheadersrrrc ng}t|tD]}t|D cgc]} | jc} |||d\} } | radj | D cgc]8} d| j dd| j dij d :c} } t d | || z }|Dcic]'}|jjj|)c}g}|D]N}|j d *tjd |djd >|j|Pt|dk(rtjdydfd }tjr=tjdt|dt!|dD] }|| yt|dk(r!tjd||dytjdt|d|dt#||dt|d|t ycc} wcc} wcc}w)a2 Uploads the content of `additions` to the Hub using the large file storage protocol. Relevant external documentation: - LFS Batch API: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: additions (`List` of `CommitOperationAdd`): The files to be uploaded repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. headers (`Dict[str, str]`): Headers to use for the request, including authorization headers and user agent. num_threads (`int`, *optional*): The number of concurrent threads to use when uploading. Defaults to 5. revision (`str`, *optional*): The git revision to upload to. Raises: [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError) If an upload failed for any reason [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the server returns malformed responses [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the LFS batch endpoint returned an HTTP error.  chunk_sizeN) upload_infosrrrrrtoken z$Encountered error for file with OID oidz: `errormessagez$LFS batch endpoint returned errors: actionszContent of file z/ is already present upstream - skipping upload.rzNo LFS files to upload.c |d}t||y#t$r}tdjd|d}~wwxYw)Nr)rlfs_batch_actionrrzError while uploading 'z ' to the Hub.)r Exception RuntimeErrorr3) batch_actionrrorr oid2addops r?_wrapped_lfs_uploadz._upload_lfs_files.._wrapped_lfs_uploadsZ i!,u"56I \SZem n i!89O9O8PP]^_eh h is A>Az Uploading z* LFS files to the Hub using `hf_transfer`.zhuggingface_hub.lfs_upload)namerzUploading 1 LFS file to the Hubz" LFS files to the Hub using up to z threads concurrentlyzUpload z LFS files)desc max_workers tqdm_classr)r"UPLOAD_BATCH_MAX_NUM_FILESrrVjoingetr<rrloggerdebugr3appendlenrHF_HUB_ENABLE_HF_TRANSFERhf_tqdmr)rrrrrrr batch_actionschunkopbatch_actions_chunkbatch_errors_chunkerrradd_opfiltered_actionsactionrrs `` @r?_upload_lfs_filesrds^V!#M 6PQ-2E389R"..93 // ii 2;3775>:J#cggV]_aNbNfNfgpNqMrsG DWINO O,, +-,HQQV##**..0&8QI, ::i ( LL"9VE]#;#H#H"IJ77   # #F +, ! ./i** z#&6"7!88bcd.5QR (F  ' (  ! # 67,Q/0 -.//QR]Q^^s t    3/01<#  q:RsH( =H- 5,H2)rr create_prrc Vt|dk(ryddlm}m} t t j rddind} | j} | j| jf} d ttt fffd } t#j$t|t&z }t!t#j(|d z}t+t-|t&D]Z\}}|Dcgc]}|}}|Dcgc]}t/|j0t2s|!}}|Dcgc]%}t/|j0tt4fs$|'}}t7d||zD}|d kDr%dt|d zj9|d |d}nd}t;||ddddt<j?}|5dt ffd }t|dkDr.||Dcgc]}t|j0c}| | | |t|dkDr%||Dcgc]}|j0c}| | | |ddd]y#t$r2} | jjdk(rtdd d | d} ~ wwxYwcc}wcc}wcc}wcc}wcc}w#1swYxYw)aQ Uploads the content of `additions` to the Hub using the xet storage protocol. This chunks the files and deduplicates the chunks before uploading them to xetcas storage. Args: additions (`List` of `CommitOperationAdd`): The files to be uploaded. repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. headers (`Dict[str, str]`): Headers to use for the request, including authorization headers and user agent. endpoint: (`str`, *optional*): The endpoint to use for the xetcas service. Defaults to `constants.ENDPOINT`. revision (`str`, *optional*): The git revision to upload to. create_pr (`bool`, *optional*): Whether or not to create a Pull Request with that commit. Raises: [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError) If an upload failed for any reason. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the server returns malformed responses or if the user is unauthorized to upload to xet storage. [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the LFS batch endpoint returned an HTTP error. **How it works:** The file download system uses Xet storage, which is a content-addressable storage system that breaks files into chunks for efficient storage and transfer. `hf_xet.upload_files` manages uploading files by: - Taking a list of file paths to upload - Breaking files into smaller chunks for efficient storage - Avoiding duplicate storage by recognizing identical chunks across files - Connecting to a storage server (CAS server) that manages these chunks The upload process works like this: 1. Create a local folder at ~/.cache/huggingface/xet/chunk-cache to store file chunks for reuse. 2. Process files in parallel (up to 8 files at once): 2.1. Read the file content. 2.2. Split the file content into smaller chunks based on content patterns: each chunk gets a unique ID based on what's in it. 2.3. For each chunk: - Check if it already exists in storage. - Skip uploading chunks that already exist. 2.4. Group chunks into larger blocks for efficient transfer. 2.5. Upload these blocks to the storage server. 2.6. Create and upload information about how the file is structured. 3. Return reference files that contain information about the uploaded files, which can be used later to download them. rN) upload_bytes upload_filesr1 token_typerrrrrparamsiz2You are unauthorized to upload to xet storage for r7zX. Please check that you have configured your access token with write access to the repo.r]c ttjrddind}| td|j|j fS)NrrrzFailed to refresh xet token)r#r!WRITEr access_tokenexpiration_unix_epoch)new_xet_connectionrrrrrrs r?token_refresherz*_upload_xet_files..token_refresher,s^E#)))2K%   %&'DE E!..0B0X0XXXrArrc3HK|]}|jjywrP)rVsize)rrs r?rz$_upload_xet_files..AsPBBNN//Ps "zUploading Batch [z]...z Uploading...BTzhuggingface_hub.xet_put)rtotalinitialunit unit_scaler log_level incrementc(j|yrP)update)rprogresss r?update_progressz*_upload_xet_files..update_progressRs *rA) rhf_xetrrr#r!rrresponse status_coderrrrrrFrmathceilrlog10 enumerater"r:rSrfrsumzfillr+rgetEffectiveLevel)rrrrrrrrrxet_connection_infoe xet_endpointaccess_token_infor num_chunksnum_chunks_num_digitsirr_chunk bytes_ops paths_ops expected_size description progress_cmrrs `````` @r?_upload_xet_filesrs~ 9~1F#)))2K% "'//L,99;N;d;de YU38_ Y Y3y>,FFGJ : 67!;nYC]^_*5$%"%%"(RBJr7I7I5,QRR R"(XBJr7I7ICQU;,WRX XP)i:OPP >-c!a%j.>.>?T.U-VVWXbWccghK(K/*..0  H +3 +9~!7@AS++,A %## 9~!2;>FG GH rAgitignore_contentc ||ntj}i}i} i} t|dD]P} d| D cgc]\} | jt j | j jjd| j jd^c} i} ||| d<tj|d|d|d || ||rd d ind }t|t|j}|jdi|dDcic] }|d |d c}| jdi|dDcic] }|d |d c}| jdi|dDcic]}|d |j!dc}S|D]>}||j|_| |j|_| |j|_@|D]#}|j jdk(sd|_%ycc} wcc}wcc}wcc}w)a Requests the Hub "preupload" endpoint to determine whether each input file should be uploaded as a regular git blob, as a git LFS blob, or as a XET file. Input `additions` are mutated in-place with the upload mode. Args: additions (`Iterable` of :class:`CommitOperationAdd`): Iterable of :class:`CommitOperationAdd` describing the files to upload to the Hub. repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. headers (`Dict[str, str]`): Headers to use for the request, including authorization headers and user agent. revision (`str`): The git revision to upload the files to. Can be any valid git revision. gitignore_content (`str`, *optional*): The content of the `.gitignore` file to know which files should be ignored. The order of priority is to first check if `gitignore_content` is passed, then check if the `.gitignore` file is present in the list of files to commit and finally default to the `.gitignore` file already hosted on the Hub (if any). Raises: [`~utils.HfHubHTTPError`] If the Hub API returned an error. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the Hub API response is improperly formatted. Nr0rascii)r`sampler gitIgnorez/api/zs/z /preupload/rr)jsonrrr`r shouldIgnorerrr/rH)rENDPOINTr"r3rzr{rVr decoderr$postr%rr rrrXrYrZ)rrrrrrrr upload_modesshould_ignore_infooid_inforrpayloadresprrvadditions r?_fetch_upload_modesrys/N$/xY5G5GH+-L*,)+H 3/_     OO$..r~~/D/DELLWUNN//    (#4GK }!!ji[7);xj I)2K% "  D!1$))+> eN[bLcdDtF|T,-??de!!!mTbcjTk$lDT&\43G%G$lm^^T[E\]T4<%8]^1_6? ,X-B-B C"4X5J5J"K'(=(=>?.    $ $ )$-H !.A(e$l]sA!G'G,-G1G6copiesr,c ddlm}m}|||}i} i} |D cgc]} | j} } t dt | t D]P} |j|| | | t z||}|D]*}t||r|j| |j|f<,Rt|dD]\}}t|}|D cgc]} | j}} t dt |t D]} |j||| | t z|xs||}|D]}t||r td|j| |j|f<|jr|| |j|f<St!||||xs||j }t#j%|| }t'||j(| |j|f<|D]}|j|f| vr t+d |jd |xs|d | j%|j|j,f|_| j%|j|f|_| Scc} wcc} w)a; Fetch information about the files to copy. For LFS files, we only need their metadata (file size and sha256) while for regular files we need to download the raw content from the Hub. Args: copies (`Iterable` of :class:`CommitOperationCopy`): Iterable of :class:`CommitOperationCopy` describing the files to copy on the Hub. repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. headers (`Dict[str, str]`): Headers to use for the request, including authorization headers and user agent. revision (`str`): The git revision to upload the files to. Can be any valid git revision. Returns: `Dict[Tuple[str, Optional[str]], Union[RepoFile, bytes]]]` Key is the file path and revision of the file to copy. Value is the raw content as bytes (for regular files) or the file information as a RepoFile (for LFS files). Raises: [`~utils.HfHubHTTPError`] If the Hub API returned an error. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the Hub API response is improperly formatted. r)HfApi RepoFolder)rrr)rpathsrrc|jSrP)rL)rs r?z&_fetch_files_to_copy.. s 2??rA)keyz$Copying a folder is not implemented.)rrrrfilename)rz Cannot copy z at revision z: file is missing on repo.)hf_apirrr3rangerFETCH_LFS_BATCH_SIZEget_paths_infor:blob_idr`rrrKNotImplementedErrorr.rr$rr%contentrrLrMrN)rrrrrrrrr" files_to_copyrr dest_pathsoffsetdest_repo_filesrvrLr src_pathssrc_repo_files src_repo_fileurlrrs r?_fetch_files_to_copyr1sN* Hg 6FOQM?AH,23b"//3J33z?,@A ? //Vf/C&CD 0  $ ?DdJ/26,,$))X./ ? ?%,F8R$S'S j*% 3=>RR((> >As9~/CD YF#222F)FG%1# 3N"0 Y mZ8-.TUU?L?T?T-,,l;< $$HUM=#5#5|"DE%!)"+ '!-!9!.!3!3 C +}00g0FH'1HPHXHXM=#5#5|"DE% Y Y:$ SI**L9N("9#=#=">m#/x00JL"*y/I/I9KaKa.b!cI "*,, 0F0F/Q"RI  SA'SP k4?s IIr)commit_messagecommit_description parent_commitc #K||nd}||d}|||d<d|dd}|D]}t|tr5|jr)tj d|j d |d z }It|trA|j d k(r2d |jj|j d ddt|tr]|j dk(rNd|j d|jjj|jjddt|tr#|jrdndd|j id:t|tr||j |j"f}t|t$r8d t'j(|j|j d dd|j*r+d|j d|j*jddt-dt-d|dt/|dd|dkDrtj1d|dyyw)aG Builds the payload to POST to the `/commit` API of the Hub. Payload is returned as an iterator so that it can be streamed as a ndjson in the POST request. For more information, see: - https://github.com/huggingface/huggingface_hub/issues/1085#issuecomment-1265208073 - http://ndjson.org/ N)summaryr parentCommitheader)r valuerzSkipping file 'z(' in commit (ignored by gitignore file).rr/rvrz)r(r`encodingr.lfsFiler)r`algorr deletedFolder deletedFiler`)r`r=rz_Malformed files_to_copy (should be raw file content as bytes or RepoFile objects with LFS info.z(Unknown operation to commit. Operation: z. Upload mode: rXzSkipped z/ file(s) in commit (ignored by gitignore file).)r:rRrYrrr3rXr~rrVrrrr2r5rJrKrLrfrzr{r.r<getattrinfo) rr)r2r3r4 header_valuenb_ignored_filesr file_to_copys r?_prepare_commit_payloadrE4s"0B/M+SU .>PQL '4 ^$\ 22 @ i!3 49Q9Q LL?9+A+A*BBjk l  !   i!3 49O9OS\9\(335<<>%22 (  #5 69;Q;QUZ;Z %22$$0077;;=%1166   #8 9*3*=*== )"8"89   #6 7()*D*DiF\F\)]^L,.!#)#3#3L#A#H#H#J ) 6 6$,!!$ ) 6 6 (+//66!u :9+FI~t<=? {@D! h/00_`asI I )NFNrP)NN)UrErzrdrr_r collectionsr contextlibr dataclassesrr itertoolsrpathlibrr typingr r r r rrrrrrrtqdm.contrib.concurrentrr6rerrorsrrrr file_downloadrr.rrrutilsr r!r"r#r$r%r&r'r(r)r*r utils.tqdmr+r"r, get_loggerrBrrr$rr2rJrRrFr8CommitOperationrrrr;rrrrrfr1rErHrAr?rSs #%('pppp.cc%<<   #1    H %% &    <  F F  FF b1b1 b1J"*,?AVVW+_0E+$+\#"k &'k k  k #s(^ k sm k k smk k \#" $S &'S S  S #s(^ S sm S smS ~S S l T #'+Q.*+Q.Q.Q.#s(^ Q.  Q. sm Q.Q. }Q. Q.Q.h# b ( )bbb#s(^ b  b sm b %Xc]" #U:u+<%= =>bbR)-#' _b)_bc8C=015U9J3KKL_b_b! _b C= _b d38n _brA