Spaces:

---

Duplicated from  huggingface-projects/repo_duplicator

John6666

/

repo_duplicator_mod

like 8

Running

This program is not suitable for downloading single files. Also, the original duplication function uses a hidden function on the HF side, so it doesn't matter what the file size is, but the download function is done on its own, so it can't handle files that are too large.
However, if it's any other error, it's an unknown bug, so I'd like to know the details. It would be easier if I knew which repo you were trying to duplicate.
As for single file downloads, I think it's possible to add them, as long as it's not a function that's impossible due to CPU space performance.

for example im trying to duplicate sulph/illustriousMerges it has 14 models but it only copies 7 models, half, then it gives an error, im trying to overwrite or update the whole folder but it hangs

Thank you. I'll try to verify it. If the error actually appears, there are many cases where the rest can be dealt with.

Edit:
If you want to copy the whole repo, the official HF one is faster. My version still works, but it has some quirks.
https://huggingface.co/spaces/huggingface-projects/repo_duplicator

I fixed the bug. It seems to have been fixed after changing the path where the download files are stored.🤔

Edit:
I've modified it so that you can also specify files.
I don't think there are many places where you would use it though...

its working now, thank you very much, i will continue backing up safetensors models and loras with these tools, thank yo so much for the tools

hi john, repo gives Failed to retrieve error logs: SSE is not enabled

hi. Thank you. I'm not sure what caused the error, but I think I've fixed it for now...

Hi John,I've noticed two things with the repo duplicator:

(i think this only happens with profiles or accounts with only 1 folder without multiple links of folders,nvm happens with all profiles no matter what, i think its just an old dup method vs new dup method).
It creates a folder with the username inside the already existing model folder (which already had its subfolders organized). This results in duplicate folders, making it impossible to update directly and forcing the user to either move everything manually or end up with a second folder.

Would it be possible to add an option or a button to control this behavior? For example, a button to update already existing folders, and if it's not pressed, it defaults to creating the user-named folder inside the existing one? Or is there another way this could be fixed?

I think that's definitely possible, and since it's a broad definition of a bug, I'd like to fix it, but it would be helpful if you could provide the original report and a concrete example of what you expect the final result to look like (it doesn't have to be the actual thing).😆

By the way, currently, if you specify John6666/foo/bar, it's supposed to upload to the bar folder inside John6666/foo (if there are no bugs).

Never mind, I kind of figured it out. I was using all the options for everything except some folders—it was the 'Create Subfolders' option, i thinks its just a confusion as one to one includes the create subdirectories option and multi to one no. I'm just updating some duplicated folders, but overwriting takes time and can't be stopped. I'm using only the 'Remove NFAA' tag without Create Subfolders option with that folder it's working again, but I'm not sure if it's actually verifying and just updating or overwriting and updating.

Here:

Cant update without force overwriting or it skipes files?, can we add an update repo button?. Multi to one creates a minaiosu/foo/foo/

DEFINITELY BUG.😆 Ill see...

Yeah multi to one should not create minaiosu/foo/foo/ or a new folder with the same dup name folder, but i think i and many duplicated profiles like that, i think it creates subdirectores automatically.
Also can't one to one without just uploading the subdirectory file directly to the main folder, the only option to prevent it is to create subdirectories, so its kinda buggy,it should create the subdirectory folder to the main dup profile instead just the file

I made test version for now. Try testing... If there are no serious bugs, I will update here.
https://huggingface.co/spaces/John6666/repo_duplicator_mod_dupdeltest

testing:
1.one to one failed,
-selected foldername and imported the file instead of the folder.
-selected create subdirectory created minaiosu/foo/foo/foldername/file

2.multi to one failed,
-selected avoid creating duplicated directory and created minaiosu/foo/foo/foldername/file
-avoid creating duplicated directory must be implemented to one to one too

3.Option Stop works pretty well

1
2

Oh no, I deleted the wrong duplicate... I think I fixed it.🥶

testing:
1.one to one, nothing changed, still importing the file directly to folder, instead of creating a folder with the files or file
2.multi to one, nothing changed, now it puts the file outside the duplicated folder inside the duplicated name folder

There was a critical error where subdirectories in the destination folder were being ignored, so I fixed it. This may have been the cause...
Also, something was acting strangely, so I disabled the duplicate deletion feature for now.
I added debug messages (will remove later).
https://huggingface.co/spaces/John6666/repo_duplicator_mod_dupdeltest

with some repos i get this error: "Error occured: 'NoneType' object has no attribute 'sha256'"

Hmm... Maybe fixed.

thank you very much, working pretty well now 😄