Triangle104/Llama-3.1-1-million-ctx-DeepHermes-Deep-Reasoning-8B-Q5_K_M-GGUF

This model was converted to GGUF format from DavidAU/Llama-3.1-1-million-ctx-DeepHermes-Deep-Reasoning-8B using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

---

Context : 1,000,000 tokens.

Required: Llama 3 Instruct template.

The Deep Hermes 8B Preview model (reasoning), [ https://huggingface.co/NousResearch/DeepHermes-3-Llama-3-8B-Preview ] converted to 1 million context using Nvidia's Ultra Long 1 million 8B Instruct model.

The goal of this model was to stablize long generation and long context "needle in a haystack" issues.

According to Nvidia there is both a bump in general performance, as well as perfect "recall" over the entire 1 million context.

[ https://huggingface.co/nvidia/Llama-3.1-8B-UltraLong-1M-Instruct ]

Additional changes:

Model appears to be de-censored / more de-censored. Output generation is improved. Creative output generation is vastly improved.

NOTE: Higher temps will result in deeper, richer "thoughts"... and frankly more interesting ones too.

The "thinking/reasoning" tech (for the model at this repo) is from the original Llama 3.1 "DeepHermes" model from NousResearch:

[ https://huggingface.co/NousResearch/DeepHermes-3-Llama-3-8B-Preview ]

---

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo Triangle104/Llama-3.1-1-million-ctx-DeepHermes-Deep-Reasoning-8B-Q5_K_M-GGUF --hf-file llama-3.1-1-million-ctx-deephermes-deep-reasoning-8b-q5_k_m.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo Triangle104/Llama-3.1-1-million-ctx-DeepHermes-Deep-Reasoning-8B-Q5_K_M-GGUF --hf-file llama-3.1-1-million-ctx-deephermes-deep-reasoning-8b-q5_k_m.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo Triangle104/Llama-3.1-1-million-ctx-DeepHermes-Deep-Reasoning-8B-Q5_K_M-GGUF --hf-file llama-3.1-1-million-ctx-deephermes-deep-reasoning-8b-q5_k_m.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo Triangle104/Llama-3.1-1-million-ctx-DeepHermes-Deep-Reasoning-8B-Q5_K_M-GGUF --hf-file llama-3.1-1-million-ctx-deephermes-deep-reasoning-8b-q5_k_m.gguf -c 2048