Mistral-Large-Instruct-2407-GGUF

Instructions to use lmstudio-community/Mistral-Large-Instruct-2407-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use lmstudio-community/Mistral-Large-Instruct-2407-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Use Docker

docker model run hf.co/lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use lmstudio-community/Mistral-Large-Instruct-2407-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "lmstudio-community/Mistral-Large-Instruct-2407-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "lmstudio-community/Mistral-Large-Instruct-2407-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Ollama
How to use lmstudio-community/Mistral-Large-Instruct-2407-GGUF with Ollama:
```
ollama run hf.co/lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use lmstudio-community/Mistral-Large-Instruct-2407-GGUF with Docker Model Runner:
```
docker model run hf.co/lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M
```

Lemonade

How to use lmstudio-community/Mistral-Large-Instruct-2407-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull lmstudio-community/Mistral-Large-Instruct-2407-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Mistral-Large-Instruct-2407-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

bartowski commited on Aug 29, 2024

Commit

baa4d98

1 Parent(s): 40d2738

Revert "Upload folder using huggingface_hub"

Browse files

This reverts commit 34db3359133f5e6d16ded8f14ea756f9f1976c1a.

Files changed (13) hide show

.gitattributes +0 -10
Mistral-Large-Instruct-2407-IQ2_M.gguf +2 -2
Mistral-Large-Instruct-2407-IQ3_M-00001-of-00002.gguf +2 -2
Mistral-Large-Instruct-2407-IQ3_M-00002-of-00002.gguf +1 -1
Mistral-Large-Instruct-2407-IQ4_XS-00001-of-00002.gguf +2 -2
Mistral-Large-Instruct-2407-IQ4_XS-00002-of-00002.gguf +1 -1
Mistral-Large-Instruct-2407-Q2_K.gguf +2 -2
Mistral-Large-Instruct-2407-Q3_K_M-00001-of-00002.gguf +2 -2
Mistral-Large-Instruct-2407-Q3_K_M-00002-of-00002.gguf +1 -1
Mistral-Large-Instruct-2407-Q4_0_4_4-00001-of-00002.gguf +0 -3
Mistral-Large-Instruct-2407-Q4_0_4_4-00002-of-00002.gguf +0 -3
Mistral-Large-Instruct-2407-Q4_K_M-00001-of-00002.gguf +2 -2
Mistral-Large-Instruct-2407-Q4_K_M-00002-of-00002.gguf +1 -1

.gitattributes CHANGED Viewed

@@ -50,13 +50,3 @@ Mistral-Large-Instruct-2407-IQ1_M.gguf filter=lfs diff=lfs merge=lfs -text
 Mistral-Large-Instruct-2407-IQ4_XS/Mistral-Large-Instruct-2407-IQ4_XS-00001-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
 Mistral-Large-Instruct-2407-IQ4_XS/Mistral-Large-Instruct-2407-IQ4_XS-00002-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
 Mistral-Large-Instruct-2407.imatrix filter=lfs diff=lfs merge=lfs -text
-Mistral-Large-Instruct-2407-IQ3_M-00001-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
-Mistral-Large-Instruct-2407-IQ3_M-00002-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
-Mistral-Large-Instruct-2407-IQ4_XS-00001-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
-Mistral-Large-Instruct-2407-IQ4_XS-00002-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
-Mistral-Large-Instruct-2407-Q3_K_M-00001-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
-Mistral-Large-Instruct-2407-Q3_K_M-00002-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
-Mistral-Large-Instruct-2407-Q4_0_4_4-00001-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
-Mistral-Large-Instruct-2407-Q4_0_4_4-00002-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
-Mistral-Large-Instruct-2407-Q4_K_M-00001-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
-Mistral-Large-Instruct-2407-Q4_K_M-00002-of-00002.gguf filter=lfs diff=lfs merge=lfs -text

 Mistral-Large-Instruct-2407-IQ4_XS/Mistral-Large-Instruct-2407-IQ4_XS-00001-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
 Mistral-Large-Instruct-2407-IQ4_XS/Mistral-Large-Instruct-2407-IQ4_XS-00002-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
 Mistral-Large-Instruct-2407.imatrix filter=lfs diff=lfs merge=lfs -text

Mistral-Large-Instruct-2407-IQ2_M.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f03f00b628e20c734e94b8817bb72c4a7419f3bb44adb2bc5999580e6d758bb2
-size 41619609248

 version https://git-lfs.github.com/spec/v1
+oid sha256:e10f3009dbe5d748febf627919e69ed0f7257f41828f130b9d894a9dcbf60a7b
+size 41619605248

Mistral-Large-Instruct-2407-IQ3_M-00001-of-00002.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:98f01a7809824930873a0c3fd009e11bb7cb8b5ecc20fd28c0be26d8f129a3a1
-size 39991632768

 version https://git-lfs.github.com/spec/v1
+oid sha256:b7133f979b18c8a7d9f30833f113dcd87971f44f4b451274d5cb7f4de642cba2
+size 39991628736

Mistral-Large-Instruct-2407-IQ3_M-00002-of-00002.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0e2db1b8d1eae390e0f10a1f8162623edf58a674b55998e14993256b75dac40d
 size 15284761600

 version https://git-lfs.github.com/spec/v1
+oid sha256:41dfc51318b6f6440275ab3d9ce630a7b65c5898353e44cca53cc5f0e7f5769f
 size 15284761600

Mistral-Large-Instruct-2407-IQ4_XS-00001-of-00002.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f4650d7056da931ceb755d5a76aa7ff0c6b3bd786c813498d1858ca6d2196e02
-size 39944244896

 version https://git-lfs.github.com/spec/v1
+oid sha256:b3cbd710bb9697e8ce798286cfddf32acd76a1a302bd53874445113637f71b8e
+size 39944240896

Mistral-Large-Instruct-2407-IQ4_XS-00002-of-00002.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7c464dd9a24d57edf82a1ac2e3396a6dd1a11472adea5c56c73dfcf3e04a28ab
 size 25490098368

 version https://git-lfs.github.com/spec/v1
+oid sha256:0168a3e134ab6fe2450e709c79a30777f6c8a1d95b9ffe52f9dcdbe23bb25115
 size 25490098368

Mistral-Large-Instruct-2407-Q2_K.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f830171738e052c7af7149f6d0e05270ffa091d50f9e4185b68f3a992e94d242
-size 45196301984

 version https://git-lfs.github.com/spec/v1
+oid sha256:29359b45bdb5cb69403fa92ad5424b26742267daa396c5248e231f441e415244
+size 45196297984

Mistral-Large-Instruct-2407-Q3_K_M-00001-of-00002.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e7a750cdf87a2366340d9a731c179b3101a0a6d0fdad8d8ab4dba4c25977f9f3
-size 39895636768

 version https://git-lfs.github.com/spec/v1
+oid sha256:4967ce61417b08f604d264e1c10b8db7f7e81cc4c0ebfee88efcdfd6221abd9f
+size 39895632736

Mistral-Large-Instruct-2407-Q3_K_M-00002-of-00002.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:181d2c6c8147e17dda006a4ce87ebf4f2e85ec4a1872e7214bb96def16f2f4ad
 size 19207142496

 version https://git-lfs.github.com/spec/v1
+oid sha256:fd974053a0a5463e88f57b784e8735d4ec89fd24ed876a86a221e8a791a552af
 size 19207142496

Mistral-Large-Instruct-2407-Q4_0_4_4-00001-of-00002.gguf DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:25303974ef9b8cfadd77905c4e19ead32f2303cef2d8566e03db581d009c8b04
-size 39939279104

Mistral-Large-Instruct-2407-Q4_0_4_4-00002-of-00002.gguf DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:38e4a5dfa44ffe21c2828c6072a880a9cbe0ed907471440c92b830e57f0a374e
-size 29140962912

Mistral-Large-Instruct-2407-Q4_K_M-00001-of-00002.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fbb8f14d7af39355c93ec98001cbde60f63ad2396331f240e6daaeef65c44d92
-size 39965180608

 version https://git-lfs.github.com/spec/v1
+oid sha256:dc2597365128eed7af9b82579aa1d54a81470ff6e7d013effe2535daabbd0b7c
+size 39965176608

Mistral-Large-Instruct-2407-Q4_K_M-00002-of-00002.gguf CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0b4558293226dd4e24a1d757c8f174fb245f582c2d3d891062b6c852bb7c8de5
 size 33254446272

 version https://git-lfs.github.com/spec/v1
+oid sha256:7744ddb4c64a384f0ab74f58f7f232dc22eebda093f3f87b46a7ecae04728328
 size 33254446272