lucyknada

/

Gryphe_Pantheon-RP-1.8-24b-Small-3.1-exl2

like 3

Tabbyapi config.yaml

network:
  host: 0.0.0.0
  port: 5000
  disable_auth: true
  send_tracebacks: false
  api_servers: ["OAI"]

logging:
  log_prompt: false
  log_generation_params: false
  log_requests: false

model:
  model_dir: models
  inline_model_loading: false
  use_dummy_models: false
  model_name: lucyknada_Gryphe_Pantheon-RP-1.8-24b-Small-3.1-exl2_6.0bpw
  use_as_default: []
  max_seq_len: 4096
  tensor_parallel: true
  gpu_split_auto: true
  rope_scale: 8.0
  rope_alpha: auto
  cache_mode: Q4
  chunk_size: 1024
  fasttensors: false

draft_model:
  draft_model_dir: models
  draft_rope_scale: 1.0
  draft_rope_alpha: 1.0
  draft_cache_mode: FP16

lora:
  lora_dir: loras
  loras:

embeddings:
  embedding_model_dir: models
  embeddings_device: cpu
  embedding_model_name:

sampling:

developer:
  unsafe_launch: false
  disable_request_streaming: false
  cuda_malloc_backend: false
  uvloop: true
  realtime_process_priority: false

I haven't used Tabby in a long time, so I might be forgetting something important. I updated it and the requirements but had no luck.
I use sillytavern with text completion.
Output:

and  values
 / [Link  with which ( (  as  1 total a.
 image  division ( ( ( ( in "# with
 orientation
 is
,  .. — and
 updated hearttechn
 en original
 + work  after
  call and (
 PH
 Two system Hall a
 convention WHt
 (  FFFS $;
 ep and .\ astra a.