WizardLM-2-22B-RP-control-vectors

gghfez/WizardLM-2-22B-RP-control-vectors

Creative Writing control-vectors for gghfez/WizardLM-2-22B-RP

Usage example

llama.cpp

llama-server --model  wizardlm-2-22b-rp-q6_k.gguf [other CLI arguments] \
    --control-vector  WizardLM-2-22B-RP-character_focus__debias.gguf \
    --control-vector  WizardLM-2-22B-RP-character_focus__narration.gguf

Exllamav2

Clone this repo into the same directory as the exl2 model but with -vectors appended:
`huggingface-cli download gghfez/WizardLM-2-22B-RP-control-vectors --local-dir WizardLM-2-22B-RP-exl2-vectors

So you should have something like
/models/WizardLM-2-22B-RP-exl2
/models/WizardLM-2-22B-RP-exl2-vectors

Follow instructions the instructions below for TabbyAPI or Exui:

TabbyAPI Fork

1. git clone https://github.com/gapeleon/tabbyAPI-control-vectors
2. cd tabbyAPI-control-vectors
3. ./start.sh #let it build the venv, etc then exit
4. source venv/bin/activate #to enter the venv
5. pip install gguf #to read the vectors
6. cp config_sample_control-vectors.yml config.yml
7. set the model_dir and model_name in config.yml
8. tweak the control-vectors how you want them
9. ./start.sh

Exui Fork

1. git clone https://github.com/gapeleon/exui-control-vectors
2. Set it up / install it the same as regular exui
3. pip install gguf if you don't have it already
4. Start exui python server.py
5. Configure the model as usual.
6. Tick "Enable control vectors" and wait for them to appear in the UI
7. Enable the vectors + direction + scaling you want, then hit "Load Model"

Based off Version v3.0 of creative-writing-control-vectors-v3.0.

See here for more info about control-vectors creative-writing-control-vectors-v3.0