library_name: transformers
license: gemma
base_model: unsloth/gemma-3-12b-it
tags:
- axolotl
- generated_from_trainer
datasets: - ToastyPigeon/unalign-v2
model-index: - name: g3-12b-it-unalign
results: []
g3-12b-it-unalign
This model is a fine-tuned version of unsloth/gemma-3-12b-it on the ToastyPigeon/unalign-v2 dataset.
It achieves the following results on the evaluation set:
- Loss: 1.4684
So, it seems alright. I noticed however that the responses got pretty short at the end of the 2nd epoch. Not like, unusably short, but generally shorter than I personally like.
The epoch 1 test gguf is based on this commit.
I personally prefer epoch 1 to epoch 2, and will likely update this or make a second proper commit for epoch 1.
Update: I did indeed make a second commit for the epoch 1 checkpoint.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 69
- optimizer: Use apollo_adamw_layerwise with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=proj=random,rank=1,scale=128.0,scale_type=tensor,update_proj_gap=200
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 5
- num_epochs: 2.0
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 7.8965 | 0.0118 | 1 | 6.4897 |
| 4.4934 | 0.2 | 17 | 4.0497 |
| 3.9523 | 0.4 | 34 | 3.7484 |
| 3.5624 | 0.6 | 51 | 3.3152 |
| 2.7168 | 0.8 | 68 | 2.4773 |
| 2.1303 | 1.0 | 85 | 1.9483 |
| 1.8215 | 1.2 | 102 | 1.7577 |
| 1.7199 | 1.4 | 119 | 1.6561 |
| 1.5771 | 1.6 | 136 | 1.5611 |
| 1.5599 | 1.8 | 153 | 1.5124 |
| 1.4831 | 2.0 | 170 | 1.4684 |
Framework versions
- Transformers 4.50.0.dev0
- Pytorch 2.5.1+cu124
- Datasets 3.4.1
- Tokenizers 0.21.1