Adapter-based fine-tuning for text generation

Llama2-7b Fine-Tuning 4bit (QLoRA)¶

This example shows how to fine-tune Llama2-7b to follow instructions. Instruction tuning is the first step in adapting a general purpose Large Language Model into a chatbot.

This example uses no distributed training or big data functionality. It is designed to run locally on any machine with GPU availability.

Prerequisites¶

HuggingFace API Token
Access approval to Llama2-7b-hf
GPU with at least 12 GiB of VRAM (in our tests, we used an Nvidia T4)

Running¶

Install Ludwig¶

pip install ludwig ludwig[llm]

Command Line¶

Set your token environment variable from the terminal, then run the API script:

export HUGGING_FACE_HUB_TOKEN="<api_token>"
./run_train.sh

Python API¶

Set your token environment variable from the terminal, then run the API script:

export HUGGING_FACE_HUB_TOKEN="<api_token>"
python train_alpaca.py

Upload to HuggingFace¶

You can upload to the HuggingFace Hub from the command line:

ludwig upload hf_hub -r <your_org>/<model_name> -m <path/to/model>