Clear domain
AI Act, compliance and internal procedures for Italian SMEs, with outputs designed for operational support.
AI case study • LLM fine tuning
LLora: LLM fine tuning for business with QLoRA for AI Act and compliance on a small locally runnable model.
LLora documents a complete domain adaptation pipeline: curated JSONL dataset, QLoRA training, intermediate checkpoints, final LoRA adapter and metric interpretation.
50
JSONL instruction/output examples
40
best checkpoint identified
52.6%
training loss reduction
Overview
A documented technical experiment, not a generic AI demo.
LLora was built to specialize a compact model on regulatory and operational content related to the European AI Act. The goal was not a general purpose model, but a realistic domain adaptation workflow.
The value is traceability: dataset, configuration, metrics, checkpoints and final adapter are presented as parts of a verifiable ML system.
Lightweight setup
Qwen2.5 1.5B base model in 4-bit, separate LoRA adapter and a workflow suitable for local reuse.
Challenge
Turning a complex regulatory topic into a training-ready dataset.
The challenge was not only training a model. It was defining a clean information boundary, with coherent examples, consistent responses and a credible specialization goal.
- small but curated dataset
- unique instructions and coherent outputs
- regulatory topic with risk of generic answers
- need to read metrics and checkpoints correctly
Solution
A complete, measured and presentable QLoRA pipeline.
The solution uses QLoRA on a 4-bit quantized model, rank 16, lightweight adapter and intermediate checkpoints. The best checkpoint is identified at step 40, avoiding the mistake of treating the final step as the best model.
- instruction tuning on question/answer pairs
- LoRA applied to attention and MLP modules
- training loss from 2.395 to 1.136
- minimum eval loss 1.500 at step 40
Technical stack
Key components of the LLora workflow.
The case study highlights ML design, data preparation, controlled training and metric interpretation.
Qwen2.5 1.5B 4-bit
Compact instruction tuned and quantized model selected for efficient experimentation and local reuse.
50 JSONL examples
Instruction/output records with 40/10 split, responses close to 100 words on average and consistent structure.
Rank 16 adapter
Efficient fine tuning without fully retraining the base model, with LoRA alpha 16 and dropout 0.
Controlled run
Effective batch 4, 5 epochs, learning rate 0.0002, short warmup and linear scheduler.
Best checkpoint
Best eval loss at step 40, with critical reading of the final plateau and generalization gap.
Exported adapter
Reusable PEFT output, intermediate checkpoints and technical reporting ready for review.
Impact
What LLora demonstrates in the AI portfolio.
LLora shows the ability to design an applied LLM workflow: data, method, training, metrics and technical documentation. The result is a credible operational demo, not a generic claim about artificial intelligence.
Domain specialization
Adaptation of a compact model to a specific regulatory perimeter.
Training loss reduction
52.6% improvement across the documented run.
Non-trivial evaluation
Best checkpoint identified before the final step, with plateau interpretation.
Technical reuse
Exported LoRA adapter, lightweight and suitable for local experimentation scenarios.
CTA
A case study for companies that want controllable proprietary AI.
LLora positions ZenkeiX as a studio able to build and document applied LLM workflows: from dataset to training, technical evaluation and artifact reuse.