Fine-Tuning Qwen3-VL with LoRA: Price Lists to JSON

A supplier sends you a PDF. You need article numbers, descriptions and prices in your software. The PDF has merged cells, prices per 100 pieces, and a footnote explaining that the number you just extracted means something else entirely.

This is the kind of AI application i find interesting. Small problem, concrete output, very little room for creative writing.

My price-list demo on Hugging Face turns PDFs and images into JSON, CSV and DATANORM 4. The current demo runs the stock Qwen3-VL-8B-Instruct model. It isn’t a fine-tune yet. Here’s how to use that baseline for Qwen3-VL fine-tuning with LoRA, teaching a vision-language model to extract structured JSON from price-list images.

From PDF to JSON: start with a baseline

The current path is straightforward:

PDF page or image
    -> Qwen3-VL + extraction instructions
    -> JSON schema validation
    -> human review and corrections
    -> Python DATANORM exporter

Each page is processed separately. The prompt asks for article numbers, prices, units, evidence and uncertainties. Missing prices stay null. A price per 100 pieces and a box containing 100 pieces are different fields. Mix those up and your perfectly valid JSON is still wrong.

The model proposes data. Pydantic checks its structure. The exporter checks what it can actually represent and writes the file without another GPU call.

Before training anything, run the base model on a fixed set of documents and keep its mistakes. Otherwise you have no way to tell whether your expensive new adapter improved anything.

Preparing the fine-tuning dataset

For this task, a training example is a page image, the extraction instruction, and the corrected JSON answer. The model learns to produce that answer from the image and instruction. That’s supervised fine-tuning.

For a fictional row reading AB-1042 | Sechskantschraube M8 | 12,50 EUR / 100 Stück, part of the target could look like this:

{
  "article_number": "AB-1042",
  "description": "Sechskantschraube M8",
  "price": "12.50",
  "currency": "EUR",
  "price_quantity": "100",
  "pack_quantity": null,
  "unit": "Stück"
}

That’s a position excerpt, not the complete response schema. In the dataset, keep the full schema consistent, including warnings and uncertainty fields. Don’t label a price as net if the page doesn’t establish that.

Include awkward examples: unreadable cells, repeated headers, variants, quantity breaks and pages with no articles. Training only on clean tables teaches the model that every page has a convenient answer.

Model-generated labels can give you a head start. They still need checking against the page. Feeding uncorrected output back into training is a good way to make the same mistake more confidently.

Split by supplier or document family before training. Pages from the same catalogue belong together. A random page split can put almost identical layouts into training and evaluation, then reward you for memorizing them.

Fine-tuning Qwen3-VL with LoRA

You don’t need to pretrain a vision model from scratch. LoRA adds trainable low-rank adapters while keeping the original weights frozen. You still load the base model; the adapter contains the learned changes.

Hugging Face’s TRL SFTTrainer supports image datasets and PEFT adapters. The training core can look like this:

from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen3-VL-8B-Instruct",
    args=SFTConfig(
        output_dir="price-list-adapter",
        model_init_kwargs={"dtype": "bfloat16"},
        max_length=None,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=8,
        learning_rate=1e-4,
        num_train_epochs=1,
        report_to="none",
    ),
    peft_config=LoraConfig(
        r=16, lora_alpha=32,
        target_modules="all-linear",
        task_type="CAUSAL_LM",
    ),
    train_dataset=train_ds,
)
trainer.train()
trainer.save_model("price-list-adapter")

This is an illustrative starting configuration, not a training run behind the demo. train_ds must contain decoded page images in an image column and conversational prompt/completion fields: the user prompt includes the image placeholder and extraction instruction; the assistant completion contains the corrected JSON. TRL’s prompt-completion format trains on the completion by default.

Use a CUDA training environment with BF16 support and enough VRAM. Pin compatible versions of PyTorch, Transformers, TRL and PEFT, and record the base-model revision with your dataset revision. The demo’s serving requirements aren’t a training environment.

One detail worth paying attention to: blindly truncating a multimodal sample can cut out image tokens. max_length=None avoids that truncation, but doesn’t make memory unlimited. Bound image resolution and document size, then run a small batch through the whole training path before committing to a longer job. If memory is tight, investigate QLoRA with a quantized base model rather than assuming this example fits your GPU.

Evaluate the extracted prices, not just the loss

Run the base model and adapter on the same held-out documents. Match the extracted rows to the reference rows and compare article numbers, exact prices and their quantity basis. Count missing rows and invented rows too. A lower training loss doesn’t tell you whether 12.50 EUR / 100 Stück became 12.50 EUR / Stück.

Keep a validation set for choosing checkpoints and a separate test set for the final comparison. Only put the adapter into the application if it earns its place there. Deployment also needs the adapter loaded on top of the matching base model, or deliberately merged weights; changing a model-name string isn’t enough for the current Space loader.

And keep the DATANORM writer as normal code. A model can help read a messy table. It doesn’t need to improvise the file format your accounting software imports.