KIEFERSA/Sophea-RAG-Nemo3

Sophea-RAG-Nemo3

Greek + English grounded RAG reader — fine-tuned NVIDIA Nemotron-3 Nano 30B-A3B

Sophea-RAG-Nemo3 is a Greek + English RAG reader: it answers strictly from the retrieved documents you put in its context, cites the documents it used as [n], and abstains when the answer is not there. It is the generation stage of a Greek RAG stack whose retrieval stages are Sophea-Nemo-Embedding and Sophea-Nemo-Reranker.

• Creator: Kiefer SA
• Base model: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 (NemotronHForCausalLM, hybrid Mamba + attention MoE, 30B total / A3B active; trust_remote_code)
• Languages: Greek + English
• Behaviour: grounded answer → [n] citation → abstain when unsupported
• Training: LoRA (r 16, α 32) on attention + MLP + Mamba in_proj, 1 epoch, 8k context, assistant-only loss, bf16, cosine LR 1e-4, grad-clip 0.3, 4×B200 — merged into the base weights, so this is a standalone checkpoint (no adapter needed)

The base model is not a grounded reader. Off the shelf it answers from parametric memory: it hallucinates instead of abstaining (abstains on ~1% of unanswerable questions) and its answers are faithful to the context only a quarter of the time. This fine-tune teaches grounding, citation and selective prediction.

Why it matters — base vs fine-tuned

Measured on a Greek Wikipedia long-retrieval RAG benchmark (4,946 questions, up to 40 documents / ~24k tokens of context, 25% deliberately unanswerable, L1 factoid → L5 multi-hop synthesis). Identical prompts and greedy decoding for both models; correctness and faithfulness are LLM-judged.

metric	base	fine-tuned	Δ
Answer correctness (LLM-judge)	29.4%	66.9%	+37.5
Faithfulness / groundedness	25.2%	84.5%	+59.3
Citation F1	41.1%	67.8%	+26.7
— citation precision	43.2%	73.4%	+30.2
— citation recall	46.8%	65.0%	+18.2
Cited all gold documents	41.0%	56.5%	+15.5
Exact citation set	30.1%	56.5%	+26.4
Abstention recall (unanswerable refused)	1.2%	30.5%	+29.3

Read. Correctness 29.4% → 66.9% and faithfulness 25.2% → 84.5% — the fine-tune answers from the documents instead of from memory. Citation F1 rises 41.1 → 67.8 and the model returns the exact gold citation set on 56.5% of items (base 30.1%). Abstention recall goes from ~zero to 30.5%: the base essentially never refuses, so its "faithfulness" on unanswerable questions is accidental.

Robustness to retrieval noise

More retrieved documents = more distractors. Correct / faithful (%), by number of documents in context:

metric (documents in context)	8	12	16	20	28	40
correct — base	30.8	32.4	29.1	29.0	26.1	29.8
correct — fine-tuned	76.5	72.7	67.2	67.8	63.6	59.3
faithful — base	24.3	23.0	24.1	23.7	28.1	29.0
faithful — fine-tuned	86.1	85.2	86.7	86.5	82.1	79.4

Read. The fine-tune leads at every context size, from 76.5% correct at 8 documents to 59.3% at 40 — it degrades gracefully as noise grows, while faithfulness stays ≥79% throughout. The base is flat at ~29% because it is not reading the context in the first place.

Position of the gold document ("lost in the middle")

metric (gold position in context)	early	middle	late
correct — base	31.9	29.4	26.1
correct — fine-tuned	65.8	65.0	70.3
faithful — base	21.9	27.9	26.7
faithful — fine-tuned	84.2	83.1	86.4

Read. The base is best when the gold document is early (31.9%) and worst when it is late (26.1%) — the classic position bias. The fine-tune is strongest on late gold (70.3%) and varies by only ~5 points across positions.

By question difficulty (L1 factoid → L5 multi-hop synthesis)

metric (question difficulty)	L1	L2	L3	L4	L5
correct — base	33.9	28.3	22.1	32.2	34.6
correct — fine-tuned	77.7	73.6	60.0	54.3	57.3
faithful — base	30.0	25.2	23.1	20.4	24.1
faithful — fine-tuned	91.3	90.3	85.4	68.6	70.2

Read. The gain holds at every level — largest on the retrieval-shaped levels (L1 +43.8, L2 +45.3) and still +22 on the hardest multi-hop synthesis questions, where faithfulness rises from ~20-24% to ~69-70%.

Domain RAG evaluation

A second, independent evaluation on in-house Greek + English domain documents (800 questions across energy, finance, legal and medical; 187 of them unanswerable):

metric	base	fine-tuned	Δ
Answer correctness (LLM-judge)	31.2%	61.0%	+29.8
Faithfulness / groundedness	23.0%	66.2%	+43.2
Citation hit (cited a gold document)	40.0%	60.1%	+20.1
Abstention on unanswerable items (n=187)	8.6%	80.2%	+71.6

metric	Energy	Finance	Legal	Medical
correct — base	37.5	22.5	42.0	23.0
correct — fine-tuned	62.0	57.0	72.0	53.0
faithful — base	25.0	21.0	23.0	23.0
faithful — fine-tuned	67.0	64.5	70.5	63.0
cite-hit — base	35.5	40.5	44.0	40.0
cite-hit — fine-tuned	60.0	57.5	71.0	52.0

Read. The fine-tune wins on every domain and every metric, strongest on legal (72.0% correct / 70.5% faithful) and weakest on medical (53.0% / 63.0%) — the same ordering the retrieval stages show on these domains.

Selective prediction

On the 187 unanswerable items the correct action is to refuse. The base answers anyway 91% of the time; the fine-tune abstains on 80.2% of them — this is the mechanism behind the faithfulness gain.

Position invariance (answerable subset, n=613)

metric (gold document position)	1	2	3	4	5	6
correct — base	45.4	38.6	48.5	32.0	36.2	32.4
correct — fine-tuned	78.3	77.3	84.3	76.7	84.5	76.5
faithful — base	33.6	30.3	34.3	20.4	25.9	32.4
faithful — fine-tuned	84.2	85.6	90.3	86.4	86.2	85.3

Read. The base decays as the gold document moves deeper into the context (45.4% at position 1 → 32.4% at position 6). The fine-tune is essentially flat at 77-85% across all six positions.

Usage (transformers)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

name = "KIEFERSA/Sophea-RAG-Nemo3"
tok = AutoTokenizer.from_pretrained(name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    name, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True).eval()

SYSTEM = (
    "Είσαι βοηθός που απαντά ΑΠΟΚΛΕΙΣΤΙΚΑ με βάση τα παρεχόμενα έγγραφα. Χρησιμοποίησε μόνο τις πληροφορίες των εγγράφων, ΜΗΝ επινοείς. Ανέφερε σε αγκύλες τον αριθμό του εγγράφου/των εγγράφων που χρησιμοποίησες, π.χ. [2]. Αν η απάντηση ΔΕΝ υπάρχει στα έγγραφα, πες το καθαρά. Απάντησε στα ελληνικά, σύντομα και ουσιαστικά."
)

docs = ["Η Αθήνα είναι η πρωτεύουσα της Ελλάδας.", "Το Παρίσι είναι η πρωτεύουσα της Γαλλίας."]
question = "Ποια είναι η πρωτεύουσα της Ελλάδας;"
context = "\n".join(f"[{i}] {d}" for i, d in enumerate(docs, 1))

msgs = [{"role": "system", "content": SYSTEM},
        {"role": "user", "content": f"Έγγραφα:\n{context}\n\nΕρώτηση: {question}"}]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
enc = tok(text, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**enc, max_new_tokens=200, do_sample=False)
print(tok.decode(out[0][enc.input_ids.shape[1]:], skip_special_tokens=True))
# -> "Η Αθήνα. [1]"

Number the documents [1..n] in the user turn and keep the system message above — the model was trained to cite those numbers and to say the answer is absent when it is. Greedy decoding (do_sample=False) is what the numbers above were measured with. Needs trust_remote_code=True.

Serving with vLLM

vllm serve KIEFERSA/Sophea-RAG-Nemo3 --trust-remote-code --max-model-len 40960

curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "KIEFERSA/Sophea-RAG-Nemo3",
  "temperature": 0,
  "messages": [
    {"role": "system", "content": "Απάντησε μόνο από τα έγγραφα και ανέφερε [n]."},
    {"role": "user", "content": "Έγγραφα:\n[1] Η Αθήνα είναι η πρωτεύουσα της Ελλάδας.\n\nΕρώτηση: Ποια είναι η πρωτεύουσα της Ελλάδας;"}
  ]
}'

License

Fine-tune of nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 — governed by the NVIDIA Open Model License; commercial use permitted under those terms (note: not Apache-2.0). Review the base model's license before use.