Work · LLM post-training research · 6 min read
A domain-specific LLM for reliability engineering
There was no training data for reliability engineering, so we made some. We turned 56 textbook problems into 866 verified question-answer pairs, and an 8B model went from 63.6% to 82.2% on them. The results I find most interesting are the ones about what didn't work.
The problem
Reliability engineering (Weibull and exponential lifetimes, mean time to failure, k-out-of-n systems, fault trees, Bayesian updating of failure rates) had no large public question-answer dataset and no established benchmark. The lab had also found frontier models unreliable on its own hand-written problems. We wanted to know whether a general 8B model could be adapted to the domain starting from a handful of textbook exercises, on a student's compute budget.
Making the data
We digitised two textbooks (32 chapters) and mined them with regular expressions plus a language-model validator, which gave 56 curated seed triplets of question, worked reasoning and numeric answer. A Self-Instruct loop then generated new problems in the style of the seeds. The one change that mattered was the filter. Instead of the usual same-model consistency check, a second frontier model solved every generated problem independently, and we kept a problem only if the two answers agreed within 5% (with a semantic fallback for non-numeric cases). That produced 158 verified items.
We merged them with the textbook seeds and cleaned the lot (splitting multi-part problems, deduplication, unit stripping), which gave 256 entries. The 215 with numeric answers became the first training set. A "hard generation" round added 65 questions. A MetaMath-style paraphrase round, where the numbers and formulas are frozen and only the wording changes, added 586 more, for 866 in total.
We learned two things about synthetic data along the way. The first is a self-instruct ceiling. Generators write problems inside their own competence, so baseline models scored 92 to 97% on the generated questions but only 68 to 76% on the textbook ones. Verification doesn't fix that. It only removes wrong answers. The second is that surface diversity beat difficulty: paraphrases helped more than harder questions did. The 65 hard questions moved accuracy by +2.0 points (p = 0.69) and the 586 paraphrases moved it by +18.6 (p = 0.002).
Fine-tuning
I fine-tuned Qwen3-8B with 4-bit QLoRA through Unsloth, with LoRA rank 16 and alpha 32 on all attention and MLP projections, NEFTune noise and thinking mode off. Evaluation used greedy decoding, a fresh base model per fold, and a Wilcoxon signed-rank test over the per-fold accuracies. Early stopping with patience two picked epoch four whatever maximum I set. On the 866-example set with 10 folds the base model answers 63.6% correctly and the fine-tuned model 82.2%. Every fold improved, by 12.6 to 26.4 points, and 206 questions flipped from wrong to right against 45 the other way.
The choice of base model mattered more than any other decision. On the same 215 numeric questions Qwen3-8B scored 70.7% zero-shot against 41.4% for Llama 3.1 8B (37.5% over the full 256-question cleaned set, which also includes formula and text items).
Two evaluation lessons came out of the early Qwen3-8B runs. One early run reported +7.0 points. Re-run with greedy decoding, the same configuration gained only +0.9, and the best greedy configuration, trained for one more epoch, +2.3. The rest was sampling variance. Every cross-validation number from that re-run on, the +18.6 included, is greedy. The other lesson was that fine-tuning a thinking model on polished non-thinking solutions collapsed its reasoning. The fine-tuned model emitted empty thinking blocks and answers eight times shorter than the base model's.
Where it stands against frontier models
I ran the frontier models on a 500-question sample of the same benchmark, through an API at temperature 0.1 with no extended thinking requested (o3-mini at its lowest reasoning effort, and Gemini 3.1 Pro, whose thinking can't be switched off, at its default level). Claude Sonnet 4.6 scored 92.4%, o3-mini 86.4% and Gemini 3.1 Pro 76.4% of the answers they returned in a parseable form. Counted over all 500 questions instead, the figures are 85.4% and 61.6%, and Gemini's thinking mode left 97 answers unparseable. The fine-tuned 8B model's 82.2% sits between them. The comparison is informative but it isn't clean. The small model's number is in-distribution cross-validation on a paraphrase-heavy set, and the frontier numbers are zero-shot.
Reinforcement learning, and a caveat
My co-author led the second stage, which applied GRPO with graded verifiable rewards and DAPO-style dynamic sampling on the 266 questions the base model solves inconsistently (one to three of four samples). It started from a separate fine-tuning checkpoint trained only on the other 600 questions (the ones screened out of the GRPO set), not from the 82.2% headline adapter. Accuracy on those 266 questions rose from 50.8% to 57.9% at step 80 (McNemar p = 0.056, not significant at 5%), then declined with more steps. Direct preference optimisation had hurt earlier (68.0% against 69.8% for the base model, on the four folds that finished), and a first GRPO attempt with a judge-model reward added only +0.5.
The +18.6 points is an in-distribution number, and that caveat belongs next to the headline. The cross-validation folds were shuffled at the question level, so paraphrases of a training question can sit in the test fold. On a 54-question independent holdout the picture is weaker. A fine-tuned adapter trained on a 600-question subset fell to 33.3% against the base model's 53.7% (p = 0.019). The headline adapter, evaluated greedily, scored 55.6% against 59.3%, which is within noise. The GRPO variants stayed near the base model on that holdout. We read this as a small-data forgetting problem. Grouped folds would settle it, and we don't claim any transfer until they've been run.
What I would do differently
- Group the folds by source question from the first experiment, so paraphrases never leak across the split.
- Hold out an independent test set before generating any data, and report it beside the cross-validation number in every table.
- Decide the decoding protocol before the first result. Greedy evaluation would have saved the retracted +7.0.
Credits
This was a joint MSc research project with Elora Drouilhet, supervised by Dr. Zhiguo Zeng at CentraleSupélec's LGI laboratory. I led the data pipeline (seed extraction, cross-model verification, paraphrase augmentation), the supervised fine-tuning, the frontier-model baselines and the cluster work. The evaluation framework was shared work. My co-author led the reinforcement-learning stage, with my support.