Blog Post

Microsoft Foundry Blog
14 MIN READ

When Does RL Actually Help Fine-Tuning? A Difficulty-Controlled Study on Structured Generation

shihyaolin's avatar
shihyaolin
Icon for Microsoft rankMicrosoft
Aug 05, 2026

Author: Shih-Yao Lin

The uncomfortable question

Reinforcement learning is often the finishing move of the modern fine-tuning stack: run SFT first, then add RL (GRPO, PPO, DPO) to squeeze out the last few points. In practice the return is wildly inconsistent — sometimes a real jump, sometimes nothing after a burned GPU budget. The folk rule "RL helps when the task is hard" is directionally right but too vague to budget against: it doesn't say how much, which fields, or how to check in advance. This post turns that intuition into a decision rule you can apply before committing to an RL run.

TL;DR. In a controlled text-to-JSON experiment, GRPO added only +0.0111 absolute after SFT on a straightforward formulation where target attributes were explicitly provided. On a harder formulation requiring attributes to be inferred and normalized from prose, the weighted score increased from 0.6983 to 0.7498 — +0.0515 absolute, or approximately +7.4% relative. The largest gains occurred in Semantic-IoU (SIoU)-scored attribute fields, while cosine-scored free-text fields improved less. The practical takeaway is that low post-SFT performance alone is not enough to justify RL: first confirm that SFT has plateaued, that the missing information is recoverable, and that the reward can provide meaningful sequence-level credit. Data preparation and the evaluation methodology are reproducible on public ABO data. Reproducing the reported model scores requires either published predictions or access to Project Loom for retraining.

The setup: one task, two difficulties

The task is a large class of enterprise fine-tuning jobs: turn semi-structured product data into a fixed 8-field JSON object. Four fields are free-text editorial descriptions scored by cosine similarity; four are inference-oriented attribute fields scored by Semantic-IoU (SIoU) (fuzzy set overlap): three lists (vibe, quick_semantic, and trend_signals) plus the short overall_vibe string, which the scorer tokenizes on whitespace. The reported metric is the weighted aggregate of these eight per-field scores. GRPO uses the same underlying cosine and SIoU scoring functions as evaluation; during training both arms apply the same reward shaping (threshold annealing and per-field weighting), while final test reporting uses a fixed SIoU threshold (τ = 0.7) and equal field weights.

We use Amazon Berkeley Objects (ABO) (public, CC BY 4.0) as a stand-in for a proprietary product catalog, and build two arms of the same task:

  • Easy — the target attributes (productType / color / material) are handed to the model in the prompt, so the gold is copyable. SFT can reformat its way to a high score; RL should have little to add.
  • Hard — those same label lines are hidden in prose (the info still lives in the product name and bullets), so the gold must be inferred and normalized, not copied.

Both arms use the same open-weights Qwen3-32B base and the identical SFT-then-GRPO champion recipe — same LoRA config, learning rates, sampling settings, reward functions, reward curriculum (threshold annealing), and per-field weighting. The only thing that changes is whether the target attributes are exposed in the prompt or must be inferred from prose. Train and test are disjoint product sets, so the reported gains are measured on held-out items the model did not see in training.

What the two arms look like (same product)

A representative pair makes the difficulty knob concrete — the rows are the same item, differing only in whether the structured attributes are exposed:

Easy — attributes exposed (gold is copyable):

Every gold token already appears in the prompt, so SFT can reformat its way to a high score — RL has little to add.

Hard — the same three label lines are hidden (gold must be inferred):

The attributes still live in the product name ("yellow gold", "earrings"), so the model must locate and normalize them from prose instead of copying a labeled field — and the free-text gold is rephrased so the bullet-copy shortcut is closed too. That is where SFT leaves more headroom and GRPO produces a larger improvement.

The headline result

TaskSFTSFT → GRPOΔ absoluteNotes
Easy (attributes exposed)0.88850.8996+0.0111small gain after SFT
Hard (attributes hidden)0.69830.7498+0.0515 (≈ +7.4% rel.)larger gain; concentrated in SIoU fields
 

Two things stand out. First, post-SFT performance under the fixed recipe depends strongly on the task formulation: exposing the attributes lets SFT learn a largely copy-and-format transformation, reaching 0.8885; hiding them drops it to 0.6983 because the model can no longer copy. Second, the GRPO gain is much larger on the hard formulation (+0.0515) than the easy one (+0.0111).

The pattern in one line: headroom is necessary but not sufficient — GRPO helped most on recoverable, inference-heavy fields that remained weak after SFT and the task-level reward provided direct semantic credit. In this experiment the size of the GRPO gain was strongly associated with the headroom remaining after SFT, particularly for inference-oriented fields.

Figure 1. Weighted held-out scores before and after GRPO. The easy formulation improves by +0.0111 absolute, while the hard formulation improves by +0.0515 absolute, equivalent to approximately +7.4% relative.

Where the gain lands: it's the field type, not just the score

If the pattern holds, the hard-arm gain should concentrate in the SIoU inference fields — the ones that require inference — and touch the already-strong cosine fields less. That is what we observe.

FieldTypeEasy SFTEasy GRPOΔ easyHard SFTHard GRPOΔ hard
quick_summarycosine1.0001.000+0.0000.8320.856+0.024
rich_semanticcosine0.9950.995−0.0000.7880.804+0.016
summarycosine0.9590.962+0.0030.8900.903+0.013
trendcosine0.8840.894+0.0100.8190.858+0.039
quick_semanticSIoU0.8810.892+0.0120.5880.694+0.105
trend_signalsSIoU0.7060.724+0.0180.4710.532+0.061
overall_vibeSIoU0.8610.886+0.0250.6650.717+0.052
vibeSIoU0.8230.846+0.0230.5330.635+0.101
weighted_overall 0.88850.8996+0.01110.69830.7498+0.0515

Notice the contrast. On the hard arm, the four cosine free-text fields also have headroom (rich_semantic 0.788, trend 0.819), yet GRPO produced smaller and less consistent gains there (e.g., trend +0.039, quick_summary +0.024). The four SIoU inference fields, sitting at 0.47–0.67 after SFT, are where the largest gains land: quick_semantic +0.105, vibe +0.101, trend_signals +0.061, overall_vibe +0.052.

Figure 2. Per-field GRPO-minus-SFT differences. The largest hard-arm improvements occur in SIoU-scored attribute fields, especiallyquick_semanticandvibe.

 

Why? Standard SFT optimizes token-level likelihood against one reference output. Although it can generalize beyond exact memorization, the objective does not directly assign sequence-level partial credit based on the downstream SIoU or cosine metric. GRPO instead evaluates sampled outputs with the task-level grader and reinforces higher-scoring generations. This gives the optimizer a direct signal for recovering semantically correct attributes even when the generated sequence differs from the reference wording. In this experiment, GRPO's gains concentrated where the missing information was recoverable from the input but poorly served by token-level imitation — the hard-arm SIoU fields.

A pre-flight check: should you run RL at all?

You don't need to run RL to estimate whether it will help. Because GRPO's gains here concentrated where SFT left recoverable headroom, a short SFT-baseline inspection is informative. But mind the trap: a low score is necessary, not sufficient. The check is three cheap steps:

  1. Always run SFT first. RL isn't a replacement for SFT — it comes after it. So the question is never "SFT vs RL," but "after SFT, is it worth spending more on RL?" SFT is relatively cheap, so do it first.
  2. Score the SFT checkpoint per-field on a held-out test set (not just the aggregate). This is nearly zero-cost — a single inference pass plus grading.
  3. Apply the checklist below to each low-scoring field.

When to consider RL. Consider RL when SFT has genuinely plateaued, important fields remain below target, the missing information is recoverable from the input, sampled outputs can meaningfully vary, and a reliable task-level reward can distinguish partial improvements. In this study that reward is SIoU-based semantic attribute matching — a concrete choice for this task, not a universal requirement.

Post-SFT conditionDiagnosisRecommended action
Aggregate and field scores are already near targetLimited headroomKeep SFT; skip RL
Important fields remain low and the information is recoverablePotential reward-credit gapConsider RL
Scores are low because labels are noisy or unrecoverableData or reward problemFix data/reward first
SFT validation is still improvingSFT not convergedContinue tuning SFT
Reward does not reliably distinguish better outputsWeak RL signalImprove the grader before RL

The point is that a low score alone is not enough. GRPO helped here specifically where the missing information was recoverable and the task-level reward could credit partial progress. A low free-text (cosine) score, a noisy label, or an unconverged SFT are different problems with different fixes, and GRPO should not be assumed to solve them automatically.

 

Figure 3. Exploratory relationship between post-SFT field score and subsequent GRPO gain. Each point is one field on one arm: x = its post-SFT score (inspectable on your own checkpoint), y = the GRPO gain it received. Color = arm (orange = hard, blue = easy); shape = field type (circle = inference / SIoU, square = free-text / cosine). In this experiment the largest gains occur among low-scoring SIoU attribute fields, while free-text fields generally show smaller gains over the observed score range. (Headroom size alone still correlates with gain, pooled R²≈0.74 — the field-type axis is what a headroom-only reading misses.)

A prerequisite: make sure you’ve actually maxed out SFT

The whole diagnostic assumes the post-SFT score reflects a well-tuned, converged baseline, not an under-trained run. Hyperparameters absolutely matter here: if SFT is left under-tuned, its "headroom" is illusory — RL will appear to help when a better-tuned SFT would have closed the same gap for far less compute. So before you read any field as "stranded," confirm SFT has converged:

  • Train to a plateau, then measure. Push epochs/steps until the held-out score stops improving. A still-rising validation curve means you're measuring an unfinished SFT, not headroom.
  • Sweep the few hyperparameters that move the needle — learning rate, epochs, and (for LoRA) rank — and keep the best-validation checkpoint. You don't need an exhaustive grid; you need evidence the val metric is no longer sensitive to these.
  • Sanity-check the copyable fields. Fields whose answer is directly copyable from the input should saturate near 1.0. If even those are low, SFT is under-trained — fix that first; recovering copyable content is SFT's job, not RL's.

In this study both arms share one SFT recipe (Qwen3-32B, LoRA rank 32, lr 5e-4). On the easy arm it reaches 0.8885 with many test items at exact 1.0, evidence that the recipe can recover copyable fields. We do not treat the hard-arm score as a proven global SFT ceiling; it is the post-SFT performance under this fixed recipe. With the SFT, GRPO, and reward-shaping settings held constant, the observed difference is associated with the two task formulations, though the bundled changes prevent attributing it to a single difficulty factor.

What we can conclude — and what remains open

Shows: in this experiment, GRPO's post-SFT gain was strongly associated with the headroom remaining after SFT, particularly for inference-oriented fields — small on the easy formulation (+0.0111) and larger on the hard one (+0.0515), concentrated in the SIoU attribute fields.

Does not establish: that this holds unchanged for hard-verifiable rewards (math, code) or other task families. The gold is a rule-derived silver label, not a human-preference target. The easy +0.0111 is a small gain, and the reported numbers use a fixed 48-example subset of the 600-example test split with a single seed, so we do not treat it as a significance test; the hard +0.0515 is larger and lands on the predicted field-level structure, but additional seeds and a larger test set would be needed to quantify uncertainty.

Confounded factors. In this study, inference requirement, output format, and grader type are partly coupled: three hidden attribute fields are list-valued, overall_vibe is a short tokenized string, and all four are scored with SIoU. The experiment identifies a useful empirical pattern but does not fully disentangle these factors. The hard formulation also removes explicit attribute lines and reduces direct lexical overlap with the target output, so it changes more than a single microscopic difficulty knob. The GRPO recipe (its reward curriculum and per-field weights) was tuned on the harder task and applied unchanged to both arms; holding it fixed keeps the arm-to-arm comparison clean, but the recipe is not independently optimized for the easy arm.

What’s next

  • Extend to hard-verifiable rewards (math-correctness, code-execution) to test whether the observed headroom–gain relationship holds under other reward types — the most important open question.
  • Explore a predictor for whether the post-SFT per-field distribution can forecast the GRPO gain.
  • Sweep more difficulty points (a graded reveal-fraction) to trace the full curve.

Implementation guide: reproduce it on public data (CC BY 4.0)

Data preparation and the evaluation methodology use the public Amazon Berkeley Objects dataset (Collins et al., CVPR 2022; abo-listings text metadata only) and public sample code. Reproducing the reported model scores requires either published model predictions or Project Loom access to retrain the checkpoints. The steps below cover the public data pipeline and evaluation contract; only model retraining needs the preview.

  1. Clone the sample and run its offline check. The adapter requires Python 3.10 or later and uses only the standard library. Clone the public samples repository, enter the sample directory, and verify the installation before downloading any data:
    git clone https://github.com/microsoft-foundry/foundry-samples.git
    cd foundry-samples/samples/python/finetuning/abo-rl-headroom
    python abo_prep.py --selftest

    The last line should be SELFTEST PASS. This check is completely offline and validates the output schema, paired product IDs, and the absence of hidden labels from hard prompts.

  2. Download the data and build both arms. Download the public ABO listings metadata; the command requires the AWS CLI but does not require an AWS account or credentials:

    aws s3 cp --no-sign-request --recursive s3://amazon-berkeley-objects/listings/ data/abo-listings/listings/

    Then filter to fashion listings and map each product's real attributes (brand, product_type, color, material, style, pattern, bullet points, keywords) into the 8-field editorial JSON — four free-text fields + four SIoU-scored inference fields. Build both arms aligned by product_id, differing by the difficulty knob: easy exposes the productType / color / material lines in the prompt (gold is copyable); hard hides those label lines (the words still live in the product name and bullets) and rephrases the free-text gold so the copy shortcut is closed. Split disjointly by product_id (4,800 / 600 / 600 train / val / test). Because of inference and grading cost, the numbers in this post are computed on a fixed 48-example subset of the 600-example test split (seed 42), used for every checkpoint (both arms, SFT and GRPO).
    The standalone adapter and complete runbook are in the ABO RL headroom sample. Generate the datasets from the sample directory:

    python abo_prep.py \
      --abo-glob "data/abo-listings/listings/metadata/listings_*.json.gz" \
      --out-dir data/processed --category fashion \
      --difficulty both --max 6000 --seed 42

    Using --difficulty both is important: the adapter selects and shuffles product pairs once, then writes aligned easy/ and hard/ train, validation, and test files. Each JSONL row includes product_id, student_prompt, completion, and chat-format messages.

  3. Check the generated data before training. A successful 6,000-pair run reports 4,800 training rows, 600 validation rows, and 600 test rows for each arm. Confirm that corresponding easy and hard splits contain the same product IDs:
    python - <<'PY'
    import json
    from pathlib import Path
    
    root = Path("data/processed")
    for split, expected in (("train", 4800), ("val", 600), ("test", 600)):
        ids = {}
        for arm in ("easy", "hard"):
            path = root / arm / f"{split}.jsonl"
            ids[arm] = [json.loads(line)["product_id"] for line in path.open(encoding="utf-8")]
            assert len(ids[arm]) == expected, (path, len(ids[arm]))
        assert ids["easy"] == ids["hard"], f"unaligned {split} split"
        print(f"{split}: {expected} aligned pairs")
    PY

    Treat the generated completion as a silver label derived deterministically from ABO metadata, not as a human annotation or an LLM-generated ground truth.

  4. Verify predictions. The evaluation grader is fully specified in the Appendix: cosine similarity on the four free-text fields + SIoU (embedding overlap at τ = 0.7, text-embedding-3-small) on the four inference fields, averaged with equal weights into weighted_overall. To construct the fixed evaluation subset exactly, load each 600-row test.jsonl, shuffle with seed 42, and select the first 48 shuffled rows. Use those same product IDs for every checkpoint. Given predictions from the four checkpoints, score them on that subset using a text-embedding-3-small deployment; the reference results are Easy SFT ≈ 0.89, Easy GRPO ≈ 0.90, Hard SFT ≈ 0.70, and Hard GRPO ≈ 0.75. Predictions are not included in the public sample, so this step validates available predictions rather than independently reproducing the reported scores.
  5. Rerun the training (needs preview access). The fine-tuning itself uses the Microsoft Foundry Interactive Post-Training API, currently in private preview.

    1. Submit the preview intake form. Submitting the form acknowledges the preview Terms of Use.
    2. After approval, follow the private-preview onboarding material to configure the Loom cookbook and your assigned Foundry project.
    3. Point the cookbook's SFT and RL dataset builders at one generated arm at a time: use data/processed/easy/ for the easy run and data/processed/hard/ for the hard run.

The base model, SFT configuration, GRPO configuration, and reward shaping are identical across arms; only the data (attributes exposed vs. inferred) changes:

  • SFT: Qwen3-32B LoRA r32, lr 5e-4, batch 32, 1 epoch, 4,800 train examples.
  • GRPO (from the SFT checkpoint): lr 2e-5, temperature 1.0, group_size 8, groups_per_batch 8, importance-sampling loss, 64 steps, seed 42. Both arms anneal the SIoU threshold 0.5→0.7 and up-weight the inference-type SIoU fields (overall_vibe/trend_signals ×3, vibe ×2, quick_semantic ×1.5).

Score both checkpoints with the same evaluation grader (τ = 0.7, equal weights) and compare the SFT vs GRPO deltas. This is a custom reward on public data, not a preset recipe.

Bottom line

Run and tune SFT first. Then inspect held-out performance by field rather than relying only on an aggregate score. When SFT has saturated on a copy-and-format task, RL may offer little additional value. When important, recoverable attributes remain difficult and a reliable task-level reward can distinguish partial semantic improvements, RL becomes a more promising investment. These results establish that pattern for one structured-generation task; testing whether it generalizes to math, code, preference optimization, and other reward types remains future work.

Appendix — Scoring metrics

This is the evaluation grader used to report all test metrics in this post. Each of the eight fields is scored in [0, 1] by one of two functions, then averaged into weighted_overall. Training additionally applies reward shaping (threshold annealing and per-field weights, the same on both arms); the numbers reported here use the fixed evaluation configuration below (τ = 0.7, equal weights). All embeddings use text-embedding-3-small; emb(·) is the embedding of a normalized (lowercased, whitespace-collapsed) string.

  1. Cosine fields (quick_summary, rich_semantic, summary, trend) — free-text descriptions. Gold and prediction are each reduced to one string, embedded, and compared:
  1. Semantic-IoU fields (quick_semantic, trend_signals, overall_vibe, vibe) — three sparse attribute lists plus the short overall_vibe string. Let G, P be the gold and predicted item sequences; list values remain items and string values are split into whitespace tokens. Embed every item, greedily match pairs whose cosine ≥ τ (neither already matched), and let m be the number of matched pairs (the semantic intersection |G ∩ P|):
    This is the Jaccard index |G ∩ P| / |G ∪ P|, but intersection is defined by embedding similarity ≥ τ (τ = 0.7) instead of exact string match — so "sneakers" matches "shoes" without being identical.
  2. Aggregation. With per-field scores sₖ and equal weights (wₖ = 1 over all 8 fields at scoring time):

The denominator is fixed at the full field count (8), so a missing or malformed field contributes 0 to the numerator but still counts in the denominator — the model can't inflate its score by emitting only the easy fields, and output that fails to parse as JSON scores 0.

In one line: cosine fields ask "is the sentence semantically close?"; SIoU fields ask "how much do the two attribute sets overlap, allowing synonyms?" The hard task's headroom lives almost entirely in the second question — which is exactly where GRPO delivers its gain.

References

This work uses the Amazon Berkeley Objects (ABO) dataset (Collins et al., ABO: Dataset and Benchmarks for Real-World 3D Object Understanding, CVPR 2022, arXiv:2110.06199). Data © Amazon.com, used under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). We use only the abo-listings text-metadata subset (not the 3D assets), which we reshaped and adapted into the text-to-JSON task described above.

Disclaimer

© 2026 Microsoft Corporation. All rights reserved. This document is provided "as-is." Information and views expressed in this document, including URL and other internet website references, may change without notice. You bear the risk of using it.

This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.

Some examples are for illustration only and are fictitious. No real association is intended or inferred.

Updated Oct 21, 2026
Version 3.0
No CommentsBe the first to comment