Between SFT and the final model, we ran reinforcement learning (RL) in four stages: instruction following and math, code, assistant behavior, and finally agentic tasks. For each prompt, the model generates several responses, and the ones with the highest score are reinforced. The score comes from a program when the answer can be checked automatically, and from an LLM judge otherwise.
Safety is present to some extent from the first stage, but most of the work happens in the “assistant behavior” stage, using two datasets:
- 23,754 prompts drawn from public jailbreak and prompt-injection benchmarks, plus scenarios where the user tries to get around a system prompt’s instructions, sometimes over multiple turns (repeated asks, gradual escalation). They span five languages and mix attacks with legitimate requests;
- 1,112 cases, each scored against its own rubric. These notably include the weaknesses found through manual red-teaming: role-play, professional pretexts, and multi-turn escalation.
The key point is how responses are scored. The judge penalizes both helping with harmful use and over-refusal. And nearly half of the prompts are requests where the model should help. So the model can’t “win” by refusing everything.
| Metric |
Olala-v1.0-mini-7A1B-SFT |
Olala-v1.0-mini-7A1B |
| Attacks n = 5,701 |
| Guard ASR (%)↓ |
12.5 |
6.3 |
| Guard ASR, strict (%)↓ |
24.6 |
13.5 |
| Harmful, rubric judge (%)↓ |
34.9 |
22.2 |
| Benign n = 299 |
| Over-refusal (%)↓ |
7.0 |
10.7 |
| All prompts |
| Reasoning overflow (%)↓ |
11.8 |
15.0 |
↓ lower is better. Bold = better of the two.
That explains the shape of the results: far fewer harmful responses, including outside English (likely thanks to the multilingual data), with only a modest increase in over-refusals. The other stages (code, agentic) may also have played a role, since the comparison covers the full RL cascade.