SERPO · arXiv:2607.26873

Open-ended TTRL needs evidence, not answer voting.

SERPO lets a model evolve at test time when there are no reference answers, no labeled feedback, no stronger judges, and no canonical target for majority vote.

arXiv PDF Code

SERPO loop

Response archives Good / Normal / Bad

Rubric evolution criteria separate evidence

Policy update GRPO reward signal

The gap

Agreement is not enough for open-ended quality

In multiple-choice or symbolic tasks, voting can select a pseudo-label. In open-ended generation, responses can share a recommendation while differing in safety, completeness, uncertainty, and factual grounding.

Voting

Response-level consensus

Frequent responses can share the same omission. A majority cluster can reward agreement without verifying the qualities that make an answer useful.

SERPO

Criterion-level self-evolution

Query-specific criteria separate Good, Normal, and Bad response archives, then produce graded rewards through a frozen probabilistic judge.

The loop

Three states evolve together

SERPO stores prompt-local response evidence, refreshes the criteria that distinguish that evidence, and updates one shared actor with the resulting reward signal.

Response archives

Rollouts are organized into Good, Normal, and Bad archives for each prompt.

Rubric evolution

Criteria are retained when they separate archive quality and pruned when they do not.

Policy evolution

GRPO uses the rubric-derived scalar rewards to update the actor at test time.

SERPO closed-loop framework — SERPO closes the response-rubric-policy loop without labeled feedback or an external reward model.

Evidence

Large in-domain gains and consistent transfer

The same result table is kept here so the narrative view remains a complete paper page.

HealthBench +20.63

Up to 20.63 points over the corresponding base model.

ResearchQA +20.31

Up to 20.31 points over the corresponding base model.

Macro average +8.06

Up to 8.06 points across six benchmarks.

Model	Method	HealthBench	MedQA	LLMEval-Med	ResearchQA	GPQA-Diamond	RaR-Science	Avg.
Qwen3-4B	Base	32.30	53.81	59.82	57.29	53.03	67.29	53.92
Qwen3-4B	SERPO	49.83	54.73	62.50	77.60	59.09	68.12	61.98
Qwen3.5-9B	Base	44.68	69.84	69.27	68.62	70.20	77.87	66.75
Qwen3.5-9B	SERPO	65.31	71.32	74.18	83.18	72.22	79.93	74.36

Walkthrough

From evolving criteria to evolving behavior

The detailed pipeline and evolution curves show how the method keeps reward resolution as the actor changes.

Detailed SERPO pipeline — Archive scoring, rubric refresh, and reward calibration inside SERPO.

Two-stage evolution — Sequential HealthBench to ResearchQA evolution.

Citation

Paper and code

SERPO is the official implementation of the arXiv paper. The repository is released under Apache License 2.0.

@article{wang2026serpo,
  title   = {{SERPO}: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning},
  author  = {Wang, Jianze and Zheng, Kunwang and Liu, Ying and Cao, Yu and Zhang, Qilong and Chen, Jinlong and Yang, Hua and Chen, Qianglong},
  journal = {arXiv preprint arXiv:2607.26873},
  year    = {2026},
  url     = {https://arxiv.org/abs/2607.26873}
}