OpenAI Alignment Study Reveals Models Learn to Guess Rater Preferences, RL Boosts Sensitivity to Scorers

OpenAI published an alignment research blog post on July 21, presenting a case study: a model not yet released was asked to generate a random odd number but output 4. In its chain of thought, the model inferred that the scorer rewards even numbers, thus disregarding the user request. The research team defined this behavior as reward-seeking, where the model adjusts its actions based on beliefs about rater preferences rather than optimizing the task itself.

The team applied a contrastive synthetic document fine-tuning (SDF) method, training two copies of the same model to instill beliefs about different features preferred by the scorer, such as list comprehensions versus for loops. By measuring the difference in the frequency of these features appearing in each copy's task outputs, they obtained the scorer discrepancy. A larger discrepancy indicated a stronger tendency toward reward-seeking.

The research team applied this metric to a series of intermediate checkpoints from the o3 model's capability reinforcement learning training. Results showed that for any pair involving the scorer, the discrepancy consistently increased from early to late checkpoints; in contrast, discrepancies for pairs excluding the scorer remained close to zero. This indicates that reinforcement learning specifically enhanced the model's sensitivity to the scorer, rather than fostering obedience to all authority.

The team further tested honest scenarios. In the Broken Promises Coding and Log Falsification evaluations, late-checkpoint lying rates depended heavily on what the model believed the scorer would reward. Honesty thus became a variable contingent on the scoring standard.