Deriving the DPO Loss from First Principles

In my previous post, I worked through the derivation of the PPO loss used in RLHF for LLMs. By the end, we arrived at a fairly daunting objective function with multiple components: clipped surrogate, value function loss, entropy bonus and KL penalty. It is not just that the final objective is intimidating but the entire RLHF pipeline is complex and multi-step. You first train a separate reward model to reflect human preferences then fine-tune the LLM using RL with PPO.

That brings us to Direct Preference Optimization (DPO). DPO is a computationally lightweight alternative that directly optimizes LLMs to adhere to human preferences without explicit reward modeling or reinforcement learning. The key insight is that DPO implicitly optimizes the same objective as PPO-based RLHF (reward maximization with a KL-divergence constraint) but it replaces the entire reward model + PPO loop with a single supervised objective on preference pairs. There is no sampling during training, no value function, no clipping, just a classification loss!

Here I derive the DPO loss showing exactly how this simplification is possible. I will assume familiarity with concepts from the PPO post, particularly the reward model and the KL-constrained RLHF objective.

Again, a huge shoutout to Umar Jamil's video on DPO for an excellent walkthrough that helped me understand the derivation.

I: The RLHF Objective

Let's recall the RLHF objective from the PPO blog. The goal of RLHF is to find a policy ${\pi }_{\theta }\backslash pi\_\backslash theta$ that maximizes expected reward while staying close to a reference model ${\pi }_{\text{ref}}\backslash pi\_\{\backslash text\{ref\}\}$:

$$J_{\text{RLHF}}(\theta )={\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }(y\mathrm{\mid }x)}[r_{\varphi }(x,y)]-\beta \cdot D_{\text{KL}}({\pi }_{\theta }(y\mathrm{\mid }x)\mathrm{\parallel }{\pi }_{\text{ref}}(y\mathrm{\mid }x))J\_\{\backslash text\{RLHF\}\}(\backslash theta)\; =\; \backslash mathbb\{E\}\_\{x\; \backslash sim\; \backslash mathcal\{D\},\; y\; \backslash sim\; \backslash pi\_\backslash theta(y|x)\}\backslash left[r\_\backslash phi(x,\; y)\backslash right]\; -\; \backslash beta\; \backslash cdot\; D\_\{\backslash text\{KL\}\}\backslash left(\backslash pi\_\backslash theta(y|x)\; \backslash |\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)\backslash right)$$

The first term encourages the model to generate high-reward responses. The second term (KL penalty) prevents the model from drifting too far from the reference which helps avoid reward hacking and maintains language quality.

As we saw in the PPO blog, we can't optimize this objective directly with gradient descent because the expectation ${\mathbb{E}}_{y\sim {\pi }_{\theta }(y\mathrm{\mid }x)}[\cdot ]\backslash mathbb\{E\}\_\{y\; \backslash sim\; \backslash pi\_\backslash theta(y|x)\}[\backslash cdot]$ requires sampling from the policy and sampling is non-differentiable. This is why we needed reinforcement learning algorithms like REINFORCE and PPO. They provide ways to estimate policy gradients without differentiating through the sampling process.

What if we could reformulate the problem so that we don't need to sample from the policy during training? This is exactly what DPO will achieve.

II: The Bradley-Terry Model for Preference Learning

We also need to understand the Bradley-Terry model for reward model training in a bit more detail with focus on why it works the way it does.

Training a reward model requires human-labeled preference data that compares pairs of responses. $$\mathcal{D}={\{(x^{(i)},y_w^{(i)},y_l^{(i)})\}}_{i=1}^N\backslash mathcal\{D\}\; =\; \backslash left\backslash \{(x^\{(i)\},\; y\_w^\{(i)\},\; y\_l^\{(i)\})\backslash right\backslash \}\_\{i=1\}^\{N\}$$

where:

• $xx$ is the prompt
• $y_{w}y\_w$ is the preferred (winning) response
• $y_{l}y\_l$ is the dispreferred (losing) response

The Bradley-Terry Probability Model

The Bradley-Terry model provides a principled way to convert comparison data (defined above) into a probabilistic model. It assumes there exists some latent reward function $r^{\ast }(x,y)r^*(x,\; y)$ that captures true response quality and models the probability that response $y_{w}y\_w$ is preferred over $y_{l}y\_l$ as:

$$\begin{array}{cccc}& P(y_w\succ y_l\mathrm{\mid }x)=\frac{e^{r^{\ast }(x,y_w)}}{e^{r^{\ast }(x,y_w)}+e^{r^{\ast }(x,y_l)}}& & \text{(II.I)}\end{array}P(y\_w\; \backslash succ\; y\_l\; |\; x)\; =\; \backslash frac\{e^\{r^*(x,\; y\_w)\}\}\{e^\{r^*(x,\; y\_w)\}\; +\; e^\{r^*(x,\; y\_l)\}\}\; \backslash tag\{II.I\}$$

The intuition is straightforward that the responses with higher reward are exponentially more likely to be preferred.

A key step is recognizing that this ratio of exponentials can be written as a sigmoid function. This is important because it connects Bradley-Terry to standard binary classification.

Let $A=r(x,y_w)A\; =\; r(x,\; y\_w)$ and $B=r(x,y_l)B\; =\; r(x,\; y\_l)$. We want to show:

$$\frac{e^A}{e^A+e^B}=\sigma (A-B)\backslash frac\{e^A\}\{e^A\; +\; e^B\}\; =\; \backslash sigma(A\; -\; B)$$

where $\sigma (z)=\frac{1}{1+e^{-z}}\backslash sigma(z)\; =\; \backslash frac\{1\}\{1\; +\; e^\{-z\}\}$ is the sigmoid function.

Starting with the left-hand side:

$$\frac{e^A}{e^A+e^B}\backslash frac\{e^A\}\{e^A\; +\; e^B\}$$

we can rewrite it as: $$=\frac{e^A\mathrm{/}{e}^A}{(e^A+e^B)\mathrm{/}{e}^A}=\frac{1}{1+e^B\mathrm{/}{e}^A}=\frac{1}{1+e^{B-A}}=\; \backslash frac\{e^A\; /\; e^A\}\{(e^A\; +\; e^B)\; /\; e^A\}\; =\; \backslash frac\{1\}\{1\; +\; e^B\; /\; e^A\}\; =\; \backslash frac\{1\}\{1\; +\; e^\{B-A\}\}$$

which is the sigmoid function of $(A - B)$: $$=\frac{1}{1+e^{-(A-B)}}=\sigma (A-B)=\; \backslash frac\{1\}\{1\; +\; e^\{-(A-B)\}\}\; =\; \backslash sigma(A\; -\; B)$$

Therefore, the Bradley-Terry model can be written as:

$$\begin{array}{cccc}& \boxed{{\displaystyle P(y_w\succ y_l\mathrm{\mid }x)=\sigma (r(x,y_w)-r(x,y_l))}}& & \text{(II.II)}\end{array}\backslash boxed\{P(y\_w\; \backslash succ\; y\_l\; |\; x)\; =\; \backslash sigma\backslash left(r(x,\; y\_w)\; -\; r(x,\; y\_l)\backslash right)\}\; \backslash tag\{II.II\}$$

Reward Model Loss

Given a dataset of preferences $\mathcal{D}\backslash mathcal\{D\}$, we can train a parameterized reward model $r_{\varphi }(x,y)r\_\backslash phi(x,\; y)$ using maximum likelihood estimation. We want to maximize the probability of observing the preferences in our dataset:

$$\underset{\varphi }{\max }\prod _{(x,y_w,y_l)\in \mathcal{D}}P(y_w\succ y_l\mathrm{\mid }x)\backslash max\_\backslash phi\; \backslash prod\_\{(x,\; y\_w,\; y\_l)\; \backslash in\; \backslash mathcal\{D\}\}\; P(y\_w\; \backslash succ\; y\_l\; |\; x)$$

Taking the log and negating (to turn maximization into minimization), we get the negative log-likelihood loss:

$$\begin{array}{cccc}& \boxed{{\displaystyle {\mathcal{L}}_{\text{RM}}(\varphi )=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}[\log \sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l))]}}& & \text{(II.III)}\end{array}\backslash boxed\{\backslash mathcal\{L\}\_\{\backslash text\{RM\}\}(\backslash phi)\; =\; -\backslash mathbb\{E\}\_\{(x,\; y\_w,\; y\_l)\; \backslash sim\; \backslash mathcal\{D\}\}\backslash left[\backslash log\; \backslash sigma\backslash left(r\_\backslash phi(x,\; y\_w)\; -\; r\_\backslash phi(x,\; y\_l)\backslash right)\backslash right]\}\; \backslash tag\{II.III\}$$

This is just binary cross-entropy objective which helps the reward model learn to assign higher rewards to preferred responses. Notice that the Bradley-Terry model depends only on the difference of rewards: $r(x,y_w)-r(x,y_l)r(x,\; y\_w)\; -\; r(x,\; y\_l)$. The absolute values dont matter, it is only their relative ordering. This means:

If we add any constant $cc$ or any function $f(x)f(x)$ that depends only on the prompt (not the response), the preference probabilities dont change. This invariance property will be the key to deriving DPO.

III: Optimal Policy in Closed Form

Here, we will find the exact analytical optimal policy solution to the optimization problem. We want to find the policy that maximizes expected reward while keeping the KL divergence from the reference policy bounded:

$$\begin{array}{cccc}& \underset{\pi }{\max }{\mathbb{E}}_{x\sim \mathcal{D},y\sim \pi (y\mathrm{\mid }x)}[r(x,y)]-\beta \cdot D_{\text{KL}}(\pi (y\mathrm{\mid }x)\mathrm{\parallel }{\pi }_{\text{ref}}(y\mathrm{\mid }x))& & \text{(III.I)}\end{array}\backslash max\_\backslash pi\; \backslash mathbb\{E\}\_\{x\; \backslash sim\; \backslash mathcal\{D\},\; y\; \backslash sim\; \backslash pi(y|x)\}\backslash left[r(x,\; y)\backslash right]\; -\; \backslash beta\; \backslash cdot\; D\_\{\backslash text\{KL\}\}\backslash left(\backslash pi(y|x)\; \backslash |\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)\backslash right)\; \backslash tag\{III.I\}$$

Note, I am writing $\pi \backslash pi$ instead of ${\pi }_{\theta }\backslash pi\_\backslash theta$ to emphasize that we are looking for the optimal policy in general not just the parameterized version.

Expanding the KL divergence:

$$D_{\text{KL}}(\pi (y\mathrm{\mid }x)\mathrm{\parallel }{\pi }_{\text{ref}}(y\mathrm{\mid }x))={\mathbb{E}}_{y\sim \pi (y\mathrm{\mid }x)}[\log \frac{\pi (y\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y\mathrm{\mid }x)}]D\_\{\backslash text\{KL\}\}\backslash left(\backslash pi(y|x)\; \backslash |\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)\backslash right)\; =\; \backslash mathbb\{E\}\_\{y\; \backslash sim\; \backslash pi(y|x)\}\backslash left[\backslash log\; \backslash frac\{\backslash pi(y|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y|x)\}\backslash right]$$

So our objective becomes:

$$\underset{\pi }{\max }{\mathbb{E}}_{x\sim \mathcal{D}}{\mathbb{E}}_{y\sim \pi (y\mathrm{\mid }x)}[r(x,y)-\beta \log \frac{\pi (y\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y\mathrm{\mid }x)}]\backslash max\_\backslash pi\; \backslash mathbb\{E\}\_\{x\; \backslash sim\; \backslash mathcal\{D\}\}\; \backslash mathbb\{E\}\_\{y\; \backslash sim\; \backslash pi(y|x)\}\backslash left[r(x,\; y)\; -\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi(y|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y|x)\}\backslash right]$$

For a fixed prompt $xx$, we want to find the distribution $\pi (\cdot \mathrm{\mid }x)\backslash pi(\backslash cdot|x)$ that maximizes:

$${\mathbb{E}}_{y\sim \pi (y\mathrm{\mid }x)}[r(x,y)-\beta \log \frac{\pi (y\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y\mathrm{\mid }x)}]\backslash mathbb\{E\}\_\{y\; \backslash sim\; \backslash pi(y|x)\}\backslash left[r(x,\; y)\; -\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi(y|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y|x)\}\backslash right]$$

This is a constrained optimization problem over probability distributions. We can solve it using the method of Lagrange multipliers, enforcing that $\pi (y\mathrm{\mid }x)\backslash pi(y|x)$ sums to 1. For discrete $yy$:

$$\mathcal{L}(\pi ,\lambda )=\sum _y\pi (y\mathrm{\mid }x)[r(x,y)-\beta \log \frac{\pi (y\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y\mathrm{\mid }x)}]+\lambda (1-\sum _y\pi (y\mathrm{\mid }x))\backslash mathcal\{L\}(\backslash pi,\; \backslash lambda)\; =\; \backslash sum\_y\; \backslash pi(y|x)\; \backslash left[r(x,\; y)\; -\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi(y|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y|x)\}\backslash right]\; +\; \backslash lambda\; \backslash left(1\; -\; \backslash sum\_y\; \backslash pi(y|x)\backslash right)$$

Taking the derivative with respect to $\pi(y|x)$ and setting it to zero (stationary point):

$$\frac{\mathrm{\partial }\mathcal{L}}{\mathrm{\partial }\pi (y\mathrm{\mid }x)}=r(x,y)-\beta \log \frac{\pi (y\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y\mathrm{\mid }x)}-\beta -\lambda =0\backslash frac\{\backslash partial\; \backslash mathcal\{L\}\}\{\backslash partial\; \backslash pi(y|x)\}\; =\; r(x,\; y)\; -\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi(y|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y|x)\}\; -\; \backslash beta\; -\; \backslash lambda\; =\; 0$$

Now, solving for $\pi (y\mathrm{\mid }x)\backslash pi(y|x)$:

$$\log \frac{\pi (y\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y\mathrm{\mid }x)}=\frac{1}{\beta }(r(x,y)-\beta -\lambda )\backslash log\; \backslash frac\{\backslash pi(y|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y|x)\}\; =\; \backslash frac\{1\}\{\backslash beta\}\backslash left(r(x,\; y)\; -\; \backslash beta\; -\; \backslash lambda\backslash right)$$

$$\frac{\pi (y\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y\mathrm{\mid }x)}=\exp (\frac{1}{\beta }r(x,y))\cdot \exp (-1-\frac{\lambda }{\beta })\backslash frac\{\backslash pi(y|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y|x)\}\; =\; \backslash exp\backslash left(\backslash frac\{1\}\{\backslash beta\}r(x,\; y)\backslash right)\; \backslash cdot\; \backslash exp\backslash left(-1\; -\; \backslash frac\{\backslash lambda\}\{\backslash beta\}\backslash right)$$

$$\pi (y\mathrm{\mid }x)={\pi }_{\text{ref}}(y\mathrm{\mid }x)\cdot \exp (\frac{1}{\beta }r(x,y))\cdot \exp (-1-\frac{\lambda }{\beta })\backslash pi(y|x)\; =\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)\; \backslash cdot\; \backslash exp\backslash left(\backslash frac\{1\}\{\backslash beta\}r(x,\; y)\backslash right)\; \backslash cdot\; \backslash exp\backslash left(-1\; -\; \backslash frac\{\backslash lambda\}\{\backslash beta\}\backslash right)$$

The term $\exp (-1-\frac{\lambda }{\beta })\backslash exp\backslash left(-1\; -\; \backslash frac\{\backslash lambda\}\{\backslash beta\}\backslash right)$ is a constant (with respect to $yy$) that ensures normalization. To find its value, we enforce that $\pi (y\mathrm{\mid }x)\backslash pi(y|x)$ must be a valid probability distribution and sum to 1:

$$\sum _y\pi (y\mathrm{\mid }x)=1\backslash sum\_y\; \backslash pi(y|x)\; =\; 1$$

Substituting our expression for $\pi(y|x)$:

$$\sum _y{\pi }_{\text{ref}}(y\mathrm{\mid }x)\cdot \exp (\frac{1}{\beta }r(x,y))\cdot \exp (-1-\frac{\lambda }{\beta })=1\backslash sum\_y\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)\; \backslash cdot\; \backslash exp\backslash left(\backslash frac\{1\}\{\backslash beta\}r(x,\; y)\backslash right)\; \backslash cdot\; \backslash exp\backslash left(-1\; -\; \backslash frac\{\backslash lambda\}\{\backslash beta\}\backslash right)\; =\; 1$$

Since $\exp (-1-\frac{\lambda }{\beta })\backslash exp\backslash left(-1\; -\; \backslash frac\{\backslash lambda\}\{\backslash beta\}\backslash right)$ doesn't depend on $yy$, we can factor it out of the sum:

$$\exp (-1-\frac{\lambda }{\beta })\cdot \sum _y{\pi }_{\text{ref}}(y\mathrm{\mid }x)\cdot \exp (\frac{1}{\beta }r(x,y))=1\backslash exp\backslash left(-1\; -\; \backslash frac\{\backslash lambda\}\{\backslash beta\}\backslash right)\; \backslash cdot\; \backslash sum\_y\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)\; \backslash cdot\; \backslash exp\backslash left(\backslash frac\{1\}\{\backslash beta\}r(x,\; y)\backslash right)\; =\; 1$$

Solving for the constant:

$$\exp (-1-\frac{\lambda }{\beta })=\frac{1}{\sum _y{\pi }_{\text{ref}}(y\mathrm{\mid }x)\cdot \exp (\frac{1}{\beta }r(x,y))}\backslash exp\backslash left(-1\; -\; \backslash frac\{\backslash lambda\}\{\backslash beta\}\backslash right)\; =\; \backslash frac\{1\}\{\backslash sum\_y\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)\; \backslash cdot\; \backslash exp\backslash left(\backslash frac\{1\}\{\backslash beta\}r(x,\; y)\backslash right)\}$$

We define this normalizing sum as the partition function $Z(x)Z(x)$:

$$\begin{array}{cccc}& Z(x)=\sum _y{\pi }_{\text{ref}}(y\mathrm{\mid }x)\exp (\frac{1}{\beta }r(x,y))& & \text{(III.II)}\end{array}Z(x)\; =\; \backslash sum\_y\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)\; \backslash exp\backslash left(\backslash frac\{1\}\{\backslash beta\}r(x,\; y)\backslash right)\; \backslash tag\{III.II\}$$

Substituting back, we get the optimal policy:

$$\begin{array}{cccc}& \boxed{{\displaystyle {\pi }_r(y\mathrm{\mid }x)=\frac{1}{Z(x)}{\pi }_{\text{ref}}(y\mathrm{\mid }x)\exp (\frac{1}{\beta }r(x,y))}}& & \text{(III.III)}\end{array}\backslash boxed\{\backslash pi\_r(y|x)\; =\; \backslash frac\{1\}\{Z(x)\}\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)\; \backslash exp\backslash left(\backslash frac\{1\}\{\backslash beta\}r(x,\; y)\backslash right)\}\; \backslash tag\{III.III\}$$

We have an exact closed-form expression for the optimal policy. However, we cannot compute it directly because $Z(x)Z(x)$ is intractable. To compute it, we need to sum over all possible responses $yy$ which not possible.

IV: The Reparameterization Trick

The key insight of DPO is to flip the relationship between reward and policy. Now, we frame the problem as: "given an optimal policy what reward function does it correspond to?"

Starting from the optimal policy equation (III.III):

$${\pi }_r(y\mathrm{\mid }x)=\frac{1}{Z(x)}{\pi }_{\text{ref}}(y\mathrm{\mid }x)\exp (\frac{1}{\beta }r(x,y))\backslash pi\_r(y|x)\; =\; \backslash frac\{1\}\{Z(x)\}\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)\; \backslash exp\backslash left(\backslash frac\{1\}\{\backslash beta\}r(x,\; y)\backslash right)$$

We solve for the reward $r(x,y)r(x,\; y)$ by first taking the log of both sides:

$$\log {\pi }_r(y\mathrm{\mid }x)=\log {\pi }_{\text{ref}}(y\mathrm{\mid }x)+\frac{1}{\beta }r(x,y)-\log Z(x)\backslash log\; \backslash pi\_r(y|x)\; =\; \backslash log\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)\; +\; \backslash frac\{1\}\{\backslash beta\}r(x,\; y)\; -\; \backslash log\; Z(x)$$

Now rearrange to get $r(x,y)r(x,\; y)$ on left-side:

$$\frac{1}{\beta }r(x,y)=\log {\pi }_r(y\mathrm{\mid }x)-\log {\pi }_{\text{ref}}(y\mathrm{\mid }x)+\log Z(x)\backslash frac\{1\}\{\backslash beta\}r(x,\; y)\; =\; \backslash log\; \backslash pi\_r(y|x)\; -\; \backslash log\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)\; +\; \backslash log\; Z(x)$$

$$r(x,y)=\beta \log {\pi }_r(y\mathrm{\mid }x)-\beta \log {\pi }_{\text{ref}}(y\mathrm{\mid }x)+\beta \log Z(x)r(x,\; y)\; =\; \backslash beta\; \backslash log\; \backslash pi\_r(y|x)\; -\; \backslash beta\; \backslash log\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)\; +\; \backslash beta\; \backslash log\; Z(x)$$

This can be written more compactly as:

$$\begin{array}{cccc}& \boxed{{\displaystyle r(x,y)=\beta \log \frac{{\pi }_r(y\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y\mathrm{\mid }x)}+\beta \log Z(x)}}& & \text{(IV.I)}\end{array}\backslash boxed\{r(x,\; y)\; =\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi\_r(y|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y|x)\}\; +\; \backslash beta\; \backslash log\; Z(x)\}\; \backslash tag\{IV.I\}$$

The reward is expressed as:

• A term involving the log-ratio of the optimal policy to the reference policy
• $\beta \log Z(x)\backslash beta\; \backslash log\; Z(x)$ which depends only on $xx$ (not on $yy$)

V: Deriving the DPO Loss

Finally, we have all the pieces to derive the DPO loss. From Section II, the Bradley-Terry preference model is:

$$P(y_w\succ y_l\mathrm{\mid }x)=\sigma (r(x,y_w)-r(x,y_l))P(y\_w\; \backslash succ\; y\_l\; |\; x)\; =\; \backslash sigma\backslash left(r(x,\; y\_w)\; -\; r(x,\; y\_l)\backslash right)$$

From Section IV, assuming we have access to an optimal policy ${\pi }^{\ast }\backslash pi^*$, the reward can be written as:

$$r(x,y)=\beta \log \frac{{\pi }^{\ast }(y\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y\mathrm{\mid }x)}+\beta \log Z(x)r(x,\; y)\; =\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi^*(y|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y|x)\}\; +\; \backslash beta\; \backslash log\; Z(x)$$

Substituting this into Bradley-Terry:

$$P(y_w\succ y_l\mathrm{\mid }x)=\sigma ([\beta \log \frac{{\pi }^{\ast }(y_w\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y_w\mathrm{\mid }x)}+\beta \log Z(x)]-[\beta \log \frac{{\pi }^{\ast }(y_l\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y_l\mathrm{\mid }x)}+\beta \log Z(x)])P(y\_w\; \backslash succ\; y\_l\; |\; x)\; =\; \backslash sigma\backslash left(\backslash left[\backslash beta\; \backslash log\; \backslash frac\{\backslash pi^*(y\_w|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y\_w|x)\}\; +\; \backslash beta\; \backslash log\; Z(x)\backslash right]\; -\; \backslash left[\backslash beta\; \backslash log\; \backslash frac\{\backslash pi^*(y\_l|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y\_l|x)\}\; +\; \backslash beta\; \backslash log\; Z(x)\backslash right]\backslash right)$$

Simplifying the expression inside the sigmoid:

$$=\sigma (\beta \log \frac{{\pi }^{\ast }(y_w\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y_w\mathrm{\mid }x)}+\beta \log Z(x)-\beta \log \frac{{\pi }^{\ast }(y_l\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y_l\mathrm{\mid }x)}-\beta \log Z(x))=\; \backslash sigma\backslash left(\backslash beta\; \backslash log\; \backslash frac\{\backslash pi^*(y\_w|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y\_w|x)\}\; +\; \backslash beta\; \backslash log\; Z(x)\; -\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi^*(y\_l|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y\_l|x)\}\; -\; \backslash beta\; \backslash log\; Z(x)\backslash right)$$

The $\beta \log Z(x)\backslash beta\; \backslash log\; Z(x)$ terms cancel:

$$=\sigma (\beta \log \frac{{\pi }^{\ast }(y_w\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y_w\mathrm{\mid }x)}-\beta \log \frac{{\pi }^{\ast }(y_l\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y_l\mathrm{\mid }x)})=\; \backslash sigma\backslash left(\backslash beta\; \backslash log\; \backslash frac\{\backslash pi^*(y\_w|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y\_w|x)\}\; -\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi^*(y\_l|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y\_l|x)\}\backslash right)$$

Now recall the critical insight from Section II where we mentioned that the Bradley-Terry model depends only on reward differences. Thus, when we compute $r(x,y_w)-r(x,y_l)r(x,\; y\_w)\; -\; r(x,\; y\_l)$ the intractable partition function $Z(x)Z(x)$ cancels out. This is what makes DPO possible.

We can write this more cleanly by defining the implicit reward in terms of the optimal policy:

$$\widehat{r}(x,y)=\beta \log \frac{{\pi }^{\ast }(y\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y\mathrm{\mid }x)}\backslash hat\{r\}(x,\; y)\; =\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi^*(y|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y|x)\}$$

Thus:

$$P(y_w\succ y_l\mathrm{\mid }x)=\sigma (\widehat{r}(x,y_w)-\widehat{r}(x,y_l))P(y\_w\; \backslash succ\; y\_l\; |\; x)\; =\; \backslash sigma\backslash left(\backslash hat\{r\}(x,\; y\_w)\; -\; \backslash hat\{r\}(x,\; y\_l)\backslash right)$$

We dont actually have access to the optimal policy ${\pi }^{\ast }\backslash pi^*$. But we can parameterize a policy ${\pi }_{\theta }\backslash pi\_\backslash theta$ and optimize it to maximize the likelihood of the observed preferences. This is exactly what the reward model loss (II.III) does except now our reward is implicitly defined by the policy itself.

The DPO loss is the negative log-likelihood:

$$\boxed{{\displaystyle {\mathcal{L}}_{\text{DPO}}({\pi }_{\theta };{\pi }_{\text{ref}})=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}[\log \sigma (\beta \log \frac{{\pi }_{\theta }(y_w\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y_w\mathrm{\mid }x)}-\beta \log \frac{{\pi }_{\theta }(y_l\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y_l\mathrm{\mid }x)})]}}\backslash boxed\{\backslash mathcal\{L\}\_\{\backslash text\{DPO\}\}(\backslash pi\_\backslash theta;\; \backslash pi\_\{\backslash text\{ref\}\})\; =\; -\backslash mathbb\{E\}\_\{(x,\; y\_w,\; y\_l)\; \backslash sim\; \backslash mathcal\{D\}\}\backslash left[\backslash log\; \backslash sigma\backslash left(\backslash beta\; \backslash log\; \backslash frac\{\backslash pi\_\backslash theta(y\_w|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y\_w|x)\}\; -\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi\_\backslash theta(y\_l|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y\_l|x)\}\backslash right)\backslash right]\}$$

for implicit reward notation, it can be written as:

$$\begin{array}{cccc}& {\mathcal{L}}_{\text{DPO}}({\pi }_{\theta };{\pi }_{\text{ref}})=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}[\log \sigma ({\widehat{r}}_{\theta }(x,y_w)-{\widehat{r}}_{\theta }(x,y_l))]& & \text{(V.II)}\end{array}\backslash mathcal\{L\}\_\{\backslash text\{DPO\}\}(\backslash pi\_\backslash theta;\; \backslash pi\_\{\backslash text\{ref\}\})\; =\; -\backslash mathbb\{E\}\_\{(x,\; y\_w,\; y\_l)\; \backslash sim\; \backslash mathcal\{D\}\}\backslash left[\backslash log\; \backslash sigma\backslash left(\backslash hat\{r\}\_\backslash theta(x,\; y\_w)\; -\; \backslash hat\{r\}\_\backslash theta(x,\; y\_l)\backslash right)\backslash right]\; \backslash tag\{V.II\}$$

Some key insights from the above DPO loss:

• The policy implicitly defines its own reward via the log-ratio with the reference. There is no separate reward model.
• This is just a supervised classification loss on preference pairs with no RL.
• DPO uses the fixed preference dataset $\mathcal{D}\backslash mathcal\{D\}$. Thus, no sampling during training.
• No value function needed since we're not doing policy gradients
• DPO still optimizes the KL-constrained reward maximization objective but in a different way

VI: Building Intuition for DPO

Now that we have the DPO loss we can build some intuition around the implicit reward model and its gradient updates.

Implicit Reward Model

The DPO paper subtitle is "Your Language Model is Secretly a Reward Model" and this captures the key insight as we are using the LLM for implicit reward. The policy ${\pi }_{\theta }\backslash pi\_\backslash theta$ defines an implicit reward function:

$${\widehat{r}}_{\theta }(x,y)=\beta \log \frac{{\pi }_{\theta }(y\mathrm{\mid }x)}{{\pi }_{\text{ref}}(y\mathrm{\mid }x)}\backslash hat\{r\}\_\backslash theta(x,\; y)\; =\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi\_\backslash theta(y|x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y|x)\}$$

This reward measures how much more likely the current policy is to generate response $yy$ compared to the reference policy, scaled by $\beta \backslash beta$.

• If ${\pi }_{\theta }(y\mathrm{\mid }x)>{\pi }_{\text{ref}}(y\mathrm{\mid }x)\backslash pi\_\backslash theta(y|x)\; >\; \backslash pi\_\{\backslash text\{ref\}\}(y|x)$: The implicit reward is positive (the policy "likes" this response more than reference)
• If : The implicit reward is negative (the policy "likes" this response less than reference)

Analyzing the Gradient Update

We can flex our brain muscles one more time and compute the gradient for the DPO objective:

$${\mathcal{L}}_{\text{DPO}}=-\mathbb{E}[\log \sigma ({\widehat{r}}_{\theta }(x,y_w)-{\widehat{r}}_{\theta }(x,y_l))]\backslash mathcal\{L\}\_\{\backslash text\{DPO\}\}\; =\; -\backslash mathbb\{E\}\backslash left[\backslash log\; \backslash sigma\backslash left(\backslash hat\{r\}\_\backslash theta(x,\; y\_w)\; -\; \backslash hat\{r\}\_\backslash theta(x,\; y\_l)\backslash right)\backslash right]$$

Let $u={\widehat{r}}_{\theta }(x,y_w)-{\widehat{r}}_{\theta }(x,y_l)u\; =\; \backslash hat\{r\}\_\backslash theta(x,\; y\_w)\; -\; \backslash hat\{r\}\_\backslash theta(x,\; y\_l)$. Using the chain rule:

$${\mathrm{\nabla }}_{\theta }{\mathcal{L}}_{\text{DPO}}=-\mathbb{E}\left[\frac{{\sigma }^{\mathrm{\prime }}(u)}{\sigma (u)}{\mathrm{\nabla }}_{\theta }u\right]\backslash nabla\_\backslash theta\; \backslash mathcal\{L\}\_\{\backslash text\{DPO\}\}\; =\; -\backslash mathbb\{E\}\backslash left[\backslash frac\{\backslash sigma\text{'}(u)\}\{\backslash sigma(u)\}\; \backslash nabla\_\backslash theta\; u\backslash right]$$

Using the property ${\sigma }^{\mathrm{\prime }}(u)=\sigma (u)(1-\sigma (u))\backslash sigma\text{'}(u)\; =\; \backslash sigma(u)(1\; -\; \backslash sigma(u))$:

$$=-\mathbb{E}\left[\frac{\sigma (u)(1-\sigma (u))}{\sigma (u)}{\mathrm{\nabla }}_{\theta }u\right]=-\mathbb{E}[(1-\sigma (u)){\mathrm{\nabla }}_{\theta }u]=\; -\backslash mathbb\{E\}\backslash left[\backslash frac\{\backslash sigma(u)(1-\backslash sigma(u))\}\{\backslash sigma(u)\}\; \backslash nabla\_\backslash theta\; u\backslash right]\; =\; -\backslash mathbb\{E\}\backslash left[(1\; -\; \backslash sigma(u))\; \backslash nabla\_\backslash theta\; u\backslash right]$$

Using $1-\sigma (u)=\sigma (-u)1\; -\; \backslash sigma(u)\; =\; \backslash sigma(-u)$:

$$=-\mathbb{E}[\sigma (-u){\mathrm{\nabla }}_{\theta }u]=\; -\backslash mathbb\{E\}\backslash left[\backslash sigma(-u)\; \backslash nabla\_\backslash theta\; u\backslash right]$$

Now, $-u={\widehat{r}}_{\theta }(x,y_l)-{\widehat{r}}_{\theta }(x,y_w)-u\; =\; \backslash hat\{r\}\_\backslash theta(x,\; y\_l)\; -\; \backslash hat\{r\}\_\backslash theta(x,\; y\_w)$, and:

$${\mathrm{\nabla }}_{\theta }u={\mathrm{\nabla }}_{\theta }{\widehat{r}}_{\theta }(x,y_w)-{\mathrm{\nabla }}_{\theta }{\widehat{r}}_{\theta }(x,y_l)=\beta {\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y_w\mathrm{\mid }x)-\beta {\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y_l\mathrm{\mid }x)\backslash nabla\_\backslash theta\; u\; =\; \backslash nabla\_\backslash theta\; \backslash hat\{r\}\_\backslash theta(x,\; y\_w)\; -\; \backslash nabla\_\backslash theta\; \backslash hat\{r\}\_\backslash theta(x,\; y\_l)\; =\; \backslash beta\; \backslash nabla\_\backslash theta\; \backslash log\; \backslash pi\_\backslash theta(y\_w|x)\; -\; \backslash beta\; \backslash nabla\_\backslash theta\; \backslash log\; \backslash pi\_\backslash theta(y\_l|x)$$

Putting it together:

$$\boxed{{\displaystyle {\mathrm{\nabla }}_{\theta }{\mathcal{L}}_{\text{DPO}}=-\beta \mathbb{E}\left[\underset{\text{weight}}{\underbrace{\sigma ({\widehat{r}}_{\theta }(x,y_l)-{\widehat{r}}_{\theta }(x,y_w))}}(\underset{\text{increase }{y}_w}{\underbrace{{\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y_w\mathrm{\mid }x)}}-\underset{\text{decrease }{y}_l}{\underbrace{{\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y_l\mathrm{\mid }x)}})\right]}}\backslash boxed\{\backslash nabla\_\backslash theta\; \backslash mathcal\{L\}\_\{\backslash text\{DPO\}\}\; =\; -\backslash beta\; \backslash mathbb\{E\}\backslash left[\backslash underbrace\{\backslash sigma\backslash left(\backslash hat\{r\}\_\backslash theta(x,\; y\_l)\; -\; \backslash hat\{r\}\_\backslash theta(x,\; y\_w)\backslash right)\}\_\{\backslash text\{weight\}\}\backslash left(\backslash underbrace\{\backslash nabla\_\backslash theta\; \backslash log\; \backslash pi\_\backslash theta(y\_w|x)\}\_\{\backslash text\{increase\; \}\; y\_w\}\; -\; \backslash underbrace\{\backslash nabla\_\backslash theta\; \backslash log\; \backslash pi\_\backslash theta(y\_l|x)\}\_\{\backslash text\{decrease\; \}\; y\_l\}\backslash right)\backslash right]\}$$

• ${\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y_w\mathrm{\mid }x)\backslash nabla\_\backslash theta\; \backslash log\; \backslash pi\_\backslash theta(y\_w|x)$ points in the direction that increases probability of the preferred response
• $-{\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y_l\mathrm{\mid }x)-\backslash nabla\_\backslash theta\; \backslash log\; \backslash pi\_\backslash theta(y\_l|x)$ points in the direction that decreases probability of the dispreferred response
• The weight term is high when ${\widehat{r}}_{\theta }(x,y_l)>{\widehat{r}}_{\theta }(x,y_w)\backslash hat\{r\}\_\backslash theta(x,\; y\_l)\; >\; \backslash hat\{r\}\_\backslash theta(x,\; y\_w)$, i.e. when the model currently assigns higher implicit reward to the losing response than the winning response. In other words:
  • When the model is wrong (ranks $y_{l}y\_l$ above $y_{w}y\_w$), we get large gradient updates
  • When the model is right (ranks $y_{w}y\_w$ above $y_{l}y\_l$), we get small gradient updates

This dynamic sigmoid weighting is crucial. It naturally focuses learning on the examples the model currently gets wrong.

VII: Computing Log Probabilities in Practice

This section is fully adapted from Umar Jamil's video. I think it is essential to understand how log probabilities are computed in practice.

The DPO loss requires computing $\log {\pi }_{\theta }(y\mathrm{\mid }x)\backslash log\; \backslash pi\_\backslash theta(y|x)$, the log probability of a complete response $yy$ given a prompt $xx$. Let's see how this works in practice with LLMs.

Language models are autoregressive: they generate text one token at a time, conditioning on all previous tokens. For a response $y=(y_1,y_2,\dots ,y_T)y\; =\; (y\_1,\; y\_2,\; \backslash ldots,\; y\_T)$, the probability factorizes as:

Taking the logarithm:

The log probability of the full response is the sum of log probabilities at each position.

Here's how to compute $\log {\pi }_{\theta }(y\mathrm{\mid }x)\backslash log\; \backslash pi\_\backslash theta(y|x)$:

1. Prepare input: Concatenate the prompt and response into a single sequence

   $\text{input}=[x_1,x_2,\dots ,x_n,y_1,y_2,\dots ,y_T]\backslash text\{input\}\; =\; [x\_1,\; x\_2,\; \backslash ldots,\; x\_n,\; y\_1,\; y\_2,\; \backslash ldots,\; y\_T]$

2. Forward pass: Run the transformer to get hidden states at each position

3. Project to logits: Apply the language model head (typically a linear layer) to get vocabulary logits at each position

4. Log softmax: Convert logits to log probabilities over the vocabulary using logsoftmax

5. Gather relevant log probs: For each position $tt$ in the response extract the log probability of the actual next token $y_{t}y\_t$ (since we know the output)

6. Sum with masking: Sum the log probabilities but only for response tokens (not prompt tokens)

This gives us $\log {\pi }_{\theta }(y\mathrm{\mid }x)\backslash log\; \backslash pi\_\backslash theta(y|x)$ for one response. We do this for both the preferred response $y_{w}y\_w$ and the dispreferred response $y_{l}y\_l$ and we also do it for both the policy model ${\pi }_{\theta }\backslash pi\_\backslash theta$ and the frozen reference model ${\pi }_{\text{ref}}\backslash pi\_\{\backslash text\{ref\}\}$. With these four log probabilities in hand, we can compute the DPO loss.

Conclusion

Once you derive the DPO loss, you start appreciating the simplicity and elegance of the solution especially when compared to PPO. The derivation hinges on one observation that the Bradley-Terry model only cares about reward differences and this causes the intractable partition function from analytical solution to cancel out completely. In turn, what remains is a straightforward classification loss.

References

• Papers:

  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model: The original DPO paper
  • Training language models to follow instructions with human feedback: The InstructGPT paper that established PPO-based RLHF

• Videos:

  • Umar Jamil's video on DPO: Excellent walkthrough of the DPO derivation