Numerical Considerations

Differentiating an optimizer is more sensitive than evaluating its objective or primal solution. Nearly identical solutions can still produce different gradients near non-unique optima or active-set transitions.

Local solution sensitivity

KKT-based derivatives are most reliable when:

• the forward problem is feasible and bounded;
• the primal-dual solution is sufficiently accurate;
• the optimizer is locally unique;
• active constraint gradients are independent;
• the active set is stable under small parameter changes.

When these conditions fail, the solution map may be set-valued or nondifferentiable. A singular adjoint system can therefore reflect the underlying optimization problem rather than a software error.

QP and LP behavior

Positive curvature in a strictly convex QP helps select a unique primal solution. An LP has \(P=0\) and may instead have an entire face of optimal solutions. Different solvers can then return similar objective values but different optimal points, active sets, or gradients.

The optimal value can remain stable even when the optimizer is not unique. When a loss needs only that value, the value-function layer avoids the additional solution-map adjoint system.

Quadratic regularization

SolMapLayer can replace the original objective with

\[ f_{0,\epsilon}(z,\hat y) = f_0(z,\hat y) +\frac{\epsilon}{2}\sum_j\lVert z_j\rVert_2^2, \qquad \epsilon\ge0. \]

For an LP, a small \(\epsilon>0\) introduces curvature and can select a more stable optimizer. Regularization changes the mathematical problem, however, so the returned solution \(\hat z_\epsilon^\star\) and its derivative belong to the regularized problem.

Use regularization_eps deliberately and report it with experimental results. See the SolMapLayer constructor for the exact interface.

Active-set sensitivity

Reduced-KKT differentiation depends on identifying which inequalities are active at \(\tilde z^\star\). A tolerance that is too small can omit a binding constraint because of solver residuals, while a tolerance that is too large can include nearly active or dependent rows.

DiffAPQP exposes act_ineq_tol and act_ineq_tol_rel for this purpose. Their defaults and precedence are documented under SolMapLayer adjoint configuration. The underlying reduced system is derived in Solution-Map Differentiation.

Full versus reduced KKT

Formulation	Main benefit	Main sensitivity
Full KKT	Does not require active-set selection	Complementarity rows can be poorly conditioned
Reduced KKT	Smaller symmetric system	Depends on correct active-set identification

Agreement between the two formulations is useful evidence at a regular point. Disagreement should be interpreted together with solver accuracy, active-set stability, and possible non-uniqueness.

Practical guidance

For a new application:

1. Verify feasibility, objective values, and returned decisions.
2. Compare full and reduced KKT on a small problem.
3. Check gradients with finite differences where practical.
4. For LPs, compare unregularized and clearly reported regularized variants.
5. Tighten forward or adjoint tolerances only when the application requires it.

Solver defaults are listed on the solver page, while the exact adjoint options are documented in the SolMapLayer reference.