The Orthogonal Alignment

Align the embedding spaces of two languages. Make French vectors and English vectors point in the same directions for the same concepts. Now transfer a model trained on English to French. The transfer should improve — the representations are more similar.

It doesn’t. Or rather, it doesn’t reliably. Veitsman et al. show that cross-lingual alignment and task performance are “largely orthogonal.” The gradients of the alignment objective and the task objective point in nearly perpendicular directions. Improving one contributes minimally to the other.

The intuition that failed: similar representations should produce similar task behavior. This holds when the task depends on the same features that alignment preserves. For POS tagging — where the relevant features are syntactic and relatively universal — alignment sometimes helps. For sentence classification — where the relevant features are semantic and culturally variable — alignment is irrelevant or harmful.

The independence is not approximate. The gradients are close to orthogonal, meaning the optimization landscapes of alignment and task performance are genuinely separate. They share the same parameter space but occupy different dimensions of it. Moving toward better alignment moves sideways relative to better task performance, and vice versa.

The structural lesson: representation similarity is not a universal proxy for functional similarity. Two systems can represent the same concepts in the same geometric arrangement and still use those representations differently. The map can be identical while the territory of use diverges. Alignment measures the geometry of the space; task performance measures the dynamics in the space. Geometry and dynamics are independent unless the task is geometrically defined.