报告题目: On the Diverse Dynamical Behavior Arising in Deep Linear Transformers
报告人：郦思旭（Johns Hopkins University）
邀请人&主持人：黄辉
报告时间与地点：9月18日 10 am， 腾讯会议：936-386-914
报告摘要: The transformer architecture has become a fundamental component of modern machine learning and artificial intelligence, with the self-attention mechanism serving as one of its core computational components. Motivated by the broader goal of developing a mathematical understanding of this mechanism, we study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems.
In the first part of the talk, I will show that, in embedding dimension
d=2 and for arbitrary self-attention parameter matrices, the resulting dynamics can be reformulated as a generalized Kuramoto-type model with pure second-harmonic coupling. This connection makes the system amenable to Watanabe--Strogatz theory and reveals an intrinsic low-dimensional structure that is independent of the parameter choices. For a class of token initializations associated with the Ott--Antonsen (OA) manifold, we show that different parameter regimes give rise to a rich variety of long-time behaviors, including clustering, oscillations, and bifurcations. We further establish a structural stability result showing that some of these qualitative behaviors persist for initializations near the OA manifold. Numerical experiments suggest that the behaviors identified theoretically in dimension two also persist in higher-dimensional linear transformers.
In the second part of the talk, I will present new analytical techniques that extend this framework to higher dimensions, providing a route toward the systematic analysis of higher-dimensional linear self-attention dynamics.
报告人介绍：郦思旭目前是约翰斯·霍普金斯大学数学系的博士后研究员。他本科毕业于南京大学，随后在威斯康星大学麦迪逊分校获得统计学博士学位。
他的研究兴趣主要集中在相互作用粒子系统与机器学习数学基础的交叉领域。具体而言，他致力于运用偏微分方程分析、动力系统以及平均场分析等数学工具，来探究现代机器学习模型与算法背后的底层数学结构。
