Kairong Luo

Kairong Luo

1st year Phd student at Tsinghua University

CV

Education

Publications

  1. A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules
    Kairong Luo*, Haodong Wen, Shengding Hu, Zhenbo Sun, Zhiyuan Liu, Maosong Sun†, Kaifeng Lyu†, Wenguang Chen†
    • Accepted by ICLR 2025 (March 2025)
    • arXiv preprint
    • Summary: This paper introduces an empirical law to predict the pretraining loss of large language models under various learning rate schedules (e.g., constant, cosine, step decay). The proposed multi-power law combines a power law based on the sum of learning rates with additional terms to account for loss reduction due to learning rate decay. Validated across multiple model sizes and architectures, this law accurately predicts loss curves for unseen schedules and helps identify optimal schedules that outperform widely used ones like cosine. The findings provide insights into pretraining dynamics and learning rate schedule design. The automatically discovered schedule resembles the Warmup-Stable-Decay (WSD) schedule but achieves slightly better performance.
  2. DreamFuser: Value-guided Diffusion Policy for Offline Reinforcement Learning
    Kairong Luo*, Caiwei Xiao*, Zhiao Huang, Zhan Ling, Yunhao Fang, Hao Su†
    • Status: Preprint / Under Review (November 2023)
    • OpenReview
    • Summary: DreamFuser is a trajectory-based value optimization approach that integrates diffusion-based trajectory learning with efficient Q-function learning. It addresses computational challenges in action sampling during training by leveraging the Generalized Noisy Action Markov Decision Process (GNMDP), which treats the diffusion denoising process as part of the MDP transition. Empirical results show DreamFuser outperforms existing diffusion policy algorithms, particularly in low-level control tasks, and matches or exceeds state-of-the-art methods on the D4RL benchmark. The work also highlights the computational and memory advantages of DreamFuser over traditional MDP-based diffusion policies.

Teaching

Service and Leadership