Nicholas E. Corrado

TLDR: This paper answers the question: Under what conditions will model-free reinforcement learning give rise to thinking as a strategy for reward maximization?

Another special thanks to Julian Katz-Samuels. Much of what I know about LLMs comes from working with him, and that foundation made it easy to contribute to this project.

TLDR: We identify a subtle failure mode of independent on-policy MARL: on-policy sampling can produce data that deviates from the expected joint on-policy distribution, yielding inaccurate gradients that can make agents converge suboptimally—even when the expected gradient of each agent aligns with optimal behavior. We introduce adaptive sampling algorithm that reduces this sampling error w.r.t. the joint on-policy distribution, enabling agents to more reliability converge to an optimal equilibria.

TLDR: Naively aligning LLMs across many datasets each targeting different tasks often yields a model that performs well on some tasks but not others. We introduce AutoMixAlign, a theoretically-grounded data mixing algorithm that adaptively mixes datasets during training to balance performance across tasks.

Special thanks to Julian Katz-Samuels, my mentor at Amazon. I came into this project with no prior LLM experience, and Julian's guidance, support, and confidence in me made all the difference.

TLDR: On-policy learning requires on-policy data, not on-policy sampling! We introduce an adaptive, off-policy sampling algorithm that produces on-policy data more efficiently than on-policy sampling, improving data efficiency.

TLDR: While offline RL algorithms can in principle learn from highly suboptimal data, they nevertheless perform much better with near expert-quality data. Taking a "data first" perspective, we introduce a data augmentation framework that automatically generates near expert-quality synthetic data.

Special thanks to John U. Balis and Adam Labiosa for helping with the physical robotics experiments.

TLDR: Several prior works introduce data augmentation techniques that improve the data efficiency of RL. Rather than proposing yet another method, we ask: when and why does data augmentation help? Our work offers practical insights to help RL practitioners apply data augmentation more effectively.