Language models (LMs) are increasingly used to assist users in day to day tasks such as programming (Github Copilot) or search (Google's AI Overviews). But can we build language model systems that are able to autonomously complete entire tasks end-to-end? In this talk I'll discuss our efforts to build autonomous LM systems, focusing on the software engineering domain. I'll present SWE-bench, our novel method for measuring the performance of automatic programming systems on their abilities to fix real issues in popular software libraries from GitHub. I'll then discuss SWE-agent, our system for solving SWE-bench tasks. SWE-bench and SWE-agent are used by many leading AI orgs in academia and industry including OpenAI, Anthropic, Meta, and Google, and these projects show that academics on tight budgets are able to have substantial impact on steering the research community towards building autonomous systems that can complete challenging tasks.
Short bio:
Ofir Press is a postdoc at Princeton University. I previously completed my PhD at the University of Washington in Seattle, where I was advised by Noah Smith. During my PhD I spent two years at Facebook AI Research Labs.