The Taub Faculty of Computer Science Events and Talks

This talk will focus on structural representations and neural models of source code. I will present a language-agnostic approach for structural language modeling (SLM) of code. This general approach obtains state-of-the-art results in a variety of tasks including code summarization, code captioning, code completion, name prediction, and edit completion, outperforming sequence models (such as textual Transformers and LSTMs) and models based on graph neural networks (GNNs). Studying the reason why GNNs do poorly compared to SLM exposed a fundamental bottleneck that results in a phenomenon that we call "over-squashing". This bottleneck of GNNs provides a new perspective on a phenomenon that was observed for years and still affects existing GNN-based models. I will explain the GNN bottleneck problem and demonstrate its practical implications. The talk summarizes my PhD dissertation, which has been presented in both machine learning and programming language conferences.