Colloquia and Seminars

In this talk, I present an approach for interpreting the internal computation in deep vision models. I show that these interpretations can be used to detect model bugs and to improve the performance of pre-trained deep neural networks (e.g., reducing hallucinations from image captioners and detecting and removing spurious correlations in CLIP) without any additional training. Moreover, the obtained understanding of deep representations can unlock new model capabilities (e.g., novel identity editing techniques in diffusion models and faithful image inversion in GANs). I demonstrate how to find common representations across different models (discriminative and generative) and how deep representations can be adapted at test-time to improve model generalization without any additional supervision. Finally, I discuss future work on improving the presented interpretation techniques and their application to continual model correction and scientific discovery.

Bio:
Yossi is a EECS PhD at UC Berkeley, advised by Alexei Efros, and a visiting researcher at Meta. Before that, he was a member of the perception team at Google Research (now Google-DeepMind). He completed his M.Sc. at Weizmann Institute, advised by Prof. Michal Irani. His research centers around deep learning, computer vision, and mechanistic interpretability.

Image generative models are advancing rapidly, producing images of remarkable realism and fidelity. However, existing models often lack precise control over the generated content, limiting their image editing capabilities and the integration of real content into synthesized imagery. In this talk, I will demonstrate how a deep understanding of the inner mechanisms of large-scale pretrained generative models enables the design of powerful techniques for a variety of image manipulation tasks. By analyzing the semantic representations learned by these models, I will present methods that enable effective content editing. Additionally, I will discuss the challenges and trade-offs involved in manipulating real content and propose strategies to address these challenges. Finally, I will highlight recent advancements in incorporating real content, with a particular focus on techniques for injecting information into pretrained models.

Bio: Or Patashnik (https://orpatashnik.github.io/) is a Computer Science PhD candidate at Tel Aviv University, supervised by Daniel Cohen-Or. Her research focuses on computer graphics and its intersection with computer vision, with an emphasis on generative tasks such as image editing, personalization, and image inversion using large-scale pretrained models. Recently, she has been particularly interested in better understanding diffusion models for various applications.

Language models (LMs) are increasingly used to assist users in day to day tasks such as programming (Github Copilot) or search (Google's AI Overviews). But can we build language model systems that are able to autonomously complete entire tasks end-to-end? In this talk I'll discuss our efforts to build autonomous LM systems, focusing on the software engineering domain. I'll present SWE-bench, our novel method for measuring the performance of automatic programming systems on their abilities to fix real issues in popular software libraries from GitHub. I'll then discuss SWE-agent, our system for solving SWE-bench tasks. SWE-bench and SWE-agent are used by many leading AI orgs in academia and industry including OpenAI, Anthropic, Meta, and Google, and these projects show that academics on tight budgets are able to have substantial impact on steering the research community towards building autonomous systems that can complete challenging tasks. 

Short bio:
Ofir Press is a postdoc at Princeton University. I previously completed my PhD at the University of Washington in Seattle, where I was advised by Noah Smith. During my PhD I spent two years at Facebook AI Research Labs. 

Digital pathology has emerged as a transformative field, enabling automated imaging and computational analysis of thin tissue biopsy slices or body fluids. These samples are typically stained to enhance contrast in biological structures and reveal their morphology under microscopic magnification. Digitally scanning these stained samples produces gigapixel-scale images, known as whole slide images (WSIs), which pose significant challenges for computational analysis using deep learning techniques. Variability in staining protocols and scanning devices across medical centers introduces inconsistencies in WSIs, while their large size imposes substantial computational constraints. Furthermore, most slides are annotated such that the pathologist's prognosis is provided as a holistic textual report at the patient level, rather than pinpointing the specific diagnostic regions in the image that indicate the medical estimate.

To address these challenges, we propose an end-to-end approach for WSI analysis. Our method involves self-supervised pretraining on patches extracted from a large collection of WSIs to learn generic and transferable feature representations without requiring manual annotations. Subsequently, we employ a Transformer-based architecture trained in a weakly supervised manner to effectively aggregate spatial and contextual information across image regions, producing accurate slide-level predictions.

We evaluated our proposed method on multiple WSI datasets for various clinically relevant molecular profiling tasks, such as determining hormonal receptor status in invasive breast cancer, demonstrating its effectiveness and generalizability. Leveraging recent advances in computer vision, our approach addresses the challenges of WSI analysis, providing insights for processing large and complex images. By enhancing robustness and predictive accuracy in various clinical tasks within computational histopathology, our approach has the potential to improve clinical decision-making, ultimately contributing to better patient outcomes.

In a recent work, Cormode, Dall'Agnol, Gur and Hickey (CCC, 2024) introduced the model of Zero-Knowledge Streaming Interactive Proofs (zkSIPs). Loosely speaking, such proof-systems enable a prover to convince a streaming verifier that the input x, to which it has read-once streaming access, satisfies some property, in such a way that nothing beyond the correctness of the claim is revealed. Cormode et al. also gave constructions of zkSIPs to some specific and notable problems of interest.

In this work, we advance the study of zero-knowledge proofs in the streaming model, by presenting protocols that are significantly more general and more secure. We use a definition of zero-knowledge that is a variation of that used by Cormode et al., which we find more appealing but is technically incomparable.

Our main result is a zkSIP for any NP relation, that can be decided by low-depth polynomial-size circuits. We emphasize that this is the first general purpose protocol in this model, which captures, as a special case, the problems considered by the prior work. We also construct a specialized protocol for the ``polynomial evaluation'' problem considered in that work, with improved parameters.

The protocols constructed by Cormode et al. have an inverse polylogarithmic simulation error (i.e., a gap with which a bounded-space distingiusher can distinguish the simulation from a real execution). This means that their protocols are entirely insecure if run multiple times (say on different inputs). In contrast, our protocols achieve a negligible zero-knowledge error, a stronger and far more robust security guarantee.

Over the past decade, several studies have shown that DNA-based storage systems can potentially become the standard for data archival due to their high data density and durability. However, the current bottleneck involves the synthesis and sequencing costs, along with a lack of coding solutions to address the unique error characteristics of DNA-based systems.

This work tackles multiple challenges that hinder the practical implementation of DNA storage. First, we explore theoretical aspects of the deletion channel, presenting detailed findings from the maximum likelihood decoder for both single and dual-channel outputs. Next, we address the DNA reconstruction problem, aiming to accurately reconstruct a DNA sequence from multiple noisy copies. We propose several reconstruction algorithms that significantly enhance accuracy compared to previously published approaches. Furthermore, we investigate two novel synthesis methods, the composite synthesis and the combinatorial composite synthesis, highlighting their potential benefits and inherent complexities. These methods require innovative algorithmic and coding solutions, and thus we design error-correction codes specifically tailored for these technologies.

Finally, we introduce the DNA storalator, a software tool designed to simulate the biological and computational processes of DNA storage, aiding our research and facilitating further exploration within the scientific community. Overall, the results presented in this work advance several aspects of DNA data storage and promote the feasibility of this storage solution further.

Edge computing extends cloud capabilities to the proximity of end-users, offering ultra-low latency, which is essential for real-time applications. Unlike traditional cloud systems that suffer from latency and reliability constraints due to distant datacenters, edge computing employs a distributed model, leveraging local edge datacenters to process and store data.

This talk explores key challenges in edge computing across three domains: workloads, storage, and service allocation. 
The first part focuses on the absence of comprehensive edge workload datasets. Current datasets do not accurately reflect the unique attributes of edge systems. To address this, we propose a workload composition methodology and introduce WoW-IO, an open-source trace generator. The second part examines aspects of edge storage. Edge datacenters are significantly smaller than their cloud counterparts and require dedicated solutions. We analyze the applicability of a promising mathematical model for edge storage systems and raise inherent gaps between theory and practice. The final part addresses the virtual network embedding problem (VNEP). In VNEP, given a set of requests for deploying virtualized applications, the edge provider has to deploy a maximum number of them to the underlying physical network, subject to capacity constraints. We propose novel solutions, including a proactive service allocation strategy for mobile users, a flexible algorithm for service allocation that is adaptable to the underlying physical topology, and  an algorithm for scalable online service allocation.

Labeling schemes are a prevalent paradigm in various computing settings. In such schemes, an oracle is given an input graph and produces a label for each of its nodes, enabling the labels to be used for various tasks. In this talk, I will address the question of what happens in a labeling scheme if some labels are erased, e.g., due to communication loss with the oracle or hardware errors. I will present a new resilient labeling scheme which improves upon the state of the art in several computational aspects and I will show that it is nearly optimal.