1

We present the first pipeline for automatic, difficulty-controlled creation of uncheatable, realistic, unanswerable, and multi-hop queries (CRUMQs), adaptable to any corpus and domain. This offers a simple way to enhance benchmark difficulty and realism and drive development of more capable RAG systems.

Nugget-based evaluation methods have emerged as the standard for measuring relevance of long-form RAG responses. We argue that nuggets are valuable also for guiding retrieval and generation, and we present a nugget-centric RAG system that automatically constructs its own nugget bank and uses it as a control signal throughout the pipeline.

LLM benchmarks are essential for tracking progress and ensuring safety in AI, but most benchmarks don’t measure what matters, as suggested by our systematic review of 445 LLM benchmarks from top AI conferences. A taxonomy of these failures was therefore built and translated into an operational checklist to help future benchmark authors demonstrate construct validity.

Multi-document (MD) processing is crucial for LLMs to handle real-world tasks across large sets of documents. This paper introduces MDCure, an effective and scalable procedure for generating high-quality multi-document instruction tuning data to improve MD capabilities of any base LLM.

We define the Dynamical Chinese Restaurant Process (Dynamical CRP), a novel stochastic process that provides a non-stationary prior over cluster assignments and yields an efficient streaming variational inference algorithm. Experiments show the Dynamical CRP can be applied on diverse synthetic and real data with Gaussian and non-Gaussian likelihoods.

R-IBP is a novel recursive form of the Indian Buffet Process that makes feature models applicable to streaming data. It enables creation of new features online and in a probabilistic, principled manner. As a prior for feature models, R-IBP yields efficient inference over an unbounded number of latent features, with quasilinear average time complexity and logarithmic average space complexity.

The first experimentally quantitative index of metacognition of emotion. Advancing understanding of awareness toward subjective feelings.