1^st International Workshop on Data FORMATS for Modern Architectures and Workloads

Abstract. Modern AI training pipelines demand efficient data loading and preparation; the storage layout layer plays an important role in the performance of both the read and write paths. This talk explores challenges in designing and evolving storage formats optimized for training workloads, discusses feature storage and processing, implications of wide tables, and challenges with normalization. It also presents Nimble, Meta's columnar file format, some of its main design decisions, features, and discusses future challenges and areas of exploration.

About the speaker. Pedro Pedreira is a Software Engineer at Meta, where he has spent over 13 years working on large-scale compute, storage, and query processing systems. Pedro leads Velox, a cross-organizational effort involving 20+ companies aimed at unifying execution engines using an open-source library, in addition to a variety of related efforts aimed at modernizing compute engines, more recently focused on AI training. Prior to Velox, Pedro led the creation of Cubrick and has contributed to a series of data open source infrastructure projects including Presto, Spark, Gluten, Nimble, Arrow, and others. His work focuses on building high-performance, composable execution libraries that serve as the foundation for analytical and AI workloads at scale.

Abstract. The lakehouse promise is simple: persist data once in an open format, query it from anywhere. Lakehouses are a real step forward — open formats, ACID transactions, multi-engine access. But persisting data once is solved. It breaks when that data gets served.

In open lakehouses, there is no narrow API enforcing correctness inline. The data plane is format libraries embedded in N engines; the catalog sees metadata but never the bytes. Every gap in this architecture compounds silently at scale.

We describe five research frontiers surfaced by operating OpenHouse — LinkedIn's open-source declarative control plane for Apache Iceberg — across 300,000+ tables and over one exabyte of managed data. (1) Interoperability: the same bits on disk can look different when queried from different engines — types, schemas, and semantics diverge silently across the stack. (2) The online/offline divide: the industry-wide cost of duplicating data between operational and analytical systems via CDC pipelines, and the path toward WAL-centric convergence. (3) Agent guardrails: as LLM-driven agents write to production data, data engineering needs the same branch-validate-merge safety model that software engineering solved decades ago. (4) Data layout optimization: compaction, sort order, partitioning, and write throughput are coupled objectives with no single global optimum, building on our AutoComp framework (SIGMOD '25). (5) Incremental processing: real-time freshness is well-served by Kafka stream processing, and completeness is well-served by Spark batch over verified data — but the middle ground of near-real-time freshness remains underserved, where streaming becomes prohibitively expensive and batch lacks the tooling to consume incremental deltas efficiently.

For each frontier, we present the problem at production scale and pose open research questions for the data community.

About the speaker. Sumedh Sakdeo is a Principal Staff Software Engineer at LinkedIn. He has 15+ years of experience working in data infrastructure. He previously contributed to a log-structured file system at Tintri and led data infrastructure, engineering, and visualization for Lyft's self-driving division. Notably, at LinkedIn, he spearheaded the development of OpenHouse, a pioneering control plane for open-source data lakehouse deployments.