"Big Metadata": The Need for Principled Metadata Management in Big Data Ecosystems

Current big data ecosystems lack a principled approach to metadata management. This impedes large organizations' ability to share data and data preparation and analysis code, to integrate data, and to ensure that analytic code makes compatible assumptions with the data it uses. This use-case paper describes the challenges and an in-progress effort to address them. We present a real application example, discuss requirements for "big metadata" drawn from that example as well as other U.S. government analytic applications, and briefly describe an effort to adapt an existing open source metadata manager to support the needs of big data ecosystems.