Making Sense of Big Data

Hadoop is currently the large-scale data analysis “hammer” of choice, but there exist classes of algorithms that aren’t “nails” in the sense that they are not particularly amenable to the MapReduce programming model. To address this, researchers have proposed MapReduce extensions or alternative programming models in which these algorithms can be elegantly expressed. Th is article espouses a very diff erent position: that MapReduce is “good enough,” and that instead of trying to invent screwdrivers, we should simply get rid of everything that’s not a nail. To be more specifi c, much discussion in the literature surrounds the fact that iterative algorithms are a poor fi t for MapReduce. Th e simple solution is to fi nd alternative, noniterative algorithms that solve the same problem. Th is article captures my personal experiences as an academic researcher as well as a soft ware engineer in a “real-world” production analytics environment. From this combined perspective, I refl ect on the current state and future of “big data” research.