Combining R with Scalable Libraries to Get the Best of Both for Big Data

The R programming language is known for its diversity and sophistication in data analysis, however its scalability to big data has been lacking. Our project ”Programming with Big Data in R” (pbdR) is adding scalability to its list of data virtues. We have developed several packages that provide a tight coupling of R with highly scalable libraries, enabling scalability to terabytes of data on tens of thousands of cores. We also added classes and methods to handle distributed data objects needed by the libraries so that the R language syntax is largely unchanged. Our philosophy is that the R developer is not asked to deal with the details of managing the data distribution and processor communication but the developer is asked to be aware of the data distribution and provided with high-level functions to manage it if needed. Many R functions are already instrumented to handle the distributed data classes. We encourage developers of compute intensive R packages to use pbdR methods for scalability to bigger data and to bigger computing platforms.