Large atomic data in R: package 'ff' Adler, Oehlschlgel, Nenadic, - PowerPoint PPT Presentation

DISCLOSED Large atomic data in R: package 'ff' Adler, Oehlschlägel, Nenadic, Zucchini Göttingen, Munich August 2008 This report contains public intellectual property. It may be used, circulated, quoted, or reproduced for distribution as a whole. Partial citations require a reference to the author and to the whole document and must not be put into a context which changes the original meaning. Even if you are not the intended recipient of this report, you are authorized and encouraged to read it and to act on it. Please note that you read this text on your own risk. It is your responsibility to draw appropriate conclusions. The author may neither be held responsible for any mistakes the text might contain nor for any actions that other people carry out after reading this text.

SUMMARY A proof of concept for the 'ff' package has won the large data competition at useR!2007 with its C++ core implementing fast memory mapped access to flat files. In the meantime we have complemented memory mapping with other techniques that allow fast and convenient access to large atomic data residing on disk. ff stores index information efficiently in a packed format, but only if packing saves RAM. HIP (hybrid index preprocessing) transparently converts random access into sorted access thereby avoiding unnecessary page swapping and HD head movements. The subscript C-code directly works on the hybrid index and takes care of mixed packed/unpacked/negative indices in ff objects; ff also supports character and logical indices. Several techniques allow performance improvements in special situations. ff arrays support optimized physical layout for quicker access along desired dimensions: while matrices in the R standard have faster access to columns than to rows, ff can create matrices with a row-wise layout and arbitrary 'dimorder' in the general array case. Thus one can for example quickly extract bootstrap samples of matrix rows. In addition to the usual '[' subscript and assignment '[<-' operators, ff supports a 'swap' method that assigns new values and returns the corresponding old values in one access operation - saving a separate second one. Beyond assignment of values, the '[<-' and 'swap' methods allow adding values (instead of replacing them). This again saves a second access in applications like bagging which need to accumulate votes. ff objects can be created, stored, used and removed, almost like standard R ram objects, but with hybrid copying semantics, which allows virtual 'views' on a single ff object. This can be exploitet for dramatic performance improvements, for example when a matrix multiplication involves a matrix and it's (virtual) transpose. The exact behavior of ff can be customized through global and local 'options', finalizers and more. The supported range of storage types was extended since the first release of ff, now including support for atomic types 'raw', 'logical', 'integer' and 'double' and ff data structures 'vector' and 'array'. A C++ template framework has been developed to map a broader range of signed and unsigned types to R storage types and provide handling of overflow checked operations and NAs. Using this we will support the packed types 'boolean' (1 bit), 'quad' (2 bit), 'nibble' (4 bit), 'byte' and 'unsigned byte' (8 bit), 'short', 'unsigned short' (16 bit) and 'single' (32bit float) as well as support for (dense) symmetric matrices with free and fixed diagonals. These extensions should be of some practical use, e.g. for efficient storage of genomic data (AGCT as.quad) or for working with large distance matrices (i.e. symmetric matrices with diagonal fixed at zero). Source: Adler, Oehlschlägel, Nenadic, Zucchini (2008) Large atomic data in R: package 'ff' 1

FF 2.0 DESIGN GOALS: BASE PACKAGE FOR LARGE DATA • large objects (size > RAM and virtual address space large data limitations) • many objects (sum(sizes) > RAM and …) • single disk (or enjoy RAID) • single processor (or shared processing) standard HW • limited RAM (or enjoy speedups) • required RAM << maximum RAM minimal RAM • be able to process large data in background • close to in-RAM performance if size < RAM (system cache) maximum • still able to process if size > RAM performance • avoid redundant access Source: Adler, Oehlschlägel, Nenadic, Zucchini (2008) Large atomic data in R: package 'ff' 2

A SHORT FF DEMO library(ff) ffVector <- ff(0:1, length=36e6) # 0,1,0,… 4 byte integers ffVector ffMatrix <- ff(vmode="logical", dim=c(6e3,6e3)) # 2 bit logical ffMatrix ffPOSIXct <- ff(Sys.time(), length=36e6) # 8 byte double ffPOSIXct bases <- c("A","T","G","C") ffFactor <- ff("A", levels=bases, length=400e6 # 2 bit quad , vmode="quad", filename="QuadFactorDemo.ff", overwrite=TRUE) # 95 MB with quad instead of 1.5 GB with integer ffFactor # accessing parts based on memory mapping and OS file caching ffFactor[3:400e6] <- c("A","T") # quick recycling at no RAM ffFactor[1:12] Source: Adler, Oehlschlägel, Nenadic, Zucchini (2008) Large atomic data in R: package 'ff' 3

SUPPORTED DATA TYPES on CRAN prototype available vmode(x) not yet implemented boolean 1 bit logical without NA logical 2 bit logical with NA quad 2 bit unsigned integer without NA # example x <- ff(0:3 nibble 4 bit unsigned integer without NA , vmode= " quad " ) byte 8 bit signed integer with NA ubyte 8 bit unsigned integer without NA short 16 bit signed integer with NA ushort 16 bit unsigned integer without NA integer 32 bit signed integer with NA single 32 bit float double 64 bit float Compounds factor complex 2x64 bit float ordered raw 8 bit unsigned char POSIXct POSIXlt character fixed widths, tbd. Source: Adler, Oehlschlägel, Nenadic, Zucchini (2008) Large atomic data in R: package 'ff' 4

SUPPORTED DATA STRUCTURES on CRAN prototype available not yet implemented example class(x) vector ff(1:12) c("ff_vector","ff") array ff(1:12, dim=c(2,2,3)) c("ff_array","ff") c("ff_matrix","ff_array","ff") matrix ff(1:12, dim=c(3,4)) ff(1:6, dim=c(3,3) c("ff_symm","ff") symmetric matrix with free diag , symm=TRUE, fixdiag=NULL) symmetric matrix ff(1:3, dim=c(3,3) with fixed diag , symm=TRUE, fixdiag=0) distance matrix c("ff_dist","ff_symm","ff") mixed type arrays c("ff_mixed", "ff") instead of data.frames Source: Adler, Oehlschlägel, Nenadic, Zucchini (2008) Large atomic data in R: package 'ff' 5

SUPPORTED INDEX EXPRESSIONS implemeneted not implemented x <- ff(1:12, dim=c(3,4), dimnames=list(letters[1:3], NULL)) Example expression positive integers x[ 1 ,1] negative integers x[ -(2:12) ] x[ c(TRUE, FALSE, FALSE) ,1] logical x[ "a" ,1] character integer matrices x[ rbind(c(1,1)) ] hybrid index x[ hi ,1] zeros x[ 0 ] NAs x[ NA ] Source: Adler, Oehlschlägel, Nenadic, Zucchini (2008) Large atomic data in R: package 'ff' 6

FF DOES SEVERAL ACCESS OPTIMIZATIONS R frontend C interface C++ backend Hybrid Index Fast Memory Mapped Preprocessing ... access methods … Pages … • Tunable pagesize and • HIP • C-code accelerating system caching= – parsing of index is.unsorted() and rle() for integers: intisasc(), expressions instead of c("mmnoflush", memory consuming intisdesc(), "mmeachflush") • Custom datatype bit- intrle() evaluation • C-code for looping over level en/decoding , – ordering of access ‚ add ‘ arithmetics and NA positions and re-ordering hybrid index can handle of returned values mixed raw and rle packed handling • Ported to Windows, Mac – rapid rle packing of indices in arrays indices if and only if rle OS, Linux and BSDs representation uses less • Large File Support memory compared to raw (>2GB) on Linux storage • Paged shared memory • Hybrid copying semantics allows parallel – virtual dim/dimorder() processing – virtual windows vw() • Fast creation of large – virtual transpose vt() files • New generics – clone(), update(), swap(), add() Source: Adler, Oehlschlägel, Nenadic, Zucchini (2008) Large atomic data in R: package 'ff' 7

DOUBLE VECTOR CHUNKED SEQUENTIAL ACCESS TIMINGS [sec] plain R bigmemory ff2.0 ff1.0 R.huge 76 MB read by 1e6 of 1e7 0,3 4,5 0,40 0,25 165,0 76 MB write 0,3 1,1 0,20 0,70 110,0 0,75 GB read by 1e6 of 1e8 2,5 42,5 4,00 1,97 1600,0 0,75 GB write 2,5 12,3 2,00 7,57 1150,0 3,50 GB read by 1e6 of 4*1e8 failed crashed 99,78 90,00 skipped 3,50 GB write : : 188,16 420,00 : 7,50 GB read by 1e6 of 1e9 : : 229,00 skipped : 7,50 GB write : : 916,00 : : as fast as faster than in-memory older disk methods methods * HP nc6400 Notebook 2GB RAM, Windows XP, x86 dual core ~2327 Mhz (of which 50% is used) Source: Adler, Oehlschlägel, Nenadic, Zucchini (2008) Large atomic data in R: package 'ff' 8