ARC-TS Data Services Brock Palen | @brockpalen | brockp@umich.edu - PowerPoint PPT Presentation

ARC-TS Data Services Brock Palen | @brockpalen | brockp@umich.edu http://arc-ts.umich.edu | hpc-support@umich.edu

In Memory In Server Big Data ● Small to modest data ● Small to medium data ● Medium to huge data ● Interactive or batch work ● Interactive or batch work ● Batch work ● Might have many ● Hosted/shared and ● Full table scans thousands of jobs transactional data ● Hadoop, Spark, Flink ● Excel, R, SAS, Stata, ● SQL / NoSQL ● Presto, HBase, Impala SPSS ● Hosted data pipelines ● iRODS / Globus ● Document databases

Flux & Armis - High Performance Computing Batch and Interactive High Performance Platform supporting serial to highly parallel jobs Resources Key Features Software and Development 20,000+ CPU Cores 40 or 100gbit/s InfiniBand Over 100 Titles MATLAB, Mathematica Upto 1.5TB Shared Memory 1.5PB Scratch Filesystem FEA/CAE Abaqus, Ansys Upto 54 Core Shared Memory HIPAA/PHI Acceptance (Armis) SAS, STATA/MP, R Nvidia GPU’s, Intel Xeon PHI’s 160Gig Backbone Connection Compilers, Debuggers, Libs Free to Undergrads 4

https://connect.arc-ts.umich.edu/

Coming Soon: Bigger Big Data ● ~5000 CPUs ● ~24TB Memory ● ~ 3PB of HDFS/Storage ● 6x the network speed of the backbone Software ● Hadoop ● Spark ● Presto ● Etc. (Researcher Driven)

Storage Storage Options across data types Turbo Locker Data Den High speed storage with solid Cost Optimized Large File Extreme low cost state memory cache Storage Onetime costs for data protection and archive Optional Replication and Early Phase 200TB+ Snapshots Late Phase Winter 2017 7

Network Connection Improvement Program Provide uninterrupted data flow to meet needs of researchers across campus Key Features Features Ctd Partners Upgrade connections to 1, 10, and 40gbit/s network Unit IT support laboratory speeds possible ITS instruments and data flows ARC-TS representative will MSIS/MCIT help design workflows to Network need not be in achieve lab goals University building or on campus 8

Yottabyte Research Cloud - Databases and Services SQL, NoSQL, Columnar, Data Ingest - ARC-TS can consume your datasource Key Features Services Offered Services Ctd. Database/Service on demand MariaDB / PostgreSQL Impala Hardware scales as needed Redis / Kafka Hbase Elasticsearch Cassandra Faculty feedback determines InfluxDB / Grafana iRods the services we offer Apache Storm MongoDB 9

Example Workflow - Analytics ● Transportation ● Social Surveys ● Remote Sensor Nets (IOT) ● Bedside Monitors ● Remote Imaging ● Telescope Networks ● Finance and Marketing ● Social Media Mining ● Machine Learning Prediction ● Network Security Prediction

Thank You - Contact • hpc-support@umich.edu • http://arc-ts.umich.edu/ • @ARCTS_UM • brockp@umich.edu • http://myumi.ch/aV7kz 11