BenchCouncil AIBench --- A Datacenter AI Benchmark Suite Wanling Gao, Fei Tang, Jianfeng Zhan http://www.benchcouncil.org/AIBench/index.html INSTITUTE O BenchCouncil OF C Bench’19, Denver, Colorado, USA COMPUTING T TECHNOLOGY
Why Datacenter AI? § AI is widely employed to augment Internet services processing images, video, speech, and audio n n There is an urgent need for datacenter AI benchmarks AIBench Bench’19
Challenge 1# Isolation n Confidential issues of workloads and datasets n Isolation ！ n There is no publicly available industry-scale Internet service benchmark AIBench Bench’19
Challenge 2# Microservice based Architecture n A collection of loosely coupled services n Various modules and complex execution path n massive scale and complex hierarchy of infrastructure Component 1 … Request st Resp sponse se Splitted Merged suboperations results Component n End-to-end benchmark that models the critical paths and primary modules is needed AIBench Bench’19
Challenge 3# Diversity of workloads and models Bianco, S., Cadene, R., Celona, L., and Napoletano, P. Benchmark analysis of representativ e deep neural network architectures . IEEE Access, 6:64270– 64277, 2018. AIBench Bench’19
Challenge 4# Domain-specific metrics n Time-to-accuracy n State-of-the-art accuracy n Throughput n Latency n Tail latency AIBench Bench’19
Challenge 5# More vs. Less n Workload characterization n SPECCPU 2017 (43), PARSEC3.0 (30), TPC-DS (99) n Performance ranking (Benchmarketing) n top500 AIBench Bench’19
Challenge 6# Inconsistency n Inconsistent benchmarking requirements Micro or Application Portability Reality benchmark ? Component ? Earlier stage of Later stage of architecture research architecture research AIBench Bench’19
Requirements n Industry-scale n critical paths and primary modules of business AI scenario • AI-related and non AI-related components n A modular framework design n Collectively as a whole end-to-end application n Individually as a micro or component benchmark n Representativeness and coverage n Diverse AI problem domains and datasets are needed AIBench Bench’19
Outline n AIBench Overview n Tasks, Models, Datasets, Metrics n How to use AIBench n Preliminary Results n Conclusion AIBench Bench’19
BenchCouncil AIBench n A Datacenter AI Benchmark Suite n Contributors: many companies and top universities • Alibaba, Microsoft, Paypal, Tencent, etc http://www.benchcouncil.org/AIBench/index.html Wanling Gao, Fei Tang, Lei Wang, Jianfeng Zhan, Chunxin Lan, Chunjie Luo, et al. AIBench: An Industry Standard Internet Service AI Benchmark Suite. Technical Report 2019. arXiv preprint arXiv:1908.08998. AIBench Bench’19
AIBench Overview The First end-to-end industry- n standard AI benchmark suite Industry-scale Internet services n • critical paths and primary modules • AI-related and non AI-related A highly extensible, configurable, and n flexible benchmark framework 16 prominent AI problem domains n Multiple loosely coupled modules n • Individually – Micro/Component benchmarks • Collectively – Application benchmarks AIBench Bench’19
Sixteen AI Problem Domains n Text Processing (4) n Text-to-Text translation, Text summarization, Learning to rank, Recommendation n Image Processing (8) n Image classification, Image generation, Image-to-text, Image-to-Image, Face embedding, Object detection, Image compression, Spatial transformer n Audio Processing (1) n Speech recognition n Video Processing (1) n Video prediction n 3D Data Processing (2) n 3D face recognition, 3D object reconstruction AIBench Bench’19
End-to-End: E-commerce Search Query generator ： simulate concurrent users and send query requests n Online Module ： personalized searching and recommendations n Offline Module ： a training stage to generate a learning model n Data storage module ： data storage, e.g., user database, product database n AIBench Bench’19
Component Benchmark (16) AIBench Bench’19
BenchCouncil International Competitions n DC-AI-C1 Image classification n DC-AI-C8 3D face recognition n DC-AI-C10 Recommendation n Competition papers are available soon! AIBench Bench’19
Image Classification n Extract different thematic classes within an image n a supervised learning problem to define a set of target classes and train a model to recognize n ResNet neural network, Dataset ： ImageNet2012, 100GB+ AIBench Bench’19
Image Generation n Mimic the distribution of data and generate image data n Dataset ： LSUN ， about million labelled image data n Model: WGAN algorithm AIBench Bench’19
Text-to-Text Translation n Translates text from one language to another n Model: Transformer n Dataset ： WMT English-German (4.5MB training text data) AIBench Bench’19
Image-to-Image n Converts an image from one representation of a specific scene to another scene or representation n Model: cycle- GAN algorithm n Datasets ： Cityscapes from 50+ cities （ 300MB ） AIBench Bench’19
Speech-to-Text n Recognizes and translates the spoken language to text n Model ： deep speech 2 n Dataset ： LibriSpeech, 1000+ hours‘ speech data AIBench Bench’19
Object Detection n Detects the objects within an image n Model ： Faster R-CNN algorithm n Dataset ： MSCOCO2014 • 82783 training samples, 40504 Validation samples, 40775 test samples （ 20GB+ ） AIBench Bench’19
Image-to-Text n Generates the description of an image automatically n Model ： Neural Image Caption model n Dataset ： MSCOCO2014 AIBench Bench’19
Face Embedding n Transforms a facial image to a vector in embedding space n Model ： FaceNet algorithm n Dataset ： VGGFace2 • 36GB training data ， 1.9GB test data AIBench Bench’19
3D Face Recognition n Recognize the 3D facial information from an image n Model ： 3D face models n Dataset ： Intellifusion data set ， 77,715 samples from 253 face IDs AIBench Bench’19
Video Prediction n Predicts the future video through predicting previous frames transformation n Model ： motion-focused predictive models n Dataset ： Robot pushing dataset • 59000 samples, 100GB+ AIBench Bench’19
Image Compression n Full-resolution lossy image compression n Model ： recurrent neural networks n Dataset ： ImageNet2012 ， 100GB+ AIBench Bench’19
Recommendation n Collaborative filtering-based movie recommendations n Model ： Collaborative filtering algorithm n Dataset ： MovieLens • 20,000,000 movie ratings data AIBench Bench’19
3D Object Reconstruction n Predicts and reconstructs 3D objects n Model ： a convolutional encoder-decoder network n Dataset ： ShapeNet • 51,3000 different 3D data covering 5 object categories AIBench Bench’19
Text Summarization n Generate the text summary n Model ： sequence-to-sequence model n Dataset ： Gigaword • 10,000,000 text data, Four billion words AIBench Bench’19
Spatial Transformer n Performs spatial transformations n Model ： spatial transformer networks n Dataset ： MNIST • 60000 training samples, 10000 test samples AIBench Bench’19
Learning to Rank n Machine-learned ranking for recommender system n Model: ranking distillation n Dataset: Gowalla • Social network data: 196,591 nodes and 950,327 edges AIBench Bench’19
Micro Benchmark (12) AIBench Bench’19
AIBench Inference Specification n Inference System under Test Datasets Query Result Generator Outputs System under Test Concurrency Accuracy Ø Ø Ø Arriving rate Ø Latency Distribution Tail Latency Ø Ø Thinking time Throughput Ø Ø Monitoring Tools AIBench Bench’19
Inference Metrics n Online Inference ( Accuracy-ensured ) n Latency, Tail latency n Latency-bounded throughput n Offline Inference ( Accuracy-ensured ) n Throughput, Energy consumption n Accuracy-ensured: n Accuracy deviation with target accuracy is within 2% AIBench Bench’19
AIBench Training Specification n Training System under Test Datasets Result Outputs System under Test Accuracy Ø Ø Latency Tail Latency Ø Throughput Ø Monitoring Tools AIBench Bench’19
Training Metrics n Offline Training n Time-to-accuracy n Energy-to-accuracy n Throughput • Running 1000 epochs • Hyper parameter settings should be able to achieve target accuracy AIBench Bench’19
Benchmark Guideline n Online Server n Each component can be distributed deployed on a large cluster AIBench Bench’19
Benchmark Guideline n Offline Analytics n Single GPU, Multi GPUs, Distributed versions • TensorFlow implementation • PyTorch implementation n Example distributed training setting AIBench Bench’19
Outline n AIBench Overview n Tasks, Models, Datasets, Metrics n How to use AIBench n Preliminary Results n Conclusion AIBench Bench’19
Recommend
More recommend
