Distributed Bingo 1 7 -6 5 4 : Analysis of Softw are Artifacts 1 8 - PowerPoint PPT Presentation

Distributed Bingo 1 7 -6 5 4 : Analysis of Softw are Artifacts 1 8 -8 4 1 : Dependability Analysis of Middlew are Team 5: Jack Yao Bubba Beasley Kai Liao Alex Berendeyev http: / / www.ece.cmu.edu/ ~ ece846/ team5

2 Alex Berendeyev Bubba Beasley Team Members Kai Liao

3 Real-World Demonstration

Real-World Characteristics � Each player gets a Bingo card to start � A player joining mid-game can catch up with knowledge of previous draws � The host announces each draw � The winning player announces Bingo � The host verifies the win � The host announces the win 4

Baseline Application � A distributed, online version of Bingo � The clients pull data from the servers and the servers push data to the clients � DB: SQL Server 2000 , Windows XP, MSE Cave machine Servers: JBoss (JNDI, JMS) on Linux, ECE Game cluster Clients: Windows, Linux (theoretically anywhere) Automated command-line client Interactive GUI-based client 5

Publish/ Subscribe with JMS Server Server JMS JMS Client Client Client Key Client Server JMS Connection JMS 6

Baseline Architecture BingoClient Join() GetSnapshot() BingoServer DeclareBingo() HS AS JNDI JMS DBServer Key DBServer Client Server Remote Method Call JMS Connection DB Call 7

From the Real to the App � Each player gets a Bingo card to start � A player joining mid-game can catch up with knowledge of previous draws The Client asks the AnsweringServer to join and receives a bingo card and all previous draws. � The host announces each draw At regular intervals (5 seconds), the HostServer broadcasts the draws via the JMS. 8

From the Real to the App � The winning player announces Bingo � The host verifies the win � The host announces the win The Client asks the AnsweringServer to verify Bingo The AnsweringServer verifies and stores the winner's ID in the database The HostServer checks for a winner in the DB and broadcasts that there is a win 9

GUI Interface Java GUI on top of a Java command-line interface 10

Miscellany � Each game: 100 draws, rather than 75 � Approximately 1.4 x 10 30 card combinations (1.4 billion trillion trillion cards) � Duplicate cards are not a problem, so theoretically no limit on the number of players in a single game � No guarantee of fairness in declaring a winner 11

Fault Tolerance Goals (1) Server Faults “Sacred Machine” Assum ptions • JBoss Process crashes • Replication Manager • Machine crashes • Global Fault Detector Netw ork Faults • DB Server Replicas N replicas Tested with 3 replicas Round-robin recovery of JBoss servers 12

FT Baseline Architecture BingoClient BingoClient BingoClient One Connection RepMan to the primary FD (HealthChecker class) JMS Server One JBoss server up BingoServer JBoss Servers BingoServer on replicas are not running HS AS JNDI JMS Key Remote Method Call JMS Connection DB Call SSH Connection DBServer Server Client DBServer 13

Baseline Fail-Over Measurements FT Baseline Fail-Over Slow Server 90 80 70 Fast Server 60 Second 50 FD Time (seconds) SN 40 30 20 10 0 0 50 100 150 200 Number of faults injected 14

Baseline Fail-Over Measurements FT Baseline Client Time B/w Messages 120 Slow Server 100 80 Second Time (seconds) 60 40 Fast Server 20 0 0 1000 2000 3000 4000 5000 Number of faults injected 15

RT-FT Optimized 1 Architecture BingoClient BingoClient BingoClient Pre-established RepMan Connections to FD (HealthChecker class) all JMS Servers BingoServer All JBoss BingoServer servers on replicas are up HS AS JNDI JMS Key Remote Method Call JMS Connection DB Call SSH Connection DBServer Server Client DBServer 16

RT Optimized 1 Fail-Over Measurements RT Optimization 1 Fail-Over 30 Slow Server 25 20 Second FD Time (seconds) 15 SN 10 5 Fast Server 0 0 10 20 30 40 50 60 70 80 90 100 Number of faults injected 17

RT-FT Optimized 2 Architecture BingoClient BingoClient BingoClient RM__FD BingoServer BingoServer FD runs as a daemon HS AS JNDI JMS Key Remote Method Call JMS Connection DB Call SSH Connection DBServer Server Client DBServer 18

RT Optimized 2 Fail-Over Measurements RT Optimization 2 Fail-Over (Repman=1s, HealthCheckerDaemon=0.5s) 5 4.5 4 3.5 3 Time (seconds) Second FD 2.5 SN 2 1.5 1 0.5 0 0 5 10 15 20 25 30 Number of faults injected 19

RT-FT Optimized 3 Architecture BingoClient BingoClient BingoClient RM__FD BingoServer BingoServer FD runs as a Local FD daemon + HS Local FD AS JNDI JMS Key Remote Method Call JMS Connection DB Call SSH Connection DBServer Server Client DBServer 20

RT Optimized 3 Fail-Over Measurements RT Optimization 3 Fail-Over (Repman=0.1s, Local Checker=0.4s) 3.5 3 2.5 2 Time (seconds) Second FD SN 1.5 1 0.5 0 0 5 10 15 20 25 30 35 40 45 Number of faults injected 21

Measurements Insights (1) 8 6 FD (Second) 4 2 0 1.0 1.0 0.1 1.0 (MIN) 0.4 0.7 (AVG) 0.7 0.7 (MAX) 1.0 (MIN) 0.7 1.3 (AVG) 0.4 1.6 (MAX) 1.9 0.4 (MIN) 0.4 Repman Period (AVG) (MAX) (Second) Local Checker Period (Second) 22

Measurements Insights (2) Average Fault Detection Time Comparison Real Time Tuning: Local FD= 0 .7 s • Repman period Local FD= 1 .0 s Repman= 1 .0 s Repman= 0 .1 s • Local FD period • Broadcast period FD= 3.1s Local FD Period (Second) Local FD Period 1.0 (A V G ) 0.7 (A V G ) 0.4 (A V G ) 0.1 0.4 0.7 1 1.3 1.6 1.9 (Second) Repman Period (Second) Repman Period (Second) 0.4 (AVG) 0.7 (AVG) 1.0 (AVG) 23

Lessons Learned • Potentially Publish/ Subscribe (JMS) can hide server errors from clients Typical Pull Architecture Our Push Architecture FD FD Start New Start New Replica Replica Reconnection • JBoss Server should be run in the minimal configuration. (default configurations are not suited for RT) 24

25 Questions?