Choosing Middleware: Why performance and scalability do (and don’t) matter Michi Henning Chief Scientist, ZeroC, Inc.
Middleware is (almost) forever Committing to middleware represents major investment Product life cycle typically several years to a decade long (at least) Once you choose middleware you are stuck with it Standardization does not help ◦ CORBA is the most standardized middleware to date ◦ Despite standardization, changing ORBs was pragmatically impossible You had better exercise due diligence
Choose on features The one with the most features must be best, right? Middleware 1 Middleware 2 Middleware 3 Middleware 4 Feature 1 Feature 2 Feature 3 Feature 4 Feature 5
Choose on features More features is not necessarily better Unless the middleware is very carefully designed: ◦ Larger memory footprint at run time leads to fewer cache hits and reduced performance ◦ More features almost always means steeper learning curve: you almost always pay for what you do not use ◦ All the features in the world do not help if the one feature you really need is missing
Choose on price The one that’s cheapest must be best, right? Up-front licensing cost is typically 2-3% of total cost over product life cycle Development cost usually eclipses licensing cost by an order of magnitude A 5% difference in programmer productivity is likely to swamp any price difference A 5% difference in defect rate and MTBF is certain to swamp any price difference
Choose on performance The one that goes fastest must be best, right? Biggest fallacy in the history of middleware ◦ Performance only matters if you need it! ◦ Many performance gains due to selective accounting ◦ Benchmarks rarely tell the story ◦ Most non-experts are incapable of creating valid benchmarks and evaluating them correctly
Real evaluation criteria How steep is the learning curve? What is the quality of the APIs? ◦ Type-safe? ◦ Thread-safe? ◦ Exception-safe? ◦ Memory management? What is the quality of the documentation? ◦ Well indexed? ◦ Online and searchable? ◦ Non-trivial examples? ◦ Real-world example programs with real-world error handling?
Real evaluation criteria High-quality professional training? How reliable is the software? ◦ Defect rate? ◦ MTBF? Quality of support? ◦ 24x7 ◦ Guaranteed response time? ◦ Who provides the support? ◦ Consulting services? ◦ Vibrant developer community?
Real evaluation criteria What operating systems and compilers are supported? What languages are supported? Source code available? How complete is the relevant feature set? What are the licensing conditions? ◦ Up-front one-time buy out? ◦ Royalty based? ◦ GPL?
When performance doesn’t matter Performance no longer matters when the middleware is no longer noticeable Middleware is no longer noticeable if: ◦ Overall time spent in the middleware is < 5% ◦ Overall memory footprint is < 5% ◦ Realized bandwidth is within 5% of theoretical maximum For many applications, even bad middleware meets these criteria!
When performance does matter Everyone’s first benchmark: interface I { void ping(); }; Measure round-trip time over loopback to establish messaging rate compared to native sockets Sockets Ice (C++) Ice (Java) RMI Ice (.NET) WCF (bin) WCF Loopback (SOAP) 18,000 10,500 8,000 10,600 7,500 7,500 2,500 Sockets Ice (C++) Ice (Java) RMI Ice (.NET) WCF (bin) WCF (SOAP) Network 2,300 2,300 2,300 2,300 2,300 2,200 570
When performance does matter Everyone’s second benchmark: typedef sequence<byte> ByteSeq; interface I { void send(ByteSeq bs); }; Measure over loopback to establish throughput compared to native sockets (Gbit/sec) Sockets Ice (C++) Ice (Java) RMI Ice (.NET) WCF (bin) WCF Loopback (SOAP) 1.3 1.2 0.83 0.83 0.63 0.52 0.15 Sockets Ice (C++) Ice (Java) RMI Ice (.NET) WCF (bin) WCF (SOAP) Network 0.78 0.75 0.66 0.28 0.52 0.43 0.14
When performance does matter Everyone’s forgotten benchmark: typedef sequence<byte> ByteSeq; interface I { ByteSeq receive(); }; Receive slower than send anything between 20% and 300% The same middleware, compiled from the same source code, shows 20% difference on one OS, and 300% on another The same middleware, compiled from the same source code, can be faster over the network than over loopback
When performance does matter Data effects Type of data is important: ◦ A byte sequence performs very different from a structure sequence ◦ A structure sequence with fixed-length members performs very different from a structure with variable- length members
When performance does matter struct Fixed { struct Var { int i; string s; int j; double d; double d; }; }; sequence<Variable> VarSeq; sequence<Fixed> FixedSeq; interface Throughput { interface Throughput { void send(FixedSeq s); void send(VarSeq s); FixedSeq recv(); VarSeq recv(); }; }; Performance for these tests differs dramatically!
Data Effects The values in the data make a difference Wire size for Fixed structures with small values versus random values: Small Values Ice for .NET WCF (binary) WCF (SOAP) Request size 800,057 bytes 601,538 bytes 2,750,781 bytes Random Values Ice for .NET WCF (binary) WCF (SOAP) Request size 800,057 bytes 1,167,968 bytes 3,976,802 bytes
Data Effects struct Fixed { struct Fixed { int i; int int1Member; int j; int int2Member; double d; double doubleMember; }; }; sequence<Fixed> FixedSeq; sequence<Fixed> FixedSeq; interface Throughput { interface Throughput { void send(FixedSeq s); void send(FixedSeq s); FixedSeq recv(); FixedSeq recv(); }; }; Ice WCF (binary) WCF (SOAP) Request size (short names) 800,057 bytes 601,538 bytes 2,750,781 bytes Request size (long names) 800,057 bytes 600,488 bytes 3,404,862 bytes
Platform effects Same middleware (Ice), same source, same test, same hardware Different OS: Vista versus OS X C++ (byte seq) C++ (fixed seq) Java (byte seq) Java (fixed seq) Vista 1,800Mbit/s 900Mbit/s 1,250Mbit/s 620Mbit/s OS X 2,850Mbit/s 740Mbit/s 2,500Mbit/s 145Mbit/s
Performance or Scalability? Most applications do not have high performance requirements. ◦ Performance for an individual client not much of a concern But: many applications have high scalability requirements ◦ If you have an application with thousands of clients, how many servers and how much bandwidth do you need?
Performance or Scalability? High scalability requires high performance But: ◦ High performance at small scale in no way guarantees high performance at large scale Factors: ◦ Quality of implementation ◦ APIs designed for scale ◦ Threading models ◦ Connection management ◦ Use of OS resources (file descriptors, timers, memory garbage or fragmentation)
Scalability Scenario Stateful interaction of few servers with many clients Examples: ◦ Twitter ◦ Online shopping cart ◦ Secure communication ◦ Instant messaging ◦ Stock ticker ◦ Remote monitoring
Scalability Test One server, one object, one method that does nothing Lots of clients connected to server (one connection per client) Clients each send one message every four seconds Message deemed successful if it completes within 1 second. How many clients can the server support?
Scalability Test (Java) Ice for C++ Ice for Java RMI Requests/sec 20,000 20,000 7,500 # connections 80,000 80,000 30,000 CPU usage 15% 25% 30% Working set 50MB 810MB 3GB Virtual memory 2.5GB 1.4GB 37GB RMI fails catastrophically beyond 30,000 clients due to memory starvation Ice for Java continues to scale, but requests take more than 1 second due to GC passes Ice for C++ scales well beyond 120,000 clients
Scalability Test (.NET) Ice for .NET WCF (binary) WCF (SOAP) Requests/sec 7,250 4,850 8,000 # connections 29,000 19,500 32,000 CPU usage 33% 90% 40% WCF binary encoder consumes an inordinate amount of CPU WCF SOAP encoder scales better than expected, but fails on the client side
CPU Consumption CPU consumption for Fixed throughput test Fixed-length seq. Ice for .NET WCF (binary) WCF (SOAP) CPU (send) 1 68 84 CPU (receive) 1 62 236 CPU and bandwidth requirements make it unlikely that WCF will scale
Bandwith and hardware are cheap? Server has 50Mbit/sec link to Internet Clients request 50kB of data 1Mbit/sec works out to 1kB/sec of data ⇒ Server can handle 100 concurrent clients If SOAP adds only a factor of 10 in size, the same server can handle 10 concurrent clients You will need a 500Mbit/sec link instead ($$$) Single server will probably not handle 500Mbit/sec. You will need several servers ($$$)
Lessons learned Benchmarking is hard Creating relevant benchmarks is harder Interpreting the results is harder still Small scale results do not indicate large scale performance Chances are that what you measure will be irrelevant because the problem that will kill your application is one you never thought of.
Recommend
More recommend
