PRACTICAL MTLS Ying Li @cyli PROBLEM TYPICAL MICROSERVICE - PowerPoint PPT Presentation

MINIMIZING THE WINDOW OF COMPROMISE PRACTICAL MTLS Ying Li @cyli

PROBLEM TYPICAL MICROSERVICE ARCHITECTURE VPC S1 DB S2 S1 S3

PROBLEM VLAN-TASTIC MICROSERVICE ARCHITECTURE S1 DB S2 S1 S3

PROBLEM CORRECT MICROSERVICE ARCHITECTURE S1 DB S2 S1 S3

PROBLEM APPLICATION TLS LIFECYCLE Bootstrap Revoke Renew

PROBLEM BOOTSTRAP • CSR ➡ CA • Configuration

PROBLEM RENEW • Schedule

PROBLEM RENEW • Schedule • CSR ➡ CA • Configuration

PROBLEM RENEW • Schedule • CSR ➡ CA • Configuration • Restart

PROBLEM REVOKE • CRL • OCSP [Stapling]

PRINCIPLE AUTOMATE, AUTOMATE, AUTOMATE • Promotes adoption of mTLS

PRINCIPLE AUTOMATE, AUTOMATE, AUTOMATE • Promotes adoption of mTLS • Single location for private key

PRINCIPLE AUTOMATE, AUTOMATE, AUTOMATE • Promotes adoption of mTLS • Single location for private key • Shorter certificate expiry

SWARMKIT OVERVIEW https://github.com/docker/swarmkit

SWARMKIT OVERVIEW CLUSTER Worker Worker Manager Manager Manager Worker Worker Worker Worker

SWARMKIT OVERVIEW Node Node CLUSTER Worker Worker Manager Node Node Manager Manager Worker Worker Node Node Worker Worker

SWARMKIT OVERVIEW CLUSTER Worker Worker Manager raft store Manager Manager Worker Worker Worker Worker

SWARMKIT OVERVIEW CLUSTER Node Node CA raft store CA CA Node Node Node Node

SWARMKIT’S IMPLEMENTATION

SWARMKIT’S IMPLEMENTATION BOOTSTRAP Token Random Version Secret SWMTKN-1-mx8suomaom825bet6-cm6zts22rl4hly2 Known Hash Prefix of Root CA

SWARMKIT’S IMPLEMENTATION BOOTSTRAP CA 1. Retrieve, validate Root CA certificate. 1 Node

SWARMKIT’S IMPLEMENTATION BOOTSTRAP CA 1. Retrieve, validate Root CA certificate. 2 2. CSR + secret token ➡ CA. (TLS) 1 Node

SWARMKIT’S IMPLEMENTATION BOOTSTRAP CA 1. Retrieve and validate Root CA Public key material. 2 2. CSR + secret token ➡ CA. (TLS) 1 3 3. Get certificate. (TLS) Node

SWARMKIT’S IMPLEMENTATION RENEW 50% 80% Valid Valid From Until

SWARMKIT’S IMPLEMENTATION RENEW CA 1. CSR + ➡ CA. (mTLS) 2.Get certificate. (mTLS) 1 2 Node

SWARMKIT’S IMPLEMENTATION RENEW 1. Trigger extra leader election 2. Workers all need to reconnect Restart to managers 3. Reschedule work

SWARMKIT’S IMPLEMENTATION RENEW

SWARMKIT’S IMPLEMENTATION RENEW

SWARMKIT’S IMPLEMENTATION RENEW Server Existing connections New connections

SWARMKIT’S IMPLEMENTATION RENEW Client Existing connections New connections

SWARMKIT’S IMPLEMENTATION REVOKE

SWARMKIT’S IMPLEMENTATION REVOKE REMOVE CRLS, OCSP [Stapling]

SWARMKIT’S IMPLEMENTATION REMOVE NODE BLACKLIST Node ID Certificate Expiry a8h1vsk3k9o5nwea858ty9kma 2017-08-26 01:02:52 UTC k80l2au3yq9f7x6r2oca13vwt 2017-07-15 11:35:23 UTC n970d5be9ccgnreg4iti4jho3 2017-08-01 22:59:05 UTC

SWARMKIT’S IMPLEMENTATION REMOVE Worker/Manager Manager Request Validate node ID against blacklist Authorize role Perform work Response Worker/Manager Manager

SWARMKIT’S IMPLEMENTATION REMOVE BLACKLIST VS WHITELIST

SWARMKIT’S IMPLEMENTATION REMOVE Manager delayed join Manager Manager

SWARMKIT’S IMPLEMENTATION REMOVE Manager Manager Manager

PROBLEM Rotate CA

PROBLEM CA ROTATION • (conf.) All nodes: trust old and new CA 1 • (wait.) Verify all nodes

PROBLEM CA ROTATION • (conf.) All nodes: trust old and new CA 1 • (wait.) Verify all nodes • (conf.) All nodes: renew certificates 2 • (wait.) Verify all nodes

PROBLEM CA ROTATION • (conf.) All nodes: trust old and new CA 1 • (wait.) Verify all nodes • (conf.) All nodes: renew certificates 2 • (wait.) Verify all nodes • (conf.) All nodes: trust new CA only 3 • (wait.) Verify all nodes

PRINCIPLE CROSS-SIGNED INTERMEDIATE Root A Root Root B B Key Info: A Key Info: B X Signed by: A Signed by: B Leaf cert: X Signed by: B Root: B

PRINCIPLE CROSS-SIGNED INTERMEDIATE Root A Root Root B A Key Info: A Key Info: B X Signed by: A Signed by: B Leaf cert: X Signed by: B Root: A

PRINCIPLE CROSS-SIGNED INTERMEDIATE Root A Root A Key Info: A Root A Intermediate Root B’ B DN: A Signed by: A Key Info: B Key Info: B DN: B DN: B Signed by: A Signed by: B

PRINCIPLE CROSS-SIGNED INTERMEDIATE Root A Leaf cert: X Signed by: B’ Root A Intermediate B’ Root: A X

PRINCIPLE CROSS-SIGNED INTERMEDIATE Root A Leaf cert: X Signed by: B Root A Root B Intermediate B’ Root: B X

SWARMKIT’S IMPLEMENTATION CA ROTATION • (conf.) All nodes: trust old and new CA • (wait.) Verify all nodes • Generate cross-signed intermediate

SWARMKIT’S IMPLEMENTATION CA ROTATION • Generate cross-signed intermediate • (conf.) All nodes: renew certificates 1 • (wait.) Verify all nodes

SWARMKIT’S IMPLEMENTATION CA ROTATION • Generate cross-signed intermediate • (conf.) All nodes: renew certificates 1 • (wait.) Verify all nodes • (conf.) All nodes: trust new CA 2 • (wait.) Verify all nodes • Throw away cross-signed intermediate

SWARMKIT’S IMPLEMENTATION CA ROTATION: BEFORE ROTATION Node Trust Cluster Root A Root A Root: Trust Root: Root A Node TLS Cluster Root A Certificate: Cert Issuer: Root A Z

SWARMKIT’S IMPLEMENTATION CA ROTATION: START ROTATION Node Trust Cluster Root A Root A Root: Trust Root: Root A Root A Node TLS Cluster Certificate: Cert Issuer: Root A Root A Intermediate Z B’

SWARMKIT’S IMPLEMENTATION CA ROTATION: NODE CERT RENEWAL Node Trust Cluster Root A Root A Root: Trust Root: Root Root A Node TLS Cluster A Certificate: Cert Issuer: Root A Intermediate Root A B Intermediate B’ X

SWARMKIT’S IMPLEMENTATION CA ROTATION: NODE CERT RENEWAL Node1 Node2 Node3 Node4 Node5 Root A Root A Root A Root A Root A Trust Root Root Root Root Root Root A A A TLS A A Ro A Ro A Ro A Certificate Intermediate Intermediate Intermediate B B B Z Z X X X

SWARMKIT’S IMPLEMENTATION CA ROTATION: ROTATE TRUST ROOT Node Trust Cluster Root B Root B Root: Trust Root: Root A Root A Node TLS Cluster Certificate: Cert Issuer: Root A Intermediate Root B B Root A Intermediate B’ X

SWARMKIT’S IMPLEMENTATION CA ROTATION: ROTATE TRUST ROOT Node1 Node2 Node3 Node4 Node5 Root B Root B Root A Root A Root B Trust Root Root Root Root Root Root A A A A A TLS Ro A Ro A Ro A Ro A Ro A Certificate Intermediate Intermediate Intermediate Intermediate Intermediate B B B B B X X X X X

SWARMKIT’S IMPLEMENTATION CA ROTATION: FINISH ROOT ROTATION Node Trust Cluster Root B Root B Root: Trust Root: Root A Node TLS Cluster Root B Certificate: Cert Issuer: Root A Intermediate Root B B X

DEMO

SUMMARY MINIMIZING THE WINDOW OF COMPROMISE

SUMMARY MINIMIZING THE WINDOW OF COMPROMISE • automatic bootstrap, renewal • short certificate expiry

SUMMARY MINIMIZING THE WINDOW OF COMPROMISE • automatic bootstrap, renewal • short certificate expiry • certificate revocation • CA rotation

SUMMARY MORE INFORMATION https://github.com/docker/swarmkit https://diogomonica.com/2017/01/11/hitless-tls-certificate-rotation-in-go/ https://github.com/cloudflare/cfssl (@cyli)