Issuing temporary credentials for MySQL using Hashicorp Vault Walter Heck - CTO at OlinData Percona Live Europe 2017
Sounds familiar? “ Hey, Jane Doe from that department you’ve never heard of wants to do some analysis and she ‘needs’ direct access to our production database. Can you set that up in the next 30 minutes please, I know you have nothing better to do anyway. Cheers, not-your-manager.”
Suuuuureeee….
What if... ● They could self-service? ● They could request read-only credentials by authenticating against {LDAP, GitHub, AWS IAM, etc} ● Their credentials would automatically expire in 24 hours
Demo Architecture ● AWS ○ 3 az’s, 1 region ● HashiCorp Consul ● HashiCorp Vault ○ Secrets backend: consul ○ Auth backend -> github ● MySQL ○ (any flavor/version > 5.0)
Vault Vault is a tool for securely accessing secrets . A secret is The key features of Vault are: anything that you want to tightly control access to, such as ● Secure Secret Storage API keys, passwords, certificates, and more. Vault ● Dynamic Secrets provides a unified interface to any secret, while providing ● Data Encryption tight access control and recording a detailed audit log. ● Leasing and Renewal ● Revocation ● Vault is Open Source ● Enterprise support available ● The data stored with Vault is encrypted using 256-bit AES in GCM mode with a randomly generated nonce.
Workflow (vault/mysql admin) ● Setup Consul cluster ● Setup MySQL ● Setup Vault and point it at the Consul cluster Terraform (Infra as Code) Manual (One time operations) ● vault init ● Unseal the vault ● Setup GitHub auth ● Configure database secret backend ● Create one or more roles
Consul ● “A fancy Key/value store” ● Backend for our Vault cluster ○ Officially supported by Hashicorp root@ip-10-1-103-8:~# consul members Node Address Status Type Build Protocol DC Segment ip-10-1-103-104 10.1.103.104:8301 alive server 0.9.3 2 dc1 <all> ip-10-1-104-233 10.1.104.233:8301 alive server 0.9.3 2 dc1 <all> ip-10-1-105-147 10.1.105.147:8301 alive server 0.9.3 2 dc1 <all> ip-10-1-103-8 10.1.103.8:8301 alive client 0.9.3 2 dc1 <default>
root@ip-10-1-103-8:~# vault init Unseal Key 1: p5Luba1DNJcFSvThese2rj/fJ4iJQMA8bUBG5fuvIsS Vault init Unseal Key 2: 3+M+ajrPVCS96fKeysxUOEfM4JxsT40sosMVHfq1bqA Unseal Key 3: aW1qaxI2H7u57YAreG26Fuchao0XEaWq/f79dljE3iLA Unseal Key 4: tNSeeA6WWkAMK5Notjs/gEqf+8KbqQ32ypcfh3oecsfu Unseal Key 5: nkbtNRGOUxiXPiRealtNBTai9bzVaMmkkbCVRzbaoFn8 Initial Root Token: d57d945b-yoaa-f476-5660-3f6645692555 ● Initialises the vault Vault initialized with 5 keys and a key threshold of 3. Please ● Seals the vault securely distribute the above keys. When the vault is re-sealed, restarted, or stopped, you must provide at least 3 of these keys to unseal it again. ● Hands out unseal keys Vault does not store the master key. Without at least 3 keys, ○ By default 5, need 3 minimum to your vault will remain permanently sealed. unseal ( Shamir's Secret Sharing) ○ Don’t lose them, you lose everything ○ Use gpg init for easier distribution (https://www.vaultproject.io/docs/con cepts/pgp-gpg-keybase.html) ● Hands out root auth token
Sealing/Unsealing the vault root@ip-10-1-103-8:~# vault unseal 1 Key (will be hidden): Sealed: true Key Shares: 5 Key Threshold: 3 Unseal Progress: 1 ● A vault starts sealed 2 Unseal Nonce: d1747cd1-a850-bf79-9175-7ac1aaffdddd root@ip-10-1-103-8:~# vault unseal Key (will be hidden): ● If there’s ever any reason, a single Sealed: true Key Shares: 5 vault seal will seal the vault Key Threshold: 3 Unseal Progress: 2 Unseal Nonce: d1747cd1-a850-bf79-9175-7ac1aaffdddd ● Unsealing needs majority 3 out of 5 3 root@ip-10-1-103-8:~# vault unseal Key (will be hidden): keys by default Sealed: false Key Shares: 5 Key Threshold: 3 ● Sealing requires authentication first Unseal Progress: 0 Unseal Nonce: root@ip-10-1-103-8:~# vault status root@ip-10-1-103-8:~# vault seal ✓ Sealed: false Error sealing: Error making API request. Key Shares: 5 URL: PUT https://127.0.0.1:8200/v1/sys/seal Key Threshold: 3 Code: 500. Errors: Unseal Progress: 0 * 1 error occurred: Unseal Nonce: * missing client token Version: 0.8.2 root@ip-10-1-103-8:~# vault auth Cluster Name: vault-cluster-d2bd39fc Token (will be hidden): Cluster ID: d6957662-bc13-826e-5a10-1effde41a718 Successfully authenticated! You are now logged in. token: d57d945b-b0aa-f476-5660-3f6645692555 High-Availability Enabled: true token_duration: 0 Mode: standby token_policies: [root] Leader Cluster Address: https://10.1.104.220:8201 root@ip-10-1-103-8:~# vault seal Vault is now sealed.
Policies cat <<EOF | vault policy-write core-policy /dev/stdin path "sys/*" { policy = "deny" } path "database/creds/readonly" { ● Policies provide a declarative way to policy = "read" capabilities = ["list", "sudo"] grant or forbid access to certain } path "database/creds/demodb_admin" { paths and operations policy = "read" capabilities = ["list", "sudo"] } path "database/roles/*" { policy = "read" capabilities = ["read", "list"] } EOF
root@ip-10-1-103-8:~# vault auth-enable github Enable GitHub auth Successfully enabled 'github' at 'github'! root@ip-10-1-103-8:~# vault auth -methods Path Type Accessor Default TTL Max TTL Replication Behavior Description ● One of many auth mechanisms github/ github auth_github_db842730 system system replicated ○ AWS, Gcloud, LDAP, Radius, Okta and token/ token auth_token_84532020 system system more available replicated token based credentials ● Doesn’t use oauth but personal root@ip-10-1-103-8:~# vault write auth/github/config organization=olindata tokens Success! Data written to: auth/github/config ○ Beware! Losing a personal token is a root@ip-10-1-103-8:~# vault write auth/github/map/teams/core value=core-policy security risk Success! Data written to: auth/github/map/teams/core ● Access your Personal Access Tokens root@ip-10-1-103-8:~# vault auth -method=github in https://github.com/settings/tokens. token=10a8acd3f4ec0b2399146abb0ba6b70211bb6990 Successfully authenticated! You are now logged in. ○ Generate a new Token that has the scope The token below is already saved in the session. You do not need to "vault auth" again with the token. read:org . token: 58b52458-4e25-6642-f1f1-a24eda37913d token_duration: 2764799 token_policies: [core-policy default]
1 3 2
Enable MySQL secrets backend ● Secrets backends are mounts in root@ip-10-1-103-194:~# vault mount database Successfully mounted 'database' at 'database'! the tree root@ip-10-1-103-8:~# vault write database/config/mysql \ ● The database backend is generic > plugin_name=mysql-database-plugin \ for a number of database engines > connection_url="user:mypwd@tcp(perconalive.olindata.local:3306)/" \ > allowed_roles="readonly" ○ Postgres, mongo, oracle, MS The following warnings were returned from the Vault server: SQL server * Read access to this endpoint should be controlled via ACLs as it will return the connection details as is, including passwords, if any. ● The creation_statements argument for the role is flexible root@ip-10-1-103-8:~# vault write database/roles/readonly \ > db_name=mysql \ and can contain whatever SQL > creation_statements="CREATE USER '{{name}}'@'%' \ statement you want > IDENTIFIED BY '{{password}}';GRANT SELECT ON *.* TO '{{name}}'@'%';" \ > default_ttl="1h" \ ● Also see revocation_statements, > max_ttl="24h" Success! Data written to: database/roles/readonly max_open_connections and others
Demo 1
Workflow (Getting creds) 1. User auths against Vault 2. User asks vault for creds 3. Vault creates records in consul and issues a grant statement to MySQL 4. Vault returns username+password back to user … user does their thing 5. After X amount of time, vault removes grant from MySQL
GitHub Auth & Get mysql creds ● Authentication on command line is not root@ip-10-1-103-8:~# vault auth -method=github token=a8acd3f4ec0b2399146abb0ba6b70211bb699010 really useful in prod Successfully authenticated! You are now logged in. The token below is already saved in the session. You do not ○ Use HTTP API instead need to "vault auth" again with the token. token: 76ee8b56-61e5-cbcd-2710-7f7d08668568 token_duration: 2764799 token_policies: [core-policy default] root@ip-10-1-103-8:~# vault read database/creds/readonly Key Value --- ----- lease_id database/creds/readonly/1b889400-092d-e634-6444-d1217d c93690 lease_duration 1h0m0s lease_renewable true password A1a-77r41spp13x57vy5 username v-github-wal-readonly-23q9t9vxx2
Demo 2
What’s next? ● Audit backend ● Vault in HA mode ● Check other integrations ○ AWS, LDAP, Kerberos, SSH, etc.
Recommend
More recommend
