Horus: Fine-Grained Encryption- Based Security for High Performance Petascale Storage Ranjana Rajendran • Ethan L. Miller • Darrell D. E. Long Storage Systems Research Center University of California, Santa Cruz Sunday, November 13, 11
What’s the problem? • Ever-increasing volume of data • More files • Larger files • Ever-increasing threat • Intrusions • Insider attacks • Accidental data leakage • HPC systems have a lot of vulnerabilities • Storage nodes • Metadata servers • Thousands of clients • Goal: limit the risk of data leakage in an HPC system • Goal: allow protection of some parts of a file 2 Sunday, November 13, 11
Typical HPC storage environment • Clients interact with MDS to open files Client Client • Clients interact directly 1: open() Client MDS Client Client Client MDS with storage to read/write 2: capability data 4: I/O response 3: I/O request Policy control • Maat [SC07] can provide authorization • No encryption... Disk Disk Disk Disk Disk Disk Disk Disk Disk Disk 3 Sunday, November 13, 11
Threat model: leakage of confidential HPC data • Traditional encryption: one key per file • Data can be encrypted at the client • Still vulnerable to leaks • Compromised storage devices / nodes • Little risk if data is encrypted • High risk if done with other compromises • Compromised metadata servers • Potential for leaking keys • Difficult to secure given complexity • Compromised client (compute) nodes • Keys from a single client can leak the whole file! • There are thousands of clients... 4 Sunday, November 13, 11
Keyed hash trees • Solution: use keyed hash trees to generate block keys from the file key • Clients only get the block keys they need • Clients can’t encrypt / decrypt data for which they don’t have keys • Nodes at any level of the tree can be given out • Value of a key depends on parent and key’s position • Simple to derive block key from any key above it KR (file root key) K1,0 K1,1 K1,2 K2,0 K2,1 K2,2 K2,3 K2,4 K2,5 K2,6 K2,7 K2,8 K2,9 K3,0 K3,1 K3,2 K3,3 K3,4 K3,5 K3,6 K3,7 K3,8 K3,9 K3,10 K3,11 K3,12 K3,13 K3,14 K3,15 K3,16 K3,17 K3,18 K3,19 5 Sunday, November 13, 11
Keyed hash trees • Solution: use keyed hash trees to generate block keys from the file key • Clients only get the block keys they need • Clients can’t encrypt / decrypt data for which they don’t have keys • Nodes at any level of the tree can be given out • Value of a key depends on parent and key’s position • Simple to derive block key from any key above it KR (file root key) K1,0 K1,1 K1,2 K2,0 K2,1 K2,2 K2,3 K2,4 K2,5 K2,6 K2,7 K2,8 K2,9 Block K3,0 K3,1 K3,2 K3,3 K3,4 K3,5 K3,6 K3,7 K3,8 K3,9 K3,10 K3,11 K3,12 K3,13 K3,14 K3,15 K3,16 K3,17 K3,18 K3,19 keys 5 Sunday, November 13, 11
Keyed hash trees • Solution: use keyed hash trees to generate block keys from the file key • Clients only get the block keys they need • Clients can’t encrypt / decrypt data for which they don’t have keys • Nodes at any level of the tree can be given out • Value of a key depends on parent and key’s position • Simple to derive block key from any key above it KR (file root key) K1,0 K1,1 K1,2 Range keys K2,0 K2,1 K2,2 K2,3 K2,4 K2,5 K2,6 K2,7 K2,8 K2,9 K3,0 K3,1 K3,2 K3,3 K3,4 K3,5 K3,6 K3,7 K3,8 K3,9 K3,10 K3,11 K3,12 K3,13 K3,14 K3,15 K3,16 K3,17 K3,18 K3,19 5 Sunday, November 13, 11
Generating block keys • Start at root key KR (file root key) • At each level, generate K1,0 K1,1 new key from • Parent key K2,0 K2,1 K2,2 K2,3 K2,4 • Level number • “Offset” in the level K3,0 K3,1 K3,2 K3,3 K3,4 K3,5 K3,6 K3,7 K3,8 • Process can be split • Simple to go down the tree for x = start + 1 to end do • “Difficult” to go up the tree k ← keyed_hash ( k , x || ⎣ b / B x ⎦ ) (or sideways) end for return k 6 Sunday, November 13, 11
Handing out range keys KR Root key Range keys File blocks A B C D Clients • Provide only needed range keys to each client • Ranges cover any number of blocks • Ranges must be aligned to key • Hand out multiple range keys to a client if needed • Range key usage is flexible • Multiple clients can get key for a single block • Any range key that “covers” a block can be used to generate its key 7 Sunday, November 13, 11
Using Horus • Key Distribution Cluster can run Key 5: Range key(s) • Separately Distribution Cluster • Stateless: easier to reset between 3: Protected KR 4: Calculate permitted computations range key(s) 6: Calculate • On MDS block key • On nodes doling out work Client Client 1: open() Client MDS Client Client units for computation Client MDS 2: Protected KR 7 9: Decrypt data : • Keys stored using public- I / O 8 : r I e / O q u r e e s s key encryption t p o n s e • Client forwards key to KDC Disk Disk Disk Disk Disk Disk • KDC could request key from Disk Disk Disk Disk MDS directly 8 Sunday, November 13, 11
Storing file root keys • Encrypt file root keys with users’ public keys • Lockbox structure similar to those used in many secure file systems • Store file root keys in the file system • In a separate file • In extended attributes attached to the file • Alternative approach: supply file keys as part of the setup for the computation • More secure? • May require additional infrastructure 9 Sunday, November 13, 11
Using Horus as an encryption layer • In-kernel implementation HDF/ HDF/ • May be a bit faster NetCDF NetCDF • Requires OS changes Horus System calls • User-level implementation • No OS changes • Could leverage data layout System calls Horus knowledge • Divide file by content rather than by block offset File system File system OS kernel OS kernel In-kernel User-level 10 Sunday, November 13, 11
Horus security • Data is only in the clear on clients • Storage nodes can’t leak data • MDS can’t leak data (or keys) • Only a client can leak data • Client can only leak data for which it has a key • Requires large-scale client compromise to leak the entire file • Can’t leak “idle” files without obtaining user’s private key • Revocation is an issue (as with other encrypted file systems) 11 Sunday, November 13, 11
Ongoing work • User-level implementation of Horus • Layered just above system calls • Uses extended attributes for key storage • Includes protocol to communicate with KDC • Explore tradeoffs between deeper tree and wider range keys • Eventually, integrate into HPC file system such as Ceph or PVFS 12 Sunday, November 13, 11
Conclusions • Security is becoming increasingly important for HPC • Leaving data in the clear may no longer be acceptable • Horus prevents many attacks • Compromise of disks or MDS • Small-scale compromise of compute nodes & clients • Horus allows sharing differential security for portions of large files • Horus can run in the kernel or at user level ➡ Provide greater confidentiality for HPC data 13 Sunday, November 13, 11
Questions? Supported by 14 Sunday, November 13, 11
Recommend
More recommend
