Integrating Keystone with Large-scale Centralized Authentication - PowerPoint PPT Presentation

Integrating Keystone with Large-scale Centralized Authentication Ken Holden Chris Janiszewski OpenStack Solutions Architect OpenStack Solutions Architect kholden@redhat.com chris.openstack@redhat.com blog: holdenthecloud.com blog: chrisj.cloud

Agenda Level set ● Benefits ● Issues and Bottlenecks ● Best Practices ● Tuning and Results ●

Level Set Authentication and Authorization ● Keystone ● Tokens ● LDAP ● Domains ● Kerberos and SSO ●

Authentication and Authorization Keystone (OpenStack Identity Service) ● Methods of Authentication ● Authorization tokens ● Transport Layer Security (TLS) with X.509 ●

Keystone .. provides identity, token, catalog, and policy services for use specifically by services in the OpenStack family Features: External authentication ● Federation ● Multi-factor authentication ● Considerations: Invalid login attempts ● No enforcement policies on password strength, expiration ●

Tokens generated for authorization and access ● default value for expiry is one hour ● often passed within the structure of a larger ● context Token types: ● UUID ○ PKI and PKIZ ○ FERNET ○

LDAP The Lightweight Directory Access Protocol ● (LDAP) - is an open, vendor-neutral, industry standard application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network LDAP simplifies integration of Identity ● authentication Identity service MUST NOT be allowed to write ● to LDAP services

Domains Keystone V2 Keystone V3 * *Pictures found on Matt’s Dorn Blog - http://madorn.com/keystone-v3-api.html#.WwM2hXWUvgk

Kerberos Keystone can now be executed in a web ● server like Apache HTTPD The Kerberos ticket-granting ticket can ● be used to securely provide tickets for a given service Web server handle authentication is not ● exclusive Web server can provide support for X.509 ● or Kerberos authentication Keystone provides support for password ● authentication (with SQL or LDAP)

Identity Service: Authentication Mechanisms Carnival Ticket Booth vs. Kerberos (server) TGT Grants a ticket called a ● Ticket Granting Ticket (TGT) which says which services you have access to. This ticket also has an expiration called a Time-To-Live (TTL). You present the TGT to the ● service you want access to and the service will use that to determine if you can access the service.

LDAP Benefits Consolidated repository ● Simplified management ● Ideal for multiple OpenStack Deployments with ● shared resources Performance of LDAP vs SQL ● SSL/TLS ● Back-end database flexibility ● Well-defined client API ● Decoupling of the roles ●

LDAP Considerations Few people understand LDAP ● No writes allowed ● Is OpenStack a front end for your users? ● Very strict schema ● Performance considerations ●

Issues and Bottlenecks Token Performance ● LDAP Size ● Database ● Horizon Dashboard ● LDAP response ●

Generally a good idea NOT to... Enable Debug and Verbosity - Impact can be significant ● Use slow hard drives for Controller’s OS and Databases ● Use non-secured APIs, Dashboard, and LDAP queries ●

Tokens The faster you can request tokens, the faster OpenStack responds ● Fernet tokens offer superior performance over UUID ●

Just say No to UUID Who remembers the good old days when OpenStack started to crawl ? # du -sh /var/lib/mysql/keystone 149G /var/lib/mysql/keystone MariaDB [keystone]> select count(*) from token; +----------+ | count(*) | +----------+ | 20717136 | Number of UUID tokens +----------+

Where do I query? Active Directory DNS: Use Domain Name ● url = ldaps://example.com Or obtain a list of Domain Controllers from DNS via SRV Records dig -t SRV _ldap.tcp.example.com LDAP: list multiple LDAP servers ● url = ldaps://ldap1,ldaps://ldap2

Limit number of LDAP records Do all 100,000+ users REALLY need access to OpenStack? ● Keystone doesn’t always cleanup after itself ● MariaDB [keystone]> select count(*) from id_mapping; +----------+ | count(*) | +----------+ | 129052 | Number of LDAP Accounts without filtering +----------+ MariaDB [keystone]> select count(*) from id_mapping; +----------+ | count(*) | +----------+ | 129052 | Number doesn’t decrease when adding filtering later +----------+

LDAP - Filtering Filtering users and groups ● user_filter = (&(|(memberOf=CN=OpenStack-Admins)(memberOf=CN=OpenStack-Users))) group_filter = (&(objectClass=Group)(&(|(cn=OpenStack)(cn=OpenStack-Admins)(cn=OpenSt ack-Users)))) Using AND / OR ● (&(|(memberOf=foo)(memberOf=bar))) (&(memberOf=for)(memberOf=bar)) Active Directory Nested groups ● (memberOf:1.2.840.113556.1.4.1941:=cn=OpenStack)

ldapsearch Member of group1 OR group2 ● ldapsearch -LLL -H ldap://example.com -E pr=2/noprompt -b 'dc=example,dc=com' -D 'example\USER' -w PASSWORD '(&(|(memberOf=CN=OpenStack-Admins)(memberOf=CN=OpenStack-Users)))' Member of group1 AND group2 ● ldapsearch -LLL -H ldap://example.com -E pr=2/noprompt -b 'dc=example,dc=com' -D 'example\USER -w PASSWORD '(&(memberOf=CN=OpenStack-Admins)(memberOf=CN=OpenStack-Users))' Members of Active Directory Nested Group ● ldapsearch -LLL -H ldap://example.com -E pr=2/noprompt -b 'dc=example,dc=com' -D 'example\USER -w PASSWORD '(&(objectClass=organizationalPerson)(sAMAccountName=*)(memberOf: 1.2.8 40.113556.1.4.1941 :=cn=OpenStack))'

LDAP Tuning - Pagination Default value is 0 which disables paging $ ldapsearch -LLL -H ldap://ldapserver.domain.com \ -b ' dc=domain,dc=com ' -D ' DOMAIN\USERNAME ' \ -w PASSWORD |grep sAMAccountName |wc -l Size limit exceeded (4) 825 $ ldapsearch -LLL -H ldap://ldapserver.domain.com \ -E pr=1/noprompt -b ' dc=domain,dc=com ' \ -D ' DOMAIN\USERNAME ' \ -w PASSWORD |grep sAMAccountName |wc -l 10052

LDAP Debug View how keystone queries LDAP by enabling debug ● crudini --set keystone.conf ldap debug_level 3 crudini --set keystone.conf DEFAULT insecure_debug true crudini --set keystone.conf DEFAULT debug true # restart keystone Watch the magic... ● tail -f /var/log/keystone/keystone.log | grep LDAP\ search filterstr=(&(sAMAccountName=kholden)(&(|(memberOf=CN=OpenStack-Admins,OU=People, DC=lab,DC=lan)(memberOf=CN=OpenStack-Users,OU=People,DC=example,DC=com)))(object Class=person)) attrs=['sAMAccountName', 'userPassword', 'userAccountControl', 'mail', 'description'] attrsonly=0 Disable debug when done ●

Keystone Pooling Keystone Pooling ● Decrease load on Domain Controller especially with TLS ○ Separate pool for User Authentication ○ use_auth_pool requires use_pool enabled ○ crudini --set keystone.DOMAIN.conf ldap use_pool True crudini --set keystone.DOMAIN.conf ldap pool_size 200 crudini --set keystone.DOMAIN.conf ldap pool_retry_max 20 crudini --set keystone.DOMAIN.conf ldap pool_retry_delay 0.1 crudini --set keystone.DOMAIN.conf ldap pool_connection_timeout -1 crudini --set keystone.DOMAIN.conf ldap pool_connection_lifetime 600 crudini --set keystone.DOMAIN.conf ldap use_auth_pool True crudini --set keystone.DOMAIN.conf ldap auth_pool_size 1000 crudini --set keystone.DOMAIN.conf ldap auth_pool_connection_lifetime 60 # restart keystone

Keystone Caching Enable Keystone Caching with memcached backend ● systemctl enable memcached && systemctl start memcached crudini --set keystone.conf cache enabled true crudini --set keystone.conf cache backend dogpile.cache.memcached crudini --set keystone.conf cache backend_argument url: LOCAL_IP :11211 crudini --set keystone.conf catalog caching true crudini --set keystone.conf domain_config caching true crudini --set keystone.conf federation caching true crudini --set keystone.conf revoke caching true crudini --set keystone.conf role caching true crudini --set keystone.conf token caching true crudini --set keystone.conf token cache_on_issue true crudini --set keystone.conf identity caching true crudini --set keystone.conf identity cache_time 600 # restart keystone

Results

Meanwhile, in the real world... Challenges with Implementation ● Proxies in front of Domain Controllers ○ Firewalled Domain Controllers ○ Omitted Distinguished Names to save licensing costs ○ Service Account Password Change ○ Certificate / CA change ○ Changes require keystone restart ○ Fernet Token rotation ○ The Defiant Windows Admin ○

Multi-site Multiple OpenStack Environments with One LDAP ● Unique IDs for roles, Domains, and Projects are unique to each ○ site Determine Project, Domain and Role IDs from one site and ○ update other sites with same IDs (BACKUP DB FIRST) select * from project where name='PROJECT_NAME'; select * from project where name='DOMAIN_NAME'; select * from project where name='ROLE_NAME'; update project set id='SITE1_PROJECT_ID' where name='PROJECT_NAME'; update project set id='SITE1_DOMAIN_ID' where name='DOMAIN_NAME'; update project set id='SITE1_ROLE_ID' where name='ROLE_NAME';