scripts.mit.edu Quentin Smith scripts@mit.edu Student Information - - PowerPoint PPT Presentation

▶
scripts mit edu
SMART_READER_LITE
LIVE PREVIEW

scripts.mit.edu Quentin Smith scripts@mit.edu Student Information - - PowerPoint PPT Presentation

Mar 13, 2023 461 likes •898 views

Services Backend Further Info scripts.mit.edu Quentin Smith scripts@mit.edu Student Information Processing Board October 29, 2019 Quentin Smith scripts@mit.edu scripts.mit.edu Services Backend Further Info Outline Services 1 Web Mail

slide-1
SLIDE 1

Services Backend Further Info

scripts.mit.edu

Quentin Smith scripts@mit.edu

Student Information Processing Board

October 29, 2019

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-2
SLIDE 2

Services Backend Further Info

Outline

1

Services

Web Mail Cron (“Shortjobs”) SQL Version control

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-3
SLIDE 3

Services Backend Further Info

Outline

1

Services

Web Mail Cron (“Shortjobs”) SQL Version control

2

Backend

AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-4
SLIDE 4

Services Backend Further Info

Outline

1

Services

Web Mail Cron (“Shortjobs”) SQL Version control

2

Backend

AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

3

Further Info

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-5
SLIDE 5

Services Backend Further Info Web Mail Cron (“Shortjobs”) SQL Version control

Outline

1

Services

Web Mail Cron (“Shortjobs”) SQL Version control

2

Backend

AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

3

Further Info

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-6
SLIDE 6

Services Backend Further Info Web Mail Cron (“Shortjobs”) SQL Version control

Apache

Everyone wants Apache Apache’s default configuration isn’t safe for scripting Scripting requires code execution—mod php, mod perl, mod python, mod wsgi Apache normally runs everything as apache/nobody How to secure?

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-7
SLIDE 7

Services Backend Further Info Web Mail Cron (“Shortjobs”) SQL Version control

Apache

Everyone wants Apache Apache’s default configuration isn’t safe for scripting Scripting requires code execution—mod php, mod perl, mod python, mod wsgi Apache normally runs everything as apache/nobody How to secure? suEXEC—allows Apache to spawn a process as the user. . . . . . even for static content!

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-8
SLIDE 8

Services Backend Further Info Web Mail Cron (“Shortjobs”) SQL Version control

suEXEC

setuid program Passed the request by Apache Verifies that the script is in the web scripts directory Switches to the uid of the file and executes Even for static files!

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-9
SLIDE 9

Services Backend Further Info Web Mail Cron (“Shortjobs”) SQL Version control

Postfix

Standard Postfix server No local mailboxes All mail is passed to procmail mailbox_command = /usr/bin/procmail -t -p \

  • a "${EXTENSION}" ~/mail_scripts/procmailrc

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-10
SLIDE 10

Services Backend Further Info Web Mail Cron (“Shortjobs”) SQL Version control

procmail

Reads ~/mail_scripts/procmailrc from user’s home directory Users can do whatever they want with messages AFS causes problems—No way to know if failure is temporary (file server is down) or permanent (user isn’t signed up for mail scripts) All procmail failures are treated as temporary, so mail is queued

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-11
SLIDE 11

Services Backend Further Info Web Mail Cron (“Shortjobs”) SQL Version control

Cron (cronie)

Crontabs are currently stored locally on scripts servers cronload command loads the crontabs from ~/cron_scripts/crontab

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-12
SLIDE 12

Services Backend Further Info Web Mail Cron (“Shortjobs”) SQL Version control

Cron (cronie)

Crontabs are currently stored locally on scripts servers cronload command loads the crontabs from ~/cron_scripts/crontab Needs improvement Cron does not fail over with Web and Mail

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-13
SLIDE 13

Services Backend Further Info Web Mail Cron (“Shortjobs”) SQL Version control

sql.mit.edu

Though scripts.mit.edu makes use of sql.mit.edu, it’s a separate SIPB service with different maintainers. sql.mit.edu provides MySQL databases to scripts users and anyone else SQL data is stored locally, replicated across multiple servers Nightly backups go into AFS

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-14
SLIDE 14

Services Backend Further Info Web Mail Cron (“Shortjobs”) SQL Version control

SVN and Git hosting

Not well documented svn://username.scripts.mit.edu/ and git://username.scripts.mit.edu/ Uses suEXEC to run a svnserve / git-daemon as the user /mit/username/Scripts/{svn,git} git:// is read-only, so future plans for svn+ssh:// and git+ssh://

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-15
SLIDE 15

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

Outline

1

Services

Web Mail Cron (“Shortjobs”) SQL Version control

2

Backend

AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

3

Further Info

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-16
SLIDE 16

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

AFS access controls

AFS enforces server side access controls. On Athena systems: user’s password → Kerberos tickets → AFS tokens, which authenticate the client to the AFS server. On scripts, we don’t have the user’s password or tickets. User’s scripts are not publicly readable. Access is controlled through a single daemon.scripts AFS user.

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-17
SLIDE 17

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

Isolating users on scripts

If all users share daemon.scripts AFS tokens, how are they prevented from accessing each other’s web scripts? On scripts, we enforce additional restrictions in the AFS kernel module.

afsAccessOK() in

  • penafs/src/afs/VNOPS/afs vnop access.c

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-18
SLIDE 18

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

You can only use daemon.scripts credentials to access files in a volume with volume ID equal to your UID,

int afs_AccessOK(struct vcache *avc, afs_int32 arights, struct vrequest *areq, afs_int32 check_mode_bits) { ... + if (!(areq->realuid == avc->fid.Fid.Volume) && + !((avc->anyAccess | arights) == avc->anyAccess) && + !(arights == PRSFS_LOOKUP && areq->realuid == HTTPD_UID) && + !(arights == PRSFS_LOOKUP && areq->realuid == POSTFIX_UID) && + !(arights == PRSFS_READ && areq->realuid == HTTPD_UID && + avc->m.Mode == 0100777 || avc->apache_access) && + !(PRSFS_USR2 == afs_GetAccessBits(avc, PRSFS_USR2, areq)) && + !(PRSFS_USR3 == afs_GetAccessBits(avc, PRSFS_USR3, areq) && + areq->realuid == 0) && + !(PRSFS_USR4 == afs_GetAccessBits(avc, PRSFS_USR4, areq) && + (areq->realuid == 0 || areq->realuid == SIGNUP_UID))) { + return 0;

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-19
SLIDE 19

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

  • r the file is system:anyuser readable anyway,

int afs_AccessOK(struct vcache *avc, afs_int32 arights, struct vrequest *areq, afs_int32 check_mode_bits) { ... + if (!(areq->realuid == avc->fid.Fid.Volume) && + !((avc->anyAccess | arights) == avc->anyAccess) && + !(arights == PRSFS_LOOKUP && areq->realuid == HTTPD_UID) && + !(arights == PRSFS_LOOKUP && areq->realuid == POSTFIX_UID) && + !(arights == PRSFS_READ && areq->realuid == HTTPD_UID && + avc->m.Mode == 0100777 || avc->apache_access) && + !(PRSFS_USR2 == afs_GetAccessBits(avc, PRSFS_USR2, areq)) && + !(PRSFS_USR3 == afs_GetAccessBits(avc, PRSFS_USR3, areq) && + areq->realuid == 0) && + !(PRSFS_USR4 == afs_GetAccessBits(avc, PRSFS_USR4, areq) && + (areq->realuid == 0 || areq->realuid == SIGNUP_UID))) { + return 0;

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-20
SLIDE 20

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

  • r the apache or postfix users are doing a stat(),

int afs_AccessOK(struct vcache *avc, afs_int32 arights, struct vrequest *areq, afs_int32 check_mode_bits) { ... + if (!(areq->realuid == avc->fid.Fid.Volume) && + !((avc->anyAccess | arights) == avc->anyAccess) && + !(arights == PRSFS_LOOKUP && areq->realuid == HTTPD_UID) && + !(arights == PRSFS_LOOKUP && areq->realuid == POSTFIX_UID) && + !(arights == PRSFS_READ && areq->realuid == HTTPD_UID && + avc->m.Mode == 0100777 || avc->apache_access) && + !(PRSFS_USR2 == afs_GetAccessBits(avc, PRSFS_USR2, areq)) && + !(PRSFS_USR3 == afs_GetAccessBits(avc, PRSFS_USR3, areq) && + areq->realuid == 0) && + !(PRSFS_USR4 == afs_GetAccessBits(avc, PRSFS_USR4, areq) && + (areq->realuid == 0 || areq->realuid == SIGNUP_UID))) { + return 0;

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-21
SLIDE 21

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

  • r apache is trying to read a file with mode 777 or named “.ht*”,

int afs_AccessOK(struct vcache *avc, afs_int32 arights, struct vrequest *areq, afs_int32 check_mode_bits) { ... + if (!(areq->realuid == avc->fid.Fid.Volume) && + !((avc->anyAccess | arights) == avc->anyAccess) && + !(arights == PRSFS_LOOKUP && areq->realuid == HTTPD_UID) && + !(arights == PRSFS_LOOKUP && areq->realuid == POSTFIX_UID) && + !(arights == PRSFS_READ && areq->realuid == HTTPD_UID && + avc->m.Mode == 0100777 || avc->apache_access) && + !(PRSFS_USR2 == afs_GetAccessBits(avc, PRSFS_USR2, areq)) && + !(PRSFS_USR3 == afs_GetAccessBits(avc, PRSFS_USR3, areq) && + areq->realuid == 0) && + !(PRSFS_USR4 == afs_GetAccessBits(avc, PRSFS_USR4, areq) && + (areq->realuid == 0 || areq->realuid == SIGNUP_UID))) { + return 0;

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-22
SLIDE 22

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

  • r the root or signup users are accessing file with the special C,

D, or E bits.

int afs_AccessOK(struct vcache *avc, afs_int32 arights, struct vrequest *areq, afs_int32 check_mode_bits) { ... + if (!(areq->realuid == avc->fid.Fid.Volume) && + !((avc->anyAccess | arights) == avc->anyAccess) && + !(arights == PRSFS_LOOKUP && areq->realuid == HTTPD_UID) && + !(arights == PRSFS_LOOKUP && areq->realuid == POSTFIX_UID) && + !(arights == PRSFS_READ && areq->realuid == HTTPD_UID && + avc->m.Mode == 0100777 || avc->apache_access) && + !(PRSFS_USR2 == afs_GetAccessBits(avc, PRSFS_USR2, areq)) && + !(PRSFS_USR3 == afs_GetAccessBits(avc, PRSFS_USR3, areq) && + areq->realuid == 0) && + !(PRSFS_USR4 == afs_GetAccessBits(avc, PRSFS_USR4, areq) && + (areq->realuid == 0 || areq->realuid == SIGNUP_UID))) { + return 0;

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-23
SLIDE 23

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

Serving static content

The apache user does not have permission to read the user’s files directly. Both static and dynamic content is served through suEXEC.

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-24
SLIDE 24

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible 1 /etc/httpd/conf.d/execsys.conf is configured to serve

static content with the cgi-script handler.

<FilesMatch ‘‘(?i)\.(cgi|exe|php|pl|py|scm)$’’> SetHandler cgi-script Options +ExecCGI </FilesMatch> <FilesMatch ‘‘(?i)\.(avi|css|doc|docm|docx|...|zip)$’’> SetHandler cgi-script Options +ExecCGI </FilesMatch> ...

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-25
SLIDE 25

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible 3 httpd/support/suexec.c is modified to dispatch static

content to /usr/local/bin/static-cat.

+#define STATIC_CAT_PATH "/usr/bin/static-cat" +static const char *static_extensions[] = { + "html", + "css", + ... +} + int main(int argc, char *argv[]) { ... + + if (is_static_extension(cmd)) { + if (setenv("PATH_TRANSLATED", cmd, 1) != 0) { + log_err("setenv failed\n"); + exit(255); + } + execl(STATIC_CAT_PATH, STATIC_CAT_PATH, (const char *)NULL); + log_err("(%d)%s: static-cat exec failed (%s)\n", errno,

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-26
SLIDE 26
slide-27
SLIDE 27

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

Group locker support

“Users” on scripts are actually lockers. User IDs are actually locker volume IDs.

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-28
SLIDE 28

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

Group locker support

“Users” on scripts are actually lockers. User IDs are actually locker volume IDs. Kerberos is modified to let users SSH in as any locker they administrate.

Replaced the .k5login mechanism: krb5 kuserok() in krb5/src/lib/krb5/os/kuserok.c Calls a Perl script /usr/local/sbin/admof to do the actual check.

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-29
SLIDE 29

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

krb5_boolean KRB5_CALLCONV krb5_kuserok(krb5_context context, krb5_principal principal, const char *luser) { ... + if ((pid = fork()) == -1) + goto cleanup; + if (pid == 0) { +#define ADMOF_PATH "/usr/local/sbin/ssh-admof" + exec(ADMOF_PATH, ADMOF_PATH, (char *) luser, princname, NULL); + exit(1); + } + if (waitpid(pid, &status, 0) > 0 && WIFEXITED(status) && + WEXITSTATUS(status) == 33) { + result = ACCEPT; + } ... }

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-30
SLIDE 30

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

LDAP architecture

All user-specific information is stored in LDAP records scripts-ldap-1 through scripts-ldap-3 run LDAP daemons with multi-master replication Each realserver runs a read-only local caching LDAP daemon

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-31
SLIDE 31

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

LDAP data

Each user has a scriptsAccount and at least one scriptsVhost Users can request additional virtual hosts using “pony” scriptsAccount is consulted by Postfix for mail routing (so accounts can be blocked) scriptsVhost is consulted by a cron job for SSL certificates

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-32
SLIDE 32

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

Apache modules

We make it easy to do authentication against MIT certificates. Both https://scripts-cert.mit.edu, and port 444 on any scripts hostname, are configured to request client certificates. mod ssl provides the SSL CLIENT S DN Email environment variable, but does not integrate with the Apache authentication and authorization framework. Wrote a collection of Apache modules to make this cleaner.

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-33
SLIDE 33

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

mod auth sslcert

mod auth sslcert passes the SSL CLIENT S DN Email variable to the Apache authorization handlers. AuthType SSLCert AuthSSLCertVar SSL_CLIENT_S_DN_Email AuthSSLCertStripSuffix "@MIT.EDU"

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-34
SLIDE 34

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

mod authz afsgroup

mod authz afsgroup does Apache authorization based on AFS groups. Require afsgroup system:scripts-team

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-35
SLIDE 35

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

mod auth optional

mod auth optional subverts the authorization process to allow you to serve different pages to users with certificates and users without certificates.

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-36
SLIDE 36

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

Linux Virtual Server

Provides high availability and load balancing Pacemaker provides failover between LVS “directors” ldirectord keeps track of online scripts servers and chooses destination server for each request

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-37
SLIDE 37

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

Load Balancing

Users are assigned to scripts servers based on IP Works around bugs in scripts that assume a single web server

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-38
SLIDE 38

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

Load Balancing status

http://scripts.mit.edu:78/ shows the current load Or you can finger @scripts.mit.edu for more detail

$ finger @scripts [scripts.mit.edu] IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags

  • > RemoteAddress:Port

Forward Weight ActiveConn InActConn FWM 1 wrr

  • > CATS-WHISKERS.MIT.EDU:0

Route 1 13 6 FWM 2 wlc persistent 600

  • > SHINING-ARMOR.MIT.EDU:0

Route 4096 53 855

  • > BEES-KNEES.MIT.EDU:0

Route 4096 50 2140

  • > CATS-WHISKERS.MIT.EDU:0

Route 1024 17 53

  • > BUSY-BEAVER.MIT.EDU:0

Route 4096 54 641

  • > PANCAKE-BUNNY.MIT.EDU:0

Route 4096 52 1039 FWM 3 wlc persistent 600

  • > SHINING-ARMOR.MIT.EDU:25

Route 4096

  • > BEES-KNEES.MIT.EDU:25

Route 4096 1

  • > CATS-WHISKERS.MIT.EDU:25

Route 1024 1

  • > BUSY-BEAVER.MIT.EDU:25

Route 4096 1

  • > PANCAKE-BUNNY.MIT.EDU:25

Route 4096 2 Quentin Smith scripts@mit.edu scripts.mit.edu

slide-39
SLIDE 39

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

Configuration Management

Ansible is a tool for declarative configuration management We can install LVS, syslog, and real servers using Ansible Server configuration is modular, so each feature can be separately developed Hopefully will make future version upgrades easier

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-40
SLIDE 40

Services Backend Further Info AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

Example Role - real-fuse

  • name: Install fuse.conf

copy: dest: /etc/fuse.conf content: | user_allow_other

  • name: Load fuse kernel module

copy: dest: /etc/modules-load.d/fuse.conf content: | fuse notify: load modules

  • name: Immediately load new modules

meta: flush_handlers

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-41
SLIDE 41

Services Backend Further Info

Outline

1

Services

Web Mail Cron (“Shortjobs”) SQL Version control

2

Backend

AFS suEXEC Kerberos LDAP Apache modules LVS Ansible

3

Further Info

Quentin Smith scripts@mit.edu scripts.mit.edu

slide-42
SLIDE 42

Services Backend Further Info

Further Info

Trac: https://scripts.mit.edu/trac/wiki/StarterTickets GitHub: https://github.com/mit-scripts/ Zephyr: -c scripts (http://sipb.mit.edu/doc/zephyr/) These slides: https: //web.mit.edu/scripts/doc/cluedump/slides.pdf

Quentin Smith scripts@mit.edu scripts.mit.edu