Developing your own Swift middleware OpenStack Summit Atlanta, May - PowerPoint PPT Presentation

Developing your own Swift middleware OpenStack Summit Atlanta, May 2014

About me ● Christian Schwede ● Developer @ eNovance ● Mostly working on Swift, testing and automation ● Started using Swift in 2012

Agenda ● Using middlewares to extend Swift functionality ● Introduction to wsgi, middlewares and paste.deploy ● Developing our own middleware for Swift Proxy ● Testing & Packaging ● References

Writing Swift middlewares?

Swift middlewares? ● Most of Swift features are implemented as a middleware ○ logging, tempurl, dlo, ratelimit, tempauth, quotas, ... ● Flexibility to extend existing functionality ● No need to fork or modify existing code ● Several additional middlewares outside of Swift

wsgi and middlewares?

wsgi ● simple interface between web servers & web applications ● defined in PEP0333 and PEP3333 ● application ○ callable object (with a __call__ method) ● server ○ invokes the callable once for each client request Client Server App

application objects ● must accept two positional arguments ○ environ: Python dictionary ○ start_response: callable ■ status = „200 OK“ ■ headers = [(„header_name“, „header_value“), ]

environ ● REQUEST_METHOD ● PATH_INFO ● QUERY_STRING ● HTTP_HEADERNAME ● wsgi.input

Sample server from wsgiref.simple_server import make_server def myapp(environ, start_response): body = [] body.append("PATH_INFO: '%s'\n" % environ.get('PATH_INFO')) headers = [('Content-Type', 'text/plain')] start_response('200 OK', headers) return body srv = make_server('localhost', 8000, myapp) srv.serve_forever()

middleware ● can act as a server for an application and vice versa ● run multiple applications side-by-side ● authentication ● rerouting a request ● content processing Client Server Middleware App

Sample middleware class SummitMiddleware(object): def __init__(self, app, *args, **kwargs): self.app = app def __call__(self, env, start_response): response = self.app(env, start_response) if env.get('PATH_INFO') == '/echo': length = int(env.get('CONTENT_LENGTH') or 0) return env.get('wsgi.input').read(length) return response srv = make_server(‘localhost’, 8000, SummitMiddleware(myapp))

Testing, packaging & deploying

Testing a middleware class FakeApp(object): def __call__(self, env, start_response): start_response('200 OK', []) return "" class TestSummitMiddleware(unittest.TestCase): def test_simple_request(self): environ = {'REQUEST_METHOD': 'PUT'} req = Request.blank('/echo', environ, body="Hello World") mw = SummitMiddleware(FakeApp()) resp = req.get_response(mw) self.assertEqual("Hello World", resp.body)

paste.deploy ● load WSGI applications and servers from URI ● uses INI-style configuration files ● separates config from code ● Paste Script can serve applications from config files ● widely used in Openstack

config.ini for paste.deploy [app:sample] use = egg:sample#app [filter:middleware] use = egg:sample#middleware suffix = /echoresponse [pipeline:main] pipeline = middleware sample [server:main] use = egg:Paste#http port = 8000

setup.py setup(name='sample', packages=['sample', ], zip_safe=False, entry_points={ 'paste.app_factory': ['app=sample.app:app_factory'], 'paste.filter_factory': ['middleware = sample.middleware: filter_factory'] })

Writing a Swift middleware

Preview middleware ● PUT ○ create a small preview image and store it separate ● GET ○ return preview if QUERY_STRING contains “preview” ● DELETE ○ also delete preview if exists

Useful helpers ● swift.common.utils.split_path path = /v1/AUTH_account/test/img.jpg ver, acc, cont, obj = split_path(path) ● swift.common.wsgi.make_subrequest ○ add middleware after authentication middleware ● swift.common.swob.wsgify ○ decorator

wsgify? @wsgify def __call__(self, request): # request.params # request.path_info # request.method # request.environ # request.body return self.app

Return preview @wsgify def __call__(self, req): try: (ver, acc, con, obj) = split_path(req.path_info, 4, 4, True) except ValueError: return self.app preview_path= '/%s/%s/%s_%s/%s' % (ver, acc, con, self.suffix, obj) if req.method == 'GET' and request.params.has_key('preview'): req.path_info = preview_path

Extract preview if req.method == 'PUT': preview = create_preview(request.body) if preview: sub = wsgi.make_subrequest( request.environ, path=preview_path, body=preview) sub.get_response(self.app)

Delete preview if req.method == 'DELETE': sub = wsgi.make_subrequest(req.environ, path=preview_path) sub.get_response(self.app) return self.app

References

3rd party middlewares ● swauth - github.com/gholt/swauth ● swift3 - github.com/stackforge/swift3 ● CDMI - github.com/osaddon/cdmi ● Swift informant - github.com/pandemicsyn/swift-informant ● Swift Origin Server - github.com/dpgoetz/sos ● Ceilometer - ceilometer/objectstore/swift_middleware.py

Something to read ● github.com/enovance/swift-middleware-sample ● docs.openstack.org/developer/swift/ ○ middleware.html ○ development_middleware.html ○ development_auth.html ○ associated_projects.html ● legacy.python.org/dev/peps/pep-3333

THANK YOU! christian@enovance.com | @cschwede_de | OpenStack Juno Summit | May 2014, Atlanta