DIY IoT Backend Voon Siong WONG greenfields greenfields 4 years - PowerPoint PPT Presentation

DIY IoT Backend Voon Siong WONG

greenfields

greenfields 4 years ago, wanted IoT platform no clear requirements grew organically flaws in hindsight never in production, never fixed hobby project

revisited the project made list of good, bad, and how it can be better development ~2 weeks, ~800 LOC Ruby runs on 1x t2.small instance response time average 50ms, 95% < 80ms learnt from a lot of bad experiences

disclaimer not necessarily advocating DIY outsourcing to 3rd party vendors can be cheaper depends on in-house capabilities

Let's Get Started

item 1: time series don't DIY time series v1 had to do it, not many open source v2 used open source database, threw out a ton of code

read/write optimisation is hard use 1-D sort for index optimisation usually time as primary index overwrite or append for same timestamp? aggregating across multiple tables? geo-spatial capabilities eg. measurements in this area at this time 4-D "sort"

we tried building one based on HBase careful planning of Row-key & Column-key only supported overwrite, no appends fixed time intervals aggregation were scheduled tasks, rather than queries poor performance

unless your job is to build databases, don't outside of your control if you're using vendor

item 2: authentication treat browsers and iot devices equally as first class citizens accessible with fixed or variable API keys optionally with username/password

avoid client SSL cumbersome to generate/manage impractical to install on browsers thus requires proxy server use API Keys fixed API keys variable API key, HMAC request and time deploy with environment variables interactive logins username/password oauth

item 3: expect change plan for it very relevant when DIY but use as criteria for judging vendors many vendors are still young

ensure APIs are versioned using path, headers, request params, etc. deprecations devices don't / can't understand warning messages warn developers directly return success on dead APIs don't care about their behaviour any more avoid old devices's retry logic from hammering it

API ownership APIs can make or break your application one of strongest reasons to DIY don't forget that it changes!

Item 4: model associations don't enforce a schema for model relationships embrace schema-less, because sensors are

schema / syntactic fixed associations, inflexible need to know "position" in hierarchy / grouping conflicts when reusing names schema-less / semantic uniqueness by credentials and name hierarchies or grouping stored in text, eg. "state/suburb/street", arbitrarily delimited

item 5: data flows trickles from device to server streams from server to advanced clients

use regular session-less HTTP for small volumes for greater volume, use a pub-sub mechanism firehose server , subscribe to desired channels modern clients use (uni-directional) websockets, resume on connection failures older clients fallback to long-polling

item *: nuts & bolts measurement units overwrite vs append don't require SNI prefer time since epoch text over binary

IoT Backend: Solved Problem?

IoT for hire authentication, data aggregation, remote control, etc. for hire still maturing sector...

focus on value this list apply whether you're DIY backend, or evaluating vendors not suggesting one or the other, that's up to you suggest you think about value, rather than technical capability I can, should I?

worth it? in our case, DIY was worth it (re)development effort was small lessons from previous generation in-house knowledge on how to fix things

LiveVu Realtime energy monitoring

SmokeAlert Air Quality Monitoring

disclaimer (repeat) not necessarily advocating DIY outsourcing to 3rd party vendors can be cheaper depends on in-house capabilities

Thanks!