Prophet a path out of the cloud http://syncwith.us jesse@bestpractical.com 1
You may know me from... RT (Request Tracker) Jifty SVK Hiveminder Perl 6 Shirts 2
I’ve been hacking on an open source database called “Prophet” 3
It has an API like Amazon SimpleDB or Google App Engine’s... 4
It’s designed for “team-scale” apps 5
It’s built for P2P replication and disconnected use 6
App #1 is the canonical “offline bug tracker” 7
App #2 will probably be a BBS you can sync over sneakernet 8
But first, a brief digression... 9
...about cloud computing 10
Living in the cloud = sharecropping 11
(That’s bad) 12
This is a rant 13
The bad old days: 14
Pic of sharecroppers 15
You farmed land you didn’t own... 16
...with tools you couldn’t really afford 17
You paid for it with part of your harvest... 18
It sounded like a pretty sweet deal... 19
...until things got bad 20
(Things always got bad) 21
In a bad year, you got further in debt to the land owner 22
23
The (more recent) bad old days: 24
pic of mainframes 25
You ran code you didn’t own on hardware you didn’t own 26
Things got a little better: 27
Pic of PCs 28
Things weren’t all rosy: 29
Pic of BSOD 30
Sometimes new versions of software killed features... 31
...so you were locked in to old versions 32
pic of win 31? 33
Things got ‘better’: 34
rms che 35
Now, things are getting worse again... 36
37
What happens when your favorite service goes down? 38
pic of twitter being down 39
...or stops accepting new signups? 40
41
...or gives all your data to the secret police? 42
Pic of yahoo.cn 43
...or starts making arbitrary choices about what’s ‘safe’ content? 44
45
You don’t own the services you use 46
When the service provider cuts you off, that’s it. No recourse. 47
Not so secret shame: I’m a really bad zealot 48
My calendar lives at google.com 49
50
I make a web 2.0 tasklist service called Hiveminder.com 51
pic of hiveminder 52
Using hosted apps is going to hurt you 53
Data access is important 54
APIs are great 55
...but easy access to a service just makes it easier to get locked in 56
What about Google Gears, Adobe Air, etc? 57
Great. now you can use your word processer while you’re offline! 58
Pic of wordperfect 59
Real offline apps shouldn’t need servers 60
Real offline apps should sync like you do 61
I might be a nut job 62
...but smart people seem to agree with me 63
If we want people to have the same degree of user autonomy as we've come to expect from the world, we may have to sit down and code alternatives to Google Docs, Twitter, and EC3 that can live with us on the edge, not be run by third parties. - Danny O’Brien http://www.oblomovka.com/entries/2008/07/16 64
Back to that database thing... 65
Jesse Vincent 66
Chia-liang Kao 67
We work together 68
CL lives in Taipei Jesse lives in Boston 69
Sometimes we need to work face to face 70
TPE - BOS: 9410 mi TPE - HNL: 5,095 mi BOS - HNL: 5,069 mi 71
Our Plan Step 1: Go to Hawaii for “work” Step 2: ??? Step 3: Prophet! 72
The Plan Backfired We were there for 8 days We wrote 8000 lines of Perl We figured out step 2 73
Step 2: Build a Disconnected Syncable Database 74
Prophet 75
Getting Prophet Prophet http://syncwith.us/prophet SD http://syncwith.us/sd 76
Prophet A grounded, semirelational, peer to peer replicated, disconnected, versioned, property database with self-healing conflict resolution 77
What do all those buzzwords mean? 78
grounded Runs here 79
grounded Not here 80
grounded Runs at the edge Doesn’t need to run in the cloud Syncs with services you already use (We call the adaptors “Foreign Replicas”) 81
semirelational Joins are expensive (They’re still possible) 82
peer-to-peer replicated Update any replica Pull from any replica Push to any replica Publish a replica Changes will propagate 83
disconnected Real-time replication is hard to scale It only “works” with constant connectivity I don’t have constant connectivity Neither do you Prophet sync can happen whenever 84
versioned Every update is recorded as a change set Change sets don’t lose any data (so you can use them to go backwards) All history is introspectable Replication just replays changesets 85
property database Atomic operations CREATE, READ, UPDATE, DELETE, SEARCH Record types can have optional validation and canonicalization Records of the same type do not need to have the same properties Add and remove properties at will 86
self-healing conflict resolution Remembers all conflict resolutions Syncs all resolutions with your peers Detects identical conflicts Uses your peers’ resolutions to “vote” for the winner of a conflict 87
Working with Prophet 88
RESTy API GET /records.json GET /records/Cars.json GET /records/Cars/716499-5F9-4AC4-827.json GET /records/Cars/716499-5F9-4AC4-827/wheels.json POST /records/Cars.json POST /records/Cars/716499-5F9-4AC4-827.json POST /records/Cars/716499-5F9-4AC4-827/wheels.json 89
RESTy API Yes, we should be using PUT and DELETE Yes, you can have a commit bit and help us fix it :) 90
Native API (Yes, the core is Perl.) my $cli = Prophet::CLI->new(); my $cxn = $cli->app_handle->handle; my $record = Prophet::Record->new( handle => $cxn, type => 'Person' ); my $uuid = $record->create( props => { name => 'Jesse', age => 31 } ); $record->set_prop( name => 'age', value => 32 ); my $people = Prophet::Collection->new( handle => $cxn, type => 'Person' ); $people->matching( sub { shift->prop('species') ne 'cat' } ); 91
What could you build with Prophet? 92
sd A bug tracker: “simple defects” • id. Status, Summary • (Arbitrary other properties too) • History • Comments • Attachments 93
Create ./bin/sd ticket create -- summary="Can't sync sd with Google Code" status=new Created ticket 5 (93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4) 94
List and Search ./bin/sd ticket search --regex publish 29 } new the online help doesn't describe publish 34 } new publish a static html view of records 35 } new publish should create a static rss file 95
Updates ./bin/sd ticket update --uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 -- status=resolved 96
Bugs on my laptop aren’t interesting. 97
Sync! Jesse sd publish --to fsck.com:public_html/sd/ CL sd pull --from http://my.com/~jesse/sd 98
My project has a bug tracker 99
My project has a bug tracker Actually, mine use two: • RT • hiveminder.com 99
Recommend
More recommend
