Go London User Group - 21 st November 2018 ● Rclone “rsync for cloud storage” – https://rclone.org – https://github.com/ncw/rclone ● Talk by – Nick Craig-Wood – T witter: @njcw – Email: nick@craig-wood.com 1 rclone.org Nick Craig-Wood
About me ● Nick Craig-Wood – CTO of Memset Ltd by day – Open Source coder by night – Keen interest in storage, data integrity – Reformed data hoarder (ha!) 2 rclone.org Nick Craig-Wood
Contents ● About Me ● What Rclone Is ● History ● How it works ● Some code ● Testing ● Libraries 3 rclone.org Nick Craig-Wood
Rclone - “rsync for cloud storage” ● Rclone is a command line program to sync fjles and directories to and from cloud providers ● MD5/SHA1 hashes checked at all times for fjle integrity ● Timestamps preserved on fjles ● Copy mode to just copy new/changed fjles ● Sync (one way) mode to make a directory identical ● Check mode to check for fjle hash equality ● Can sync to and from network, eg two difgerent cloud accounts ● Encryption backend ● Cache backend ● Optional FUSE mount (rclone mount) 4 rclone.org Nick Craig-Wood
Rclone vs Rsync ● rsync is a utility for effjciently transferring ✓ and synchronizing fjles across computer From Wikipedia systems, by checking the timestamp and size of fjles. ● It is commonly found on Unix-like systems ✓ and functions as both a fjle synchronization and fjle transfer program. ✗ ● The rsync algorithm is a type of delta encoding, and is used for minimizing network usage. 5 rclone.org Nick Craig-Wood
Cloud providers supported by rclone ● Amazon Drive ● Microsoft Azure Blob Storage ● Microsoft OneDrive ● Amazon S3 ● Minio ● Backblaze B2 ● Nextcloud ● Box ● OVH ● Ceph ● OpenDrive ● DigitalOcean Spaces ● Openstack Swift ● Dreamhost ● Oracle Cloud Storage ● Dropbox ● ownCloud ● FTP ● pCloud ● Google Cloud Storage ● put.io ● Google Drive ● QingStor ● HTTP ● Rackspace Cloud Files ● Hubic ● SFTP ● Jottacloud ● Wasabi ● IBM COS S3 ● WebDAV ● Memset Memstore ● Yandex Disk ● The local fjlesystem ● Mega 6 rclone.org Nick Craig-Wood
Rclone platforms CPU OS I ♥ Cross Compilation 7 rclone.org Nick Craig-Wood
How rclone came to be ● Started as a tool to exercise – github.com/ncw/swift – originally was “swiftsync” ● First version in 2012 – Go 1.0 – 3 backends ● Somewhat outgrew its original design! 8 rclone.org Nick Craig-Wood
Why Go? ● Single binary deploy ● Excellent concurrency ● Great cross platform ● Fast! Why? ● Standard library ● New challenge for me ● Easy for contributors to pick up 9 rclone.org Nick Craig-Wood
One tool to rule them all ● What started as a tiny exercise – 11,000 stars on Github – 200 contributors – 500 pull requests – 1,500 issues – 250,000 downloads a month – Packaged in Ubuntu, Arch, Debian, Homebrew, Chocolatey and more ● ...is now an enormous project. 10 rclone.org Nick Craig-Wood
Visualising Rclone’s History 11 rclone.org Nick Craig-Wood
Rclone becomes popular and breaks Amazon Cloud Drive ⇒ ? 12 rclone.org Nick Craig-Wood
Rclone verbs – bigger = more popular 13 rclone.org Nick Craig-Wood
rclone confjg - Confjg Wizard ● Old School Confjg Wizard – T ext based – Easy to use – Not pretty – Calls your browser to do oauth 14 rclone.org Nick Craig-Wood
rclone copy - demo ● rclone copy – Copy new fjles to destination – Don’t delete fjles from destination – Your go to rclone command! 15 rclone.org Nick Craig-Wood
rclone sync - demo ● rclone sync – Copy new fjles to destination – Delete destination fjles not in source – Use with –dry- run fjrst recommended 16 rclone.org Nick Craig-Wood
rclone copy “Source Dir” “Dest Dir” Source Dir Dest Dir Source Dir Dest Dir Copied File 1 File 1 File 1 File 2 File 2 Not T ouched File 2 File 2 File 3 Old File 3 File 3 File 3 Overwritten Not T ouched File 4 File 4 Destination includes Source Source Destination Source Destination Actions Before Before After After 18 rclone.org Nick Craig-Wood
rclone sync “Source Dir” “Dest Dir” Source Dir Dest Dir Source Dir Dest Dir Copied File 1 File 1 File 1 File 2 File 2 Not T ouched File 2 File 2 File 3 Old File 3 File 3 File 3 Overwritten Deleted File 4 Destination identical to Source Source Destination Source Destination Actions Before Before After After 19 rclone.org Nick Craig-Wood
rclone mount remote:path /mount/point ● FUSE Filesystem – Linux, macOS, FreeBSD – Windows va WinFSP ● Optional caching layer – Needed as can’t write to middle of object – Or read and write together ● Can run as daemon 21 rclone.org Nick Craig-Wood
rclone ncdu This displays a text based user interface allowing the navigation of a Remote. It is most useful for answering the question: What is using all my disk space? 22 rclone.org Nick Craig-Wood
Backend interface 23 rclone.org Nick Craig-Wood
Object interface 24 rclone.org Nick Craig-Wood
Optional interfaces for Fs 25 rclone.org Nick Craig-Wood
Using an optional interface Do a type assertion for – the interface to see if it exists. But what if this is a – wrapper backend wrapping a backend that doesn’t support Purge? And if we need to know – in advance?... 26 rclone.org Nick Craig-Wood
The solution 27 rclone.org Nick Craig-Wood
T esting ● How to test ● Unit test what we can – 27 backends – Some things are easy – x 50 commands – Who wants to write mocks for 27 difgerent cloud – x 8 OSes providers? – x 6 CPU Architectures ● Integration test – x 4 Go versions? – Integration tests use go ● 69k lines of code test framework ● 26k lines of test code – Run daily 28 rclone.org Nick Craig-Wood
CI – Unit testing and build ● CI Pipeline Push – Runs all non Pull Request integration tests – T ests mount – Builds for all – Makes binaries – Uploads to beta Push release Pull Request 29 rclone.org Nick Craig-Wood
Integration testing ● Integration test Integration T est Server – Run daily Subset of cloud providers Daily Pull – T oo expensive to At least one per backend run on every push Cost ~ 30p ● Time ~ 1 Hour ● – Creates fancy report – Not integrated with Github (yet) FTP SFTP HTTP Crypt 30 rclone.org Nick Craig-Wood
Integration tests ● Problems – Cloud providers aren’t perfectly reliable – Eventual consistency – Networking ● Solution – Retries, Retries, Retries – Lots of work getting it right 31 rclone.org Nick Craig-Wood
Retrying integration tests ● test_all framework Attempt 1/5 ./operations.test – Runs standard go tests -test.v -test.timeout 30m0s – Runs lots of tests in parallel -remote TestAzureBlob: – Provides fmags as specifjed in a confjg fjle – Parses the output of the tests Attempt 2/5 ./operations.test – Retries the just the failing tests -test.v – Should probably become an -test.timeout 30m0s -remote TestAzureBlob: opensource package in its own -test.run '^(TestPurge| right! TestRmdirsNoLeaveRoot)$' 32 rclone.org Nick Craig-Wood
Integration tests for backends ● Backend integration tests – Easy to add thanks to go1.6 nested tests – Give a recipe to follow when making a new backend – Just make the integration tests pass – Originally done with code gen pre go1.6 33 rclone.org Nick Craig-Wood
Integration tests elsewhere ● You can add fmags to tests – Rclone uses this with a “-remote” fmag to signal that the test should be done remotely – There are other fmags for debugging and more in depth tests 34 rclone.org Nick Craig-Wood
Standing on the shoulders of giants ● Rclone ● Rclone’s libraries – 95,000 lines of code – 520,000 lines of code – 450 source fjles – 1,100 fjles – Not including “vendor” – All stored in “vendor” All build on top of the excellent standard library 35 rclone.org Nick Craig-Wood
Favourite libraries and tools: golang.org/x/tools/cmd/goimports – Get it in your editor – never type an import statement again – Run it as a save hook – it will `go fmt` your code too 36 rclone.org Nick Craig-Wood
github.com/spf13/cobra ● Make commands with subcommands ● Very fmexible / extensible ● Used by Kubernetes / Hugo / Docker ● POSIX fmags `--fmag` with spf13/pfmag ● Creates bash completion scripts ● Creates docs ● Makes cofgee and cleans the kitchen. 37 rclone.org Nick Craig-Wood
Documentation with github.com/spf13/cobra …becomes -h output… …and markdown for web. Go code defjnes help… 38 rclone.org Nick Craig-Wood
github.com/pkg/errors ● Turns an error like this – “unexpected EOF” ● Into – “NewFs creating backend: couldn’t connect SSH: unexpected EOF” 39 rclone.org Nick Craig-Wood
What to do if your open source project takes ofg... ● Don’t Panic! Rclone Star History ● Open a forum (Discourse is good) ● Ask everyone who makes an issue for help ● Recruit pull requesters as contributors Front Page of Hacker News ● Make good contributing docs ● Get octobox.io 40 rclone.org Nick Craig-Wood
Recommend
More recommend
