KolmoLD LD: Data Modelling for the Modern In Inter ernet* Dmitry Borzov, Huawei Canada Tim Tingqiu Yuan, Huawei Mikhail Ignatovich, Huawei Canada Jian Li, Futurewei *work performed before May 2019
Challenges: Peak k Traffic Composition 73% 26% Content: sizable, faned-out, static Streaming Services Everything else Software File storage (Netflix, Hulu, (Instant Messaging, distribution services YouTube, Spotify) VoIP, Social Media) [1] Source: Sandvine Global Internet Phenomena reports for 2009, 2010, 2011, 2012, 2013, 2015, 2016, October 2018
[1] https://qz.com/1001569/the-cdn-heavy-internet-in-rich-countries-will-be-unrecognizable-from-the-rest-of-the-worlds-in-five-years/
Technologies to define the revolution of the internet Te ChunkStream Founded in 2016 Video codec Content-addressable Browser-targeted Based on a 2014 MIT Founded in 2014 Runtime network protocol based research paper Content-addressable on cryptohash naming network protocol based scheme Implemented and on cryptohash naming supported by all major Based on the scheme Open source project browsers, an IETF cryptohash naming standard P2P project scheme Founding company is a YCombinator graduate YCombinator graduate with backing of high profile SV investors
Ou Our Proposa sal: A data model for interoperable pr protocols ls KolmoLD Content addressing through hashes has become a widely-used means of Addressable: connecting layer, inspired by connecting data in distributed the principles of Kolmogorov complexity systems, from the blockchains that theory run your favorite cryptocurrencies, to the commits that back your code, to Compossible: sending data as code, where the web’s content at large. code efficiency is theoretically bounded by Kolmogorov complexity Yet, whilst all of these tools rely on Computable: sandboxed computability by some common primitives, their treating data as code specific underlying data structures Democratizing networking of generic ICT are not interoperable . devices with a principled approach
Ko KolmoLD: Ko Kolmogorov Linked Data Kolmogorov Content- Ko Turin ing-co complete te addressable Data co composability programmability Users care about what they send a way to reproduce data, Sandboxed computability by want, not who they get it from not data itself treating data as code
Con Content-ad addres essab able le revolu lutio tion in in th the e mak akin ing ChunkStream KolmoLD - Content-addressable cryptohash naming - - Composability - - - Turing-complete Programmability
Ko KolmoLD Ko Kolmogorov Content- from fr Turing-complete addres ad essab able le programmability Data composability Users care abou Us out t what t th they send a way to reproduce data, Sandboxed computability by want, not who they get it wa treating data as code not data itself
Content-addressable networking: Streaming an Olympics game (Before) Internet Service Autonomous Provider Systems (ISP) (ASs) Content Provider Server
Content-addressable networking: Streaming an Olympics game (After) Internet Service Autonomous Provider Systems (ISP) (ASs) Content Provider Server
Ho Host-ad addres essab able le netw tworks kolmoblocks.org GET /video1 kolmoblocks.org Content-addressable networks GET 42FBCC0D60EADA7 Announce interest
How to represent data content and DIKW in general? Content-addressable networking 1) Consumers specify what they want, not who they need it from 2) The solution for content distribution
Claude Shannon Andrei Kolmogorov Alan Turing (1916-2001) (1903-1987) (1912-1954) Kolmogorov complexity: The shortest unambiguous algorithm, a computer program or code, that will output a given data string. Shannon entropy = expected [ Kolmogorov complexity ]
Kolmogorov complexity metric of the given string of data is the size of the shortest algorithm that outputs that data Kolmogorov complexity https://xkcd.com/1155/
1414213562373095048801688724209698078569671875376948073176 As an algorithm Calculate the first 100 60 terms of the series: 2) Calculate the ratio of the circle surface area to its radius squared
KolmoLD Ko Kolmogorov Content- Turing-complete Da Data comp ompos osability addressable programmability Users care about what they Sandboxed computability by se send a way to reproduce want, not who they get it from treating data as code data, no da not da data itsel elf
Data composability 1: Going through the EMUI example A security patch is A patched 200Mb A new version of the issued that flips a A 200Mb distro file is distro file is treated OS is released single byte at distributed across as completely new 0xfa24d4588 20M devices across by the network the world Dear phones, The new version of the distro can be composed out of the original one by flipping the bit at 0xfa24d4588. Love, the dev team
Data composability 2: don’t send data, send a way to reproduce data Al Algorithm thm FD FD62862 62862 A1 A1EF EF919 Concatenate images A1EF919, FD62862
Data composability 3: Reusing the encoding table 1Mb, book on potatoes 3Mb, book on cabbages 2Mb, book on tomatoes Huffman Tree Huffman Tree Huffman Tree Huffman-Encoded Text Huffman-Encoded Text Huffman-Encoded Text
Data Composability 4: Quantifying the impact Assumptions: • Huffman encoding, • Poisson distribution of the distinctly originating content blocks matching the statistical profile of the reported Internet traffic • Entropy of the English language text
Da Data Comp ompos osability 4: Quantifyi ying g the impact
Data composability 1. Data blocks are identified by cryptographic hash functions 2. Global address space of data 3. Compose data blocks out of other data blocks You don’t need to send it, just send a way to reproduce the data
Ko KolmoLD Turing-co Tu complete Kolmogorov Content- Data composability addressable send a way to reproduce data, not pr programmabi bility data itself Users care about what they want, Sandboxed ed computability by not who they get it from trea eating data as code
Turing-complete programmability 1 Encode data as programs that output given data # SHA256: CC646 ABABABF def render(): return “ABABABF” # SHA256: CC646 def render(): return “AB”*3+”F”
Turing-complete programmability 2 Referencing other kolmoblocks ABABABF-FBABABA # SHA256: F3025 def render(): # SHA256: CC646 p = dep(“CC646”) ABABABF return p+”-”+p[::-1]
Turing-complete programmability 3 Reference other data chunks based on cryptohash naming ABABABF-FBABABA # kolmoblock # target block: F3025 # dependancy blocks: 77650 # lambdablock 77650 def render(): def f(huffman_tree, encoded_string): huffman_decode = eval(dep('77650')) output = '' huffman_tree = ['AB', cur = 0 ['BA', while (cur < len(encoded_string)): ['F', '-'] node = huffman_tree ] while type(node) is list: ] bit = encoded_string[cur] encoded = 0b000110111110101010 cur += 1 return node = node[0] if bit else huffman_decode(huffman_tree, node[1] encoded) output += node return output
Turing-complete programmability 4 Domain-specific video codecs Self-driving car Academic videos Surveillance video LIDAR footage (lecture videos, talks and tutorials) Application-specific Some objects (people/ car E.g. depth resolution Text needs to be readable requirements plates) are higher priority Probabilistic profile Static scenery, Surrounding traffic video Emphasis on text, Traffic & weather Illustrations, screencasting Community / engineering Billion dollar industry Self-driving car industry Academic & Huawei resources to support it community
Turing-complete programmability 5 Timelines of adoption of new codecs is in decades
Turing-complete programmability 6 Vendoring software Dynamic c libraries Go compiler er binaries es: Co Containerizati tion: .dl dll / / .so so Ev Everything is statically Dock cker, Un Unikernels et etc incl cluded # SHA256: CC646: # Fibonacci def fibonacci(n): fib = [0] * (n + 1) fib[0] = 0 fib[1] = 1 for i in range(2, n + 1): fib[i] = fib[i - 1] + fib[i - 2] return fib[n]
Turing-complete programmability 1. Send data as programs that output the given data 2. Implement a sandboxed runtime that is secure and deterministic 3. Distribute the code along with the data Data as code: distribute the data and the code that reads it along the same channel
s and live demos @ @ https://kolmoblocks.org/ Mo More details
mple : Ex Exampl kolmoblock type: application/wasm+kolmold wasm: cat Hello world! datablock datablock type: text/plain type: text/plain datablock type: text/plain kolmoblock Hello world! type: application/wasm+kolmold wasm: cat He llo world! datablock datablock type: text/plain type: text/plain
Ex Exampl mple:
Kolmoblock, cid:”cat” Globals of wasm module’s instance type: application/wasm+kolmold dep1.low=0 def main(): dep1.high=6 out.low = dep1.low dep2.low=6 out.high = dep2.high dep2.high=12 out.low out.high Linear memory of the wasm module’s instance 9 0 3 6 10 11 1 2 4 5 7 8 H e l l o o r d w l ! Hello world! dep2, dependency datablock 2 dep1, dependency datablock 1
Recommend
More recommend
