Geoapplications development http://rgeo.wikience.org Higher School - PowerPoint PPT Presentation

Geoapplications development http://rgeo.wikience.org Higher School of Economics, Moscow, www.cs.hse.ru

2 Agenda

3 Agenda (cont.) → → →

4 Netty Netty is a NIO client-server framework, which enables quick and easy development of network applications, such as protocol servers and clients.

5 Interactive mapping service Problem : 1. Web application displays an interactive map in a browser 2. It views additional layers on top of the map, say air temperature 3. User pans and zooms in/out the map (applies synchronously for all layers ) 4. Client (software) cannot store all data at its end (e.g., due to data volume) 5. Thus, client must be able to dynamically obtain portions of data to display for a given geographical area and scale (after each pan/zoom) The typical model is as follows (more details are being added slide by slide) : Internet

6 Network protocol & messages 1. We need to exchange data over network, we want a primitive for this 2. Call it message – data that can be sent/received over network 3. Think of it as a mail/parcel, you can fill it in with data or command 4. “message” = client query or server response 5. Network protocol – fixed set of possible message types , their format (internal structure) and, possibly, rules (in what order certain types of messages are allowed to be sent) Internet

Protocol definition (in Russian) Протокол передачи данных — набор соглашений, которые определяют обмен данными между различными программами. Сетевой протокол — набор правил и действий (очерёдности действий), позволяющий осуществлять соединение и обмен данными между двумя и более включёнными в сеть устройствами. In English: https://en.wikipedia.org/wiki/Communications_protocol

8 WCS (Web Coverage Service) 1. Retrieve raster data over network from a server that supports WCS 2. Built on top of HTTP 3. Allows to subset a given dataset (by time, extent, etc.) 4. Raster is returned as Coverage – raster with its metadata (extent, projection, etc.), ISO standard: https://en.wikipedia.org/wiki/Coverage_data 5. Key command: GetCoverage 6. Set of supported output formats depends on server implementation, usually GeoTIFF, NITF, HDF, JPEG, JPEG2000, PNG. Internet GeoTIFF, PNG, …

9 OSI model What is “on top of HTTP”?

OSI model: protocols

11 WCS – GetCoverage GetCoverage is an HTTP GET request (spaces are not allowed, they are for clarity only!) <URL> ? <key> = <value> {& <key> = <value>}* BNF style /thredds/wcs/galeon/striped.nc Server URL ?request=GetCoverage Command name &version=1.0.0 WCS protocol version &service=WCS Server-specific (use WCS) &format=GeoTIFF Output format &coverage=ta Dataset name &time=2005-05-10T00:00:00Z Time (subset) &vertical=100.0 Vertical level (subset) &bbox=-134,11,-47,57 Bounding box (extent, subset) https://www.unidata.ucar.edu/software/thredds/current/tds/reference/WCS.html

12 WCS – more details Pros : very simple, output format contains a lot of metadata Cons : formats too complex for web applications (only PNG usually used) Desktop GIS (e.g. Quantum GIS) easily accept NetCDF and other formats Situation changes now : geotiffjs, xdrjs , … Current versions: 1.0.0, 1.1.0, 1.1.1, 1.1.2 and 2.0.1. Types of queries ( except GetCoverage – see prev. slides ) • GetCapabilities – get XML with available coverages, supported operations (usually GetCoverage and DescribeCoverage), some server metadata <server_url>?request=GetCapabilities&version=1.0.0&service=WCS • DescribeCoverage – get XML with metadata for a given coverage <server_url>?request=DescribeCoverage&version=1.0.0&service=WC S&coverage=ta

13 WCS – TDS implementation Again: WCS is a simple protocol with some very complex output formats • Clone or browse https://github.com/Unidata/thredds • See readings section at the end

14 WFS – Web Feature Service ≈ WCS, but for vector data with some minor differences: • supports both reading and writing data (bidirectional) • provides basic processing capabilities (filtering, reprojection, etc.) • Output formats: GML, GeoJSON, CSV, etc. • etc.: see links in readings WPS – Web Processing Service Typical workflow: Get capabilities, Submit task, Poll for completeness, Retrieve Results The name tells for itself, we do not delve into, see readings section

15 WMS – Web Mapping Service One of the first geospatial protocols • Basic notions are renderer, layer, style • Delivers images rendered according to given style (e.g. PNG, but PDF, SVG, SWF may also be supported as output formats) • Vector data may be one of the layers and rendered as images as well • Allows basic processing like reprojection “on the fly” Pros : OK both for web and desktop apps Cons : very slow

16 Playing with WCS, WMS, OpenDAP metadata Climate Forecast System Reanalysis data are also published with TDS (this server will be covered later) which supports those protocols: http://nomads.ncdc.noaa.gov/data.php?name=access#CFSR-data

17 Event-driven architecture 1. The moments at which user pans or zooms are unknown 2. Thus, the moments at which server receives requests are also unknown 3. Let’s generalize user activity: call pan/zoom/etc. as GUI events 4. Let’s generalize network I/O: client receives a response = I/O event, server receives request = I/O event . 5. We will use on<EventName>(<params>) notation for GUI events 6. We will use onMessageReceived(Message msg) for network events onPan(double distance) onMessageReceived onZoom(int level) (Message msg) Internet onMessageReceived (Message msg)

18 Event-driven architecture (2) Q : what another events do we need? • onConnect, onDisconnect, onIdle, etc. • I/O pattern: • connect, query* N times, disconnect • I/O pattern for connectionless protocols (e.g. HTTP): • connect, 1 query, disconnect (overhead for connecting each time, but there are keepAlive and similar optimizations) • Abrupt disconnect, connection time outs, disconnect on idle Q: what reacts on events? • Event handlers • They are methods named after event names: onConnect, etc. Called by a framework (yours or 3 rd party) when event occurs •

19 Client and server (asynchronous, keep conn.) Server Client startup { onGUIEvent (Event e) { listen(IP:PORT) connect(IP:PORT) } // connects to server // if not already connected // event handlers onMessageReceived(Message msg) { sendQuery(msg) // asynchronous: if (msg instanceof GetCoverage) { // immediately exits // not best way } // to check message type } // I/O event handler } onMessageReceived(Message msg) { if (msg instanceof ResponseMsg) { onConnect(clientIP, clientPort) { } LOG.info(…); } }

20 Multi-user service • In real world, there are 100s or 1000s of concurrent clients • They generate 100s of queries per sec • Server is a multithreaded application, each thread is assigned a query upon its arrival (typically threads# = k*CPU cores) • A load balancer, query queues may be used; Q : why? thread occupied with thread query processing Internet time

21 Sockets • How to maintain connections from 1000s of users? • Blocking (wait an operation to complete) and non-blocking sockets: https://www.scottklement.com/rpg/socktut/nonblocking.html • Java networking model:

22 Server public class Main extends Thread { static int server_port = 1000 ; int mode = 0 ; static BufferedReader bir = null; static BufferedWriter biw = null; public static void main ( String [] args ) { try { ServerSocket ss = new ServerSocket ( server_port ); Socket s = ss . accept (); bir = new BufferedReader ( new InputStreamReader ( s . getInputStream ())); biw = new BufferedWriter ( new OutputStreamWriter ( s . getOutputStream ()));

23 Server ExecutorService execs = Executors . newFixedThreadPool ( 2 ); execs . submit (new Main ( 0 )); execs . submit (new Main ( 1 )); bir . close (); biw . close (); ss . close (); } catch ( IOException ex ) { ex . printStackTrace (); } // execs.shutdown(); }

24 Server public Main ( int mode ){ this. mode = mode ; } public void run (){ try { String line = null; while ( ( line = bir . readLine ()) != null ){ String ready = DateFormat . getDateTimeInstance (). format (new Date ()); System . out . println ( line ); System . out . println ( ready ); } } catch ( IOException ex ) { ex . printStackTrace (); } } }

Why do we need buffering (and small messages)? MTU by Jeffry Dean (Google)

26 Client public class Main extends Thread { static final int server_port = 1000 ; int mode = 0 ; // send/receive static BufferedWriter bf = null; static BufferedReader br = null; public static void main ( String [] args ) { try { Socket s = new Socket ( "127.0.0.1" , server_port ); bf = new BufferedWriter (new OutputStreamWriter ( s . getOutputStream ())); br = new BufferedReader (new InputStreamReader ( s . getInputStream ()));