Dynamic analysis, reporting and visualization of data catalogs Use cases INSPIRE Dashboard for all Member States and EEA MedSea project at Ifremer
INSPIRE Directive in Europe INSPIRE is based on the infrastructures for spatial information established and operated by the 28 Member States of the European Union Yearly report made to the Commission. > INSPIRE MIWP-16 group goal is “to support MS in making annual report to INSPIRE” > INSPIRE dashboard prototype
Another need - MedSea checkpoint
Scenario #1 Collect #2 Analyze #3 Visualize & DSS
Collecting INSPIRE Monitoring XML MS CSW Metadata MS record CSW MS CSW XML
Analyzing
Reporting
Which tool for that ? We need ● a search engine ● a simple (& fast) harvester ● analysis framework ● reporting
Which tool for that ? We need ● a search engine ● a simple & fast harvester ● analysis framework ● dashboards banana/silk project
daobs - data analyzer and observer opensource on github
Collecting
Collecting through CSW Compute MS number of Index CSW pages ? p1 GetRecords with no hits to get number of records p2 p3 GetRecords
Harvesting fast Parallel MS Index CSW processing MS CSW configurable MS CSW
Harvester module Support CSW protocol Tested on 30 CSW endpoints from different MS harvesting more than 200K records Quite fast Based on Camel routing engine
Analyzing
Analysis - 3 types indexing background tasks reports
Analyzing / Indexing
First pass analysis Made when indexing records ● should be quick to not slow down imports ● based on Solr/Lucene analysis features ● basic multilingual support
Generic indexing with dynamic fields eg. categorize keywords
Keep word filter view download discovery OGC:WMS ISO19139 transformation view metadata view invoke Index record other KeepWordFilter
Synonym analysis inspireTheme=Ortofotosnímky xpath = index field Orthoimagery > Annex II Ortofotosnímky inspireTheme_syn=Orthoimagery inspireAnnex=II ISO19139 metadata Index record ManagedSynonymFilterFactory Synonyms
Analyzing / Background tasks
INSPIRE validation task (3) Save validation report and completeness indicator task (1) Get documents (2) Validate to validate
Indexing related documents Catalog of Document metadata record ● html page ● pdf ● spreadsheet ● dbf (in a shp) ● ... 4. Search on metadata and references a related documents 1.Get document URLs 3.Analyze and index extract content
Indexing Geographic Features Catalog of metadata record 4. Search on metadata and references a features 1.Get service URLs WFS 3.Analyze services and index 2.GetFeature
Tasks chaining JMS Messaging bus - topic & queue
Analyzing / Reports
Configure reports Variables = query Variables Variable Report configuration file = mathematical Indicators Indicators expression based on Indicator variables
Configure INSPIRE report INSPIRE monitoring reporting http://inspire.ec.europa.eu/index.cfm/pageid/5022 DSv_Num1 MDv11 +(resourceType:dataset resourceType:series) MDi11 +inspireAnnex:i MDv11 / DSv_Num1 DSv_Num DSv_Num MDi1 DSv_Num DSv_Num (MDv11 + MDv12 + MDv13 + MDv14) / (DSv_Num + SDSv_Num)
INSPIRE Report Monitoring INSPIRE configuration Monitoring file
Analysis & data visualization
Based on banana/silk project A dashboard web creation tool on top of Solr
A dashboard is a set of rows
Different widgets to build dashboard
TagClouds Graphs
Heatmaps
Maps
Dynamic filtering Click on items to apply filters
Dynamic filtering ●
Examples http://apps.titellus.net/dashboard/ https://inspire-dashboard.eea.europa.eu/dashboard http://sextant.ifremer.fr/dashboard/
Technical overview Report module Main app Spring Security Camel harvesters Solr index (AngularJS app) Spring MVC Can run in a webapp or as CLI - data - dashboard Banana (AngularJS app) Camel tasks Can run in a webapp or as CLI Messaging - ActiveMQ - JMS
Roadmap & future ● Reporting ○ history of records ○ scheduled analysis to follow the trends ● Indexing ○ geographic features for improving search ● Dashboard: migration to Silk (next banana version)
Recommend
More recommend
