The Latest Progress of Chinese-Italian Culture Heritage Project - - - PowerPoint PPT Presentation
Xian Jiaotong University Xian Jiaotong University Xian Jiaotong University The Latest Progress of Chinese-Italian Culture Heritage Project - presented by Sijin QIAN (Peking University) on behalf of Chinese partners in the project
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Modules of basic information database for relics 1 Collaboration platform for the experts of ceramic painting conservation 2 Automated reparation of murals 3 Virtual display of relics 4
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Metadata technology The project information organization and retrieval norms research group ROSS (in Taiwan's National Science Council Digital Museum) has also made some researches on museums information metadata Chinese heritage central organizations of information consultation proposed << Standards of information index system for museum collections >> Meta search technology Standard vector definition of theme Chinese word segmentation
InfoGrid、Infonetware RealTerm Search
ICTCLAS word segmentation system designed and realized by the computing institute of
Chinese Academia Sinica
Content-Based Image Retrieval
QBIC、Photobook、CBIC on Web
Theme meta search engine
Tsinghua University SEGTAG system
Chinese word segmentation system in the Microsoft Research’s multi-language processing
system
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Mainly including 4 aspects: system architecture, data storage model, theme meta search engine, content-based image retrieval 1
Analysis of needs Design of the overall framework Data storage model
2
Digitalization of relics information Relics resources Applications and storage
3
Theme meta search engine Query Agent Web page parser Chinese word segmentation system Theme correlation calculation Standard vector definition of theme
4
Content-based image retrieval Selection of the underlying descriptor Improvement of color histogram Similarity computation User feedback mechanisms
Heritage data-oriented storage and retrieval system
System architecture
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Three-tier structure: (1) User requests and feedback layer, (2) Application logic layer and (3) Data layer
Browser
Users
Request / Display conversio n
Controller JSPs User requests and feedback control Heritage survival data repository Heritage data retrieval related library
. . .
Heritage theme search engine Data processing controller Heritage Data Storage
content- based image retrieval
. . . Application logic layer Data Layer Display Control
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
User requests and feedback control layer: The user requests and feedback control layer is the portal to interact with the user, it is responsible for handling the user's request and showing the results. As the middle layer between user and system applicative function, it shields specific implementary details of system for users, provides services for users with a unified storage and retrieval interface. Application logic layer: It includes applicative logic of three parts: the heritage data storage application, the heritage data retrieval application, and heritage theme meta search engine. Data layer: It is at the bottom of the whole system and is responsible for data storage, access, management; it is the base of entire system’s data.
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
文物元数据
文物基本信息
文物管理信息
文物图像信息
文物视频信息
文物三维数据信息
文物相关文档信息 0..1 +基本信息 0..n +图像信息 +基本信息 +管理信息 +文档信息 +三维信息 0..n 0..n 0..n 0.. 1
文物音频信息 0..n +基本信息
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Theme meta search engine
Correlation measure Subject standard vector Query agent Web page parser Chinese word segmentation Communication module
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Color histogram describes the global color information of image, color information of each pixel has the same importance, but through the analysis of image, we will find that, in general, the color information of the image edge is less important for the whole image.
user
Color histogram
Content-based Heritage Image Retrieval Interface
Color layout Edge Histogram
Calculation of the total feature similarity
Features Library Feedback
Color histogram Color layout Edge Histogram
Image Library
Heritage Resources
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
（a）待测试文本 （b） 测试结果
排 序 壁画主题 陶器主题 青铜器主题 主题词 Tf Df 权重 主题词 Tf Df 权重 主题词 Tf Df 权重 1 壁画 2158 111 11.7 陶器 3626 124 12.34 青铜器 4729 135 12.71 2 唐代 1288 108 11.1 文化 801 87 10.48 青铜 1487 117 11.41 3 艺术 667 86 10.2 中国 553 91 10.15 中国 867 110 10.81 4 敦煌 720 57 9.95 遗址 458 73 9.749 西周 747 87 10.43 5 研究 428 70 9.63 瓷器 477 69 9.733 商代 718 71 10.19 6 佛教 624 46 9.59 发现 381 80 9.657 纹饰 660 76 10.17 7 绘画 305 74 9.35 出土 330 75 9.448 文化 549 90 10.16 8 文化 302 69 9.27 彩陶 400 61 9.434 时期 592 81 10.13 9 发展 378 54 9.25 艺术 261 76 9.227 铭土 605 73 10.04 1 中国 272 72 9.21 新石器时代 263 70 9.152 出土 511 82 9.995
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
（a）公交车检索效果 （b）恐龙检测效果 （c）动物检索效果 （d）花朵检索效果
Public Transport Animal Dinosaur Flower
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Modules of basic information database for relics 1 Collaboration platform for the experts of ceramic painting conservation 2 Automated reparation of murals 3 Virtual display of relics 4
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Experts Case base Rule base Other data Knowledge base Reasoning machine Acquisition of Knowledge module Explanation module Man-machine interface Context database Users
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University Human machine interface module
It is the interactive interface between the expert system and users to transmit information . It is used to input the initial information of system, and output the final conclusions of system; display the explanation of reasoning process and all the man-machine dialogue during system is running ; print out the necessary information.
Context database
It is used to store the original features of the problem, the middle information generated in the process of reasoning and the final conclusions. It's another name is a comprehensive database.
Explanation module
It is used to answer users’ questions, to explain the reasoning conclusion.
Knowledge acquisition module
Accept the domain knowledge descripted by experts in the field in the form of natural language, convert it into expression of a standard case, rules and facts, then put it into knowledge base.
Inference engine
It is the core of expert system. It can use the knowledge in knowledge base, and a certain reasoning strategy (case-based reasoning, rule-based reasoning) according to the initial input information of system, to handle the problems to be solved better, give the appropriate conclusions, in order to achieve knowledge-based process of reasoning and solving.
Knowledge base
In this system, knowledge base consists of two parts, one is case base, stored related cases of the field of ceramic painting conservation summed up by the conservation experts, the other is a database storing information on a variety of documents and other data (the rule base in the graph does not realized, we may expand this function in the future).
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Modules of basic information database for relics 1 Collaboration platform for the experts of ceramic painting conservation 2 Automated reparation of murals 3 Virtual display of relics 4
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Chinese traditional art together .In terms of form and style ，It is the product of combinatting several aspects including line drawing style, decorative composition, color meticulous, writting to god etc.
hollowing , falling, powdering , fading, discoloration and others.
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Sample block Sample block model model
Broken features of mural in tombs of Tang Dynasty BSCD model BSCD model TV model TV model
Image Image decompositi decompositi
Criminisi Algorithm is used as the research base
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Fill Image
Search the best matching block
Calculate the priority of target blocks
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
1、Method for the new target block priority calculation
( ) ( )
p p p
I n K d D p α
⊥
∇
=
2、Strategy for new target block repairing order selection
Having calculated the priority of the object block formed by all border points in patch , the strategy of original algorithm is to update the block boundary as soon as dealling with the sample with the biggest priority, calculate the priority of a new set of target blocks. The algorithm mentioned in this paper is to update the boundary after having finished takled with this group of blocks in the order of priority, and calculate priority of a new set of target block.
3、The selection strategy of the best new matching block
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
1、Effect comparison between improved algorithm models
(a) With the original algorithm (b) With only the improved priority (c) With the algorithm described in last slides
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
(a) Original picture (c) Effect of the Criminisi algorithm (b) Broken picture (d) Effect of the algorithm described in last slides
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
3、Testing experiment of objective evaluation criteria (a) Broken picture (b) Criminisi algo. (c) The algo. in last slides
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
图片 T1 T2 T3 T4 T5 图5-1（a） 195.744s 115.728s 103.151s 114.814s 120.595s 图5-1（b） 268.751s 164.998s 148.610s 164.839s 168.063s 图5-1（c） 351.726s 224.601s 183.298s 226.593s 239.422s 图5-1（d） 437.154s 282.614s 235.859s 282.145s 306.597s
T1: time-spent when using serial algorithm T2: time-spent when using parallel algorithm and two nodes T3: time-spent when using parallel algorithm and three nodes T4: time-spent when using parallel algorithm and four nodes T5: time-spent when using parallel algorithm and five nodes
(Number of pixels to be repaired)
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Modules of basic information database for relics 1 Collaboration platform for the experts of ceramic painting conservation 2 Automated reparation of murals 3 Virtual display of relics 4
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Panorama display
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Features ： a) SIFT features are local features of the image, Its rotation, scaling, brightness variations remain invariant. On the angle changes, affine transformation, noise ,it also keeps a certain degree of stability . b) Unique, informative, applicable to fast and accurate matching in the massive feature database. c) Large quantity, even a handful of objects can also generate a large number of SIFT feature vectors. d) High speed, the optimized SIFT matching algorithm can even achieve real- time requirements . e) Scalability, can be very convenient to join with feature vectors in other forms.
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
1、Detection of feature points
2、The feature match for the candidate matching points (ongoing ) 3、Elimination of mismatch (ongoing )
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Example of the feature points detection for a mural of a polo game
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
3Ds Max model
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University
Xi’an Jiaotong University Xi’an Jiaotong University Xi’an Jiaotong University