Learning Spatial Relationships between Samples of Patent Image - PowerPoint PPT Presentation

Learning Spatial Relationships between Samples of Patent Image Shapes J. Castorena, M. Bhattarai, D. Oyen June, 2020 Managed by Triad National Security, LLC for the U.S. Department of Energy’s NNSA

Introduction • USPTO includes in their guidelines for patent application design that ”the drawing disclosure is the most important element of the application. • Most widely available methods for classification and retrieval are text based and a patent agents reviews results. If we can cut times and make the process more efficient by automatically classifying and retrieving patent documents based on images it would be highly beneficial. • Challenges: Image generation mechanism variances. • Lack of texture or color: binary images • Presence of multiple components • View-point • Drawing style of artisan. Figure 1: Examples of patent images.

This work: Focus: • Learning based methods to classify/retrieve patent images while being more effective in the binary image domain and at disentangling the learned functionals from some of the variations in the image generation mechanisms. Idea: • Analyze shape instead of image texture and learn geometrical features characterizing shapes. Overall method: • Extract samples from image shapes and learn spatial relationships between the points that characterize shapes [Castorena and Oyen, 2020]. Figure 2: Learning spatial relationships characterizing image shapes

Method • Image is sampled by clustering pixel shape locations and computing representative point. • Algorithm: Mini-batch K-means [Sculley, 2010] . • Orders of magnitude more computationally efficient than standard k-means with only small penalties in performance. • Learning node and spatial relationship features characterizing shape samples. • Dynamic graph CNN’s [Wang et al., Figure 4: Learning spatial relationships characterizing image shapes 2019]

Experiments I • Classification against the effect of data transformations plausibly influencing the patent image generation mechanisms. • First evaluation: on MNIST because it allows simple Figure 5: 1st row: the MNIST dataset and transformation manipulations 2nd row sampling of the MNIST digits. • We want to demonstrate the inability of standard CNN’s operating in the Euclidean domain to cope with data transformations in the absence of sufficient data characterizing these. A matter of crucial importance to build effective classification/retrieval solutions in the patent application realm. • Two experimental cases: • (1) training examples include images subjected to the corresponding transformations. • (2) training does not include examples of the transformed data. Transformations are drawn uniformly from : ±90◦ rotation, ± 9 pixels translations and [0.2,1] scaling. • Results show that our proposed framework remains more or less invariant to the aforementioned transformations in comparison to LeNet-5 and that it outperforms it in most cases in its own intended benchmark dataset.

Experiments II • Classification evaluated against the CLEF-IP 9- class dataset, image shapes from an unknown and varying viewpoint, presence of cross-class components and style. • Empirical results reflect that CNN’s operating in the Euclidean domain are not powerful enough to disentangle performance from generation mechanisms in binary patent imagery. • Method based on the Fisher vector representation outperforms standard CNN methods. • Our proposed method seems to better cope with Figure 6: Examples of the CLEF-IP 9-class images . the plausible variations in the image generation mechanism and outperforms all compared methods which includes to the best of our knowledge state of the art.

References • [Castorena and Doyen, 2020] Castorena, J., & Oyen, D. Learning Shapes on Image Sampled Points with Dynamic Graph CNNs. In 2020 IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI) (pp. 1-4). • [Csurka et al. 2011]. Csurka, G., Renders, J. M., & Jacquet, G. (2011). XRCE's Participation at Patent Image Classification and Image-based Patent Retrieval Tasks of the Clef-IP 2011. In CLEF (Notebook Papers/Labs/Workshop) (Vol. 2, No. 4). • [He et al., 2016] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778). • [Lecun et al., 1998 ] Y. Lecun, L. Bottou, Y. Bengio and P. Haffner, "Gradient-based learning applied to document recognition," in Proceedings of the IEEE , vol. 86, no. 11, pp. 2278-2324, Nov. 1998. • [Piroi et al., 2011] F. Piroi, et al. "CLEF-IP 2011: Retrieval in the Intellectual Property Domain." CLEF (notebook papers/labs/workshop) . 2011. • [Sculley, 2010] D. Sculley. 2010. Web-scale k-means clustering. In Proceedings of the 19th international conference on World wide web (WWW ’10). Association for Computing Machinery, New York, NY, USA, 1177–1178. • [Wang et al., 2019] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. 2019. Dynamic Graph CNN for Learning on Point Clouds. ACM Trans. Graph. 38, 5, Article 146 (October 2019).