Clustering-Based, Fully Automated Mixed-Bag Jigsaw Puzzle Solving Zayd Hammoudeh Chris Pollett Department of Computer Science San José State University San José, CA USA 17 th International Conference on Computer Analysis of Images and Patterns August 22-24, 2017
Introduction Jigsaw Puzzles Clustering-Based, Fully Automated Mixed-Bag Jigsaw Puzzle Solving ◮ First jigsaw puzzle introduced in the 1760s Hammoudeh & Pollett Introduction 1 Mixed-Bag Solver ◮ First computational jigsaw puzzle solver introduced Segmentation Stitching in 1964 [4] Hierarchical Clustering Quantifying Quality Direct Accuracy Experimental Results ◮ Solving a jigsaw puzzle is NP-complete [1, 3]. Input Puzzle Count Solver Comparison References ◮ Example Applications: DNA fragment reassembly, shredded document reconstruction, and speech descrambling ◮ Generally, the ground-truth source is unknown. Dept. of Computer Science San José State University 20
Introduction Mixed-Bag Puzzles Clustering-Based, Jig Swap Puzzles : Variant of the traditional jigsaw puzzle Fully Automated Mixed-Bag Jigsaw ◮ All pieces are equal-sized squares. Puzzle Solving Hammoudeh & Pollett ◮ Piece rotation, puzzle dimensions, and ground-truth input Introduction 2 contents are all unknown. Mixed-Bag Solver Segmentation Stitching “ Mixed-Bag ”: Simultaneous solving of multiple jig swap puzzles Hierarchical Clustering ◮ The number of inputs may be unknown. Quantifying Quality Direct Accuracy Experimental Results Input Puzzle Count Solver Comparison References Randomized Solver Input – 2,017 Pieces Solver Output #1 Solver Output #2 Solver Output #3 Dept. of Computer Science 805 Pieces 540 Pieces 672 Pieces San José State University 20
Summary of Key Contributions Clustering-Based, Fully Automated Mixed-Bag Jigsaw Puzzle Solving Hammoudeh & Pollett ◮ Primary Contribution: Novel mixed-bag puzzle solver Introduction 3 that outperforms the current state of the art [6] by: Mixed-Bag Solver Segmentation ◮ Requiring no external “oracle” information Stitching Hierarchical Clustering ◮ Generating superior reconstructed outputs Quantifying Quality Direct Accuracy ◮ Supporting more simultaneous inputs Experimental Results Input Puzzle Count Solver Comparison References ◮ Additional Contribution : Define the first metrics that quantify the quality of outputs from a multi-puzzle solver Dept. of Computer Science San José State University 20
Our Contribution: The Mixed-Bag Solver
Mixed-Bag Solver Overview Clustering-Based, Fully Automated Basis of the Mixed-Bag Solver: Human puzzle solving Mixed-Bag Jigsaw Puzzle Solving strategy to: Hammoudeh & Pollett ◮ Correctly assemble small puzzle regions (i.e., segments) Introduction Mixed-Bag Solver 4 ◮ Iteratively merge smaller regions to form larger ones Segmentation Stitching Hierarchical Clustering Quantifying Quality Direct Accuracy Simplified Algorithm Flow: Experimental Results Input Puzzle Count Solver Comparison References ... Hierarchical Final Segmentation Stitching Segment Assembly Clustering Mixed Bag Dept. of Computer Science San José State University 20
Segmentation Mixed-Bag Solver Stage #1 Clustering-Based, Fully Automated Mixed-Bag Jigsaw Puzzle Solving Hammoudeh & Pollett Introduction ◮ Segment : Partial puzzle assembly where this is a high Mixed-Bag Solver degree of confidence pieces are placed correctly Segmentation 5 Stitching ◮ Each piece is assigned to at most one segment. Hierarchical Clustering Quantifying Quality Direct Accuracy Experimental Results Input Puzzle Count Solver Comparison References ◮ Role of Segmentation : Provide structure to the set of puzzle pieces by partitioning them into disjoint segments Dept. of Computer Science San José State University 20
Segmentation Algorithm Overview Clustering-Based, Fully Automated Mixed-Bag Jigsaw Puzzle Solving Hammoudeh & Pollett ◮ Iterative process consisting of one or more rounds. Introduction Mixed-Bag Solver ◮ In each round, any pieces not already assigned to a Segmentation 6 segment pieces are assembled into a single puzzle. Stitching Hierarchical Clustering ◮ This assembly is then segmented based on inter-piece Quantifying Quality Direct Accuracy similarity (i.e., the “best buddies” principle). Experimental Results ◮ Segments of sufficient size are saved for use in later Input Puzzle Count Solver Comparison Mixed-Bag Solver stages. References ◮ Segmentation terminates when an assembly has no segments whose size exceeds a minimum threshold (e.g., 7). Dept. of Computer Science San José State University 20
Segmentation First-Round Example Clustering-Based, Fully Automated Mixed-Bag Jigsaw Puzzle Solving Hammoudeh & Pollett Introduction Mixed-Bag Solver Segmentation 7 Stitching Hierarchical Clustering Quantifying Quality Direct Accuracy Experimental Results Input Puzzle Count Solver Comparison References Ground-Truth Solver Output Segmented Output Inputs Dept. of Computer Science San José State University 20
Stitching Mixed-Bag Solver Stage #2 Clustering-Based, Fully Automated Mixed-Bag Jigsaw Puzzle Solving Hammoudeh & Pollett ◮ Role of Stitching : Quantify the extent that any pair of Introduction segments is related Mixed-Bag Solver Segmentation Stitching 8 ◮ Mini-Assembly : Places a pre-defined, fixed number Hierarchical Clustering Quantifying Quality (e.g., 100) of pieces Direct Accuracy Experimental Results Input Puzzle Count ◮ Stitching Piece : A piece near the boundary of a segment Solver Comparison that is used as the seed of a single mini-assembly References ◮ Segment Overlap : Inter-segment affinity score based on the composition of a segment’s mini-assembly Dept. of Computer Science San José State University 20
Stitching Example – Single Input Image Clustering-Based, Fully Automated Stitching Mixed-Bag Jigsaw Puzzle Solving Ground Truth Segmenter Output PiecesMini- Hammoudeh & Pollett Assembly Introduction Mixed-Bag Solver Segmentation Stitching 9 Hierarchical Clustering Quantifying Quality Direct Accuracy Experimental Results Input Puzzle Count Solver Comparison References Stitching piece selected from upper-right corner of the top segment Dept. of Computer Science San José State University 20
Hierarchical Segment Clustering Mixed-Bag Solver Stage #3 Clustering-Based, Fully Automated ◮ A single ground-truth image may be comprised of multiple Mixed-Bag Jigsaw Puzzle Solving segments. Hammoudeh & Pollett Introduction Mixed-Bag Solver ◮ Role of Hierarchical Clustering : Estimate the number of Segmentation Stitching inputs by grouping together all segments from the same Hierarchical Clustering 10 ground-truth image. Quantifying Quality Direct Accuracy Experimental Results Input Puzzle Count ◮ Single-Link Clustering : Inter-cluster similarity equals the Solver Comparison similarity of their most similar respective members References Dept. of Computer Science San José State University 20
Terminating the Solver Building the final outputs Clustering-Based, Fully Automated Mixed-Bag Jigsaw Puzzle Solving Hammoudeh & Pollett Introduction ◮ The solver continues merging segment clusters until one Mixed-Bag Solver Segmentation of two criteria is satisfied: Stitching Hierarchical Clustering 11 ◮ Only a single segment cluster remains Quantifying Quality Direct Accuracy ◮ Maximum similarity between any segment clusters is below Experimental Results a predefined threshold Input Puzzle Count Solver Comparison References ◮ Final Assembly : Builds the final solver outputs are built using the cluster membership results Dept. of Computer Science San José State University 20
Quantifying Solver Performance
Quantifying Solver Performance Clustering-Based, Fully Automated Mixed-Bag Jigsaw Puzzle Solving Hammoudeh & Pollett ◮ Metrics quantify the quality of the solver outputs as the Introduction reconstructions may not be reconstructions. Mixed-Bag Solver Segmentation ◮ Two Primary Quality Metrics : Range [0,1] Stitching Hierarchical Clustering ◮ Direct Accuracy Quantifying Quality 12 Direct Accuracy ◮ Neighbor Accuracy ( not discussed in this presentation ) Experimental Results Input Puzzle Count Solver Comparison ◮ Disadvantages of Current Metrics : Neither account for References issues unique to mixed-bag puzzles including: ◮ Pieces from one input misplaced in multiple output puzzles ◮ Pieces from multiple inputs in the same output Dept. of Computer Science San José State University 20
Direct Accuracy Overview of the Current Standard Clustering-Based, Fully Automated Mixed-Bag Jigsaw Puzzle Solving Hammoudeh & Pollett Standard Direct Accuracy : Fraction of pieces, c placed in the Introduction Mixed-Bag Solver same location in both the ground-truth and solved puzzles Segmentation versus the total number of pieces, n Stitching Hierarchical Clustering Quantifying Quality Direct Accuracy 13 Experimental Results Input Puzzle Count Solver Comparison Formal Definition : References DA = c (1) n Dept. of Computer Science San José State University 20
Recommend
More recommend
