Lecture 24: Barycentric Coordinates ECE 417: Multimedia Signal - PowerPoint PPT Presentation

Overview of MP4 Barycentric Coordinates Conclusion Lecture 24: Barycentric Coordinates ECE 417: Multimedia Signal Processing Mark Hasegawa-Johnson University of Illinois 11/14/2018

Overview of MP4 Barycentric Coordinates Conclusion Overview of MP4 1 Barycentric Coordinates 2 Conclusion 3

Overview of MP4 Barycentric Coordinates Conclusion Outline Overview of MP4 1 Barycentric Coordinates 2 Conclusion 3

Overview of MP4 Barycentric Coordinates Conclusion Goal of MP4: Generate video frames (right) by warping a static image (left).

Overview of MP4 Barycentric Coordinates Conclusion MP4 full outline

Overview of MP4 Barycentric Coordinates Conclusion How it is done (Full walkthrough: Tuesday November 27) lip height,width = NeuralNet (MFCC) out triangs = LinearlyInterpolate (inp triangs,lip height,width) inp coord = BaryCentric (out coord,inp triangs,out triangs) out image = BilinearInterpolate (inp coord,inp image)

Overview of MP4 Barycentric Coordinates Conclusion Outline Overview of MP4 1 Barycentric Coordinates 2 Conclusion 3

Overview of MP4 Barycentric Coordinates Conclusion

Overview of MP4 Barycentric Coordinates Conclusion Piece-wise affine transform OK, so somebody’s given us a lot of points, arranged like this in little triangles. We know that we want a DIFFERENT AFFINE TRANSFORM for EACH TRIANGLE. For the k th triangle, we want to have   a k b k c k A k = d k e k f k   0 0 1

Overview of MP4 Barycentric Coordinates Conclusion Piece-wise affine transform     x u  , output point: � x = input point: � u = y v    1 1 x is in the k th triangle in the output image , then Definition : if � we want to use the k th affine transform: u = A − 1 � x = A k � u , � k � x

Overview of MP4 Barycentric Coordinates Conclusion u = A − 1 If it is known that � k � x for some unknown affine transform matrix A k , then the method of barycentric coordinates finds � u without ever finding A k .

Overview of MP4 Barycentric Coordinates Conclusion Barycentric Coordinates Barycentric coordinates turns the problem on its head. Suppose � x is in a triangle with corners at � x 1 , � x 2 , and � x 3 . That means that � x = λ 1 � x 1 + λ 2 � x 2 + λ 3 � x 3 where 0 ≤ λ 1 , λ 2 , λ 3 ≤ 1 and λ 1 + λ 2 + λ 3 = 1

Overview of MP4 Barycentric Coordinates Conclusion Barycentric Coordinates Suppose that all three of the corners are transformed by some affine transform A , thus � u 1 = A � x 1 , u 2 = A � � x 2 , � u 3 = A � x 3 Then if If: � x = λ 1 � x 1 + λ 2 � x 2 + λ 3 � x 3 Then: = � u A � x = λ 1 A � x 1 + λ 2 A � x 2 + λ 3 A � x 3 = u 1 + λ 2 � u 2 + λ 3 � λ 1 � u 3 In other words, once we know the λ ’s, we no longer need to find A . We only need to know where the corners of the triangle have moved.

Overview of MP4 Barycentric Coordinates Conclusion Barycentric Coordinates If � x = λ 1 � x 1 + λ 2 � x 2 + λ 3 � x 3 Then � u = λ 1 � u 1 + λ 2 � u 2 + λ 3 � u 3

Overview of MP4 Barycentric Coordinates Conclusion How to find Barycentric Coordinates But how do you find λ 1 , λ 2 , and λ 3 ?       λ 1 x 1 x 2 x 3 λ 1  = x = λ 1 � � x 1 + λ 2 � x 2 + λ 3 � x 3 = [ � x 1 , � x 2 , � x 3 ] λ 2 y 1 y 2 y 3 λ 2      λ 3 1 1 1 λ 3 Write this as: x = X � � λ Therefore � λ = X − 1 � x This always works: the matrix X is always invertible, unless all three of the points � x 1 , � x 2 , and � x 3 are on a straight line.

Overview of MP4 Barycentric Coordinates Conclusion How do you find out which triangle the point is in? Suppose we have K different triangles, each of which is characterized by a 3 × 3 matrix of its corners X k = [ � x 1 , k , � x 2 , k , � x 3 , k ] x m , k is the m th corner of the k th triangle. where � Notice that, for any point � x , for ANY triangle X k , we can find λ = X − 1 k � x However, the coefficients λ 1 , λ 2 , and λ 3 will all be between 0 and 1 if and only if the point � x is inside the triangle X k . Otherwise, some of the λ ’s must be negative.

Overview of MP4 Barycentric Coordinates Conclusion The Method of Barycentric Coordinates To construct the animated output image frame O ( x , y ), we do the following things: First, for each of the reference triangles U k in the input image I ( u , v ), decide where that triangle should move to. Call the new triangle location X k . Second, for each output pixel ( x , y ): For each of the triangles, find � λ = X − 1 � x . k Choose the triangle for which all of the λ coefficients are 0 ≤ λ ≤ 1. u = U k � Find � λ . Estimate I ( u , v ) using bilinear interpolation. Set O ( x , y ) = I ( u , v ).

Overview of MP4 Barycentric Coordinates Conclusion Outline Overview of MP4 1 Barycentric Coordinates 2 Conclusion 3

Overview of MP4 Barycentric Coordinates Conclusion lip height,width = NeuralNet (MFCC) out triangs = LinearlyInterpolate (inp triangs,lip height,width) inp coord = BaryCentric (out coord,inp triangs,out triangs) out image = BilinearInterpolate (inp coord,inp image)