E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 Some Results on Threshold Separability of Boolean Functions Endre Boros Giovanni Felici RUTCOR Istituto di Analisi dei Sistemi ed Informatica Rutgers University Consiglio Nazionale delle Ricerche New Jersey United States Roma, Italy Endre.Boros@rutcor.rutgers.edu giovanni.felici@iasi.cnr.it 1
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 Supervised Learning and Data Mining • given disjoint sets of points in multidimensional space, find a set of meaningful rules that is able to separate records of different sets with high precision S R n , S = P N, P N = • difficult easy 2 Standard methods = {decision trees, SVM, logistic regression }
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 Binarization of real valued data Each real valued dimension is mapped into a set of intervals, to achieve: • Control of noise effect • Simplification of separating rules • Use of for models in logic form Binarizion is very important: is at the top of the analysis hierarchy For the i-th coordinate, define a set of cutponts T i = {t i1 ,t i2 , …, t i , ki }, t ij < t i,j+1 Define binary z ij , j = 1,…, k i associated with T i where z ij = 1 if t ij-1 x i t ij 0 otherwise A binarization induces a set of boxes B in the space of S, each defined by the interserction of the subspaces parallel to the axes that intersect the cutpoints. With proper assumptions and bounding on the cutpoint sets, each box is closed and the set of boxes is finite . 3
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 Definitions • A binarization for S is represented equivalently by a set of box B or by the cutpoint set T. • A binarization T is separable if all the non-empty boxes defined by its cutpoints either contain positive or negative points From P N = , we know that there always exists a separable binarization • A partially defined boolean function (PDBF) is a set of boolean vectors divided into two non intersecting subsets Z is a PDBF T is separable • a simple way to obtain a separable binarization is to insert a cutpoint between each pair of consecutive points of different class Trivial conditions on set S guarantee the existance of a separable T-binarization and thus of a PDBF on Z 4
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 Linear Threshold Boolean Functions • A Linear Threshold Boolean Function (LTF) is a boolean function f:{0,1} n → {-1,1} expressed as f(x) = sgn(c 0 +c 1 x 1 +…+ c n x n ) • Given S, a binarization T and the resulting set Z, a LTF f : Z → {-1,1} is separating for S if and only if f(z) = 1 for z in P and f(z) = -1 for z in N. • We also say that Z admits a LTF extension We are interested in the conditions for Z to admit a LTF extension • Once a binarization has been applied, we can keep 1 point in each box. Separability and existance of LTF-extensions are preserved 5
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 Main Topics: 1. conditions for the existence of a Separating Linear Threshold Function 2. combinatorial necessary and sufficient condition for the existence of such function when points belong to the plane 3. Minimal forbidden structures for the existance of separating LTFs 4. insights for the problem in larger dimensions that can be practically used in distcretization / binarization algorithms to find good LTF. 6
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 When a LTF cannot be obtained in 2 dimensions z 21 =1 z 11 =1 Any set composed of 3 boxes is LTF- The Forbidden Cross (FC) extendable (trivial ) : 1 B w w z w z w 00 0 11 11 21 21 0 : 1 B w w z w z w w 10 0 11 11 21 21 0 11 1 , 2 , 2 w w w 0 11 21 : 1 B w w z w z w w 01 0 11 11 21 21 0 21 : 1 B w w z w z w w w 11 0 11 11 21 21 0 11 21 Positive Equivalent representations: Negative 7
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 Minor of a Box Set A Minor is composed by the 4 “corner” boxes of a larger subset of B Observation . Given S R 2 , a T-binarization and the related box set B, then Z admits a LTF representation if B does not contain any Forbidden Cross (FC) as a minor We like minimal structures that break the LTF representability of Z; this would be very useful to determine algorithmically the right T for S 8
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 The Spanning Operation and LTF-maximal Box Sets The spanning operation : each time an empty box is the corner of a potential FC, we assign that box to the only class (color) that avoids the FC A set of boxes is LTF-maximal when no spanning can be applied the sequence of spanning operations does not affect the existence of a LTF-extension Claim 1: Z admits no LTF-extensions iif its LTF-maximal box sets contains a FC 9
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 LTF-separability in the plane and alternating cycles Given a box set B, construct the following graph G=(V,E): • assing a vertex v to each non-empty box in the plane • Draw an edge between each positive box and each negative box in the same column or row In the example: an alternating cycle of length 4 corresponds to a FC. What happens to longer alt-cycles ? 10
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 Spanning and alternating cycles Any alt-cycle of length 2n in G can generate an alt-cycle of length 2(n-1) by a spanning operation, down to a cycle of length 4 (i.e., a FC) 1 cycle of + 1 cycle of length 6 length 4 + another + another cycle of cycle of length 4… length 4 Claim 2: if G contains an alternating cycle, then no LTF-extension is possible 11
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 Important questions : is spanning allowed ? What are its relation with LTF-extensions? If spanning preserves the LTF extendability, then: Claim 1 : Z admits no LTF-extension iif its LTF-maximal box sets contains a FC Claim 2 : if G contains an alternating cycle, then no LTF-extension is possible are theorems… To make sure, we need some linear programming. 12
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 LTF-separabilty and Linear Programming (in the plane, for the time being) Given S R 2 , T, and Z: for , 1 a z s P j k 1 ij ij i Remember that we may use for , 1 b z s P k j k only one point in each box … 1 2 ij ij i ' for , 1 a z s N j k 1 ij ij i ' for , 1 b z s N k j k ij ij i 1 2 max 1 x and y are the weights that are k k 1 2 associated with the cutpoints in 1 i 1,..., P a a x b y 0 ij j ij j the two coordinates to 0 0 j j determine the LTF. So far, we k k are interested only in feasibility 1 2 ' ' 1 i 1,..., N a a x b y 0 ij j ij j 0 0 j j k k , x R y R 1 2 (1) 13
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 LTF Existence Conditions (u,v) are the dual variables associated with In compact form constraints on P and N boxes max 1 x min min u v u v ( , ) 1 a A B 0 that has P y ' 0 ' 0 uA vA uA vA empty And the ' 0 ' 0 uB vB uB vB feasible x dual is: ( ' , ' ) 1 a A B region if t t t t 0 0 0 e u e v e u e v P y this has solution: ( , ) 0 ( , ) 0 u v u v 1 , k k x R y R 2 (1) (2) (3) Z admits a LTF representation (1) has solution i. (1) has solution (2) has solution ii. If (3) has solution, (2) has no solution iii. iv. A feasible solution of (3) represents a forbidden structure for LTF-extension 14
E. Boros, G.Felici Boolean Seminar, Liblice, April 2013 Conditions that define a Forbidden Minor Consider (3): min u v ' 0 uA vA ' 0 uB vB t t 0 e u e v ( , ) 0 u v | P | | N | ' 0 a u a v ij i ij i In detail: 1 1 i i In a “slice” of the plane the sum of the weights of positive boxes is equal to the sum of the i. weights of negative boxes. ii. At least 2 v and 2 u must be > 0 iii. Must be valid for at least one horizontal and one vertical slice iv. In any slice, if a positive box is present, also a negative one must appear, and viceversa v. When this happens, no LTF extension is possible 15
Recommend
More recommend