Sum to zero constraints of logistic model tree in WEKA

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Sum to zero constraints of logistic model tree in WEKA

Li Ning

I am working on two data sets using LMT in WEKA. The training results are pretty good. As I need functions for each class of the leaves, so I check all the coefficients and intercepts of the two models.

For two classes model: Number of Leaves : 10

Size of the Tree : 19

LM_1: Class 0 : -8.58 + [d1] * -0.03 + [d2] * 1.22 + [d3] * 0.12 + [d4] * -0.11 + [d5] * 0.51

Class 1 : 8.58 + [d1] * 0.03 + [d2] * -1.22 + [d3] * -0.12 + [d4] * 0.11 + [d5] * -0.51

LM_2: Class 0 : 498.57 + [d1] * -0 + [d2] * -0.04 + [d3] * -101.31 + [d4] * 0.59 + [d5] * 0.32

Class 1 : -498.57 + [d1] * 0 + [d2] * 0.04 + [d3] * 101.31 + [d4] * -0.59 + [d5] * -0.32

For three classes model: Number of Leaves : 8

Size of the Tree : 15

LM_1: Class 0 : -0.91 + [w] * 0.06 + [h] * -0.02 + [r] * 1.61 + [UCS] * 0.03 + [p] * -0.12

Class 1 : 3.03 + [w] * -0.06 + [h] * 0.04 + [r] * 0.14 + [UCS] * -0 + [p] * -0.07

Class 2 : 10.15 + [w] * -0.01 + [h] * 0.01 + [r] * -18.09 + [UCS] * -0.03 + [p] * 0.25

LM_2: Class 0 : -0.29 + [w] * 0.07 + [h] * -0.21 + [r] * 0.07 + [UCS] * 0.06 + [p] * -0.39

Class 1 : 97.07 + [w] * -1.26 + [h] * -0.02 + [r] * -7.19 + [UCS] * 0 + [p] * -1.55

Class 2 : -93.43 + [w] * 1.18 + [h] * 0.07 + [r] * 4.74 + [UCS] * -0.04 + [p] * 1.63

As the basic idea of LMT is using the LogitBoost algorithm that need to keep the sum of the functions with different classes as zero, which can also be regards as sum to zero constraints. 



However, only the binary classes model could maintain the requirement. The three classes model is strange, though the predictive performance is satisfying. I also try to use the data sets provided by WEKA, like glass and iris, the problem is still the same.

So how could I solve this problem?

Thank you very much.



_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sum to zero constraints of logistic model tree in WEKA

Eibe Frank-3
Good observation!

When outputting linear models, LMT (and SimpleLogistic) do not centre the base models' predictions for each iteration. Instead, for the output that you show in your email, the linear models are computed as \sum_i f_{ij} * (m - 1) / m, where f_{ij} is the simple linear regression model for class j built in iteration i (in the case of m classes).

Strictly speaking, according to the LogitBoost paper, the individual iteration i models should be centred: \sum_i (f_{ij} - \sum{k=1}{m}f_{ik})(m - 1) / m should be used instead.

This is actually what happens when predictions are calculated in the LMT (and SimpleLogistic) code. However, if I'm not mistaken, applying the first term in your Equation (40) to calculate probability estimates should give the same result for both expressions, uncentered and centred (modulo numerical issues), because centring is used in both the numerator and the denominator.

Anyway, perhaps we should change the output showing the linear models to avoid confusion?

Cheers,
Eibe



On Fri, Aug 11, 2017 at 9:04 PM, Li Ning <[hidden email]> wrote:

I am working on two data sets using LMT in WEKA. The training results are pretty good. As I need functions for each class of the leaves, so I check all the coefficients and intercepts of the two models.

For two classes model: Number of Leaves : 10

Size of the Tree : 19

LM_1: Class 0 : -8.58 + [d1] * -0.03 + [d2] * 1.22 + [d3] * 0.12 + [d4] * -0.11 + [d5] * 0.51

Class 1 : 8.58 + [d1] * 0.03 + [d2] * -1.22 + [d3] * -0.12 + [d4] * 0.11 + [d5] * -0.51

LM_2: Class 0 : 498.57 + [d1] * -0 + [d2] * -0.04 + [d3] * -101.31 + [d4] * 0.59 + [d5] * 0.32

Class 1 : -498.57 + [d1] * 0 + [d2] * 0.04 + [d3] * 101.31 + [d4] * -0.59 + [d5] * -0.32

For three classes model: Number of Leaves : 8

Size of the Tree : 15

LM_1: Class 0 : -0.91 + [w] * 0.06 + [h] * -0.02 + [r] * 1.61 + [UCS] * 0.03 + [p] * -0.12

Class 1 : 3.03 + [w] * -0.06 + [h] * 0.04 + [r] * 0.14 + [UCS] * -0 + [p] * -0.07

Class 2 : 10.15 + [w] * -0.01 + [h] * 0.01 + [r] * -18.09 + [UCS] * -0.03 + [p] * 0.25

LM_2: Class 0 : -0.29 + [w] * 0.07 + [h] * -0.21 + [r] * 0.07 + [UCS] * 0.06 + [p] * -0.39

Class 1 : 97.07 + [w] * -1.26 + [h] * -0.02 + [r] * -7.19 + [UCS] * 0 + [p] * -1.55

Class 2 : -93.43 + [w] * 1.18 + [h] * 0.07 + [r] * 4.74 + [UCS] * -0.04 + [p] * 1.63

As the basic idea of LMT is using the LogitBoost algorithm that need to keep the sum of the functions with different classes as zero, which can also be regards as sum to zero constraints. 



However, only the binary classes model could maintain the requirement. The three classes model is strange, though the predictive performance is satisfying. I also try to use the data sets provided by WEKA, like glass and iris, the problem is still the same.

So how could I solve this problem?

Thank you very much.



_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html



_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Loading...