Evaluation
We have two tracks in this competition with different cost function.
Track 0
For the this track of the competition, your classifier g for an example (x, y)) will be evaluated by the following pointwise error function:
- if y = g(x), no error
- otherwise (y != g(x)), if they still correspond to the same number, no error
- else, an error of 1
effectively, it means we use 0/1 error on the 22 effective classes, one for each zodiac and one for each number, regardless of cases.
Track 1
Final Project Track 1 Phase 0
For this track of the competition, your classifier g(x) will be evaluated as follows:
- if y = g(x), no error
- number/zodiac error. otherwise (y != g(x)), if one of them is a zodiac and the other is a number, an error of 4. E.g., "mouse" predicted as "one" or "two" predicted as "cow"
- within zodiac error. otherwise ( y != g(x)), if one of them is a zodiac and the other is a zodiac, an error of 2. E.g., "mouse" predicted as "cow"
- case error. otherwise ( y != g(x)), if one of them is a number and the other is the same number with a different case, an error of 1. E.g., upper-case "one" predicted as lower-case "one"
- within number error. otherwise ( y != g(x)), an error of 2. E.g., upper-case "one" predicted as "five"
The error function above can be put in a simpler way. The usual error is 2; case error is considered light and penalized less (1); number/zodiac error is considered serious and penalized more (4). This kind of multiclass classification problem is often called cost-sensitive classification, in case you want to look up references online.