Score Tables
The following tables give the scores for each data set in the CoNLL 2007 shared task. The "Average" column contains the average score for all ten languages, which determines the ranking in the multilingual track. The ranking for the domain adaptation track is determined based on the pchemtb results only. The CHILDES data set was optional.
The numbers in parentheses next to each score gives the rank and the p-value of the difference with the next rank. The p-values for the "Average" column were computed using a standard z-test for proportions. All the other p-values were computed using version 1.12 of eval07.pl (with the -b option) and Dan Bikel's Randomized Parsing Evaluation Comparator compare.pl (Statistical Significance Tester for evalb Output) (with a default 10,000 iterations). The label accuracy scores are not used for ranking and no significance scores were computed for them. Differences are taken to be significant if p<0.05.
The answers submitted by all the systems can be downloaded as answers.tar.gz. To respect the treebank copyrights only columns 7 and 8 (HEAD and DEPREL) have been included.
Multilingual Track: Labeled Attachment Score
Team |
Average |
Arabic |
Basque |
Catalan |
Chinese |
Czech |
English |
Greek |
Hungarian |
Italian |
Turkish |
Nilsson et al. |
80.32(1,0.448) |
76.52(1,0.011) |
76.94(1,0.024) |
88.70(1,0.075) |
75.82(15,0.112) |
77.98(3,0.488) |
88.11(5,0.011) |
74.65(2,0.353) |
80.27(1,0.078) |
84.40(1,0.141) |
79.79(2,0.052) |
Nakagawa |
80.29(2,0.059) |
75.08(2,0.330) |
72.56(7,0.377) |
87.90(3,0.384) |
83.84(2,0.305) |
80.19(1,0.036) |
88.41(3,0.492) |
76.31(1,0.002) |
76.74(8,0.042) |
83.61(3,0.377) |
78.22(5,0.070) |
Titov et al. |
79.90(3,0.494) |
74.12(6,0.124) |
75.49(3,0.258) |
87.40(6,0.218) |
82.14(7,0.207) |
77.94(4,0.250) |
88.39(4,0.347) |
73.52(10,0.194) |
77.94(4,0.117) |
82.26(6,0.415) |
79.81(1,0.498) |
Sagae |
79.90(4,0.354) |
74.71(4,0.451) |
74.64(6,0.000) |
88.16(2,0.293) |
84.69(1,0.063) |
74.83(8,0.095) |
89.01(2,0.102) |
73.58(8,0.477) |
79.53(2,0.007) |
83.91(2,0.294) |
75.91(10,0.448) |
Hall, J. et al. |
79.80(5,0.002) |
74.75(3,0.476) |
74.99(5,0.259) |
87.74(4,0.381) |
83.51(3,0.150) |
77.22(6,0.018) |
85.81(12,0.001) |
74.21(6,0.347) |
78.09(3,0.431) |
82.48(5,0.404) |
79.24(3,0.090) |
Carreras |
79.09(6,0.001) |
70.20(11,0.455) |
75.75(2,0.349) |
87.60(5,0.352) |
80.86(10,0.163) |
78.60(2,0.238) |
89.61(1,0.128) |
73.56(9,0.473) |
75.42(9,0.449) |
83.46(4,0.084) |
75.85(11,0.130) |
Attardi et al. |
78.27(7,0.205) |
72.66(8,0.135) |
69.48(12,0.274) |
86.86(7,0.386) |
81.50(8,0.360) |
77.37(5,0.422) |
85.85(10,0.499) |
73.92(7,0.313) |
76.81(7,0.469) |
81.34(8,0.179) |
76.87(7,0.245) |
Chen |
78.06(8,0.084) |
74.65(5,0.218) |
72.39(8,0.037) |
86.66(8,0.078) |
81.24(9,0.328) |
73.69(10,0.101) |
83.81(13,0.091) |
74.42(3,0.422) |
75.34(10,0.092) |
82.04(7,0.154) |
76.31(9,0.247) |
Duan(1) |
77.70(9,0.001) |
69.91(13,0.000) |
71.26(9,0.231) |
84.95(10,0.242) |
82.58(6,0.268) |
75.34(7,0.256) |
85.83(11,0.487) |
74.29(4,0.437) |
77.06(5,0.323) |
80.75(9,0.465) |
75.03(12,0.071) |
Hall, K. et al. |
76.91(10,0.003) |
73.40(7,0.170) |
69.81(11,0.290) |
82.38(14,0.137) |
82.77(4,0.410) |
72.27(12,0.046) |
81.93(15,0.295) |
74.21(5,0.487) |
74.20(11,0.342) |
80.69(10,0.408) |
77.42(6,0.186) |
Schiehlen |
76.18(11,0.065) |
70.08(12,0.431) |
66.77(14,0.053) |
85.75(9,0.114) |
80.04(11,0.362) |
73.86(9,0.434) |
86.21(9,0.338) |
72.29(12,0.171) |
73.90(12,0.001) |
80.46(11,0.011) |
72.48(15,0.000) |
Johansson et al. |
75.78(12,0.000) |
71.76(9,0.395) |
75.08(4,0.435) |
83.33(12,0.433) |
76.30(14,0.276) |
70.98(13,0.341) |
80.29(17,0.150) |
72.77(11,0.230) |
71.31(13,0.324) |
77.55(14,0.001) |
78.46(4,0.344) |
Mannem |
74.54(13,0.000) |
71.55(10,0.043) |
65.64(15,0.000) |
84.47(11,0.049) |
73.76(17,0.000) |
70.68(14,0.000) |
81.55(16,0.056) |
71.69(13,0.070) |
70.94(14,0.027) |
78.67(13,0.062) |
76.42(8,0.416) |
Wu |
73.02(14,0.035) |
66.16(14,0.117) |
70.71(10,0.098) |
81.44(15,0.005) |
74.69(16,0.152) |
66.72(16,0.000) |
79.49(18,0.001) |
70.63(14,0.401) |
69.08(15,0.443) |
78.79(12,0.428) |
72.52(14,0.486) |
Nguyen |
72.53(15,0.000) |
63.58(16,0.424) |
58.18(17,0.200) |
83.23(13,0.133) |
79.77(12,0.000) |
72.54(11,0.330) |
86.73(6,0.489) |
70.42(15,0.009) |
68.12(17,0.043) |
75.06(16,0.290) |
67.63(17,0.000) |
Maes et al. |
70.66(16,0.000) |
65.12(15,0.087) |
69.05(13,0.000) |
79.21(16,0.000) |
70.97(18,0.232) |
67.38(15,0.200) |
69.68(21,0.000) |
68.59(16,0.000) |
68.93(16,0.165) |
73.63(18,0.000) |
74.03(13,0.008) |
Canisius et al. |
66.99(17,0.000) |
59.13(18,0.000) |
63.17(16,0.000) |
75.44(17,0.000) |
70.45(19,0.000) |
56.14(17,0.085) |
77.27(19,0.000) |
60.35(18,0.000) |
64.31(19,0.000) |
75.57(15,0.217) |
68.09(16,0.231) |
Jia |
63.00(18,0.000) |
63.37(17,0.000) |
57.61(18,0.000) |
23.35(20) |
76.36(13,0.474) |
54.95(18,0.091) |
82.93(14,0.113) |
65.45(17,0.000) |
66.61(18,0.004) |
74.65(17,0.167) |
64.68(18,0.000) |
Zeman |
54.87(19,0.156) |
46.06(20) |
50.61(20) |
62.94(19,0.000) |
54.49(20,0.000) |
50.21(20) |
53.59(22,0.095) |
55.29(19,0.160) |
55.24(20,0.000) |
62.13(19,0.003) |
58.10(19,0.074) |
Marinov |
54.55(20,0.000) |
54.00(19,0.000) |
51.24(19,0.229) |
69.42(18,0.000) |
49.87(21) |
53.47(19,0.001) |
52.11(23) |
54.33(20) |
44.47(21) |
59.75(20) |
56.88(20) |
Duan(2) |
24.62(21,0.000) |
|
|
|
82.64(5,0.499) |
|
86.69(7,0.371) |
|
76.89(6,0.461) |
|
|
Nash |
8.65(22,0.000) |
|
|
|
|
|
86.49(8,0.331) |
|
|
|
|
Shimizu et al. |
7.20(23) |
|
|
|
|
|
72.02(20,0.005) |
|
|
|
|
Multilingual Track: Unlabeled Attachment Score
Team |
Average |
Arabic |
Basque |
Catalan |
Chinese |
Czech |
English |
Greek |
Hungarian |
Italian |
Turkish |
Nakagawa |
86.55(1,0.000) |
86.09(1,0.265) |
81.04(5,0.201) |
92.86(4,0.165) |
88.88(2,0.035) |
86.28(1,0.059) |
90.13(2,0.273) |
84.08(1,0.000) |
82.49(3,0.364) |
87.91(1,0.414) |
85.77(3,0.125) |
Nilsson et al. |
85.71(2,0.336) |
85.81(2,0.000) |
82.84(1,0.066) |
93.12(3,0.269) |
84.52(12,0.266) |
83.59(4,0.379) |
88.93(5,0.490) |
81.22(4,0.480) |
83.55(1,0.473) |
87.77(2,0.406) |
85.77(2,0.490) |
Titov et al. |
85.62(3,0.067) |
83.18(7,0.175) |
81.93(2,0.116) |
93.40(1,0.456) |
87.91(4,0.342) |
84.19(3,0.204) |
89.73(4,0.120) |
81.20(5,0.493) |
82.18(4,0.314) |
86.26(6,0.342) |
86.22(1,0.230) |
Sagae |
85.29(4,0.012) |
84.04(4,0.166) |
81.19(3,0.457) |
93.34(2,0.249) |
88.94(1,0.456) |
81.27(8,0.312) |
89.87(3,0.425) |
80.37(11,0.134) |
83.51(2,0.078) |
87.68(3,0.173) |
82.72(9,0.312) |
Carreras |
84.79(5,0.408) |
81.48(10,0.292) |
81.11(4,0.438) |
92.46(5,0.293) |
86.20(9,0.351) |
85.16(2,0.132) |
90.63(1,0.210) |
81.37(3,0.418) |
79.92(9,0.213) |
87.19(4,0.085) |
82.41(10,0.374) |
Hall, J. et al. |
84.74(6,0.000) |
84.21(3,0.371) |
80.61(6,0.385) |
92.20(6,0.087) |
87.60(5,0.123) |
82.35(6,0.238) |
86.77(12,0.001) |
80.66(9,0.414) |
81.71(6,0.072) |
86.26(5,0.485) |
85.04(5,0.331) |
Attardi et al. |
83.96(7,0.001) |
82.53(8,0.112) |
76.88(11,0.080) |
91.41(7,0.208) |
86.73(8,0.258) |
83.40(5,0.057) |
86.99(10,0.489) |
80.75(8,0.449) |
81.81(5,0.442) |
85.54(8,0.257) |
83.56(7,0.109) |
Chen |
83.22(8,0.266) |
83.49(5,0.474) |
78.65(8,0.446) |
90.87(8,0.480) |
85.91(10,0.292) |
80.14(11,0.028) |
84.91(13,0.187) |
81.16(6,0.263) |
79.25(11,0.197) |
85.91(7,0.308) |
81.92(12,0.101) |
Hall, K. et al. |
83.08(9,0.093) |
83.45(6,0.322) |
78.55(9,0.054) |
87.80(15,0.014) |
87.91(3,0.496) |
78.47(12,0.083) |
83.21(15,0.280) |
82.04(2,0.185) |
79.34(10,0.445) |
84.81(9,0.394) |
85.18(4,0.418) |
Duan(1) |
82.77(10,0.070) |
79.04(13,0.010) |
77.59(10,0.143) |
89.71(12,0.029) |
86.88(7,0.402) |
80.82(10,0.197) |
86.97(11,0.409) |
80.77(7,0.476) |
80.66(7,0.374) |
84.20(11,0.026) |
81.03(13,0.123) |
Schiehlen |
82.42(11,0.000) |
81.07(11,0.427) |
73.30(14,0.292) |
90.79(10,0.065) |
85.45(11,0.149) |
81.73(7,0.242) |
88.91(6,0.062) |
80.47(10,0.451) |
78.61(12,0.270) |
84.54(10,0.313) |
79.33(15,0.035) |
Johansson et al. |
81.13(12,0.000) |
80.91(12,0.008) |
80.43(7,0.002) |
88.34(13,0.359) |
81.30(15,0.027) |
77.39(13,0.401) |
81.43(18,0.000) |
79.58(12,0.150) |
75.53(15,0.431) |
81.55(15,0.061) |
84.80(6,0.024) |
Mannem |
80.30(13,0.108) |
81.56(9,0.465) |
72.88(15,0.000) |
89.81(11,0.429) |
78.84(17,0.003) |
77.20(14,0.000) |
82.81(16,0.093) |
78.89(13,0.139) |
75.39(16,0.002) |
82.91(12,0.307) |
82.74(8,0.485) |
Nguyen |
80.00(14,0.000) |
73.46(18,0.000) |
69.15(18,0.000) |
88.12(14,0.358) |
84.05(13,0.191) |
80.91(9,0.468) |
88.01(7,0.419) |
77.56(15,0.019) |
78.13(13,0.145) |
80.40(16,0.012) |
80.19(14,0.095) |
Jia |
78.46(15,0.464) |
74.20(17,0.216) |
70.24(16,0.448) |
90.83(9,0.468) |
83.39(14,0.004) |
70.41(18,0.000) |
84.37(14,0.079) |
75.65(16,0.288) |
77.19(14,0.021) |
82.36(14,0.154) |
75.96(17,0.131) |
Wu |
78.44(16,0.000) |
77.05(14,0.401) |
75.77(12,0.237) |
85.85(16,0.030) |
79.71(16,0.172) |
73.07(16,0.126) |
81.69(17,0.382) |
78.12(14,0.251) |
72.39(18,0.000) |
82.57(13,0.371) |
78.15(16,0.000) |
Maes et al. |
76.60(17,0.000) |
75.47(16,0.055) |
75.27(13,0.005) |
84.35(17,0.000) |
76.57(18,0.007) |
74.03(15,0.138) |
71.62(21,0.000) |
75.19(17,0.000) |
72.93(17,0.273) |
78.32(18,0.000) |
82.21(11,0.330) |
Canisius et al. |
74.83(18,0.000) |
76.89(15,0.025) |
70.17(17,0.065) |
81.64(18,0.000) |
74.81(19,0.000) |
72.12(17,0.025) |
78.23(19,0.342) |
72.46(18,0.000) |
67.80(19,0.000) |
79.08(17,0.232) |
75.14(18,0.000) |
Zeman |
62.02(19,0.000) |
58.55(20) |
57.42(20) |
68.50(20) |
62.93(20,0.000) |
59.19(20) |
58.33(22,0.000) |
62.89(19,0.045) |
59.78(20,0.000) |
68.27(19,0.002) |
64.30(19,0.417) |
Marinov |
60.83(20,0.000) |
64.27(19,0.000) |
58.55(19,0.107) |
74.22(19,0.000) |
56.09(21) |
59.57(19,0.360) |
54.33(23) |
61.18(20) |
50.39(21) |
65.52(20) |
64.13(20) |
Duan(2) |
25.53(21,0.000) |
|
|
|
86.94(6,0.499) |
|
87.87(8,0.396) |
|
80.53(8,0.229) |
|
|
Nash |
8.77(22,0.000) |
|
|
|
|
|
87.71(9,0.145) |
|
|
|
|
Shimizu et al. |
7.79(23) |
|
|
|
|
|
77.91(20,0.000) |
|
|
|
|
Multilingual Track: Label Accuracy
Team |
Average |
Arabic |
Basque |
Catalan |
Chinese |
Czech |
English |
Greek |
Hungarian |
Italian |
Turkish |
Sagae |
87.29(1) |
85.42(5) |
80.85(6) |
92.03(5) |
87.06(1) |
83.38(7) |
93.12(2) |
84.30(2) |
90.62(2) |
90.05(1) |
86.04(5) |
Hall, J. et al. |
87.02(2) |
85.73(2) |
80.98(5) |
92.19(4) |
86.03(3) |
84.55(5) |
90.53(12) |
84.16(4) |
89.98(3) |
88.83(6) |
87.24(2) |
Nakagawa |
86.94(3) |
85.03(6) |
78.81(8) |
92.22(3) |
86.22(2) |
86.56(1) |
92.40(3) |
85.47(1) |
88.26(9) |
89.19(4) |
85.20(9) |
Titov et al. |
86.87(4) |
85.60(4) |
81.22(4) |
91.33(8) |
85.06(7) |
84.78(4) |
92.38(4) |
83.51(9) |
89.52(5) |
88.95(5) |
86.33(4) |
Carreras |
86.83(5) |
83.00(11) |
83.62(1) |
92.24(2) |
84.11(9) |
85.41(2) |
93.76(1) |
83.74(8) |
88.30(8) |
89.87(2) |
84.27(11) |
Nilsson et al. |
86.67(6) |
86.55(1) |
82.52(2) |
93.02(1) |
78.78(15) |
84.25(6) |
92.16(5) |
81.64(16) |
90.85(1) |
89.62(3) |
87.33(1) |
Attardi et al. |
86.00(7) |
84.41(7) |
76.36(13) |
91.95(6) |
84.31(8) |
85.29(3) |
91.33(8) |
83.83(7) |
88.44(7) |
88.21(8) |
85.89(6) |
Chen |
85.62(8) |
85.64(3) |
78.20(9) |
91.89(7) |
83.82(10) |
81.71(11) |
89.57(13) |
84.16(3) |
87.00(13) |
88.68(7) |
85.58(8) |
Hall, K. et al. |
84.96(9) |
84.21(8) |
76.09(14) |
89.27(15) |
85.62(4) |
82.22(9) |
88.35(15) |
84.14(5) |
87.96(10) |
87.48(9) |
84.27(10) |
Duan(1) |
84.87(10) |
81.40(15) |
77.35(11) |
89.55(14) |
85.35(6) |
83.17(8) |
90.57(11) |
83.87(6) |
88.92(6) |
87.32(11) |
81.17(15) |
Schiehlen |
84.68(11) |
82.32(12) |
76.64(12) |
91.29(9) |
83.28(11) |
82.03(10) |
90.89(10) |
83.16(11) |
87.60(11) |
87.44(10) |
82.18(14) |
Johansson et al. |
84.54(12) |
84.07(10) |
81.34(3) |
89.65(13) |
79.98(13) |
81.24(12) |
87.37(17) |
83.39(10) |
86.18(16) |
85.30(14) |
86.88(3) |
Mannem |
83.61(13) |
84.09(9) |
74.14(15) |
90.35(11) |
77.10(17) |
80.82(13) |
88.13(16) |
82.81(12) |
86.57(14) |
86.38(13) |
85.75(7) |
Wu |
83.46(14) |
81.95(14) |
79.87(7) |
90.07(12) |
78.55(16) |
78.07(16) |
85.97(18) |
82.45(13) |
87.20(12) |
87.21(12) |
83.23(12) |
Maes et al. |
82.23(15) |
82.08(13) |
77.40(10) |
89.04(16) |
75.12(18) |
80.42(15) |
82.27(20) |
82.12(14) |
86.36(15) |
84.56(15) |
82.92(13) |
Nguyen |
81.44(16) |
80.11(16) |
66.73(18) |
90.57(10) |
83.16(12) |
80.78(14) |
91.59(7) |
81.99(15) |
81.32(18) |
83.12(16) |
75.01(18) |
Canisius et al. |
75.94(17) |
71.10(19) |
71.48(16) |
83.05(17) |
73.61(19) |
64.86(19) |
84.77(19) |
70.50(20) |
82.30(17) |
81.89(17) |
75.85(17) |
Zeman |
71.98(18) |
69.13(20) |
70.00(17) |
79.15(19) |
62.14(20) |
62.85(20) |
71.64(22) |
72.04(18) |
80.41(19) |
74.98(19) |
77.49(16) |
Jia |
71.03(19) |
79.51(17) |
63.01(20) |
25.30(20) |
78.82(14) |
67.93(18) |
88.69(14) |
75.96(17) |
78.88(20) |
81.48(18) |
70.77(20) |
Marinov |
70.94(20) |
74.38(18) |
64.79(19) |
82.26(18) |
59.76(21) |
72.93(17) |
69.02(23) |
71.96(19) |
67.35(21) |
74.47(20) |
72.50(19) |
Duan(2) |
26.72(21) |
|
|
|
85.41(5) |
|
92.04(6) |
|
89.73(4) |
|
|
Nash |
9.13(22) |
|
|
|
|
|
91.31(9) |
|
|
|
|
Shimizu et al. |
7.90(23) |
|
|
|
|
|
78.99(21) |
|
|
|
|
Domain Adaptation Track: Labeled Attachment Score
Team |
pchemtb_closed |
pchemtb_open |
Sagae |
81.06(1,0.185) |
|
Attardi et al. |
80.40(2,0.408) |
|
Dredze et al. |
80.22(3,0.099) |
|
Nguyen |
79.50(4,0.000) |
|
Jia |
76.48(5,0.000) |
|
Bick |
71.81(6,0.000) |
78.48(1,0.000) |
Shimizu et al. |
64.15(7,0.000) |
63.49(2,0.341) |
Zeman |
50.61(8) |
|
Schneider et al. |
|
63.01(3,0.000) |
Watson et al. |
|
55.47(4) |
Domain Adaptation Track: Unlabeled Attachment Score
Team |
pchemtb_closed |
pchemtb_open |
CHILDES_closed |
CHILDES_open |
Sagae |
83.42(1,0.481) |
|
|
|
Dredze et al. |
83.38(2,0.337) |
|
61.37(1) |
|
Attardi et al. |
83.08(3,0.076) |
|
58.67(3) |
|
Nguyen |
82.04(4,0.000) |
|
|
|
Jia |
78.92(5,0.000) |
|
57.43(5) |
|
Bick |
74.71(6,0.000) |
81.62(1,0.000) |
58.07(4) |
62.49(1) |
Shimizu et al. |
71.25(7,0.000) |
70.01(2,0.002) |
|
|
Zeman |
54.57(8) |
|
58.89(2) |
|
Schneider et al. |
|
66.53(3,0.001) |
|
60.27(2) |
Watson et al. |
|
62.79(4) |
|
45.61(3) |
Wu |
|
|
52.89(6) |
|
Domain Adaptation Track: Label Accuracy
Team |
pchemtb_closed |
pchemtb_open |
Sagae |
88.28(1) |
|
Dredze et al. |
87.44(2) |
|
Attardi et al. |
87.40(3) |
|
Nguyen |
86.56(4) |
|
Jia |
84.08(5) |
|
Bick |
80.78(6) |
87.02(1) |
Shimizu et al. |
72.51(7) |
72.07(4) |
Zeman |
66.83(8) |
|
Schneider et al. |
|
76.62(2) |
Watson et al. |
|
72.87(3) |