Note: this is no longer a wiki, only a static archive of the orginal!

Home

Score Tables

The following tables give the scores for each data set in the CoNLL 2007 shared task. The "Average" column contains the average score for all ten languages, which determines the ranking in the multilingual track. The ranking for the domain adaptation track is determined based on the pchemtb results only. The CHILDES data set was optional.

The numbers in parentheses next to each score gives the rank and the p-value of the difference with the next rank. The p-values for the "Average" column were computed using a standard z-test for proportions. All the other p-values were computed using version 1.12 of eval07.pl (with the -b option) and Dan Bikel's Randomized Parsing Evaluation Comparator compare.pl (Statistical Significance Tester for evalb Output) (with a default 10,000 iterations). The label accuracy scores are not used for ranking and no significance scores were computed for them. Differences are taken to be significant if p<0.05.

The answers submitted by all the systems can be downloaded as answers.tar.gz. To respect the treebank copyrights only columns 7 and 8 (HEAD and DEPREL) have been included.

Multilingual Track: Labeled Attachment Score

Team

Average

Arabic

Basque

Catalan

Chinese

Czech

English

Greek

Hungarian

Italian

Turkish

Nilsson et al.

80.32(1,0.448)

76.52(1,0.011)

76.94(1,0.024)

88.70(1,0.075)

75.82(15,0.112)

77.98(3,0.488)

88.11(5,0.011)

74.65(2,0.353)

80.27(1,0.078)

84.40(1,0.141)

79.79(2,0.052)

Nakagawa

80.29(2,0.059)

75.08(2,0.330)

72.56(7,0.377)

87.90(3,0.384)

83.84(2,0.305)

80.19(1,0.036)

88.41(3,0.492)

76.31(1,0.002)

76.74(8,0.042)

83.61(3,0.377)

78.22(5,0.070)

Titov et al.

79.90(3,0.494)

74.12(6,0.124)

75.49(3,0.258)

87.40(6,0.218)

82.14(7,0.207)

77.94(4,0.250)

88.39(4,0.347)

73.52(10,0.194)

77.94(4,0.117)

82.26(6,0.415)

79.81(1,0.498)

Sagae

79.90(4,0.354)

74.71(4,0.451)

74.64(6,0.000)

88.16(2,0.293)

84.69(1,0.063)

74.83(8,0.095)

89.01(2,0.102)

73.58(8,0.477)

79.53(2,0.007)

83.91(2,0.294)

75.91(10,0.448)

Hall, J. et al.

79.80(5,0.002)

74.75(3,0.476)

74.99(5,0.259)

87.74(4,0.381)

83.51(3,0.150)

77.22(6,0.018)

85.81(12,0.001)

74.21(6,0.347)

78.09(3,0.431)

82.48(5,0.404)

79.24(3,0.090)

Carreras

79.09(6,0.001)

70.20(11,0.455)

75.75(2,0.349)

87.60(5,0.352)

80.86(10,0.163)

78.60(2,0.238)

89.61(1,0.128)

73.56(9,0.473)

75.42(9,0.449)

83.46(4,0.084)

75.85(11,0.130)

Attardi et al.

78.27(7,0.205)

72.66(8,0.135)

69.48(12,0.274)

86.86(7,0.386)

81.50(8,0.360)

77.37(5,0.422)

85.85(10,0.499)

73.92(7,0.313)

76.81(7,0.469)

81.34(8,0.179)

76.87(7,0.245)

Chen

78.06(8,0.084)

74.65(5,0.218)

72.39(8,0.037)

86.66(8,0.078)

81.24(9,0.328)

73.69(10,0.101)

83.81(13,0.091)

74.42(3,0.422)

75.34(10,0.092)

82.04(7,0.154)

76.31(9,0.247)

Duan(1)

77.70(9,0.001)

69.91(13,0.000)

71.26(9,0.231)

84.95(10,0.242)

82.58(6,0.268)

75.34(7,0.256)

85.83(11,0.487)

74.29(4,0.437)

77.06(5,0.323)

80.75(9,0.465)

75.03(12,0.071)

Hall, K. et al.

76.91(10,0.003)

73.40(7,0.170)

69.81(11,0.290)

82.38(14,0.137)

82.77(4,0.410)

72.27(12,0.046)

81.93(15,0.295)

74.21(5,0.487)

74.20(11,0.342)

80.69(10,0.408)

77.42(6,0.186)

Schiehlen

76.18(11,0.065)

70.08(12,0.431)

66.77(14,0.053)

85.75(9,0.114)

80.04(11,0.362)

73.86(9,0.434)

86.21(9,0.338)

72.29(12,0.171)

73.90(12,0.001)

80.46(11,0.011)

72.48(15,0.000)

Johansson et al.

75.78(12,0.000)

71.76(9,0.395)

75.08(4,0.435)

83.33(12,0.433)

76.30(14,0.276)

70.98(13,0.341)

80.29(17,0.150)

72.77(11,0.230)

71.31(13,0.324)

77.55(14,0.001)

78.46(4,0.344)

Mannem

74.54(13,0.000)

71.55(10,0.043)

65.64(15,0.000)

84.47(11,0.049)

73.76(17,0.000)

70.68(14,0.000)

81.55(16,0.056)

71.69(13,0.070)

70.94(14,0.027)

78.67(13,0.062)

76.42(8,0.416)

Wu

73.02(14,0.035)

66.16(14,0.117)

70.71(10,0.098)

81.44(15,0.005)

74.69(16,0.152)

66.72(16,0.000)

79.49(18,0.001)

70.63(14,0.401)

69.08(15,0.443)

78.79(12,0.428)

72.52(14,0.486)

Nguyen

72.53(15,0.000)

63.58(16,0.424)

58.18(17,0.200)

83.23(13,0.133)

79.77(12,0.000)

72.54(11,0.330)

86.73(6,0.489)

70.42(15,0.009)

68.12(17,0.043)

75.06(16,0.290)

67.63(17,0.000)

Maes et al.

70.66(16,0.000)

65.12(15,0.087)

69.05(13,0.000)

79.21(16,0.000)

70.97(18,0.232)

67.38(15,0.200)

69.68(21,0.000)

68.59(16,0.000)

68.93(16,0.165)

73.63(18,0.000)

74.03(13,0.008)

Canisius et al.

66.99(17,0.000)

59.13(18,0.000)

63.17(16,0.000)

75.44(17,0.000)

70.45(19,0.000)

56.14(17,0.085)

77.27(19,0.000)

60.35(18,0.000)

64.31(19,0.000)

75.57(15,0.217)

68.09(16,0.231)

Jia

63.00(18,0.000)

63.37(17,0.000)

57.61(18,0.000)

23.35(20)

76.36(13,0.474)

54.95(18,0.091)

82.93(14,0.113)

65.45(17,0.000)

66.61(18,0.004)

74.65(17,0.167)

64.68(18,0.000)

Zeman

54.87(19,0.156)

46.06(20)

50.61(20)

62.94(19,0.000)

54.49(20,0.000)

50.21(20)

53.59(22,0.095)

55.29(19,0.160)

55.24(20,0.000)

62.13(19,0.003)

58.10(19,0.074)

Marinov

54.55(20,0.000)

54.00(19,0.000)

51.24(19,0.229)

69.42(18,0.000)

49.87(21)

53.47(19,0.001)

52.11(23)

54.33(20)

44.47(21)

59.75(20)

56.88(20)

Duan(2)

24.62(21,0.000)

82.64(5,0.499)

86.69(7,0.371)

76.89(6,0.461)

Nash

8.65(22,0.000)

86.49(8,0.331)

Shimizu et al.

7.20(23)

72.02(20,0.005)

Multilingual Track: Unlabeled Attachment Score

Team

Average

Arabic

Basque

Catalan

Chinese

Czech

English

Greek

Hungarian

Italian

Turkish

Nakagawa

86.55(1,0.000)

86.09(1,0.265)

81.04(5,0.201)

92.86(4,0.165)

88.88(2,0.035)

86.28(1,0.059)

90.13(2,0.273)

84.08(1,0.000)

82.49(3,0.364)

87.91(1,0.414)

85.77(3,0.125)

Nilsson et al.

85.71(2,0.336)

85.81(2,0.000)

82.84(1,0.066)

93.12(3,0.269)

84.52(12,0.266)

83.59(4,0.379)

88.93(5,0.490)

81.22(4,0.480)

83.55(1,0.473)

87.77(2,0.406)

85.77(2,0.490)

Titov et al.

85.62(3,0.067)

83.18(7,0.175)

81.93(2,0.116)

93.40(1,0.456)

87.91(4,0.342)

84.19(3,0.204)

89.73(4,0.120)

81.20(5,0.493)

82.18(4,0.314)

86.26(6,0.342)

86.22(1,0.230)

Sagae

85.29(4,0.012)

84.04(4,0.166)

81.19(3,0.457)

93.34(2,0.249)

88.94(1,0.456)

81.27(8,0.312)

89.87(3,0.425)

80.37(11,0.134)

83.51(2,0.078)

87.68(3,0.173)

82.72(9,0.312)

Carreras

84.79(5,0.408)

81.48(10,0.292)

81.11(4,0.438)

92.46(5,0.293)

86.20(9,0.351)

85.16(2,0.132)

90.63(1,0.210)

81.37(3,0.418)

79.92(9,0.213)

87.19(4,0.085)

82.41(10,0.374)

Hall, J. et al.

84.74(6,0.000)

84.21(3,0.371)

80.61(6,0.385)

92.20(6,0.087)

87.60(5,0.123)

82.35(6,0.238)

86.77(12,0.001)

80.66(9,0.414)

81.71(6,0.072)

86.26(5,0.485)

85.04(5,0.331)

Attardi et al.

83.96(7,0.001)

82.53(8,0.112)

76.88(11,0.080)

91.41(7,0.208)

86.73(8,0.258)

83.40(5,0.057)

86.99(10,0.489)

80.75(8,0.449)

81.81(5,0.442)

85.54(8,0.257)

83.56(7,0.109)

Chen

83.22(8,0.266)

83.49(5,0.474)

78.65(8,0.446)

90.87(8,0.480)

85.91(10,0.292)

80.14(11,0.028)

84.91(13,0.187)

81.16(6,0.263)

79.25(11,0.197)

85.91(7,0.308)

81.92(12,0.101)

Hall, K. et al.

83.08(9,0.093)

83.45(6,0.322)

78.55(9,0.054)

87.80(15,0.014)

87.91(3,0.496)

78.47(12,0.083)

83.21(15,0.280)

82.04(2,0.185)

79.34(10,0.445)

84.81(9,0.394)

85.18(4,0.418)

Duan(1)

82.77(10,0.070)

79.04(13,0.010)

77.59(10,0.143)

89.71(12,0.029)

86.88(7,0.402)

80.82(10,0.197)

86.97(11,0.409)

80.77(7,0.476)

80.66(7,0.374)

84.20(11,0.026)

81.03(13,0.123)

Schiehlen

82.42(11,0.000)

81.07(11,0.427)

73.30(14,0.292)

90.79(10,0.065)

85.45(11,0.149)

81.73(7,0.242)

88.91(6,0.062)

80.47(10,0.451)

78.61(12,0.270)

84.54(10,0.313)

79.33(15,0.035)

Johansson et al.

81.13(12,0.000)

80.91(12,0.008)

80.43(7,0.002)

88.34(13,0.359)

81.30(15,0.027)

77.39(13,0.401)

81.43(18,0.000)

79.58(12,0.150)

75.53(15,0.431)

81.55(15,0.061)

84.80(6,0.024)

Mannem

80.30(13,0.108)

81.56(9,0.465)

72.88(15,0.000)

89.81(11,0.429)

78.84(17,0.003)

77.20(14,0.000)

82.81(16,0.093)

78.89(13,0.139)

75.39(16,0.002)

82.91(12,0.307)

82.74(8,0.485)

Nguyen

80.00(14,0.000)

73.46(18,0.000)

69.15(18,0.000)

88.12(14,0.358)

84.05(13,0.191)

80.91(9,0.468)

88.01(7,0.419)

77.56(15,0.019)

78.13(13,0.145)

80.40(16,0.012)

80.19(14,0.095)

Jia

78.46(15,0.464)

74.20(17,0.216)

70.24(16,0.448)

90.83(9,0.468)

83.39(14,0.004)

70.41(18,0.000)

84.37(14,0.079)

75.65(16,0.288)

77.19(14,0.021)

82.36(14,0.154)

75.96(17,0.131)

Wu

78.44(16,0.000)

77.05(14,0.401)

75.77(12,0.237)

85.85(16,0.030)

79.71(16,0.172)

73.07(16,0.126)

81.69(17,0.382)

78.12(14,0.251)

72.39(18,0.000)

82.57(13,0.371)

78.15(16,0.000)

Maes et al.

76.60(17,0.000)

75.47(16,0.055)

75.27(13,0.005)

84.35(17,0.000)

76.57(18,0.007)

74.03(15,0.138)

71.62(21,0.000)

75.19(17,0.000)

72.93(17,0.273)

78.32(18,0.000)

82.21(11,0.330)

Canisius et al.

74.83(18,0.000)

76.89(15,0.025)

70.17(17,0.065)

81.64(18,0.000)

74.81(19,0.000)

72.12(17,0.025)

78.23(19,0.342)

72.46(18,0.000)

67.80(19,0.000)

79.08(17,0.232)

75.14(18,0.000)

Zeman

62.02(19,0.000)

58.55(20)

57.42(20)

68.50(20)

62.93(20,0.000)

59.19(20)

58.33(22,0.000)

62.89(19,0.045)

59.78(20,0.000)

68.27(19,0.002)

64.30(19,0.417)

Marinov

60.83(20,0.000)

64.27(19,0.000)

58.55(19,0.107)

74.22(19,0.000)

56.09(21)

59.57(19,0.360)

54.33(23)

61.18(20)

50.39(21)

65.52(20)

64.13(20)

Duan(2)

25.53(21,0.000)

86.94(6,0.499)

87.87(8,0.396)

80.53(8,0.229)

Nash

8.77(22,0.000)

87.71(9,0.145)

Shimizu et al.

7.79(23)

77.91(20,0.000)

Multilingual Track: Label Accuracy

Team

Average

Arabic

Basque

Catalan

Chinese

Czech

English

Greek

Hungarian

Italian

Turkish

Sagae

87.29(1)

85.42(5)

80.85(6)

92.03(5)

87.06(1)

83.38(7)

93.12(2)

84.30(2)

90.62(2)

90.05(1)

86.04(5)

Hall, J. et al.

87.02(2)

85.73(2)

80.98(5)

92.19(4)

86.03(3)

84.55(5)

90.53(12)

84.16(4)

89.98(3)

88.83(6)

87.24(2)

Nakagawa

86.94(3)

85.03(6)

78.81(8)

92.22(3)

86.22(2)

86.56(1)

92.40(3)

85.47(1)

88.26(9)

89.19(4)

85.20(9)

Titov et al.

86.87(4)

85.60(4)

81.22(4)

91.33(8)

85.06(7)

84.78(4)

92.38(4)

83.51(9)

89.52(5)

88.95(5)

86.33(4)

Carreras

86.83(5)

83.00(11)

83.62(1)

92.24(2)

84.11(9)

85.41(2)

93.76(1)

83.74(8)

88.30(8)

89.87(2)

84.27(11)

Nilsson et al.

86.67(6)

86.55(1)

82.52(2)

93.02(1)

78.78(15)

84.25(6)

92.16(5)

81.64(16)

90.85(1)

89.62(3)

87.33(1)

Attardi et al.

86.00(7)

84.41(7)

76.36(13)

91.95(6)

84.31(8)

85.29(3)

91.33(8)

83.83(7)

88.44(7)

88.21(8)

85.89(6)

Chen

85.62(8)

85.64(3)

78.20(9)

91.89(7)

83.82(10)

81.71(11)

89.57(13)

84.16(3)

87.00(13)

88.68(7)

85.58(8)

Hall, K. et al.

84.96(9)

84.21(8)

76.09(14)

89.27(15)

85.62(4)

82.22(9)

88.35(15)

84.14(5)

87.96(10)

87.48(9)

84.27(10)

Duan(1)

84.87(10)

81.40(15)

77.35(11)

89.55(14)

85.35(6)

83.17(8)

90.57(11)

83.87(6)

88.92(6)

87.32(11)

81.17(15)

Schiehlen

84.68(11)

82.32(12)

76.64(12)

91.29(9)

83.28(11)

82.03(10)

90.89(10)

83.16(11)

87.60(11)

87.44(10)

82.18(14)

Johansson et al.

84.54(12)

84.07(10)

81.34(3)

89.65(13)

79.98(13)

81.24(12)

87.37(17)

83.39(10)

86.18(16)

85.30(14)

86.88(3)

Mannem

83.61(13)

84.09(9)

74.14(15)

90.35(11)

77.10(17)

80.82(13)

88.13(16)

82.81(12)

86.57(14)

86.38(13)

85.75(7)

Wu

83.46(14)

81.95(14)

79.87(7)

90.07(12)

78.55(16)

78.07(16)

85.97(18)

82.45(13)

87.20(12)

87.21(12)

83.23(12)

Maes et al.

82.23(15)

82.08(13)

77.40(10)

89.04(16)

75.12(18)

80.42(15)

82.27(20)

82.12(14)

86.36(15)

84.56(15)

82.92(13)

Nguyen

81.44(16)

80.11(16)

66.73(18)

90.57(10)

83.16(12)

80.78(14)

91.59(7)

81.99(15)

81.32(18)

83.12(16)

75.01(18)

Canisius et al.

75.94(17)

71.10(19)

71.48(16)

83.05(17)

73.61(19)

64.86(19)

84.77(19)

70.50(20)

82.30(17)

81.89(17)

75.85(17)

Zeman

71.98(18)

69.13(20)

70.00(17)

79.15(19)

62.14(20)

62.85(20)

71.64(22)

72.04(18)

80.41(19)

74.98(19)

77.49(16)

Jia

71.03(19)

79.51(17)

63.01(20)

25.30(20)

78.82(14)

67.93(18)

88.69(14)

75.96(17)

78.88(20)

81.48(18)

70.77(20)

Marinov

70.94(20)

74.38(18)

64.79(19)

82.26(18)

59.76(21)

72.93(17)

69.02(23)

71.96(19)

67.35(21)

74.47(20)

72.50(19)

Duan(2)

26.72(21)

85.41(5)

92.04(6)

89.73(4)

Nash

9.13(22)

91.31(9)

Shimizu et al.

7.90(23)

78.99(21)

Domain Adaptation Track: Labeled Attachment Score

Team

pchemtb_closed

pchemtb_open

Sagae

81.06(1,0.185)

Attardi et al.

80.40(2,0.408)

Dredze et al.

80.22(3,0.099)

Nguyen

79.50(4,0.000)

Jia

76.48(5,0.000)

Bick

71.81(6,0.000)

78.48(1,0.000)

Shimizu et al.

64.15(7,0.000)

63.49(2,0.341)

Zeman

50.61(8)

Schneider et al.

63.01(3,0.000)

Watson et al.

55.47(4)

Domain Adaptation Track: Unlabeled Attachment Score

Team

pchemtb_closed

pchemtb_open

CHILDES_closed

CHILDES_open

Sagae

83.42(1,0.481)

Dredze et al.

83.38(2,0.337)

61.37(1)

Attardi et al.

83.08(3,0.076)

58.67(3)

Nguyen

82.04(4,0.000)

Jia

78.92(5,0.000)

57.43(5)

Bick

74.71(6,0.000)

81.62(1,0.000)

58.07(4)

62.49(1)

Shimizu et al.

71.25(7,0.000)

70.01(2,0.002)

Zeman

54.57(8)

58.89(2)

Schneider et al.

66.53(3,0.001)

60.27(2)

Watson et al.

62.79(4)

45.61(3)

Wu

52.89(6)

Domain Adaptation Track: Label Accuracy

Team

pchemtb_closed

pchemtb_open

Sagae

88.28(1)

Dredze et al.

87.44(2)

Attardi et al.

87.40(3)

Nguyen

86.56(4)

Jia

84.08(5)

Bick

80.78(6)

87.02(1)

Shimizu et al.

72.51(7)

72.07(4)

Zeman

66.83(8)

Schneider et al.

76.62(2)

Watson et al.

72.87(3)

AllScores (last edited 2007-05-15 09:48:06 by DenizYuret)