Skip to main content

Linguistic Diagnostics for Word Embeddings

Intrinsic task performance leaderobard (all vector size data)

Intrinsic task performance leaderobard (all vector size data)

Intrinsic task datasets:

  • WordSim353 (Finkelstein et al., 2002), together with its split into similarity and relatedness sections (Agirre et al., 2009);
  • RareWords (Luong et al., 2013);
  • MTurk (Radinsky et al., 2011);
  • MEN (Bruni et al., 2014);
  • SimLex999 (Hill et al., 2015);
  • Bigger Analogy Test Set (BATS) (Gladkova et al., 2016), split into sections for inflectional and derivational morphology, lexicographic and encyclopedic semantics.
Model Context Dims. MEN Mturk Rare Words WS353 WS353 rel. WS353 sim. Sim999 BATS (Infl.) BATS (Der.) BATS (Lex.) BATS (Enc.) BATS (avg)
CBOW structured_deps 100 0.57 0.56 0.28 0.48 0.39 0.62 0.27 71.75 14.37 10.51 17.44 28.07
CBOW structured_deps 25 0.41 0.44 0.19 0.27 0.23 0.39 0.21 38.29 2.68 6.73 7.44 13.55
CBOW structured_deps 250 0.60 0.60 0.22 0.54 0.45 0.66 0.30 72.58 17.12 10.12 20.97 29.78
CBOW structured_deps 50 0.50 0.50 0.21 0.40 0.33 0.53 0.22 56.2 7.97 9.23 13.76 21.41
CBOW structured_deps 500 0.61 0.61 0.22 0.56 0.47 0.68 0.32 69.19 16.53 8.05 19.43 27.9
GloVe structured_deps 100 0.50 0.55 0.27 0.47 0.40 0.58 0.26 76.16 7.52 7.15 12.15 25.18
GloVe structured_deps 25 0.43 0.47 0.19 0.38 0.34 0.49 0.15 55.94 2.9 4.29 5.03 16.63
GloVe structured_deps 250 0.51 0.52 0.28 0.48 0.38 0.61 0.31 77.9 9.39 5.09 13.94 26.06
GloVe structured_deps 50 0.48 0.48 0.25 0.42 0.35 0.55 0.20 69.93 5.33 5.5 7.08 21.46
GloVe structured_deps 500 0.52 0.50 0.28 0.52 0.42 0.66 0.34 69.62 5.97 2.66 12.77 22.3
SkipGram structured_deps 100 0.62 0.59 0.36 0.51 0.42 0.65 0.31 78.35 22.38 12.57 23.39 33.63
SkipGram structured_deps 25 0.46 0.51 0.29 0.29 0.25 0.39 0.24 49.19 4.32 5.79 10.87 17.31
SkipGram structured_deps 250 0.66 0.63 0.39 0.58 0.46 0.70 0.36 85.67 26.09 16.72 26.86 38.28
SkipGram structured_deps 50 0.55 0.56 0.34 0.43 0.36 0.55 0.27 69.2 13.69 10.66 16.01 26.93
SkipGram structured_deps 500 0.67 0.62 0.39 0.61 0.48 0.74 0.41 86.07 24.41 10.31 24.3 35.72
CBOW structured_linear 100 0.53 0.47 0.25 0.35 0.30 0.44 0.28 70.3 20.4 8.81 20.85 29.63
CBOW structured_linear 25 0.28 0.28 0.19 0.13 0.15 0.18 0.17 21.12 4.03 6.47 7.6 9.68
CBOW structured_linear 250 0.60 0.53 0.27 0.44 0.36 0.54 0.35 80.28 26.83 10.29 29.76 36.29
CBOW structured_linear 50 0.42 0.36 0.21 0.24 0.23 0.32 0.22 49.87 11.48 8.59 15.76 21.12
CBOW structured_linear 500 0.63 0.56 0.28 0.51 0.43 0.61 0.38 81.49 26.27 6.35 30.21 35.6
GloVe structured_linear 100 0.51 0.53 0.27 0.45 0.42 0.52 0.23 68.96 10.86 7.99 14.95 25.22
GloVe structured_linear 25 0.41 0.53 0.19 0.29 0.30 0.36 0.13 35.57 2.34 3.44 6.85 11.82
GloVe structured_linear 250 0.53 0.54 0.27 0.50 0.44 0.59 0.28 75.51 15.4 5.54 23.19 29.41
GloVe structured_linear 50 0.48 0.54 0.23 0.42 0.41 0.48 0.17 57.21 5.85 5.67 8.73 18.98
GloVe structured_linear 500 0.53 0.54 0.27 0.51 0.45 0.60 0.31 74.87 13.63 2.45 17.74 26.64
SkipGram structured_linear 100 0.62 0.58 0.39 0.50 0.45 0.59 0.36 74.62 24.25 11.13 23 32.79
SkipGram structured_linear 25 0.47 0.51 0.29 0.33 0.31 0.42 0.25 36.84 3.15 4.95 10.07 13.55
SkipGram structured_linear 250 0.67 0.59 0.42 0.58 0.50 0.68 0.41 84.61 31.19 13.56 33.82 40.26
SkipGram structured_linear 50 0.56 0.58 0.34 0.44 0.40 0.54 0.31 63.3 12.53 6.36 15.54 24.1
SkipGram structured_linear 500 0.69 0.59 0.43 0.62 0.52 0.72 0.44 88.03 34.57 10.84 33.42 41.17
CBOW word_deps 100 0.60 0.57 0.21 0.43 0.36 0.53 0.27 61.3 17.27 5.5 16.58 24.78
CBOW word_deps 25 0.40 0.39 0.23 0.26 0.21 0.32 0.23 23.79 4.22 3.47 6.25 9.22
CBOW word_deps 250 0.65 0.54 0.31 0.51 0.40 0.62 0.35 76.1 24.1 9.01 28.41 33.92
CBOW word_deps 50 0.49 0.44 0.24 0.36 0.28 0.46 0.23 43.17 12.17 6.15 13.39 18.45
CBOW word_deps 500 0.68 0.58 0.32 0.55 0.44 0.65 0.38 75.44 26.36 6.54 28.56 33.74
GloVe word_deps 100 0.58 0.59 0.23 0.45 0.45 0.53 0.20 68.25 14.7 10.02 30.63 30.36
GloVe word_deps 25 0.45 0.54 0.15 0.32 0.37 0.38 0.12 34.36 6.94 5.55 9.27 13.7
GloVe word_deps 250 0.62 0.57 0.26 0.50 0.48 0.58 0.24 73.77 18.81 11.39 42.34 36.02
GloVe word_deps 50 0.53 0.56 0.19 0.41 0.43 0.48 0.15 56.37 9.3 7.8 19.32 22.74
GloVe word_deps 500 0.64 0.58 0.26 0.51 0.49 0.59 0.26 75.33 18.38 7.51 44.89 35.96
SkipGram word_deps 100 0.66 0.60 0.38 0.55 0.47 0.66 0.36 75.98 30.16 13.49 35.83 38.36
SkipGram word_deps 25 0.53 0.54 0.30 0.42 0.39 0.50 0.26 31.69 10.56 5.54 17.7 16.15
SkipGram word_deps 250 0.70 0.61 0.41 0.63 0.52 0.75 0.41 85.36 38.87 15.99 42.07 45.01
SkipGram word_deps 50 0.60 0.59 0.34 0.50 0.45 0.60 0.30 62.15 20.17 7.77 26.43 28.75
SkipGram word_deps 500 0.72 0.62 0.43 0.66 0.55 0.76 0.44 88.27 36.07 12.43 39 43.37
CBOW word_linear 100 0.61 0.58 0.31 0.53 0.49 0.62 0.29 72.12 24.91 9.88 28.7 33.43
CBOW word_linear 25 0.39 0.44 0.24 0.43 0.39 0.49 0.20 33.13 6.38 5.15 13.24 14.28
CBOW word_linear 250 0.67 0.61 0.34 0.60 0.54 0.67 0.37 82.68 33.16 11.29 35.41 40.09
CBOW word_linear 50 0.53 0.51 0.27 0.45 0.43 0.53 0.26 56.92 17.4 8.04 21.63 25.68
CBOW word_linear 500 0.69 0.61 0.35 0.63 0.54 0.70 0.40 83.29 32.68 6.98 34.44 38.78
GloVe word_linear 100 0.54 0.56 0.23 0.45 0.48 0.51 0.18 62.66 12.88 8.02 30.02 27.98
GloVe word_linear 25 0.42 0.51 0.13 0.31 0.36 0.35 0.08 23.02 3.28 2.66 10.86 9.81
GloVe word_linear 250 0.58 0.59 0.25 0.48 0.50 0.53 0.23 68.38 16.77 5.88 42.73 32.95
GloVe word_linear 50 0.49 0.53 0.17 0.39 0.44 0.45 0.13 44.37 8.26 5.3 18.93 18.93
GloVe word_linear 500 0.60 0.58 0.24 0.50 0.50 0.56 0.25 68.63 13.96 3.87 44.3 32.18
SkipGram word_linear 100 0.64 0.58 0.38 0.55 0.51 0.64 0.33 73.43 27.42 9.68 31.31 35.01
SkipGram word_linear 25 0.50 0.52 0.29 0.40 0.40 0.46 0.25 29.66 6.24 4.54 21.08 15.19
SkipGram word_linear 250 0.69 0.61 0.41 0.61 0.53 0.70 0.40 83.22 36.13 12.1 39.15 42.11
SkipGram word_linear 50 0.58 0.54 0.33 0.49 0.48 0.56 0.28 57.7 13.73 7.37 26.42 25.97
SkipGram word_linear 500 0.71 0.63 0.42 0.64 0.55 0.72 0.44 87.62 38.4 11.42 39.19 43.61