Skip to main content

Linguistic Diagnostics for Word Embeddings

LD scores leaderoard (full data)

LD scores leaderoard (full data)

All embeddings were trained on English Wikipedia of August 2013, as described in this paper. Two parameters are systematically varied: dimensionality and type of syntactic context (bound vs unbound, linear vs dependency-based). All embeddings are size 500.

LD scores quick reference (details):

  • SharedMorphForm: the two words are both lemmas/ comparatives/etc.
  • SharedPOS: the two words overlap in lists of their possible POSes;
  • SharedDerivation: shared affixes or stems;
  • Associations: words are associations according to EAT or USF-FAN;
  • Synonyms, Antonyms, etc: the amount thereof in the neighborhoods;
  • LowFreqNeighbors, HighFreqNeighbors: the frequency of the neighbor in the source corpus is under/above 10,000;
  • NeighborsInGDeps: the two words co-occur in the Google dependency ngrams;
  • NonCooccurring: the number of word pairs that do not co-occur in the source corpus;
  • CloseNeighbors / FarNeighbors: the number of top 100 neighbors with cosine distance to the target word over 0.8 / less than 0.7;
  • ShortestPath: minimum path between synsets of two words in the WordNet ontology.
Model Context Dims. Shared Form Shared Deriv. Shared POS Proper Nouns Num-bers Foreign Words Mis-spell. Associ-ations Syno-nyms Anto-nyms Mero-nyms Hyper-nyms Hypo-nyms Other Shortest Path Gdeps LowFreq Neighb. HighFreq Neighb. NonCooc. Close Neighb. Far Neighb.
CBOW structured_deps 100 76.05 18.71 71.42 17.96 2.14 1.06 7.85 2.061 1.21 0.454 0.033 0.022 0.108 0.032 0.083 29.943 87.034 8.579 81.677 94.053 0
CBOW structured_deps 25 79.718 18.06 75.79 18.26 1.61 0.91 5.69 1.618 0.81 0.329 0.018 0.015 0.091 0.028 0.077 31.166 82.861 12.172 81.441 100 0
CBOW structured_deps 250 76.716 18.90 71.64 17.70 2.14 1.04 7.92 1.811 1.17 0.429 0.028 0.017 0.1 0.035 0.083 28.369 90.429 6.636 83.437 78.394 0.568
CBOW structured_deps 50 78.605 18.54 74.29 17.92 1.79 0.95 6.50 1.87 1.04 0.387 0.023 0.019 0.086 0.036 0.077 30.979 85.132 10.294 81.037 99.512 0
CBOW structured_deps 500 76.244 18.79 70.81 17.53 2.26 1.07 8.30 1.666 1.16 0.431 0.03 0.012 0.084 0.033 0.083 27.416 91.816 5.82 84.398 70.868 2.923
GloVe structured_deps 100 60.927 6.31 55.63 31.77 2.32 3.25 8.64 2.976 1.23 0.386 0.028 0.036 0.102 0.042 0.071 31.513 75.849 16.969 77.338 4.288 46.135
GloVe structured_deps 25 60.017 4.87 53.56 34.10 2.79 3.17 7.82 2.13 0.81 0.281 0.019 0.026 0.08 0.021 0.071 29.931 75.31 17.766 78.901 92.078 0
GloVe structured_deps 250 60.633 6.86 54.17 30.73 2.56 3.33 10.43 3.045 1.37 0.392 0.033 0.033 0.089 0.04 0.071 30.532 78.072 15.527 78.353 0.464 87.558
GloVe structured_deps 50 60.553 5.54 55.13 32.79 2.52 3.29 8.13 2.688 1.07 0.329 0.026 0.031 0.083 0.033 0.071 30.96 75.385 17.462 77.635 22.727 0.018
GloVe structured_deps 500 59.983 7.49 52.04 30.30 2.93 3.53 12.60 2.87 1.37 0.405 0.026 0.034 0.107 0.039 0.077 28.222 81.715 12.853 80.319 0.099 93.639
SkipGram structured_deps 100 76.611 18.69 70.79 18.05 1.92 1.19 7.20 2.248 1.41 0.491 0.035 0.033 0.122 0.041 0.083 31.815 86.879 8.934 80.993 95.675 0
SkipGram structured_deps 25 76.484 5.32 77.60 18.40 0.93 1.96 3.25 0.78 0.98 0.231 0 0.044 0.011 0 0.1 24.692 90.165 7.89 93.132 100 0
SkipGram structured_deps 250 74.826 18.65 67.78 18.11 2.07 1.28 8.33 2.099 1.47 0.502 0.029 0.032 0.12 0.045 0.083 28.277 90.836 6.009 83.741 74.395 0.124
SkipGram structured_deps 50 77.31 18.29 72.54 18.12 1.95 1.09 6.50 2.2 1.26 0.455 0.035 0.026 0.119 0.037 0.077 33.254 84.277 10.791 79.881 99.79 0
SkipGram structured_deps 500 73.924 18.58 65.54 18.12 2.22 1.31 9.14 1.834 1.41 0.484 0.03 0.029 0.109 0.037 0.083 25.228 93.762 3.876 86.152 54.411 2.069
CBOW structured_linear 100 74.079 17.93 70.34 18.78 2.25 1.14 7.34 2.042 1.25 0.47 0.029 0.026 0.119 0.042 0.077 33.448 85.193 10.807 79.593 96.216 0
CBOW structured_linear 25 74.117 16.54 72.32 19.46 1.88 0.89 6.05 1.486 0.85 0.366 0.018 0.019 0.081 0.029 0.077 32.081 84.066 11.688 81.426 100 0
CBOW structured_linear 250 72.547 18.16 67.00 19.16 2.49 1.38 8.69 2.133 1.43 0.491 0.026 0.028 0.114 0.036 0.083 30.662 88.586 8.307 81.494 68.711 0.644
CBOW structured_linear 50 75.272 17.42 72.11 19.01 2.07 0.90 6.49 1.878 1.06 0.432 0.02 0.024 0.103 0.042 0.077 34.01 84.094 11.562 79.71 99.927 0
CBOW structured_linear 500 71.539 17.99 64.56 19.47 2.65 1.46 9.66 1.941 1.46 0.487 0.034 0.025 0.102 0.039 0.083 27.974 91.757 6.035 83.815 50.185 7.078
GloVe structured_linear 100 71.712 14.04 72.93 21.79 1.51 1.11 4.22 3.417 1.22 0.43 0.023 0.034 0.114 0.045 0.071 43.27 63.257 24.373 68.317 10.138 8.184
GloVe structured_linear 25 69.671 11.22 69.84 22.22 1.66 1.20 4.30 2.564 0.81 0.325 0.012 0.023 0.079 0.024 0.071 40.716 62.837 25.229 70.827 99.792 0
GloVe structured_linear 250 70.871 14.45 72.32 21.36 1.72 1.11 4.80 3.7 1.42 0.471 0.025 0.037 0.116 0.047 0.071 43.397 64.383 23.328 68.381 1.218 74.321
GloVe structured_linear 50 71.773 13.09 72.23 22.29 1.59 1.16 4.15 3.026 0.98 0.374 0.017 0.03 0.094 0.033 0.071 41.973 62.972 24.682 69.248 56.192 0
GloVe structured_linear 500 70.891 14.70 70.86 21.02 2.07 1.26 5.48 3.733 1.53 0.474 0.026 0.034 0.116 0.042 0.071 42.743 66.312 22.256 69.05 0.318 92.957
SkipGram structured_linear 100 75.274 18.76 70.68 17.77 1.97 1.08 7.20 2.2 1.43 0.536 0.026 0.028 0.123 0.048 0.077 35.373 85.274 10.765 78.491 92.876 0
SkipGram structured_linear 25 74.22 16.93 71.08 18.33 1.88 0.95 6.55 1.645 1.00 0.403 0.017 0.018 0.095 0.036 0.077 32.465 85.15 10.547 80.76 100 0
SkipGram structured_linear 250 73.409 18.67 67.05 17.92 2.17 1.20 8.32 2.374 1.62 0.562 0.029 0.034 0.129 0.048 0.083 33.275 87.618 8.937 79.845 53.876 0.143
SkipGram structured_linear 50 76.217 18.05 71.79 17.88 1.97 1.04 6.59 1.941 1.20 0.479 0.031 0.03 0.122 0.041 0.077 34.791 84.932 10.79 79.202 99.983 0
SkipGram structured_linear 500 71.971 18.31 63.46 18.31 2.38 1.27 9.46 2.27 1.66 0.547 0.029 0.032 0.117 0.047 0.083 30 90.198 6.88 82.1 26.699 5.949
CBOW word_deps 100 69.387 14.85 65.00 21.25 1.96 1.12 5.06 2.613 1.22 0.406 0.033 0.023 0.112 0.039 0.077 39.558 75.144 15.253 70.834 79.081 0.173
CBOW word_deps 25 67.52 13.43 65.16 21.66 1.96 0.94 4.09 1.882 0.89 0.318 0.024 0.017 0.091 0.026 0.071 40.884 69.336 20.014 70.267 100 0
CBOW word_deps 250 68.672 16.49 63.33 19.93 2.27 1.22 6.30 2.952 1.60 0.527 0.036 0.03 0.132 0.043 0.077 40.629 74.416 14.956 69.803 48.75 5.256
CBOW word_deps 50 69.473 15.11 66.96 20.54 1.96 1.02 4.34 2.422 1.16 0.399 0.033 0.033 0.117 0.035 0.071 43.299 66.105 21.384 67.838 97.835 0
CBOW word_deps 500 68.298 16.66 61.22 20.07 2.48 1.28 7.30 2.737 1.60 0.523 0.045 0.03 0.139 0.045 0.077 37.442 80.398 11.015 72.866 37.711 19.124
GloVe word_deps 100 55.829 6.05 46.45 28.58 2.38 2.31 7.54 4.24 1.31 0.426 0.035 0.041 0.128 0.036 0.071 45.361 62.73 20.227 62.059 15.032 10.747
GloVe word_deps 25 55.565 4.81 44.93 28.21 2.56 2.25 7.35 2.983 0.92 0.3 0.021 0.03 0.099 0.028 0.071 41.818 64.756 21.043 67.059 100 0
GloVe word_deps 250 56.069 6.86 46.48 27.71 2.50 2.33 8.27 4.58 1.50 0.474 0.032 0.039 0.121 0.041 0.071 46.27 63.352 18.914 60.422 3.43 62.201
GloVe word_deps 50 56.081 5.32 45.67 28.71 2.52 2.33 7.42 3.704 1.12 0.395 0.028 0.032 0.129 0.036 0.071 43.84 63.295 20.656 64.164 47.823 0
GloVe word_deps 500 55.693 7.15 45.55 27.81 2.67 2.46 9.37 4.658 1.59 0.483 0.036 0.039 0.13 0.04 0.071 46.072 64.713 17.738 60.102 0.88 81.843
SkipGram word_deps 100 66.994 14.18 58.61 20.71 2.70 1.36 8.52 2.521 1.55 0.55 0.041 0.028 0.13 0.041 0.077 35.3 85.243 9.09 76.48 91.945 0
SkipGram word_deps 25 65.012 12.48 60.68 21.12 2.25 1.09 6.77 1.968 1.10 0.404 0.028 0.025 0.1 0.037 0.077 35.935 82.744 10.776 76.732 100 0
SkipGram word_deps 250 66.054 14.50 55.12 21.12 3.05 1.47 10.20 2.539 1.70 0.567 0.041 0.028 0.13 0.042 0.077 31.864 88.35 7.134 78.714 53.409 0.403
SkipGram word_deps 50 67.136 13.55 60.44 20.58 2.44 1.25 7.47 2.265 1.32 0.487 0.037 0.025 0.129 0.044 0.077 36.282 83.723 10.018 76.108 99.713 0
SkipGram word_deps 500 65.153 14.38 51.69 21.69 3.34 1.60 11.67 2.262 1.68 0.568 0.042 0.03 0.121 0.042 0.083 28.16 91.202 5.13 81.25 31.041 6.175
CBOW word_linear 100 64.989 10.92 54.06 21.86 2.81 1.22 8.41 2.78 1.43 0.509 0.03 0.03 0.131 0.041 0.071 36.659 80.133 12.883 73.063 76.876 0
CBOW word_linear 25 61.697 8.72 53.09 23.99 2.58 0.98 6.90 1.738 0.90 0.35 0.023 0.013 0.078 0.035 0.071 32.96 84.674 10.93 78.18 99.745 0
CBOW word_linear 250 63.979 11.47 51.21 21.99 2.96 1.36 9.95 2.91 1.62 0.54 0.036 0.025 0.128 0.058 0.077 34.101 83.366 10.451 74.524 40.92 7.838
CBOW word_linear 50 63.954 9.83 53.91 22.42 2.65 1.17 7.47 2.391 1.22 0.446 0.035 0.025 0.118 0.042 0.071 35.978 81.322 12.386 74.29 96.366 0
CBOW word_linear 500 62.988 11.58 48.53 22.63 3.22 1.50 11.13 2.699 1.66 0.558 0.041 0.028 0.124 0.055 0.077 30.772 87.471 7.811 77.391 31.022 28.442
GloVe word_linear 100 59.347 7.86 48.57 24.28 2.09 1.54 5.47 4.076 1.29 0.405 0.028 0.033 0.103 0.042 0.071 48.518 58.339 20.702 58.163 21.003 2.242
GloVe word_linear 25 59.12 6.44 48.13 24.27 2.33 1.45 5.59 2.895 0.84 0.294 0.015 0.024 0.087 0.026 0.071 45.183 60.567 22.521 63.992 99.992 0
GloVe word_linear 250 58.802 8.01 47.47 23.41 2.22 1.44 6.27 4.45 1.47 0.424 0.026 0.035 0.116 0.035 0.071 50.22 58.612 19.069 55.744 5.468 54.793
GloVe word_linear 50 60.435 7.88 49.09 24.37 2.14 1.41 5.12 3.526 1.06 0.37 0.025 0.033 0.1 0.033 0.071 46.934 59.103 21.575 60.751 63.16 0
GloVe word_linear 500 59.406 8.05 46.81 22.88 2.45 1.58 7.09 4.587 1.50 0.433 0.029 0.035 0.106 0.043 0.071 49.603 60.459 17.281 55.241 1.598 75.678
SkipGram word_linear 100 66.112 12.17 54.89 20.66 2.67 1.22 8.55 2.521 1.53 0.56 0.04 0.03 0.122 0.047 0.077 36.08 84.857 9.578 74.923 90.787 0
SkipGram word_linear 25 62.089 10.11 54.38 21.90 2.66 0.98 7.41 1.891 1.06 0.387 0.023 0.023 0.091 0.03 0.071 34.458 85.065 9.189 77.474 100 0
SkipGram word_linear 250 64.396 12.38 51.48 20.82 2.80 1.34 10.02 2.635 1.77 0.573 0.037 0.032 0.133 0.054 0.077 33.449 87.295 8.009 76.541 44.382 0.117
SkipGram word_linear 50 65.758 11.30 55.52 20.46 2.67 1.08 7.92 2.237 1.32 0.502 0.04 0.025 0.118 0.044 0.077 36.454 84.546 9.527 75.447 99.811 0
SkipGram word_linear 500 63.153 11.89 47.81 21.24 3.00 1.47 11.43 2.461 1.76 0.568 0.041 0.03 0.121 0.055 0.077 30.052 89.873 6.157 79.042 22.78 7.626

See also the evaluation scores for extrinsic tasks and intrinsic tasks, and how they correlate.