Extrinsic task performance leaderboard (all vector size data)
See LDT paper for implementation details on all tasks.
Extrinsic task datasets:
- POS-tag.: POS-tagging task from CoNLL 2003 dataset (Tjong Kim Sang and De Meulder, 2003);
- Chunk.: Chunking task from CoNLL 2003 dataset (Tjong Kim Sang and De Meulder, 2003);
- NER: Named Entity Recognition task from CoNLL 2003 dataset (Tjong Kim Sang and De Meulder, 2003);
- Rel.Clas.: semantic relation classification task from SemEval 2010 (Hendrickx et al., 2010);
- Subj. Clas.: Rotten Tomato dataset (Pang and Lee, 2004);
- MR: short movie review dataset for sentence-level sentiment classification (Pang and Lee, 2005);
- IMDB: Standofd IMDB movie review dataset (Maas et al., 2011);
- SNLI: Stanford Natural Language Inference dataset (Bowman et al., 2015).
| Model | Context | Dims. | POS-tag. | Chunk. | NER | Rel. Clas. | Subj. Clas. | MR | IMDB | SNLI |
|---|---|---|---|---|---|---|---|---|---|---|
| CBOW | structured_deps | 100 | 88.89 | 77.94 | 76.58 | 71.69 | 88.04 | 73.31 | 78.01 | 64.34 |
| CBOW | structured_deps | 25 | 87.68 | 72.1 | 71.28 | 68.41 | 82.88 | 68.29 | 67.74 | 54.46 |
| CBOW | structured_deps | 250 | 88.51 | 78.05 | 76.09 | 72.23 | 88.52 | 73.91 | 81.24 | 67.79 |
| CBOW | structured_deps | 50 | 88.61 | 74.58 | 74.95 | 69.99 | 85.24 | 70.93 | 74.22 | 60.44 |
| CBOW | structured_deps | 500 | 88.34 | 78.03 | 75 | 73.92 | 87.36 | 73.88 | 82.11 | 69.33 |
| GloVe | structured_deps | 100 | 87.65 | 74.86 | 72.17 | 71.47 | 86.36 | 73.59 | 79.99 | 64.02 |
| GloVe | structured_deps | 25 | 84.75 | 66.43 | 64.71 | 68.39 | 81.96 | 66.54 | 70.72 | 53.52 |
| GloVe | structured_deps | 250 | 87.61 | 75.77 | 72.1 | 72.95 | 87.24 | 74.15 | 83.03 | 67.48 |
| GloVe | structured_deps | 50 | 86.76 | 72.31 | 69.58 | 69.26 | 84 | 69.8 | 74.97 | 59.55 |
| GloVe | structured_deps | 500 | 86.98 | 75.76 | 70.05 | 72.83 | 87.76 | 74.8 | 76.08 | 68.82 |
| SkipGram | structured_deps | 100 | 88.9 | 78.35 | 76.61 | 71.39 | 88.26 | 75.51 | 78.39 | 64.66 |
| SkipGram | structured_deps | 25 | 87.81 | 72.37 | 69.85 | 69.76 | 82.96 | 69.97 | 68.07 | 53.87 |
| SkipGram | structured_deps | 250 | 88.57 | 78.88 | 75.94 | 73.76 | 89 | 75.35 | 81.07 | 68.21 |
| SkipGram | structured_deps | 50 | 88.71 | 74.05 | 74.05 | 71.14 | 87.24 | 72.81 | 74.62 | 59.64 |
| SkipGram | structured_deps | 500 | 88.44 | 78.82 | 75.86 | 74.2 | 89.56 | 75.42 | 81.71 | 69.87 |
| CBOW | structured_linear | 100 | 88.52 | 78.6 | 75.48 | 71.46 | 88.2 | 73.41 | 77.54 | 63.79 |
| CBOW | structured_linear | 25 | 86.31 | 71.25 | 71.64 | 66.69 | 84.44 | 68.64 | 63.66 | 51.28 |
| CBOW | structured_linear | 250 | 88.63 | 79.2 | 75.25 | 73.88 | 88.76 | 74.86 | 82.15 | 67.7 |
| CBOW | structured_linear | 50 | 87.96 | 75.68 | 75.38 | 71.14 | 86.92 | 71.99 | 72.35 | 58.6 |
| CBOW | structured_linear | 500 | 88.12 | 78.29 | 75.24 | 74.48 | 88.96 | 75.12 | 82.66 | 69.19 |
| GloVe | structured_linear | 100 | 88.09 | 77.2 | 75.1 | 72.12 | 87.52 | 72.81 | 77.77 | 63.99 |
| GloVe | structured_linear | 25 | 85.55 | 69.85 | 68.31 | 68.28 | 84.52 | 68.02 | 63.62 | 54.12 |
| GloVe | structured_linear | 250 | 88.16 | 78.09 | 74.69 | 72.32 | 88.44 | 74.09 | 81.38 | 67.66 |
| GloVe | structured_linear | 50 | 87.44 | 74.45 | 74.1 | 69.67 | 86.2 | 71.41 | 71.56 | 59.33 |
| GloVe | structured_linear | 500 | 87.67 | 77.45 | 74.12 | 72.57 | 89.28 | 75.59 | 82.68 | 69.11 |
| SkipGram | structured_linear | 100 | 88.56 | 78.49 | 76.45 | 73.84 | 88.84 | 74.56 | 78.75 | 64.28 |
| SkipGram | structured_linear | 25 | 86.31 | 71.29 | 72.61 | 67.34 | 85.2 | 70 | 66.36 | 53.65 |
| SkipGram | structured_linear | 250 | 88.58 | 79.15 | 75.79 | 74.7 | 89.32 | 75.82 | 81.67 | 68.37 |
| SkipGram | structured_linear | 50 | 87.98 | 74.96 | 75.7 | 71.38 | 86.68 | 74.18 | 73.83 | 60.19 |
| SkipGram | structured_linear | 500 | 88.31 | 79.14 | 75.54 | 75.6 | 90.2 | 76.91 | 82.95 | 69.42 |
| CBOW | word_deps | 100 | 87.56 | 75.8 | 73.59 | 70.41 | 88.56 | 73.24 | 75.62 | 63.4 |
| CBOW | word_deps | 25 | 82.17 | 63.36 | 68.82 | 66.08 | 85.16 | 70.33 | 61.05 | 53.03 |
| CBOW | word_deps | 250 | 88.06 | 77.84 | 76.2 | 72.62 | 89.6 | 74.15 | 79.61 | 67.56 |
| CBOW | word_deps | 50 | 86.4 | 71.55 | 73.32 | 69.93 | 86.52 | 72.73 | 69.72 | 58.26 |
| CBOW | word_deps | 500 | 87.71 | 77.56 | 73.58 | 73.53 | 89.7 | 75.46 | 81.25 | 69.14 |
| GloVe | word_deps | 100 | 83.1 | 62.79 | 68.74 | 70.43 | 88 | 72.69 | 78.14 | 64.51 |
| GloVe | word_deps | 25 | 75.96 | 51.3 | 63.37 | 66.95 | 85.96 | 68.28 | 69.49 | 55 |
| GloVe | word_deps | 250 | 84.34 | 66.86 | 69.2 | 70.47 | 88.16 | 74.31 | 81.13 | 67.3 |
| GloVe | word_deps | 50 | 80.75 | 58.08 | 66.19 | 68.82 | 86.92 | 70.49 | 73.01 | 59.42 |
| GloVe | word_deps | 500 | 84.67 | 68.36 | 68.76 | 72.02 | 88.8 | 74.69 | 82.57 | 69.28 |
| SkipGram | word_deps | 100 | 87.62 | 75.29 | 75.63 | 72.28 | 89.2 | 74.5 | 76.95 | 64 |
| SkipGram | word_deps | 25 | 82.37 | 62.58 | 69.19 | 66.52 | 85.68 | 70.66 | 68.27 | 54.24 |
| SkipGram | word_deps | 250 | 88.17 | 77.23 | 76.59 | 73.8 | 89.84 | 75.66 | 81.38 | 68.09 |
| SkipGram | word_deps | 50 | 86.44 | 69.06 | 73.48 | 71.2 | 87.48 | 73.85 | 71.41 | 59.56 |
| SkipGram | word_deps | 500 | 87.77 | 78.05 | 75.38 | 73.96 | 89.8 | 77.15 | 82.78 | 69.93 |
| CBOW | word_linear | 100 | 87.8 | 76.51 | 75.65 | 71.51 | 88.56 | 74.95 | 76.7 | 63.81 |
| CBOW | word_linear | 25 | 81.91 | 66.98 | 70.14 | 65.61 | 85.76 | 70.44 | 60.2 | 49.71 |
| CBOW | word_linear | 250 | 88.28 | 78.52 | 75.33 | 74.64 | 89.76 | 75.09 | 81.46 | 67.66 |
| CBOW | word_linear | 50 | 86.37 | 71.86 | 73.73 | 69.51 | 87.68 | 72.84 | 69.06 | 59.13 |
| CBOW | word_linear | 500 | 87.66 | 77.53 | 75.21 | 74.78 | 89.8 | 75.9 | 82.22 | 69.29 |
| GloVe | word_linear | 100 | 82.15 | 61.18 | 69.16 | 68.03 | 88.36 | 72.6 | 78.62 | 63.93 |
| GloVe | word_linear | 25 | 76.25 | 51.1 | 64.24 | 64.9 | 85.96 | 66.97 | 68.67 | 54.07 |
| GloVe | word_linear | 250 | 83.72 | 64.64 | 69.5 | 70.8 | 89.2 | 73.46 | 81.15 | 67.79 |
| GloVe | word_linear | 50 | 80.22 | 57.28 | 65.97 | 67.82 | 87.6 | 70.84 | 73.25 | 59.87 |
| GloVe | word_linear | 500 | 83.8 | 66.1 | 69.62 | 71.05 | 89.16 | 74.6 | 82.24 | 69.51 |
| SkipGram | word_linear | 100 | 87.91 | 76.46 | 76.75 | 72.83 | 89.12 | 73.79 | 77.27 | 64.17 |
| SkipGram | word_linear | 25 | 81.64 | 63.76 | 70.37 | 65.42 | 85.36 | 71.06 | 67.32 | 52.72 |
| SkipGram | word_linear | 250 | 88.34 | 78.06 | 75.3 | 74.77 | 89.68 | 76.05 | 81.12 | 68.24 |
| SkipGram | word_linear | 50 | 86.41 | 71.64 | 74.62 | 70.05 | 87.32 | 73.61 | 71.58 | 59.46 |
| SkipGram | word_linear | 500 | 87.86 | 78.23 | 75.72 | 74.8 | 89.92 | 76.86 | 82.73 | 69.74 |