The Linear Algebraic Structure of Word Meanings

Word embeddings are often constructed with discriminative models such as deep nets and word2vec. Mikolov et al (2013) showed that these embeddings exhibit linear structure that is useful in solving “word analogy tasks“. Subsequently, Levy and Goldberg (2014) and Pennington et al (2014) tried to explain why such linear structure should arise in embeddings derived from nonlinear methods. We provide a new generative model “explanation“ for various word embedding methods as well as the above-mentioned linear