It aims to learn the distributed representation for words reducing the high dimensional word representations in a large corpus. The CBOW Word2Vec model predicts the current word from a window of context words and the skip-gram model predicts the context words based on the current word. In the figure, the model consists of a vocabulary of V words, a context of C words, a dense representation of N-dimensional word vector, an embedding matrix W of dimensions VxN at the input and a context matrix W' of dimensions NxV at the output.

GloVe exploits the global word-word co-occurrence statistics in the corpus with the underlying intuition that the ratios of word-word co-occurrence probabilities encode some form of the meaning of the words. It is essentially a log-bilinear model with a weighted least-squares objective. The training objective of GloVe is to learn word vectors such that their dot product equals the logarithm of the words' probability of co-occurrence. Owing to the fact that the logarithm of a ratio equals the difference of logarithms, this objective associates (the logarithm of) ratios of co-occurrence probabilities with vector differences in the word vector space. These ratios encode some form of meaning of the words, hence, this information gets encoded as vector differences as well.

The model jointly learns word and entity embeddings from Wikipedia where similar words and entities are close to one another in the vector space. It comprises of three submodels namely Wikipedia Link Graph Model, Word-based skip-gram model, and Anchor context model.

• Wikipedia Link Graph Model comprises of an undirected link graph in which the nodes are the Wikipedia entities and there exists an edge between two nodes if the page of one entity links to the other or if both pages link each other. It learns entity embeddings by predicting neighboring entities in the graph.
• Word-based skip-gram model learns word embeddings by predicting neighboring words given each word in a text contained on a Wikipedia page.
• Anchor context model learns embeddings by predicting neighboring words given each entity and aims to place similar words and entities near one another in the vector space

Bidirectional Encoder Representations from Transformers is a contextual information based embedding approach in which pretraining on bidirectional representations from unlabeled text by using the left and the right context in all the layers is performed. BERT makes use of Transformer, an attention mechanism that learns contextual relations between words (or sub-words) in a text. In its vanilla form, Transformer includes two separate mechanisms — an encoder that reads the text input and a decoder that produces a prediction for the task. Since BERT’s goal is to generate a language model, only the encoder mechanism is necessary. The encoder reads the entire sequence of words at once. Therefore it is considered bidirectional, which allows the model to learn the context of a word based on all of its surroundings.

Character embedding represents the latent representations of characters trained over a corpus which helps in determining the vector representations of out-of-vocabulary words. Character level embedding uses one-dimensional convolutional neural network (1D-CNN) to find numeric representation of words by looking at their character-level compositions.