The MALLET topic model toolkit produces a number of useful diagnostic measures. This document explains the definition, motivation, and interpretation of these values. To generate an XML diagnostics file, use the --diagnostics-file option when training a model.

bin/mallet train-topics --input text.sequences --num-topics 50 \
 --optimize-interval 20 --optimize-burn-in 50 \
 --output-state state.gz --diagnostics-file diagnostics.xml

If you already have a saved state or serialized model, you can use this to generate a diagnostics file.

bin/mallet train-topics --input text.sequences --num-topics 50 \
 --input-state state.gz --no-inference
 --diagnostics-file diagnostics.xml

Some diagnostics are meaningful at the topic level, others are meaningful at the level of individual words within topics. The file is organized with one tag per topic, which contains tags for the top-ranked words in the topic. The number of top-ranked words displayed is controlled by the --num-top-words parameter.

These examples will refer to a sample diagnostics file from a model trained on Wikipedia articles and an interactive interface that displays data from that file.

The following metrics are defined at the level of topics.

[Notation] These descriptions are designed to make sense without mathematical definitions, but we include them for specificity. We'll use the symbols \(w\) for a word, \(k\) for a topic, and \(d\) for a document. Several metrics count tokens. For these we define variables \(N_{d,w,k}\) as the number of times word \(w\) occurs in document \(d\) assigned to topic \(k\), \(N_{d,k}\) as \(\sum_w N_{d,w,k}\). Other metrics involve document frequencies. \(D(w)\) is the number of documents that contain at least one token of type \(w\), and \(D(w_1,w_2)\) is the number of documents that contain at least one \(w_1\) and one \(w_2\). Finally, we define \(\mathcal{W}_k\) to be the \(N\) most probable words in topic \(k\). \(\mathbb{1}(p)\) is equal to 1 if predicate \(p\) is true, and 0 otherwise.

tokens. This metric measures the number of word tokens currently assigned to the topic \(N_k = \sum_d N_{d,k}\). Comparing this number to the sum of the token counts for all topics will give you the proportion of the corpus assigned to the topic. If you are using optimized hyperparameters this number will vary considerably across topics; otherwise it will be roughly the same for all topics. We usually find topics most interesting if they are not on the extremes of this range. Small numbers (relative to other topics) are often a sign that a topic may not be reliable: we don't have enough observations to get a good sense of the topic's word distribution. Large numbers may also be bad: extremely frequent topics are often "not-quite-stopwords". For example, in the sample model the largest topic is "time number term part system form".

document entropy. We usually think of the probability of a topic given a document. For this metric we calculate the probability of documents given a topic. \[P(d | k) = \frac{N_{d,k}}{\sum_{d'} N_{d',k}} \] We count the frequency of a topic over all documents, and normalize to get a distribution, then calculate the entropy of that distribution \(H(P(d|k))\). A topic with low entropy (ie high predictability) will be concentrated in a few documents, while a topic with higher entropy will be spread evenly over many documents. In the example topic model, the topics "list links history external information site" and "game team league player players football games" have roughly the same number of tokens, but the "list" topic has much higher entropy. It occurs to a small degree in many documents, while the "game" topic occurs a lot in a smaller number of documents. Low entropy isn't necessarily good: it can indicate unusual documents (did you accidentally import the Mac .DS_Store file?) or the presence of documents in other languages.

word length. The average length, in characters, of the displayed top words. \[\frac{1}{|\mathcal{W}|} \sum_{w \in \mathcal{W}} len(w) \] Words are not weighted by probablity or rank position. This is a useful proxy for topic specificity. Longer words often carry more specific meaning, so if the topic brings together lots of short words, it's probably not a very specific topic. This can also be a good way to pick up the "hey, looks like we've got some Spanish!" topic because we often pick up the short words that would be stopwords if we were modeling a corpus in that language. The example topic "de french la france spanish spain italian paris el le" has a number of this type of short words, but they seem to be due to Wikipedia articles talking about French, Spanish, and Italian things rather than actual text in those languages.

coherence. This metric measures whether the words in a topic tend to co-occur together. We add up a score for each distinct pair of top ranked words. The score is the log of the probability that a document containing at least one instance of the higher-ranked word also contains at least one instance of the lower-ranked word. \[ \sum_i \sum_{j < i} \log\frac{D(w_j,w_i) + \beta}{D(w_i)} \] To avoid log zero errors we add the "beta" topic-word smoothing parameter specified when you calculate diagnostics. Since these scores are log probabilities they are negative. Large negative values indicate words that don't co-occur often; values closer to zero indicate that words tend to co-occur more often. The least coherent topic in the sample file is "polish poland danish denmark sweden swedish na norway norwegian sk red iceland bj baltic copenhagen cave greenland krak gda faroese". This topic seems to be about Northern and Eastern European countries, but the short abbreviations "na" and "sk" and the words "red" and "cave" don't really fit.

uniform_dist. We want topics to be specific. This metric measures the distance from a topic's distribution over words to a uniform distribution. We calculate distance using Kullback-Leibler divergence. \[ \sum_w P(w|k) \log \frac{P(w|k)}{ \frac{1}{|\mathcal{V}|} } \] Larger values indicate more specificity.

corpus_dist. This metric measures how far a topic is from the overall distribution of words in the corpus — essentially what you would get if you "trained" a model with one topic. We calculate distance using Kullback-Leibler divergence. A greater distance means the topic is more distinct; a smaller distanace means that the topic is more similar to the corpus distribution. Not surprisingly, it correlates with number of tokens since a topic that makes up a large proportion of the tokens in the corpus is likely to be more similar to the overall corpus distribution. The closest topic to the corpus distribution is "time number term part system form".

effective number of words. This metric is equivalent to the effective number of parties metric in Political Science. For each word we calculate the inverse of the squared probability of the word in the topic, and then add those numbers up. Larger numbers indicate more specificity. It is similar to distance from the uniform distribution, but produces a value that may be more interpretable.

token/document discrepancy. This metric looks for "bursty" words within topics. We compare two distributions over words using Jensen-Shannon distance. The first distribution \(P(w | k) \propto N_{k,w} \) is the usual topic-word distribution proportional to the number of tokens of type \(w\) that are assigned to the topic. The second distribution \(Q(w | k) \propto \sum_d \mathbb{1}(N_{d,w,k} > 0)\) is proportional to the number of documents that contain at least one token of type \(w\) that is assigned to the topic. A words that occurs many times in only a few documents may appear prominently in the sorted list of words, but may not be a good representative word for the topic. This metric compares the number of times a word occurs in a topic (measured in tokens) and the number of documents the word occurs in as that topic (instances of the word assigned to other topics are not counted). The highest ranked topic in this metric is the "polish poland danish denmark sweden swedish na norway norwegian sk red" topic, suggesting that those ill-fitting words may be isolated in a few documents. Although this metric has the same goal as coherence, the two don't appear to correlate well: bursty words aren't necessarily unrelated to the topic, they're just unusually frequent in certain contexts.

rank 1 documents. Some topics are specific, while others aren't really "topics" but language that comes up because we are writing in a certain context. Academic writing will talk about "paper abstract data", and a Wikipedia article will talk about "list links history". The difference is often measurable in terms of burstiness. A content-ful topic will occur in relatively few documents, but when it does, it will produce a lot of tokens. A "background" topic will occur in many documents and have a high overall token count, but never produce many tokens in any single document. This metric counts the frequency at which a given topic is the single most frequent topic in a document. Specific topics like "music album band song" or "cell cells disease dna blood treatment" are the "rank 1" topic in many documents. High token-count topics often have few rank-1 documents. This metric is often useful as a way to identify corpus-specific stopwords. But rarer topics can also have few rank-1 documents: "day year calendar days years month" is a representative example — topics for days of the week and units of measurement often appear in documents as a distinct discourse, but they are rarely the focus of a document.

allocation count. This metric has a similar motivation to the rank-1-docs metric. For each document we can calculate the percentage of that document assigned to a given topic. We can then observe how often that percentage is above a certain threshold, by default 30%.

allocation ratio. This metric reports the ratio of allocation counts at two different thresholds, by default 50% and 2%.

exclusivity. This metric measures the extent to which the top words for this topic are do not appear as top words in other topics -- i.e., the extent to which its top words are 'exclusive.' The value is the average, over each top word, of the probability of that word in the topic divided by the sum of the probabilities of that word in all topics. Of the top words in the topic, how often to they occur in other topics? Exclusivity correlates (negatively) with token count, but also indicates vaguer, more general topics. "black hand back body cross man" is about the same size as "isbn book press published work books", but is much less exclusive.

For those metrics that are decomposable into word-specific terms we report the values for each individual word.