Augmenting Latent Dirichlet Allocation and Rank Threshold Detection With Ontologies

<p>In an ever-increasing data rich environment actionable information must be extracted filtered and correlated from massive amounts of disparate often free text sources. The usefulness of the retrieved information depends on how we accomplish these steps and present the most relevant information to the analyst. One method for extracting information from free text is Latent Dirichlet Allocation (LDA) a document categorization technique to classify documents into cohesive topics. Although LDA accounts for some implicit relationships such as synonymy (same meaning) it often ignores other semantic relationships such as polysemy (different meanings) hyponym (subordinate) meronym (part of) and troponomys (manner). To compensate for this de ciency we incorporate explicit word ontologies such as WordNet into the LDA algorithm to account for various semantic relationships. Experiments over the 20 Newsgroups NIPS OHSUMED and IED document collections demonstrate that incorporating such knowledge improves perplexity measure over LDA alone for given parameters.</p><p>This work has been selected by scholars as being culturally important and is part of the knowledge base of civilization as we know it. This work was reproduced from the original artifact and remains as true to the original work as possible. Therefore you will see the original copyright references library stamps (as most of these works have been housed in our most important libraries around the world) and other notations in the work.</p><p>This work is in the public domain in the United States of America and possibly other nations. Within the United States you may freely copy and distribute this work as no entity (individual or corporate) has a copyright on the body of the work.</p><p>As a reproduction of a historical artifact this work may contain missing or blurred pages poor pictures errant marks etc. Scholars believe and we concur that this work is important enough to be preserved reproduced and made generally available to the public. We appreciate your support of the preservation process and thank you for being an important part of keeping this knowledge alive and relevant.</p><br>