Document clustering

Conversation

Mathematics

Latent Dirichlet allocation

Pooling

Mathematical analysis

Philosophy

Ground truth

Artificial intelligence

Topic model

Natural language processing

Information retrieval

Computer science

Linguistics

Cluster analysis

Scheme (mathematics)

Proceedings of the International AAAI Conference on Web and Social Media

We propose a new pooling technique for topic modeling in Twitter, which groups together tweets occurring in the same user-to-user conversation. Under this scheme, tweets and their replies are aggregated into a single document and the users who posted them are considered co-authors. To compare this new scheme against existing ones, we train topic models using Latent Dirichlet Allocation (LDA) and the Author-Topic Model (ATM) on datasets consisting of tweets pooled according to the different methods. Using the underlying categories of the tweets in this dataset as a noisy ground truth, we show that this new technique outperforms other pooling methods in terms of clustering quality and document retrieval.

Topic Modeling in Twitter: Aggregating Tweets by Conversations