What's in a Word—Breaking Texts and Making Data

Before text can be analyzed computationally, it has to be taken apart — split into pieces, normalized, stripped of noise. This session introduces the preprocessing behind nearly all text analysis: tokenization, lemmatization, stemming, and stop word removal. We'll also distinguish this sense of "token" from how the word is used in large language models, a difference that reveals a lot about both. You'll leave understanding not just how to prepare text, but what each choice quietly decides.