{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n53a5KC4BnKz"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dbamman/anlp25/blob/main/1.words/EvaluateTokenizationForSentiment.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The impact of tokenization on downstream tasks\n",
    "\n",
    "Tokenization can have a big impact on downstream model performance. Here, we look at different methods for tokenization and stemming/lemmatization and evaluate how they affect the performance on a simple binary sentiment classification task.\n",
    "\n",
    "We use a train/dev deataset of 1000 reviews from the [Large Movie Review Dataset](http://ai.stanford.edu/~amaas/data/sentiment/).\n",
    "\n",
    "Each tokenization method is evaluated on the same learning algorithm ($l_2$-regularized logistic regression); the only difference is the tokenization process.\n",
    "\n",
    "For more, see: http://sentiment.christopherpotts.net/tokenizing.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jj0huAdwBs0M"
   },
   "outputs": [],
   "source": [
    "# download code and data\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/main/1.words/happyfuntokenizing.py\n",
    "\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/main/data/sentiment.1000.train.txt\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/main/data/sentiment.1000.dev.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make sure dependencies are installed\n",
    "!pip install nltk\n",
    "!pip install spacy\n",
    "!pip install scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pOu0wKUqBnK6"
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download('punkt')\n",
    "nltk.download('punkt_tab')\n",
    "\n",
    "import spacy\n",
    "from nltk.stem.porter import *\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn import linear_model\n",
    "\n",
    "\n",
    "from happyfuntokenizing import Tokenizer as potts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up evaluation\n",
    "We'll set up a class that we can use to test different tokenization methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TokenizationTest():\n",
    "    \n",
    "    def __init__(self, train_file, dev_file):\n",
    "        self.train_file = train_file\n",
    "        self.dev_file = dev_file\n",
    "        self.count_vectorizer = CountVectorizer(\n",
    "            max_features=10_000,\n",
    "            analyzer=lambda x: x,\n",
    "            lowercase=False,\n",
    "            strip_accents=None,\n",
    "            binary=True\n",
    "        )\n",
    "        self.label_encoder = LabelEncoder()\n",
    "    \n",
    "    def read_data(self, filename, tokenizer):\n",
    "        tokenized_text = []\n",
    "        labels = []\n",
    "    \n",
    "        with open(filename, encoding=\"utf-8\") as file:\n",
    "            for idx, line in enumerate(file):\n",
    "                cols = line.rstrip().split(\"\\t\")\n",
    "                label = cols[0]\n",
    "                text = cols[1]\n",
    "                tokens = list(tokenizer(text))\n",
    "                tokenized_text.append(tokens)\n",
    "                labels.append(label)\n",
    "        return tokenized_text, labels\n",
    "    \n",
    "    def evaluate(self, tokenizer):\n",
    "        train_tokens, train_labels = self.read_data(self.train_file, tokenizer)\n",
    "        dev_tokens, dev_labels = self.read_data(self.dev_file, tokenizer)\n",
    "    \n",
    "        X_train = self.count_vectorizer.fit_transform(train_tokens)\n",
    "        X_dev = self.count_vectorizer.transform(dev_tokens)\n",
    "    \n",
    "        self.label_encoder.fit(train_labels)\n",
    "        Y_train = self.label_encoder.transform(train_labels)\n",
    "        Y_dev = self.label_encoder.transform(dev_labels)\n",
    "    \n",
    "        model = linear_model.LogisticRegression(C=1.0, solver='lbfgs', penalty='l2')\n",
    "        model.fit(X_train, Y_train)\n",
    "        print(\"Function '%s' Accuracy: %.3f\" % (tokenizer.__name__, model.score(X_dev, Y_dev)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up tokenizers\n",
    "\n",
    "Now let's set up our tokenizers. Each tokenizer should take as input a string and output a list of strings. We'll try six different tokenization methods.\n",
    "\n",
    "1. Splitting on whitespace with `str.split()`\n",
    "2. Splitting on whitespace, then stemming with the [Porter stemmer](https://tartarus.org/martin/PorterStemmer/)\n",
    "3. Using [`nltk.word_tokenize`](https://www.nltk.org/api/nltk.tokenize.word_tokenize.html)\n",
    "4. Using the [`spacy` tokenizer](https://spacy.io/usage/linguistic-features#how-tokenizer-works)\n",
    "5. Using the [`spacy` tokenizer](https://spacy.io/usage/linguistic-features#how-tokenizer-works) with [lemmatization](https://spacy.io/api/lemmatizer)\n",
    "6. Using the [Potts tokenizer](http://sentiment.christopherpotts.net/tokenizing.html) (implemented for you in `happyfuntokenization.py`)\n",
    "\n",
    "Note: evaluating the spacy tokenizers might take ~1 minute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load NLTK porter stemmer\n",
    "stemmer = PorterStemmer()\n",
    "def tokenize_with_porter(data):\n",
    "    return [\n",
    "        stemmer.stem(word) for word in str.split(data)\n",
    "    ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2xxgxawHBnK7"
   },
   "outputs": [],
   "source": [
    "# spaCy lemmatization needs tagger but disable the rest\n",
    "nlp = spacy.load('en_core_web_sm', disable=['ner,parser'])\n",
    "nlp.remove_pipe('ner')\n",
    "nlp.remove_pipe('parser')\n",
    "\n",
    "def tokenize_with_spacy(data):\n",
    "    spacy_tokens = nlp(data)\n",
    "    return [token.text for token in spacy_tokens]\n",
    "\n",
    "def tokenize_with_spacy_lemma(data):\n",
    "    spacy_tokens = nlp(data)\n",
    "    return [token.lemma_ for token in spacy_tokens]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load Potts sentiment tokenizer\n",
    "potts_tokenizer = potts()\n",
    "def tokenize_with_potts(data):\n",
    "    return list(potts_tokenizer.tokenize(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing the tokenizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Iw5uOlNXBnK8"
   },
   "outputs": [],
   "source": [
    "tester = TokenizationTest(\"sentiment.1000.train.txt\", \"sentiment.1000.dev.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tester.evaluate(str.split)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tester.evaluate(tokenize_with_porter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sbGK8OqoBnK9"
   },
   "outputs": [],
   "source": [
    "tester.evaluate(nltk.word_tokenize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_5QIVUcdBnK-"
   },
   "outputs": [],
   "source": [
    "tester.evaluate(tokenize_with_spacy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MjaRF32LBnK-"
   },
   "outputs": [],
   "source": [
    "tester.evaluate(tokenize_with_spacy_lemma)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "eGGoA0TiBnK-"
   },
   "outputs": [],
   "source": [
    "tester.evaluate(tokenize_with_potts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the output of some of these tokenizers. How do different tokenizers handle some of the issues we talked about in lecture (e.g., punctuation, emoticons, casing)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenize_with_potts(\"I love INFO 256!\")  # modify this to test different tokenizers / different strings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Potts tokenizer was designed with web text in mind, with special hand-crafted rules for emoticons, HTML tags, and hashtags. Can you approach the performance of the Potts tokenizer (>0.88) by combining some of the other methods we test?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_tokenizer(data: str) -> list[str]:\n",
    "    \"\"\"Tokenize the `data` string into a list of strings.\"\"\"\n",
    "    # your code here\n",
    "    return [\"implement\", \"me\"]\n",
    "\n",
    "tester.evaluate(my_tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}