{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VrNUi8VIqMxb"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dbamman/anlp25/blob/main/2.compare/ChiSquare_Mann_Whitney.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing prevalence of terms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7tZzp1UpqMxc"
   },
   "source": [
    "This notebook examines the words that distinguish the [2024 Democrat party platform](https://www.presidency.ucsb.edu/documents/2024-democratic-party-platform) from the [2024 Republican party platform](https://www.presidency.ucsb.edu/documents/2024-republican-party-platform) (both sourced from the American Presidency Project at UCSB), using the Chi-Square test and the Mann-Whitney test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KSP3JvxoqMxd"
   },
   "outputs": [],
   "source": [
    "!wget --no-check-certificate https://raw.githubusercontent.com/dbamman/anlp25/main/data/2024_democrat_party_platform.txt\n",
    "!wget --no-check-certificate https://raw.githubusercontent.com/dbamman/anlp25/main/data/2024_republican_party_platform.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BC1YlQrCqMxd"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import math\n",
    "import operator\n",
    "import sys\n",
    "from collections import Counter\n",
    "\n",
    "import nltk\n",
    "import numpy as np\n",
    "from scipy.stats import mannwhitneyu\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "nltk.download(\"punkt\")\n",
    "nltk.download(\"punkt_tab\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pvvLkCXeqMxe"
   },
   "outputs": [],
   "source": [
    "def read(filename):\n",
    "    with open(filename, encoding=\"utf-8\") as file:\n",
    "        # lowercase text\n",
    "        return file.read().lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jqJNDe47qMxe"
   },
   "outputs": [],
   "source": [
    "democrat_text = read(\"2024_democrat_party_platform.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4zkSWZ9qqMxe"
   },
   "outputs": [],
   "source": [
    "republican_text = read(\"2024_republican_party_platform.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "USCfgZluqMxf"
   },
   "source": [
    "Explore your assumptions between the words you think will most distinguish the Democrat and Republican platforms.  Before looking at the results of the tests, what words do you think will be comparatively distinct to both?  (If you're not familiar with either, scan the platforms linked above)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KXCKW8wiqMxf"
   },
   "outputs": [],
   "source": [
    "def tokenize(data):\n",
    "    return nltk.word_tokenize(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-oO8SKYnqMxf"
   },
   "outputs": [],
   "source": [
    "def get_counts(tokens):\n",
    "    counts = Counter()\n",
    "    for token in tokens:\n",
    "        counts[token] += 1\n",
    "    return counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_rHDsT60qMxf"
   },
   "source": [
    "## $\\chi^2$ test\n",
    "\n",
    "The $\\chi^2$ test as used in the comparison of different texts is designed to measure how statistically significant the distribution of counts in a 2x2 contingency table is.  Use the following function to analyze the difference between the platforms.  How do the most distinct terms comport with your assumptions?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_contingency_table(word, word_counts_a, word_counts_b, total_count_a = None, total_count_b = None):\n",
    "    \"\"\"\n",
    "    Construct a 2x2 contingency table that takes the form of:\n",
    "\n",
    "      # word in A | # word in B\n",
    "    --------------+--------------\n",
    "     # total in A | # total in B\n",
    "    \"\"\"\n",
    "    # we can take the total counts as input if they are precomputed\n",
    "    # otherwise, we can also compute them easily by summing the word counts\n",
    "    if total_count_a is None:\n",
    "        total_count_a = sum(word_counts_a.values())\n",
    "    if total_count_b is None:\n",
    "        total_count_b = sum(word_counts_b.values())\n",
    "\n",
    "    return np.array([\n",
    "        [word_counts_a[word], word_counts_b[word]],\n",
    "        [total_count_a - word_counts_a[word], total_count_b - word_counts_b[word]]\n",
    "    ])\n",
    "\n",
    "def chi_sq(o):\n",
    "    \"\"\"Calculate chi-square value for 2x2 contingency table o\n",
    "    \n",
    "    We use the simpler form given in Manning and Schuetze (1999)\n",
    "    for 2x2 contingency tables:\n",
    "    https://nlp.stanford.edu/fsnlp/promo/colloc.pdf, equation 5.7\n",
    "    \"\"\"\n",
    "\n",
    "    N = o.sum()\n",
    "    return (N * (o[0,0] * o[1,1] - o[0,1] * o[1,0]) ** 2) / ((o[0,0] + o[0,1]) * (o[0,0] + o[1,0]) * (o[0,1] + o[1,1]) * (o[1,0] + o[1,1]))\n",
    "    \n",
    "def run_chi_square_on_corpus(word_counts_a, word_counts_b): \n",
    "    total_count_a = 0.0\n",
    "    total_count_b = 0.0\n",
    "    vocab = {}\n",
    "    for word in word_counts_a:\n",
    "        vocab[word] = 1\n",
    "        total_count_a += word_counts_a[word]\n",
    "    for word in word_counts_b:\n",
    "        vocab[word] = 1\n",
    "        total_count_b += word_counts_b[word]\n",
    "\n",
    "    total_words = total_count_a + total_count_b\n",
    "\n",
    "    chisq_vals = {}\n",
    "    for word in vocab:\n",
    "        contingency_table = get_contingency_table(\n",
    "            word,\n",
    "            word_counts_a,\n",
    "            word_counts_b,\n",
    "            total_count_a,\n",
    "            total_count_b\n",
    "        )\n",
    "\n",
    "        chisq_vals[word] = chi_sq(contingency_table)\n",
    "\n",
    "    sorted_chi = sorted(chisq_vals.items(), key=lambda x: x[1], reverse=True)\n",
    "    corpus_a_words = []\n",
    "    corpus_b_words = []\n",
    "\n",
    "    for word, chisq_val in sorted_chi:\n",
    "        if word_counts_a[word] / total_count_a > word_counts_b[word] / total_count_b:\n",
    "            corpus_a_words.append(word)\n",
    "        else:\n",
    "            corpus_b_words.append(word)\n",
    "\n",
    "    print(\"Democrat:\\n\")\n",
    "    for word in corpus_a_words[:20]:\n",
    "        print(f\"{word}\\t{chisq_vals[word]}\")\n",
    "\n",
    "    print(\"Republican:\\n\")\n",
    "    for word in corpus_b_words[:20]:\n",
    "        print(f\"{word}\\t{chisq_vals[word]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iDNMZZiKqMxg"
   },
   "outputs": [],
   "source": [
    "democrat_tokens = tokenize(democrat_text)\n",
    "democrat_counts = get_counts(democrat_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Nk5I4JoKqMxg"
   },
   "outputs": [],
   "source": [
    "republican_tokens = tokenize(republican_text)\n",
    "republican_counts = get_counts(republican_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before running chi-square on all the words, let's step through the calculation on a single word first. We begin by plotting the 2x2 contingency table of the occurrences of each word in each corpus. We normalize to get probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def plot_contingency_table(table, normalize=False):\n",
    "    if normalize:\n",
    "        table = table / table.sum(axis=0)\n",
    "\n",
    "    \n",
    "    fig, ax = plt.subplots()\n",
    "    im = ax.imshow(table, aspect='auto')\n",
    "        \n",
    "    ax.set_xticks(range(table.shape[1]))\n",
    "    ax.set_yticks(range(table.shape[0]))\n",
    "    \n",
    "    for i in range(table.shape[0]):\n",
    "        for j in range(table.shape[1]):\n",
    "            value = table.iloc[i, j] if hasattr(table, 'iloc') else table[i, j]\n",
    "            text_color = 'white' if value < table.max() * 0.5 else 'black'\n",
    "            \n",
    "            if normalize:\n",
    "                text = f'{value:.3f}'\n",
    "            else:\n",
    "                text = f'{int(value)}'\n",
    "            \n",
    "            ax.text(j, i, text, ha='center', va='center', \n",
    "                   color=text_color, fontsize=12, fontweight='bold')\n",
    "\n",
    "    title = \"Contingency Table\"\n",
    "    x_label = \"Corpus\"\n",
    "    y_label = \"Present\"\n",
    "    ax.set_title(title, fontsize=16, fontweight='bold', pad=20)\n",
    "    ax.set_xlabel(x_label, fontsize=14, fontweight='bold')\n",
    "    ax.set_ylabel(y_label, fontsize=14, fontweight='bold')\n",
    "    ax.set_xticklabels([\"Democrat\", \"Republican\"])\n",
    "    ax.set_yticklabels([\"Yes\", \"No\"])\n",
    "    # Add grid\n",
    "    ax.set_xticks(np.arange(table.shape[1]+1)-.5, minor=True)\n",
    "    ax.set_yticks(np.arange(table.shape[0]+1)-.5, minor=True)\n",
    "    ax.grid(which='minor', color='black', linestyle='-', linewidth=1)\n",
    "    \n",
    "    plt.tight_layout()\n",
    "    return ax\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "table = get_contingency_table(\"biden\", democrat_counts, republican_counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_contingency_table(table, normalize=True);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the Democrat platform mentions \"biden\" more, but is this statistically significant? We can use the chi-square test to test against the null hypothesis that the distribution of \"biden\" in the two platforms are equivalent.\n",
    "\n",
    "For two categories, we can consult a [chi-square table](https://math.arizona.edu/~jwatkins/chi-square-table.pdf) to see that a value of $>7.879$ yields a statistically significant result at $\\alpha = 0.005$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chi_sq(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's run the chi-square test on all the words in the vocabulary to see which ones are more prevalent in which platform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BqyUZIP0qMxg",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "run_chi_square_on_corpus(democrat_counts, republican_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XdvKpqB_qMxg"
   },
   "source": [
    "Are these results surprising? Examine specific words to check their frequency in both datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "X1gXHG__qMxg"
   },
   "outputs": [],
   "source": [
    "print(\"Totals: R: %s, D: %s\" % (len(republican_tokens), len(democrat_tokens)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kTz_NjjmqMxg"
   },
   "outputs": [],
   "source": [
    "word = \"climate\"\n",
    "print(\"%s -- R: %s, D: %s\" % (word, republican_counts[word], democrat_counts[word]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0t9Rx-7eqMxh"
   },
   "source": [
    "## Mann-Whitney\n",
    "\n",
    "We saw earlier that $\\chi^2$ is not a perfect estimator since doesn't account for the \"burstiness\" of language -- if we see the word \"Dracula\" in a text, we're probably going to see it again in that same text. The occurrence of words are not independent random events; they are tightly coupled with each other. If we're trying to understanding the robust differences between two corpora, we might prefer to prioritize words that show up more frequently everywhere in corpus A (but not in corpus B) over those that show up only very frequently within narrow slice of A (such as one text in a genre, one chapter in a book, or one speaker when measuring the differences between policital parties). Use the following function to execute the Mann-Whitney test to account for this phenomenon while finding distinctive terms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_differences(tokens_a, tokens_b):\n",
    "    \"\"\"Measure the difference in the frequency of each word between two corpora.\"\"\"\n",
    "    total_len_a = len(tokens_a)\n",
    "    total_len_b = len(tokens_b)\n",
    "\n",
    "    word_counts_a = Counter(tokens_a)\n",
    "    word_counts_b = Counter(tokens_b)\n",
    "\n",
    "    vocab = set(word_counts_a) | set(word_counts_b)\n",
    "\n",
    "    differences = {}\n",
    "    for word in vocab:\n",
    "        freq_a = word_counts_a[word] / total_len_a\n",
    "        freq_b = word_counts_b[word] / total_len_b\n",
    "        diff = freq_a - freq_b\n",
    "        differences[word] = diff\n",
    "\n",
    "    return differences\n",
    "\n",
    "\n",
    "def get_chunk_counts(tokens, chunk_size):\n",
    "    \"\"\"Get token counts for chunks of the corpus.\n",
    "\n",
    "    Returns a list of Counters, each with token counts for their respective chunk.\n",
    "    \"\"\"\n",
    "    chunks = []\n",
    "    for i in range(0, len(tokens), chunk_size):\n",
    "        counts = Counter()\n",
    "        for j in range(chunk_size):\n",
    "            if i + j < len(tokens):\n",
    "                counts[tokens[i + j]] += 1\n",
    "        chunks.append(counts)\n",
    "    return chunks\n",
    "\n",
    "\n",
    "def mann_whitney(tokens_a, tokens_b):\n",
    "    chunk_size = 500\n",
    "    chunks_a = get_chunk_counts(tokens_a, chunk_size)\n",
    "    chunks_b = get_chunk_counts(tokens_b, chunk_size)\n",
    "\n",
    "    pvals = {}\n",
    "    vocab = set(tokens_a + tokens_b)\n",
    "    for word in vocab:\n",
    "        a = []\n",
    "        b = []\n",
    "\n",
    "        # Note a and b can be different lengths (i.e., different sample sizes)\n",
    "        #\n",
    "        # See Mann and Whitney (1947), \"On a Test of Whether one of Two Random\n",
    "        # Variables is Stochastically Larger than the Other\"\n",
    "        # https://projecteuclid.org/download/pdf_1/euclid.aoms/1177730491\n",
    "\n",
    "        # (This is part of their innovation over the case of equal sample sizes in Wilcoxon 1945)\n",
    "\n",
    "        for chunk in chunks_a:\n",
    "            a.append(chunk[word])\n",
    "        for chunk in chunks_b:\n",
    "            b.append(chunk[word])\n",
    "\n",
    "        # Consider: what information do `a` and `b` encode?\n",
    "\n",
    "        # we use the scipy implementation of the Mann-Whitney U rank test\n",
    "        # see: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html\n",
    "        statistic, pval = mannwhitneyu(a, b, alternative=\"two-sided\")\n",
    "\n",
    "        # We'll use the p-value as our quantity of interest.  [Note in the normal appproximation\n",
    "        # that Mann-Whitney uses to assess significance for large sample sizes, the significance\n",
    "        # of the raw statistic depends on the number of ties in the data, so the statistic itself\n",
    "        # isn't exactly comparable across different words]\n",
    "        pvals[word] = pval\n",
    "\n",
    "    return pvals\n",
    "\n",
    "\n",
    "def mann_whitney_analysis(tokens_a, tokens_b):\n",
    "\n",
    "    pvals = mann_whitney(tokens_a, tokens_b)\n",
    "\n",
    "    # Mann-Whitney tells us the significance of a term's difference in two groups, but we also\n",
    "    # need the directionality of that difference (whether it's used more by group A or group B.\n",
    "\n",
    "    # Let's use our difference-in-proportions function above to check the directionality.\n",
    "    # [Note we could also measure directionality by checking whether the Mann-Whitney statistic\n",
    "    # is greater or less than the mean=len(one_chunks)*len(two_chunks)*0.5.]\n",
    "\n",
    "    differences = count_differences(tokens_a, tokens_b)\n",
    "\n",
    "    terms_a = {k : pvals[k] for k in pvals if differences[k] <= 0}\n",
    "    terms_b = {k : pvals[k] for k in pvals if differences[k] > 0}\n",
    "\n",
    "    sorted_pvals = sorted(terms_a.items(), key=lambda x: x[1])\n",
    "    print(\"More Republican:\\n\")\n",
    "    for k,v in sorted_pvals[:20]:\n",
    "        print(\"%s\\t%.15f\" % (k,v))\n",
    "\n",
    "    print(\"\\nMore Democrat:\\n\")\n",
    "    sorted_pvals = sorted(terms_b.items(), key=operator.itemgetter(1))\n",
    "    for k,v in sorted_pvals[:25]:\n",
    "        print(\"%s\\t%.15f\" % (k,v))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UxFs1GI8qMxh"
   },
   "outputs": [],
   "source": [
    "mann_whitney_analysis(democrat_tokens, republican_tokens)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}