{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dbamman/anlp25/blob/main/11.nlp/ExploreWordNet.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vPfjknHEHufX"
   },
   "source": [
    "# Exploring WordNet\n",
    "\n",
    "This notebook explores WordNet synsets, presenting a simple method for finding in a text all mentions of all hyponyms of a given node in the WordNet hierarchy (e.g., finding all *buildings* in a text).  Upload this notebook at the end of class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uJbmEjRKHufa",
    "outputId": "e6dbb497-b29a-41f5-af0c-8ad77c90fcde"
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "import nltk\n",
    "from nltk.corpus import wordnet as wn\n",
    "\n",
    "nltk.download('wordnet')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ilRVVtTBHufc"
   },
   "source": [
    "Get the synsets for a given word.  The synsets here are roughly ordered by frequency of use (in a small tagged dataset), so that more frequent senses occur first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "hIteOmmSHufc",
    "outputId": "bb5b531f-8435-4835-c1a4-3930c38c7f69"
   },
   "outputs": [],
   "source": [
    "for synset in wn.synsets('speak'):\n",
    "    print (synset, synset.definition())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hieJ1oUOHufc"
   },
   "source": [
    "Get the words/phrases in that synset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "81x_MW-CHufd",
    "outputId": "07b1a438-611e-4730-a28a-d78211bf6c1a"
   },
   "outputs": [],
   "source": [
    "for lemma in wn.synset(\"talk.v.01\").lemmas():\n",
    "    print (lemma.name())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-XGCD_7dHufd"
   },
   "outputs": [],
   "source": [
    "# Functions from http://www.nltk.org/howto/wordnet.html to get *all* of a synset's hyponym/hypernyms\n",
    "hypo = lambda s: s.hyponyms()\n",
    "hyper = lambda s: s.hypernyms()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "naS4VrplHufd"
   },
   "source": [
    "Find all of the synsets that are hyponyms of the target synset (*descendents* in the WordNet hierarchy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Jjcz06hjHufe",
    "outputId": "aaf1e5fa-02e6-47ac-9a91-7222773bf89a"
   },
   "outputs": [],
   "source": [
    "list(wn.synset(\"talk.v.02\").closure(hypo))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "otENwjO0Hufe"
   },
   "source": [
    "Find all of the synsets that are hyperyms (*ancestors* up the tree) of the target synset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vk6df6N4Hufe",
    "outputId": "89efbe27-3b1c-4e37-a086-b5a713f86518"
   },
   "outputs": [],
   "source": [
    "list(wn.synset(\"communicate.v.02\").closure(hyper))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ASo4SI0ZHufe"
   },
   "outputs": [],
   "source": [
    "def get_words_in_hypo(synset):\n",
    "    \"\"\" Returns a list of words/phrases that comprise the hyponyms of a synset.\n",
    "    \"\"\"\n",
    "    words=set()\n",
    "    hyponym_synsets=list(synset.closure(hypo))\n",
    "    hyponym_synsets.append(synset)\n",
    "    for synset in hyponym_synsets:\n",
    "        for l in synset.lemmas():\n",
    "            word=l.name()\n",
    "            word=re.sub(\"_\", \" \", word)\n",
    "            words.add(word)\n",
    "\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "C8GBMQpfHufe",
    "outputId": "adaad874-a5e6-4be1-b4f4-1d123dc58cdc"
   },
   "outputs": [],
   "source": [
    "get_words_in_hypo(wn.synset(\"color.n.01\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QC4II9IiHuff"
   },
   "outputs": [],
   "source": [
    "def find_all_words_in_text(words, spacy_tokens):\n",
    "    \"\"\" For a given set of words, find each instance among a list of tokens already\n",
    "    processed by spacy.  Returns a list of token indexes that match.  (Note this only\n",
    "    identifies single words, not multi-word phrases.)\n",
    "    \"\"\"\n",
    "    all_matches=[]\n",
    "    for idx, token in enumerate(spacy_tokens):\n",
    "        if token.lemma_ in words:\n",
    "            all_matches.append(idx)\n",
    "    return all_matches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ApGL2CU1Huff"
   },
   "outputs": [],
   "source": [
    "def print_concordance(matches, spacy_tokens, window=3):\n",
    "    \"\"\" For a given set of token indexes, prints out a window of words around each match,\n",
    "    in the style of a concordance.\n",
    "    \"\"\"\n",
    "\n",
    "    RED=\"\\x1b[31m\"\n",
    "    BLACK=\"\\x1b[0m\"\n",
    "\n",
    "    spacing=window*10\n",
    "    for match in matches:\n",
    "        start=match-window\n",
    "        end=match+window+1\n",
    "        if start < 0:\n",
    "            start=0\n",
    "        if end > len(spacy_tokens):\n",
    "            end=len(spacy_tokens)\n",
    "        pre=' '.join([token.text for token in spacy_tokens[start:match]])\n",
    "        post=' '.join([token.text for token in spacy_tokens[match+1:end]])\n",
    "        print(\"%s %s%s%s %s\" % (pre.rjust(spacing), RED, spacy_tokens[match].text, BLACK, post))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "41Cvn05kHuff",
    "outputId": "209dde02-d00a-43b2-e6d8-1e1f1865193d"
   },
   "outputs": [],
   "source": [
    "def read_text(filename):\n",
    "    \"\"\" Read a text, replacing all whitespace sequences with a single space.\n",
    "    \"\"\"\n",
    "    with open(filename, encoding=\"utf-8\") as file:\n",
    "        return re.sub(r\"\\s+\", \" \", file.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3okuJ0-4IIYq",
    "outputId": "52b63ea8-fbf8-4eba-a05f-6b4549e6d76c"
   },
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/1342_pride_and_prejudice.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yy5k5CgRHuff"
   },
   "outputs": [],
   "source": [
    "book = read_text(\"1342_pride_and_prejudice.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9I-yeSnRHufb"
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load('en_core_web_sm', disable=['ner,parser'])\n",
    "nlp.remove_pipe('ner')\n",
    "nlp.remove_pipe('parser');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EL3e3vPzHuff"
   },
   "outputs": [],
   "source": [
    "spacy_tokens = nlp(book)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ECaWDgGBHuff"
   },
   "outputs": [],
   "source": [
    "def wordnet_search(synset, spacy_tokens):\n",
    "    \"\"\" This functions searchs through all of the tokens in the spacy_tokens argument to find\n",
    "    any mention of words in the synset or any of its hyponyms.\n",
    "    \"\"\"\n",
    "    targets = get_words_in_hypo(synset)\n",
    "    matches = find_all_words_in_text(targets, spacy_tokens)\n",
    "    print_concordance(matches, spacy_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dQMVYVABHuff"
   },
   "source": [
    "**Q1.** Let's do a very coarse tagging of a document to find all of the mentions of a specific WordNet synset and all of its hyponyms. Using the functions above, find all of the color terms in *Pride and Prejudice*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bmKV0fpZHuff",
    "outputId": "29c48876-5fca-42e7-cc85-67398b1fb76c",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "wordnet_search(wn.synset(\"color.n.01\"), spacy_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mrk7KhzcHuff"
   },
   "source": [
    "**Q2.** Find all of the vehicles mentioned in *Pride and Prejudice*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sG8TL9MiHufg"
   },
   "source": [
    "**Q3.** Find all of the verbs of speaking in *Pride and Prejudice*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_NsNu5ClHufg"
   },
   "source": [
    "**Q4.** Find all of the people in *Pride and Prejudice*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kJuseUntHufg"
   },
   "source": [
    "**Q5.** The methods above all identify *any* mentions of a WordNet synset in a text  -- e.g., every instance of *bank* would be identified as a hit for query bank.n.01 (\"sloping land ...\"), even if its specific word sense in context was the financial institution (or even a verb).\n",
    "\n",
    "How might we improve on this method?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "n0YFc2NWHufg"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}