{"cells":[{"cell_type":"markdown","metadata":{"id":"G0qZfMf4Yreh"},"source":["# Few-shot OpenQA with ColBERT retrieval"]},{"cell_type":"code","execution_count":1,"metadata":{"id":"NqUPAnWuYrej","executionInfo":{"status":"ok","timestamp":1685638201668,"user_tz":420,"elapsed":21,"user":{"displayName":"Chris Potts","userId":"10671134698620403283"}}},"outputs":[],"source":["__author__ = \"Christopher Potts and Omar Khattab\"\n","__version__ = \"CS224u, Stanford, Spring 2022\""]},{"cell_type":"markdown","metadata":{"id":"R6TRA3gNYrek"},"source":["## Contents\n","\n","1. [Overview](#Overview)\n","1. [Set-up](#Set-up)\n","    1. [Google Colab set-up](#Google-Colab-set-up)\n","    1. [General set-up](#General-set-up)\n","    1. [Language model set-up](#Language-model-set-up)\n","    1. [ColBERT set-up](#ColBERT-set-up)\n","1. [Language models](#Language-models)\n","    1. [Answerhood](#Answerhood)\n","    1. [Eleuther models from Hugging Face](#Eleuther-models-from-Hugging-Face)\n","    1. [GPT-3](#GPT-3)\n","1. [SQuAD](#SQuAD)\n","    1. [SQuAD dev](#SQuAD-dev)\n","    1. [SQuAD dev sample](#SQuAD-dev-sample)\n","    1. [SQuAD train](#SQuAD-train)\n","1. [Evaluation](#Evaluation)\n","1. [Open QA with no context](#Open-QA-with-no-context)\n","1. [Few-shot QA](#Few-shot-QA)\n","1. [ColBERT](#ColBERT)\n","    1. [ColBERT parameters](#ColBERT-parameters)\n","    1. [ColBERT index](#ColBERT-index)\n","    1. [Search](#Search)\n","    1. [Retrieval evaluation](#Retrieval-evaluation)\n","1. [Open QA with ColBERT retrieval](#Open-QA-with-ColBERT-retrieval)\n","1. [Homework questions](#Homework-questions)\n","    1. [Few-shot OpenQA with no context [2 points]](#Few-shot-OpenQA-with-no-context-[2-points])\n","    1. [Few-shot OpenQA [2 points]](#Few-shot-OpenQA-[2-points])\n","    1. [Answer scoring [2 points]](#Answer-scoring-[2-points])\n","    1. [Your original system [3 points]](#Your-original-system-[3-points])\n","1. [Bake-off [1 point]](#Bake-off-[1-point])"]},{"cell_type":"markdown","metadata":{"id":"MXqIujoUYrek"},"source":["## Overview\n","\n","The goal of this homework is to explore few-shot (or, prompt-based) learning in the context of open-domain question answering. This is an exciting area that brings together a number of recent task ideas and modeling innovations.\n","\n","Our core task is __open-domain question answering (OpenQA)__. In this task, all that is given by the dataset is a question text, and the task is to answer that question. By contrast, in modern QA tasks, the dataset provides a text and a gold passage, with a guarantee that the answer will be a substring of the passage. \n","\n","OpenQA is substantially harder than standard QA. The usual strategy is to use a _retriever_ to find passages in a large collection of texts and train a _reader_ to find answers in those passages. This means we have no guarantee that the retrieved passage will contain the answer we need. If we don't retrieve a passage containing the answer, our reader has no hope of succeeding. Although this is challenging, it is much more realistic and widely applicable than standard QA. After all, with the right retriever, an OpenQA system could be deployed over the entire Web.\n","\n","The task posed by this homework is harder even than OpenQA. We are calling this task __few-shot OpenQA__. The defining feature of this task is that the reader is simply a general purpose autoregressive language model. It accepts string inputs (prompts) and produces text in response. It is not trained to answer questions per se, and nothing about its structure ensures that it will respond with a substring of the prompt corresponding to anything like an answer.\n","\n","__Few-shot QA__ (but not OpenQA!) is explored in the famous GPT-3 paper ([Brown et al. 2020](https://arxiv.org/abs/2005.14165)). The authors are able to get traction on the problem using GPT-3, an incredible finding. Our task here – __few-shot OpenQA__ – pushes this even further by retrieving passages to use in the prompt rather than assuming that the gold passage can be used in the prompt. If we can make this work, then it should be a major step towards flexibly and easily deploying QA technologies in new domains.\n","\n","In summary:\n","\n","| Task             | Passage given | Task-specific reader training |Task-specific retriever training  | \n","|-----------------:|:-------------:|:-----------------------------:|:--------------------------------:|\n","| QA               | yes           | yes                           | n/a                              |\n","| OpenQA           | no            | yes                           | maybe                            |\n","| Few-shot QA      | yes           | no                            | n/a                              |\n","| Few-shot OpenQA  | no            | no                            | maybe                            | \n","\n","Just to repeat: your mission (should you choose to accept it!) is to explore the final line in this table. The core notebook and assignment don't address the issue of training the retriever in a task-specific way, but we've given some pointers on this in the context of [the original system question at the bottom of this notebook](#Your-original-system-[3-points]).\n","\n","As usual, this notebook sets up the task and provides starter code. We proceed through a series of approaches:\n","\n","* _Open QA with no context_: the prompt consists of the question, and we just see what comes back. This is not particularly fair to the system since it doesn't unambiguously convey what we want it to do, but it's a start.\n","\n","* _Few-shot QA_: the prompt contains one or more examples formatted so as to indirectly convey what we want the system to do, and it uses the gold passage associated with the example. This is the approach of the GPT-3 paper. It works only for datasets with gold passages.\n","\n","* _Open QA with ColBERT retrieval_: This is roughly as in the previous case, but now we presume no access to a gold passage for our example. Rather, we retrieve a passage from a large corpus using the neural information retrieval model ColBERT.\n","\n","The above examples are followed by some assignment questions aimed at helping you to think creatively about the problem. These problems improve on the above approaches in various ways.\n","\n","All of this culminates in an original system question and some code and unlabeled data (here, just a list of questions) for the bake-off.\n","\n","It is a requirement of the bake-off that a pure autoregressive model be used. In particular, trained QA systems cannot be used at all. See the original system question at the bottom of this message for the full list of allowed models.\n","\n","Note: the models we are working with here are _big_. This poses a challenge that is increasingly common in NLP: you have to pay one way or another. You can pay to use the GPT-3 API, or you can pay to use an Eleuther model on a heavy-duty cluster computer, or you can pay with time by using an Eleuther model on a more modest computer. If none of these options is palatable, you might consider instead doing the [color reference assignment](hw_colors.ipynb)!"]},{"cell_type":"markdown","metadata":{"id":"dKB9zXRBYrel"},"source":["## Set-up"]},{"cell_type":"markdown","metadata":{"id":"1JwiISlVYrel"},"source":["### Google Colab set-up"]},{"cell_type":"markdown","metadata":{"id":"lbnxdvg7Yrem"},"source":["We have sought to make this notebook self-contained so that it can easily be run as a Google Colab. If you are running it in Colab, make sure to select a GPU instance. The notebook will run on a CPU-only instance or CPU-only machine, but it should be much faster with GPU support.\n","\n","The following are all installed as part of course set-up, but you'll want to run this cell if you are working in a Colab:"]},{"cell_type":"code","execution_count":2,"metadata":{"id":"P2TXnN9HYrem","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1685638246384,"user_tz":420,"elapsed":44736,"user":{"displayName":"Chris Potts","userId":"10671134698620403283"}},"outputId":"2e5d729c-11fb-4af4-f078-91e10acdec20"},"outputs":[{"output_type":"stream","name":"stdout","text":["Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","\u001b[31mERROR: Could not find a version that satisfies the requirement torch==1.10.0 (from versions: 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 2.0.0, 2.0.1)\u001b[0m\u001b[31m\n","\u001b[0m\u001b[31mERROR: No matching distribution found for torch==1.10.0\u001b[0m\u001b[31m\n","\u001b[0mLooking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Collecting ujson\n","  Downloading ujson-5.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (52 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m52.8/52.8 kB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hInstalling collected packages: ujson\n","Successfully installed ujson-5.7.0\n","Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Collecting transformers\n","  Downloading transformers-4.29.2-py3-none-any.whl (7.1 MB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m56.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.12.0)\n","Collecting huggingface-hub<1.0,>=0.14.1 (from transformers)\n","  Downloading huggingface_hub-0.15.1-py3-none-any.whl (236 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m236.8/236.8 kB\u001b[0m \u001b[31m25.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.22.4)\n","Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (23.1)\n","Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0)\n","Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2022.10.31)\n","Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.27.1)\n","Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers)\n","  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m105.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.65.0)\n","Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers) (2023.4.0)\n","Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers) (4.5.0)\n","Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (1.26.15)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2022.12.7)\n","Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.0.12)\n","Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.4)\n","Installing collected packages: tokenizers, huggingface-hub, transformers\n","Successfully installed huggingface-hub-0.15.1 tokenizers-0.13.3 transformers-4.29.2\n","Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Collecting datasets\n","  Downloading datasets-2.12.0-py3-none-any.whl (474 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m474.6/474.6 kB\u001b[0m \u001b[31m2.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from datasets) (1.22.4)\n","Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (9.0.0)\n","Collecting dill<0.3.7,>=0.3.0 (from datasets)\n","  Downloading dill-0.3.6-py3-none-any.whl (110 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m110.5/110.5 kB\u001b[0m \u001b[31m1.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (1.5.3)\n","Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (2.27.1)\n","Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (4.65.0)\n","Collecting xxhash (from datasets)\n","  Downloading xxhash-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m212.5/212.5 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hCollecting multiprocess (from datasets)\n","  Downloading multiprocess-0.70.14-py310-none-any.whl (134 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.3/134.3 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hRequirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (2023.4.0)\n","Collecting aiohttp (from datasets)\n","  Downloading aiohttp-3.8.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m12.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hRequirement already satisfied: huggingface-hub<1.0.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.15.1)\n","Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets) (23.1)\n","Collecting responses<0.19 (from datasets)\n","  Downloading responses-0.18.0-py3-none-any.whl (38 kB)\n","Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (6.0)\n","Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (23.1.0)\n","Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (2.0.12)\n","Collecting multidict<7.0,>=4.5 (from aiohttp->datasets)\n","  Downloading multidict-6.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.5/114.5 kB\u001b[0m \u001b[31m9.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hCollecting async-timeout<5.0,>=4.0.0a3 (from aiohttp->datasets)\n","  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)\n","Collecting yarl<2.0,>=1.0 (from aiohttp->datasets)\n","  Downloading yarl-1.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (268 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m13.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hCollecting frozenlist>=1.1.1 (from aiohttp->datasets)\n","  Downloading frozenlist-1.3.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (149 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m149.6/149.6 kB\u001b[0m \u001b[31m15.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hCollecting aiosignal>=1.1.2 (from aiohttp->datasets)\n","  Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n","Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets) (3.12.0)\n","Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets) (4.5.0)\n","Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (1.26.15)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (2022.12.7)\n","Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (3.4)\n","Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2)\n","Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2022.7.1)\n","Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.16.0)\n","Installing collected packages: xxhash, multidict, frozenlist, dill, async-timeout, yarl, responses, multiprocess, aiosignal, aiohttp, datasets\n","Successfully installed aiohttp-3.8.4 aiosignal-1.3.1 async-timeout-4.0.2 datasets-2.12.0 dill-0.3.6 frozenlist-1.3.3 multidict-6.0.4 multiprocess-0.70.14 responses-0.18.0 xxhash-3.2.0 yarl-1.9.2\n","Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Requirement already satisfied: spacy in /usr/local/lib/python3.10/dist-packages (3.5.2)\n","Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.0.12)\n","Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.0.4)\n","Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.0.9)\n","Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.0.7)\n","Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.0.8)\n","Requirement already satisfied: thinc<8.2.0,>=8.1.8 in /usr/local/lib/python3.10/dist-packages (from spacy) (8.1.9)\n","Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.1.1)\n","Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.4.6)\n","Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.0.8)\n","Requirement already satisfied: typer<0.8.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (0.7.0)\n","Requirement already satisfied: pathy>=0.10.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (0.10.1)\n","Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from spacy) (6.3.0)\n","Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (4.65.0)\n","Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.22.4)\n","Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.27.1)\n","Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.10.7)\n","Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.1.2)\n","Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from spacy) (67.7.2)\n","Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (23.1)\n","Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.3.0)\n","Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4->spacy) (4.5.0)\n","Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (1.26.15)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2022.12.7)\n","Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2.0.12)\n","Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (3.4)\n","Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy) (0.7.9)\n","Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.2.0,>=8.1.8->spacy) (0.0.4)\n","Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.10/dist-packages (from typer<0.8.0,>=0.3.0->spacy) (8.1.3)\n","Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->spacy) (2.1.2)\n","Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Collecting gitpython\n","  Downloading GitPython-3.1.31-py3-none-any.whl (184 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m184.3/184.3 kB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hCollecting gitdb<5,>=4.0.1 (from gitpython)\n","  Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)\n","\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n","\u001b[?25hCollecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython)\n","  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)\n","Installing collected packages: smmap, gitdb, gitpython\n","Successfully installed gitdb-4.0.10 gitpython-3.1.31 smmap-5.0.0\n"]}],"source":["!pip install torch==1.10.0\n","\n","!pip install ujson\n","\n","!pip install transformers\n","\n","!pip install datasets\n","\n","!pip install spacy\n","\n","!pip install gitpython"]},{"cell_type":"markdown","metadata":{"id":"hlk-FsfZYrem"},"source":["If you are indeed on a GPU machine, then run the following to ensure complete CUDA support:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Fl5wBNCxYrem"},"outputs":[],"source":["import torch\n","\n","if torch.cuda.is_available():\n","    !pip install cupy-cuda111"]},{"cell_type":"markdown","metadata":{"id":"Ej8kZeh6Yren"},"source":["If the above doesn't work, it might be because you don't have CUDA version 11.1. Run "]},{"cell_type":"code","execution_count":null,"metadata":{"id":"PojRXsuPYren"},"outputs":[],"source":["import torch\n","\n","if torch.cuda.is_available():\n","    !nvcc --version"]},{"cell_type":"markdown","metadata":{"id":"VfZEOAL2Yren"},"source":["and then install the corresponding `cupy-cuda`. See [this table](https://docs.cupy.dev/en/stable/install.html#installing-cupy-from-pypi) for details on which one to install for different scenarios."]},{"cell_type":"markdown","metadata":{"id":"sFM54iO7Yren"},"source":["### General set-up"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"hL9AAtTzYren"},"outputs":[],"source":["import collections\n","from contextlib import nullcontext\n","from collections import namedtuple\n","from datasets import load_dataset\n","import json\n","import numpy as np\n","import random\n","import re \n","import string\n","import torch\n","from typing import List"]},{"cell_type":"markdown","metadata":{"id":"vNXANb8fYreo"},"source":["Try to set all the seeds for reproducibility (won't extend to GPT-3):"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"MIvsYoIpYreo"},"outputs":[],"source":["seed = 1\n","\n","np.random.seed(seed)\n","random.seed(seed)\n","torch.manual_seed(seed)\n","torch.backends.cudnn.deterministic = True\n","torch.backends.cudnn.benchmark = False"]},{"cell_type":"markdown","metadata":{"id":"oKp57bQZYrep"},"source":["The following should install the version of [Faiss](https://github.com/facebookresearch/faiss) that will cooperate with your set-up:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"aGUj1O9wYrep"},"outputs":[],"source":["import torch\n","\n","if torch.cuda.is_available():\n","    !pip install faiss-gpu==1.7.0\n","else:\n","    !pip install faiss-cpu==1.7.0"]},{"cell_type":"markdown","metadata":{"id":"-mp1C-oyYreq"},"source":["### Language model set-up"]},{"cell_type":"markdown","metadata":{"id":"pEUY5P8GYreq"},"source":["To use the GPT-3 API, install the OpenAI library:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"4FclKZiuYreq"},"outputs":[],"source":["!pip install openai"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5cbE328hYrer"},"outputs":[],"source":["import openai\n","import transformers\n","from transformers import AutoTokenizer, AutoModelForCausalLM"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"TKQEIYGDYrer"},"outputs":[],"source":["transformers.logging.set_verbosity_error()"]},{"cell_type":"markdown","metadata":{"id":"axUPfnNmYrer"},"source":["### ColBERT set-up\n","\n","Our retriever will be a ColbERT-based model ([Khattab and Zaharia 2020](https://arxiv.org/abs/2004.12832)). ColBERT is a powerful neural information retrieval (Neural IR) model that has proven extremely successful in retrieval applications and as a component in a variety of different systems for OpenQA and other knowledge-intensive tasks (e.g., [Khattab et al. 2021a](https://aclanthology.org/2021.tacl-1.55/); [Khattab et al. 2021b](https://proceedings.neurips.cc/paper/2021/hash/e8b1cbd05f6e6a358a81dee52493dd06-Abstract.html); [Santhanam, Khattab, et al. 2021](https://arxiv.org/abs/2112.01488)).\n","\n","The following will clone the ColBERTv2 repository for use in this notebook:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"L5wP933JYrer"},"outputs":[],"source":["!git clone -b cpu_inference https://github.com/stanford-futuredata/ColBERT.git"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"6LWz4f-3Yres"},"outputs":[],"source":["import os\n","import sys\n","sys.path.insert(0, 'ColBERT/')\n","\n","from colbert.infra import Run, RunConfig, ColBERTConfig\n","from colbert.data import Collection\n","from colbert.searcher import Searcher\n","from utility.utils.dpr import has_answer, DPR_normalize"]},{"cell_type":"markdown","metadata":{"id":"U5ganiuNYres"},"source":["## Language models\n","\n","In few-shot OpenQA, the language model (LM) must read in a prompt and answer the question posed somewhere in the prompt. We propose two basic strategies:\n","\n","* [EleutherAI](https://www.eleuther.ai/) has released GPT-2-style models in a variety of sizes. These are free to use and easy to use via Hugging Face, and the larger ones very effective for our task, with GPT-J competitive with GPT-3 even though it has only 6B parameters (vs. 145B for GPT-3). The downside here is that the larger models in this family might be very slow and very difficult to work with unless you have access to really impressive GPU hardware. In testing with the free version of Google Colab, we were basically able to do everything we needed to do for the 1.3B parameter model, but the larger one caused too many problems to be viable.\n","\n","* OpenAI has outstanding API access to GPT-3. You can sign up for [a free account](https://beta.openai.com/signup), and, as of this writing, you get US$18 in credit when you sign up. This is more than enough for the current assignment provided you are careful about how much testing you do. The benefits here are that the API is blazingly fast and requires nothing of your computer in terms of GPU support, and you're getting responses from a 145B parameter model that is truly exceptional.\n","\n","Our suggestion is to do basic development with `\"gpt-neo-125M\"`, scale up to `\"gpt-neo-1.3B\"` once you have a sense for what your original system will be like, and then do your final bake-off entry with GPT-3. The functions `run_eleuther` and `run_gpt3` defined below are totally interchangeable, so this kind of development path should be easy to take."]},{"cell_type":"markdown","metadata":{"id":"Fca8-RXjYres"},"source":["### Answerhood"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"U6KepnjTYret"},"outputs":[],"source":["def _find_generated_answer(tokens, newline=\"\\n\" ): \n","    \"\"\"Our LMs tend to insert initial newline characters before\n","    they begin generating text. This function ensures that we \n","    properly capture the true first line as the answer while\n","    also ensuring that token probabilities are aligned.\"\"\"        \n","    answer_token_indices = []\n","    char_seen = False            \n","    for i, tok in enumerate(tokens):\n","        # This is the main condition: a newline that isn't an initial\n","        # string of newlines:\n","        if tok == newline and char_seen:\n","            break\n","        # Keep the initial newlines for consistency:\n","        elif tok == newline and not char_seen:\n","            answer_token_indices.append(i)\n","        # Proper tokens:\n","        elif tok != newline:\n","            char_seen = True\n","            answer_token_indices.append(i)\n","    return answer_token_indices "]},{"cell_type":"markdown","metadata":{"id":"klW12GkAYret"},"source":["### Eleuther models from Hugging Face"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"1-FkbTaUYret"},"outputs":[],"source":["# \"gpt-neo-125M\" \"gpt-neo-1.3B\" \"gpt-neo-2.7B\" \"gpt-j-6B\"\n","eleuther_model_name = \"gpt-neo-125M\"\n","\n","eleuther_tokenizer = AutoTokenizer.from_pretrained(\n","    f\"EleutherAI/{eleuther_model_name}\", \n","    padding_side=\"left\", \n","    padding='longest', \n","    truncation='longest_first', max_length=2000)\n","eleuther_tokenizer.pad_token = eleuther_tokenizer.eos_token\n","\n","eleuther_model = AutoModelForCausalLM.from_pretrained(\n","    f\"EleutherAI/{eleuther_model_name}\")\n","\n","device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n","eleuther_model = eleuther_model.to(device)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"rEo9YKwNYret"},"outputs":[],"source":["def run_eleuther(prompts, temperature=0.1, top_p=0.95, **generate_kwargs): \n","    \"\"\"\n","    Parameters\n","    ----------\n","    prompts : iterable of str\n","    temperature : float\n","        It seems best to set it low for this task!\n","    top_p : float\n","       \n","    For options for `generate_kwargs`, see:\n","    \n","    https://huggingface.co/docs/transformers/master/en/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate\n","    \n","    Options that are likely to be especially relevant include \n","    `temperature`, `length_penalty`, and the parameters that\n","    determine the decoding strategy. With `num_return_sequences > 1`,\n","    the default parameters in this function do multinomial sampling.\n","    \n","    Returns\n","    -------\n","    list of dicts\n","    \n","    {\"prompt\": str, \n","     \"generated_text\": str, \"generated_tokens\": list of str, \"generated_probs\": list of float,\n","     \"answer\": str, \"answer_tokens\": list of str, \"answer_probs\": list of float\n","    }\n","         \n","    \"\"\"\n","    prompt_ids = eleuther_tokenizer(\n","        prompts, return_tensors=\"pt\", padding=True).input_ids.to(device)\n","        \n","    with torch.inference_mode():\n","        # Automatic mixed precision if possible.\n","        with torch.cuda.amp.autocast() if torch.cuda.is_available() else nullcontext():\n","            model_output = eleuther_model.generate(\n","                prompt_ids,\n","                temperature=temperature,\n","                do_sample=True,\n","                top_p=top_p,           \n","                max_new_tokens=16,\n","                num_return_sequences=1,                \n","                pad_token_id=eleuther_tokenizer.eos_token_id, \n","                return_dict_in_generate=True,\n","                output_scores=True,\n","                **generate_kwargs)\n","        \n","    # Converting output scores using the helpful recipe here:\n","    # https://discuss.huggingface.co/t/generation-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2/3175\n","    gen_ids = model_output.sequences[:, prompt_ids.shape[-1] :]\n","    gen_probs = torch.stack(model_output.scores, dim=1).softmax(-1)\n","    gen_probs = torch.gather(gen_probs, 2, gen_ids[:, :, None]).squeeze(-1)\n","    \n","    # Generated texts, including the prompts:\n","    gen_texts = eleuther_tokenizer.batch_decode(\n","        model_output.sequences, skip_special_tokens=True)\n","    \n","    data = []     \n","    iterator = zip(prompts, gen_ids, gen_texts, gen_probs)    \n","    for prompt, gen_id, gen_text, gen_prob in iterator:       \n","        gen_tokens = eleuther_tokenizer.convert_ids_to_tokens(gen_id)\n","        generated_text = gen_text[len(prompt): ]\n","        gen_prob = [float(x) for x in gen_prob.cpu().numpy()] # float for JSON storage\n","        ans_indices = _find_generated_answer(gen_tokens, newline=\"Ċ\")\n","        answer_tokens = [gen_tokens[i] for i in ans_indices]\n","        answer_probs = [gen_prob[i] for i in ans_indices]\n","        answer = \"\".join(answer_tokens).replace(\"Ġ\", \" \").replace(\"Ċ\", \"\\n\")                                       \n","        data.append({\n","            \"prompt\": prompt,\n","            \"generated_text\": generated_text,\n","            \"generated_tokens\": gen_tokens,\n","            \"generated_probs\": gen_prob,\n","            \"generated_answer\": answer,\n","            \"generated_answer_probs\": answer_probs,\n","            \"generated_answer_tokens\": answer_tokens})                        \n","\n","    return data"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"7fIfuTEGYreu"},"outputs":[],"source":["eleuther_ex = run_eleuther([    \n","    \"What year was Stanford University founded?\", \n","    \"In which year did Stanford first enroll students?\"])\n","\n","eleuther_ex"]},{"cell_type":"markdown","metadata":{"id":"oiN04KynYreu"},"source":["### GPT-3"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5t5HTfRkYreu"},"outputs":[],"source":["def run_gpt3(prompts, engine=\"text-curie-001\", temperature=0.1, top_p=0.95, **gpt3_kwargs):\n","    \"\"\"To use this function, sign up for an OpenAI account at\n","        \n","    https://beta.openai.com/signup\n","    \n","    That should give you $18 in free credits, which is more than enough\n","    for this assignment assuming you are careful with testing.\n","    \n","    Once your account is set up, you can get your API key from your \n","    account dashboard and paste it in below as the value of \n","    `openai.api_key`.\n","    \n","    Parameters\n","    ----------\n","    prompts : iterable of str\n","    engine : str\n","        This has to be one of the models whose name begins with \"text\".\n","        The \"instruct\" class of models can't be used, since they seem\n","        to depend on some kinds of QA-relevant supervision.        \n","        For options, costs, and other details: \n","        https://beta.openai.com/docs/engines/gpt-3                \n","    temperature : float\n","        It seems best to set it low for this task!\n","    top_p : float\n","        \n","    For information about values for `gpt3_kwargs`, see\n","    \n","    https://beta.openai.com/docs/api-reference/completions\n","    \n","    Returns\n","    -------\n","    list of dicts   \n","    \n","    \"\"\"\n","    # Fill this in with the value from your OpenAI account. First\n","    # verify that your account is set up with a spending limit that\n","    # you are comfortable with. If you just opened your account,\n","    # you should have $18 in credit and so won't need to supply any\n","    # payment information.\n","    openai.api_key = None\n","    \n","    assert engine.startswith(\"text\"), \\\n","        \"Please use an engine whose name begins with 'text'.\"\n","        \n","    response = openai.Completion.create(\n","        engine=engine,       \n","        prompt=prompts,\n","        temperature=temperature,\n","        top_p=top_p,\n","        echo=False,   # This function will not work\n","        logprobs=1,   # properly if any of these\n","        n=1,          # are changed!\n","        **gpt3_kwargs)\n","    \n","    # From here, we parse each example to get the values\n","    # we need:\n","    data = []\n","    for ex, prompt in zip(response[\"choices\"], prompts):\n","        tokens = ex[\"logprobs\"][\"tokens\"]\n","        logprobs = ex[\"logprobs\"][\"token_logprobs\"]        \n","        probs = list(np.exp(logprobs))\n","        if \"<|endoftext|>\" in tokens:\n","            end_i = tokens.index(\"<|endoftext|>\")\n","            tokens = tokens[ : end_i]  # This leaves off the \"<|endoftext|>\"\n","            probs = probs[ : end_i]    # token -- perhaps dubious.\n","        ans_indices = _find_generated_answer(tokens)\n","        answer_tokens = [tokens[i] for i in ans_indices]\n","        answer_probs = [probs[i] for i in ans_indices]\n","        answer = \"\".join(answer_tokens)        \n","        data.append({\n","            \"prompt\": prompt,\n","            \"generated_text\": ex[\"text\"],\n","            \"generated_tokens\": tokens,\n","            \"generated_probs\": probs,\n","            \"generated_answer\": answer,\n","            \"generated_answer_tokens\": answer_tokens,\n","            \"generated_answer_probs\": answer_probs})\n","        \n","    return data             "]},{"cell_type":"code","execution_count":null,"metadata":{"id":"y3jbKTAbYrev"},"outputs":[],"source":["gpt3_ex = run_gpt3([\n","     \"What year was Stanford University founded?\",\n","     \"In which year did Stanford first enroll students?\"])\n","\n","gpt3_ex"]},{"cell_type":"markdown","metadata":{"id":"XGK3hCs9Yrev"},"source":["## SQuAD\n","\n","Our core development dataset is [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/). We chose this dataset because it is well-known and widely used, and it is large enough to support lots of meaningful development work, without, though, being so large as to require lots of compute power. It is also useful that it has gold passages supporting the standard QA formulation, so we can see how well our LM performs with an \"oracle\" retriever that always retrieves the gold passage."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"YQ_do58EYrev"},"outputs":[],"source":["squad = load_dataset(\"squad\")"]},{"cell_type":"markdown","metadata":{"id":"VbCUz66GYrev"},"source":["The following utility just reads a SQuAD split in as a list of `SquadExample` instances:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"B9-0hkxgYrew"},"outputs":[],"source":["SquadExample = namedtuple(\"SquadExample\",  \"id title context question answers\")"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"g2gt0dkeYrew"},"outputs":[],"source":["def get_squad_split(squad, split=\"validation\"):\n","    \"\"\"\n","    Use `split='train'` for the train split.\n","    \n","    Returns\n","    -------\n","    list of SquadExample named tuples with attributes\n","    id, title, context, question, answers\n","    \n","    \"\"\"    \n","    fields = squad[split].features\n","    data = zip(*[squad[split][field] for field in fields])\n","    return [SquadExample(eid, title, context, question, answers[\"text\"]) \n","            for eid, title, context, question, answers in data]"]},{"cell_type":"markdown","metadata":{"id":"slqK_3NEzTil"},"source":["### SQuAD dev"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"NatnOLsDYrew"},"outputs":[],"source":["squad_dev = get_squad_split(squad)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"jb-ZrSzoYrew"},"outputs":[],"source":["squad_dev[0]"]},{"cell_type":"markdown","metadata":{"id":"SgXgw4jezTil"},"source":["### SQuAD dev sample\n","\n","We'll use this fixed but presumably quite random set of examples for exploration and system development:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"rw3OGISgzTil"},"outputs":[],"source":["dev_exs = sorted(squad_dev, key=lambda x: hash(x.id))[: 200]"]},{"cell_type":"markdown","metadata":{"id":"kuVwpDEhzTim"},"source":["### SQuAD train\n","\n","To build few-shot prompts, we will often sample SQuAD train examples, so we load that split here:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"YwwjQiF8zTim"},"outputs":[],"source":["squad_train = get_squad_split(squad, \"train\")"]},{"cell_type":"markdown","metadata":{"id":"DZpXMk-0Yrew"},"source":["## Evaluation\n","\n","Our evaluation protocols are the standard ones for SQuAD and related tasks: exact match of the answer (EM) and token-level F1.\n","\n","We say further that the predicted answer is the first line of generated text after the prompt.\n","\n","The following evaluation code is taken from the [apple/ml-qrecc](https://github.com/apple/ml-qrecc/blob/main/utils/evaluate_qa.py) repository. It performs very basic string normalization before doing the core comparisons."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"nHHntDSSYrew"},"outputs":[],"source":["def normalize_answer(s: str) -> str:\n","    \"\"\"Lower text and remove punctuation, articles and extra whitespace.\"\"\"\n","\n","    def remove_articles(text):\n","        regex = re.compile(r'\\b(a|an|the)\\b', re.UNICODE)\n","        return re.sub(regex, ' ', text)\n","\n","    def white_space_fix(text):\n","        return ' '.join(text.split())\n","\n","    def remove_punc(text):\n","        exclude = set(string.punctuation)\n","        return ''.join(ch for ch in text if ch not in exclude)\n","\n","    def lower(text):\n","        return text.lower()\n","\n","    return white_space_fix(remove_articles(remove_punc(lower(s))))\n","\n","\n","def get_tokens(s: str) -> List[str]:\n","    \"\"\"Normalize string and split string into tokens.\"\"\"\n","    if not s:\n","        return []\n","    return normalize_answer(s).split()\n","\n","\n","def compute_exact(a_gold: str, a_pred: str) -> int:\n","    \"\"\"Compute the Exact Match score.\"\"\"\n","    return int(normalize_answer(a_gold) == normalize_answer(a_pred))\n","\n","\n","def compute_f1_from_tokens(gold_toks: List[str], pred_toks: List[str]) -> float:\n","    \"\"\"Compute the F1 score from tokenized gold answer and prediction.\"\"\"\n","    common = collections.Counter(gold_toks) & collections.Counter(pred_toks)\n","    num_same = sum(common.values())\n","\n","    if len(gold_toks) == 0 or len(pred_toks) == 0:\n","        # If either is no-answer, then F1 is 1 if they agree, 0 otherwise\n","        return int(gold_toks == pred_toks)\n","\n","    if num_same == 0:\n","        return 0\n","\n","    precision = 1.0 * num_same / len(pred_toks)\n","    recall = 1.0 * num_same / len(gold_toks)\n","    f1 = (2 * precision * recall) / (precision + recall)\n","    return f1\n","\n","\n","def compute_f1(a_gold: str, a_pred: str) -> float:\n","    \"\"\"Compute the F1 score.\"\"\"\n","    gold_toks = get_tokens(a_gold)\n","    pred_toks = get_tokens(a_pred)\n","    return compute_f1_from_tokens(gold_toks, pred_toks)"]},{"cell_type":"markdown","metadata":{"id":"lJo6C7pgYrex"},"source":["The following is our general evaluation function. We will make extensive use of it to evaluate different systems:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"bJHSxnA6Yrex"},"outputs":[],"source":["def evaluate(examples, prompts, gens):\n","    \"\"\"Generic evalution function.\n","    \n","    Parameters\n","    ----------\n","    examples: iterable of `SquadExample` instances\n","    prompts: list of str\n","    preds: list of LM-generated texts to evaluate as answers\n","    \n","    Returns\n","    -------\n","    dict with keys \"em_per\", \"macro_f1\", \"examples\", where\n","    each \"examples\" value is a dict\n","    \n","    \"\"\"        \n","    results = []\n","    for ex, prompt, gen in zip(examples, prompts, gens):\n","        answers = ex.answers\n","        pred = gen['generated_answer']\n","        # The result is the highest EM from the available answer strings:\n","        em = max([compute_exact(ans, pred) for ans in answers])\n","        f1 = max([compute_f1(ans, pred) for ans in answers])\n","        gen.update({\n","            \"id\": ex.id, \n","            \"question\": ex.question, \n","            \"prediction\": pred, \n","            \"answers\": answers, \n","            \"em\": em,\n","            \"f1\": f1\n","        })\n","        results.append(gen)\n","    data = {}        \n","    data[\"macro_f1\"] = np.mean([d['f1'] for d in results])\n","    data[\"em_per\"] = sum([d['em'] for d in results]) / len(results)\n","    data[\"examples\"] = results\n","    return data"]},{"cell_type":"markdown","metadata":{"id":"fj170C1PYrex"},"source":["Here is a highly simplified example to help make the logic behind `evaluate` clearer:    "]},{"cell_type":"code","execution_count":null,"metadata":{"id":"0bgFXLK3Yrex"},"outputs":[],"source":["ex = namedtuple(\"SquadExample\",  \"id title context question answers\")\n","\n","examples = [\n","    ex(\"0\", \"CS224u\", \n","       \"The course to take is NLU!\", \n","       \"What is the course to take?\", \n","       [\"NLU\", \"CS224u\"])]\n","\n","prompts = [\"Dear model, Please answer this question!\\n\\nQ: What is the course to take?\\n\\nA:\"]\n","\n","gens = [{\"generated_answer\": \"NLU\", \"generated_text\": \"NLU\\nWho am I?\"}]\n","\n","evaluate(examples, prompts, gens)"]},{"cell_type":"markdown","metadata":{"id":"MHZHve9NYrex"},"source":["The bake-off uses `macro_f1` as the primary metric."]},{"cell_type":"markdown","metadata":{"id":"_3LQv2lbYrex"},"source":["## Open QA with no context\n","\n","We now have all the pieces we need to begin building few-shot OpenQA systems. Our first system is the simplest and most naive: we simply feed the question text in as the prompt and hope that the model provides an answer as the first line of its generated text."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"swF4V0ngYrex"},"outputs":[],"source":["def evaluate_no_context(examples, gen_func=run_eleuther, batch_size=20):\n","    prompts = [] \n","    gens = []\n","    for i in range(0, len(examples), batch_size):\n","        ps = [ex.question for ex in examples[i: i+batch_size]]\n","        gs = gen_func(ps)        \n","        prompts += ps\n","        gens += gs    \n","    return evaluate(examples, prompts, gens)    "]},{"cell_type":"code","execution_count":null,"metadata":{"id":"aPuq5nzDYrex"},"outputs":[],"source":["%%time\n","nocontext_results = evaluate_no_context(dev_exs)\n","\n","print(nocontext_results['macro_f1'])"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"lWi1r1-oYrey"},"outputs":[],"source":["%%time\n","nocontext_results_gpt3 = evaluate_no_context(dev_exs, gen_func=run_gpt3)\n","\n","print(nocontext_results_gpt3['macro_f1'])"]},{"cell_type":"markdown","metadata":{"id":"csAimDMGYrey"},"source":["## Few-shot QA\n","\n","The above formulation is not especially fair to our model, since it doesn't convey anything about the intended structure of the prompt. We want the model to give us an answer to the input question, but we didn't specify that goal unambiguously. Perhaps we were looking for commentary on the question, or a count of the number of tokens it contains, or a passage containing the question string, or something else entirely.\n","\n","In few-shot QA, we construct a prompt that is intended to convey our intentions more clearly. The first part of the prompt gives some examples of what we want, and the final part provides the set-up for our actual question. In the current formulation, we assume access to the gold passage. For example, if our example of interest is\n","\n","```\n","Title: CS224u\n","\n","Background: The course to take is NLU!\n","\n","Q: What is the course to take?\n","```\n","\n","with gold answer ```NLU```, then we would create a prompt with, say, 2 additional examples preceding this, to yield a full prompt like this:\n","\n","```\n","Title: Pragmatics\n","\n","Background: Pragmatics is the study of language use.\n","\n","Q: What is pragmatics?\n","\n","A: The study of language use\n","\n","Title: Bert\n","\n","Background: Bert is a Muppet who is lives with Ernie.\n","\n","Q: Who is Bert?\n","\n","A: Bert is a  Muppet\n","\n","Title: CS224u\n","\n","Background: The course to take is NLU!\n","\n","Q: What is the course to take?\n","\n","A:\n","```\n","This is essentially the formulation used in the GPT-3 paper for SQuAD. The context examples are drawn randomly from the SQuAD train set. We will adopt this same protocol for now. (You might revisit this in the context of your original system.)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"lhsl9yvHYrey"},"outputs":[],"source":["def build_few_shot_qa_prompt(ex, squad_train, n_context=2, joiner=\"\\n\\n\"):\n","    segs = []\n","    train_exs = random.sample(squad_train, k=n_context)    \n","    for t in train_exs:\n","        segs += [\n","            f\"Title: {t.title}\",\n","            f\"Background: {t.context}\",\n","            f\"Q: {t.question}\",\n","            f\"A: {t.answers[0]}\"\n","        ]\n","    segs += [\n","        f\"Title: {ex.title}\",\n","        f\"Background: {ex.context}\",\n","        f\"Q: {ex.question}\",\n","        f\"A:\"\n","    ]\n","    return joiner.join(segs)                "]},{"cell_type":"markdown","metadata":{"id":"XzLghzI5Yrez"},"source":["Here's the sort of output we get with `n_context=1`:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"VEuVae4xYrez"},"outputs":[],"source":["print(build_few_shot_qa_prompt(dev_exs[0], squad_train, n_context=1))"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"VUwgM625Yrez"},"outputs":[],"source":["def evaluate_few_shot_qa(examples, squad_train, gen_func=run_eleuther, batch_size=20, n_context=2):\n","    prompts = []\n","    gens = []\n","    for i in range(0, len(examples), batch_size):\n","        batch = examples[i: i+batch_size]\n","        ps = [build_few_shot_qa_prompt(ex, squad_train, n_context=n_context) for ex in batch]        \n","        gs = gen_func(ps)       \n","        prompts += ps\n","        gens += gs\n","    return evaluate(examples, prompts, gens)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"UelblRFCYrez"},"outputs":[],"source":["%%time\n","few_shot_qa_results = evaluate_few_shot_qa(dev_exs, squad_train, n_context=1)\n","\n","print(few_shot_qa_results['macro_f1'])"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"-fdPsEbmYrez"},"outputs":[],"source":["%%time\n","few_shot_qa_results_gpt3 = evaluate_few_shot_qa(\n","    dev_exs, squad_train, n_context=1, gen_func=run_gpt3)\n","\n","print(few_shot_qa_results_gpt3['macro_f1'])"]},{"cell_type":"markdown","metadata":{"id":"WfvVdsGrYre0"},"source":["## ColBERT\n","\n","It's now just a short step to our core task, few-shot OpenQA. We just need to give up our beloved gold passage and instead try to retrieve the right passage or passages from a corpus. \n","\n","The first step is instantiating the ColBERT retriever and loading in an index. Our ColBERT retriever was initially trained on MS MARCO, and we have pre-indexed a collection of 100K documents that we know to be well-aligned with SQuAD and with the dataset used for the bake-off assessment. (See [the original system question](#Your-original-system-[3-points]) for tips on creating your own index.)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"VCt6-Oe2zTio"},"outputs":[],"source":["index_home = os.path.join(\"experiments\", \"notebook\", \"indexes\")"]},{"cell_type":"markdown","metadata":{"id":"NFYxJPpuYre0"},"source":["### ColBERT parameters"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5tnUU2UHYre0"},"outputs":[],"source":["if not os.path.exists(os.path.join(\"data\", \"openqa\", \"colbertv2.0.tar.gz\")):\n","    !mkdir -p data/openqa\n","    # ColBERTv2 checkpoint trained on MS MARCO Passage Ranking (388MB compressed)\n","    !wget https://downloads.cs.stanford.edu/nlp/data/colbert/colbertv2/colbertv2.0.tar.gz -P data/openqa/\n","    !tar -xvzf data/openqa/colbertv2.0.tar.gz -C data/openqa/"]},{"cell_type":"markdown","metadata":{"id":"cjZ1hZJnYre0"},"source":["If something went wrong with the above, you can just download the file https://downloads.cs.stanford.edu/nlp/data/colbert/colbertv2/colbertv2.0.tar.gz, unarchive it, and move the resulting `colbertv2.0` directory into the `data/openqa` directory."]},{"cell_type":"markdown","metadata":{"id":"QzRjL61eYre0"},"source":["### ColBERT index"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"jD0U5Outa9HU"},"outputs":[],"source":["if not os.path.exists(os.path.join(index_home, \"cs224u.collection.2bits.tgz\")):\n","    !wget https://web.stanford.edu/class/cs224u/data/cs224u.collection.2bits.tgz -P experiments/notebook/indexes\n","    !tar -xvzf experiments/notebook/indexes/cs224u.collection.2bits.tgz -C experiments/notebook/indexes"]},{"cell_type":"markdown","metadata":{"id":"61ergQcQYre0"},"source":["If something went wrong with the above, download the file https://web.stanford.edu/class/cs224u/data/cs224u.collection.2bits.tgz, unarchive it, and move the resulting `cs224u.collection.2bits` directory into the `experiments/notebook/indexes` directory (which you will probably need to create)."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"XtEGC6MyYre0"},"outputs":[],"source":["collection = os.path.join(index_home, \"cs224u.collection.2bits\", \"cs224u.collection.tsv\")\n","\n","collection = Collection(path=collection)\n","\n","f'Loaded {len(collection):,} passages'"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"BixmAYizYre0"},"outputs":[],"source":["index_name = \"cs224u.collection.2bits\""]},{"cell_type":"markdown","metadata":{"id":"83oO75Z1zTio"},"source":["Now we create our `searcher`:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"N6wYpChzYre1"},"outputs":[],"source":["with Run().context(RunConfig(experiment='notebook')):\n","    searcher = Searcher(index=index_name)"]},{"cell_type":"markdown","metadata":{"id":"dex5KPUTYre1"},"source":["### Search\n","\n","Now that the index is loaded, you can do searches over it. The index is limited, but retrieval is very solid!"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"-v2jKaR8Yre1"},"outputs":[],"source":["query = \"linguistics\"\n","\n","print(f\"#> {query}\")\n","\n","# Find the top-3 passages for this query\n","results = searcher.search(query, k=3) \n","\n","# Print out the top-k retrieved passages\n","for passage_id, passage_rank, passage_score in zip(*results):\n","    print(f\"\\t[{passage_rank}]\\t{passage_score:.1f}\\t {searcher.collection[passage_id]}\")"]},{"cell_type":"markdown","metadata":{"id":"w9n0_xwdYre1"},"source":["### Retrieval evaluation\n","\n","For more rigorous evaluations of the retriever alone, we can use Sucess@`k` defined relative to the SQuAD passages and answers. We say that we have a \"success\" if a passage in the top `k` retrieved passages contains any of the answers substrings, and Sucess@`k` is the percentage of such success cases. This is very heuristic (perhaps the answer string happens to occur somewhere in a completely irrelevant passage), but it can still be good guidance."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"fsNayHzdYre1"},"outputs":[],"source":["def success_at_k(examples, k=20):\n","    scores = []\n","    for ex in examples: \n","        scores.append(evaluate_retrieval_example(ex, k=5))\n","    return sum(scores) / len(scores)\n","        \n","    \n","def evaluate_retrieval_example(ex, k=20):    \n","    results = searcher.search(ex.question, k=k)\n","    for passage_id, passage_rank, passage_score in zip(*results):\n","        passage = searcher.collection[passage_id]\n","        score = has_answer([DPR_normalize(ans) for ans in ex.answers], passage)\n","        if score:\n","            return 1\n","    return 0"]},{"cell_type":"markdown","metadata":{"id":"k2U4v58dYre1"},"source":["Here is Sucess@20 for the SQuAD dev set:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"J2oEmspeYre1"},"outputs":[],"source":["%%time\n","if torch.cuda.is_available():\n","    # This will take a few hours on a CPU:\n","    print(success_at_k(squad_dev))\n","else:\n","    # This should be reasonably fast and yields the\n","    # same kind of result:\n","    print(success_at_k(dev_exs))"]},{"cell_type":"markdown","metadata":{"id":"teKobQM8Yre1"},"source":["## Zero-shot OpenQA with ColBERT retrieval\n","\n","We're now in a position to define a system that does our full few-shot OpenQA task. To get this started, we define just a version that doesn't include any SQuaD-training examples in the prompt. So this is really zero-shot OpenQA. (The homework asks you to move to the true few-shot setting.)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"jbn4BHr4Yre1"},"outputs":[],"source":["def build_zero_shot_openqa_prompt(question, passage, joiner=\"\\n\\n\"):\n","    title, context = passage.split(\" | \", 1)\n","    segs = [\n","        f\"Title: {title}\",\n","        f\"Background: {context}\",\n","        f\"Q: {question}\",\n","        \"A:\"\n","    ]\n","    return joiner.join(segs)    "]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Xjz7l_yQYre2"},"outputs":[],"source":["def evaluate_zero_shot_openqa(examples, joiner=\"\\n\\n\", gen_func=run_eleuther, batch_size=20):\n","    prompts = []\n","    gens = []\n","    for i in range(0, len(examples), batch_size):\n","        exs = examples[i: i+batch_size]\n","        results = [searcher.search(ex.question, k=1) for ex in exs]\n","        passages = [searcher.collection[r[0][0]] for r in results]\n","        ps = [build_zero_shot_openqa_prompt(ex.question, psg, joiner=joiner) \n","              for ex, psg in zip(exs, passages)]\n","        gs = gen_func(ps)       \n","        prompts += ps\n","        gens += gs\n","    return evaluate(examples, prompts, gens)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"21DfS0hHYre2"},"outputs":[],"source":["%%time\n","zero_shot_openqa_results = evaluate_zero_shot_openqa(dev_exs)\n","\n","print(zero_shot_openqa_results['macro_f1'])"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"MNhMla71Yre2"},"outputs":[],"source":["%%time\n","zero_shot_openqa_results_gpt3 = evaluate_zero_shot_openqa(dev_exs, gen_func=run_gpt3)\n","\n","zero_shot_openqa_results_gpt3['macro_f1']"]},{"cell_type":"markdown","metadata":{"id":"outblNTaYre2"},"source":["## Homework questions\n","\n","Please embed your homework responses in this notebook, and do not delete any cells from the notebook. (You are free to add as many cells as you like as part of your responses.)"]},{"cell_type":"markdown","metadata":{"id":"R55lgmnTzTip"},"source":["### Few-shot OpenQA with no context [2 points]\n","\n","In the section [Open QA with no context](#Open-QA-with-no-context) above, we simply prompted our LM with a question string and looked at what came back. This is arguably unfair to the LM, since we didn't convey anything about our intentions.\n","\n","For a fairer assessment of what the LM alone can do, we should move to the few-shot setting by giving the model a few examples of what we have in mind. The idea here is to create prompts that look like this:\n","\n","   ```   \n","   Q: What is pragmatics?\n","\n","   A: The study of language use\n","\n","   Q: Who is Bert?\n","\n","   A: Bert is one of the Muppets.\n","\n","   Q: What was Stanford University founded?\n","   \n","   A: \n","   ```\n","   \n","This question asks you to write a function for creating such prompts, using SQuAD training examples, and a second function for evaluating this approach. The goal is to have a no context baseline for the other few-shot approaches we are considering.\n","\n","__Task 1___: Complete the function `build_few_shot_no_context_prompt` so that it builds prompts like the above. You can use `test_build_few_shot_no_context_prompt` to check that your function is returning prompts in the desired format.\n","\n","__Task 2__: Complete the function `evaluate_few_shot_no_context` so that you can evaluate this approach. You can use `test_evaluator` to check that your function is performing the desired kind of evaluation."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"jV7ySPA8zTip"},"outputs":[],"source":["def build_few_shot_no_context_prompt(question, train_exs, joiner=\"\\n\\n\"):\n","    \"\"\"No context few-shot OpenQA prompts.\n","\n","    Parameters\n","    ----------\n","    question : str   \n","    train_exs : iterable of SQuAD train examples. These can be \n","        obtained via a random sample \n","        from `squad_train` as defined above.\n","    joiner : str\n","        The character to use to join pieces of the prompt into \n","        a single str.\n","\n","    Returns\n","    -------\n","    str, the prompt\n","\n","    \"\"\"\n","    ##### YOUR CODE HERE\n","\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"JqYjeY8_zTip"},"outputs":[],"source":["def test_build_few_shot_no_context_prompt(func):\n","    train_exs = [\n","        SquadExample(0, \"T1\", \"Q1\", \"C1\", [\"A1\"]),\n","        SquadExample(1, \"T2\", \"Q2\", \"C2\", [\"A2\"]),\n","        SquadExample(2, \"T3\", \"Q3\", \"C3\", [\"A3\"])]\n","    question = \"My Q\"\n","    result = func(question, train_exs, joiner=\"\\n\")\n","    expected = \"\"\n","    tests = [\n","        (1, \"\\n\", 'Q: C1\\nA: A1\\nQ: My Q\\nA:'),                \n","        (1, \"\\n\\n\", 'Q: C1\\n\\nA: A1\\n\\nQ: My Q\\n\\nA:'),\n","        (2, \"\\n\", 'Q: C1\\nA: A1\\nQ: C2\\nA: A2\\nQ: My Q\\nA:')]\n","    err_count = 0       \n","    for n_context, joiner, expected in tests:\n","        result = func(question, train_exs[: n_context], joiner=joiner)\n","        if result != expected:\n","            err_count +=1 \n","            print(f\"Error:\\n\\nExpected:\\n\\n{expected}\\n\\nGot:\\n\\n{result}\")    \n","    if err_count == 0:\n","        print(\"No errors detected in `build_few_shot_no_context_prompt`\")     "]},{"cell_type":"code","execution_count":null,"metadata":{"id":"cxKEIpXHzTiq"},"outputs":[],"source":["test_build_few_shot_no_context_prompt(build_few_shot_no_context_prompt)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"56kO5vE-zTiq"},"outputs":[],"source":["def evaluate_few_shot_no_context(\n","        examples,\n","        squad_train,\n","        batch_size=20,\n","        n_context=2,\n","        joiner=\"\\n\\n\",\n","        gen_func=run_eleuther):\n","    \"\"\"Evaluate a few-shot OpenQA with no context approach \n","    defined by `build_few_shot_no_context_prompt` and `gen_func`.\n","\n","    Parameters\n","    ----------\n","    examples : iterable of SQuAD train examples\n","        Presumably a subset of `squad_dev` as defined above.\n","    squad_train : iterable of SQuAD train examples\n","    batch_size : int\n","        Number of examples to send to `gen_func` at once.\n","    n_context : int\n","        Number of examples to draw from `squad_train`.\n","    joiner : str\n","        Used by `build_few_shot_open_qa_prompt` to join segments\n","        of the prompt into a single str.\n","    gen_func : either `run_eleuther` or `run_gpt3`\n","\n","    Returns\n","    -------\n","    dict as determined by `evaluate` above.\n","\n","    \"\"\"\n","    # A list of strings that you build and feed into `gen_func`.\n","    prompts = []\n","\n","    # A list of dicts that you get from `gen_func`.\n","    gens = []\n","\n","    # Iterate through the examples in batches:\n","    for i in range(0, len(examples), batch_size):\n","        # Sample some SQuAD training examples to use with\n","        # `build_few_shot_no_context_prompt` and `ex.question`,\n","        # run the resulting prompt through `gen_func`, and\n","        # add your prompts and results to `prompts` and `gens`.\n","\n","        ##### YOUR CODE HERE\n","\n","\n","\n","    # Return value from a call to `evalaute`, with `examples`\n","    # as provided by the user and the `prompts` and `gens`\n","    # you built:\n","    return evaluate(examples, prompts, gens)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"i0oYyNiOzTiq"},"outputs":[],"source":["def test_evaluator(func):\n","    examples = [SquadExample(0, \"T1\", \"Q1\", \"C1\", [\"A1\"])]    \n","    squad_train = [SquadExample(0, \"sT1\", \"sQ1\", \"sC1\", [\"sA1\"])] \n","    \n","    def gen_func(*prompts):\n","        return [{\n","            \"generated_answer\": \"Constant output\", \n","            \"generated_answer_tokens\": [\"Constant\", \"output\"], \n","            \"generated_answer_probs\": [0.1, 0.2]}]\n","    \n","    batch_size = 1    \n","    n_context = 1    \n","    joiner = \"\\n\"\n","    result = func(\n","        examples, \n","        squad_train, \n","        batch_size=1, \n","        n_context=1, \n","        joiner=joiner, \n","        gen_func=gen_func)\n","    expected_keys = {'em_per', 'examples', 'macro_f1'}\n","    result_keys = set(result.keys())     \n","    if expected_keys != result_keys:\n","        print(f\"Unexpected keys in result. \"\n","              f\"Expected: {expected_keys}; Got: {result_keys}\")\n","        return\n","    expected_ex_keys = {\n","        'f1', 'id', 'em', 'generated_answer_tokens', 'generated_answer_probs',\n","        'prediction', 'generated_answer', 'question', 'answers'}\n","    result_ex_keys = set(result[\"examples\"][0].keys())\n","    if expected_ex_keys != result_ex_keys:\n","        print(f\"Unexpected keys in result['examples']. \"\n","              f\"Expected: {expected_ex_keys}; Got: {result_ex_keys}\")\n","        return\n","    print(\"No errors detected in `evaluate_few_shot_open_qa`\")  "]},{"cell_type":"code","execution_count":null,"metadata":{"id":"MnZhwidzzTiq"},"outputs":[],"source":["test_evaluator(evaluate_few_shot_no_context)"]},{"cell_type":"markdown","metadata":{"id":"w2mx3Z4HYre2"},"source":["### Few-shot OpenQA [2 points]\n","\n","In the section [Few-shot QA](Few-shot-QA) above, we used SQuAD training examples to build prompts that we hope will help the model infer our intended semantics for the prompts themselves. When we moved to the open formulation of the problem, in [Open QA with ColBERT retrieval](Open-QA-with-ColBERT-retrieval), we forced the model to deal with prompts that lack these context clues. This is a \"zero-shot\" formulation of the problem. The goal of this homework problem is to improve that system so that it truly supports few-shot OpenQA.\n","\n","__Task 1__: Complete the function `build_few_shot_open_qa_prompt` so that it builds prompts from a question, a passage, and a sample of SQuAD training examples. You can use `test_build_few_shot_open_qa_prompt` to check that your function is returning prompts in the desired format.\n","\n","__Task 2__: Complete the function `evaluate_few_shot_open_qa` so that you can evaluate this approach. You can use `test_evaluator` from above to check that your function is performing the desired kind of evaluation.\n","\n","We will be checking only that the tests pass. We will not be evaluating the quality of the results you obtain using this code."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"HUuZ5l3gYre2"},"outputs":[],"source":["def build_few_shot_open_qa_prompt(question, passage, train_exs, joiner=\"\\n\\n\"):\n","    \"\"\"Few-shot OpenQA prompts.\n","\n","    Parameters\n","    ----------\n","    question : str\n","    passage : str\n","        Presumably something retrieved via search.\n","    train_exs : iterable of SQuAD train examples\n","        These can be obtained via a random sample from \n","        `squad_train` as defined above.\n","    joiner : str\n","        The character to use to join pieces of the prompt \n","        into a single str.\n","\n","    Returns\n","    -------\n","    str, the prompt\n","\n","    \"\"\"\n","    ##### YOUR CODE HERE\n","\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"vgeNwTu4Yre2"},"outputs":[],"source":["def test_build_few_shot_open_qa_prompt(func):\n","    train_exs = [\n","        SquadExample(0, \"T1\", \"Q1\", \"C1\", [\"A1\"]),\n","        SquadExample(1, \"T2\", \"Q2\", \"C2\", [\"A2\"]),\n","        SquadExample(2, \"T3\", \"Q3\", \"C3\", [\"A3\"])]            \n","    question = \"My Q\"    \n","    passage = \"Title | target passage\"    \n","    tests = [\n","        (1, \"\\n\", ('Title: T1\\nBackground: Q1\\nQ: C1\\nA: A1\\n'\n","                   'Title: Title\\nBackground: target passage\\nQ: My Q\\nA:')),\n","        (1, \"\\n\\n\", ('Title: T1\\n\\nBackground: Q1\\n\\nQ: C1\\n\\nA: A1\\n\\n'\n","                     'Title: Title\\n\\nBackground: target passage\\n\\nQ: My Q\\n\\nA:')),\n","        (2, \"\\n\", ('Title: T1\\nBackground: Q1\\nQ: C1\\nA: A1\\nTitle: T2\\n'\n","                   'Background: Q2\\nQ: C2\\nA: A2\\nTitle: Title\\n'\n","                   'Background: target passage\\nQ: My Q\\nA:'))]\n","    err_count = 0       \n","    for n_context, joiner, expected in tests:\n","        result = func(question, passage, train_exs[: n_context], joiner=joiner)\n","        if result != expected:\n","            err_count +=1 \n","            print(f\"Error:\\n\\nExpected:\\n\\n{expected}\\n\\nGot:\\n\\n{result}\")    \n","    if err_count == 0:\n","        print(\"No errors detected in `build_few_shot_open_qa_prompt`\")"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"lK991-2AYre3"},"outputs":[],"source":["test_build_few_shot_open_qa_prompt(build_few_shot_open_qa_prompt)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"DRhoMeEGYre3"},"outputs":[],"source":["def evaluate_few_shot_open_qa(\n","        examples,\n","        squad_train,\n","        batch_size=20,\n","        n_context=2,\n","        joiner=\"\\n\\n\",\n","        gen_func=run_eleuther):\n","    \"\"\"Evaluate a few-shot OpenQA approach defined by \n","    `build_few_shot_open_qa_prompt` and `gen_func`.\n","\n","    Parameters\n","    ----------\n","    examples : iterable of SQuAD train examples\n","        Presumably a subset of `squad_dev` as defined above.\n","    squad_train : iterable of SQuAD train examples\n","    batch_size : int\n","        Number of examples to send to `gen_func` at once.\n","    joiner : str\n","        Used by `build_few_shot_open_qa_prompt` to join segments\n","        of the prompt into a single str.\n","    gen_func : either `run_eleuther` or `run_gpt3`\n","\n","    Returns\n","    -------\n","    dict as determined by `evaluate` above.\n","\n","    \"\"\"\n","    # A list of strings that you build and feed into `gen_func`.\n","    prompts = []\n","\n","    # A list of dicts that you get from `gen_func`.\n","    gens = []\n","\n","    # Iterate through the examples in batches:\n","    for i in range(0, len(examples), batch_size):\n","        # Use the `searcher` defined above to get passages\n","        # using `ex.question` as the query, and use your\n","        # `build_few_shot_open_qa_prompt` to build prompts.\n","\n","        ##### YOUR CODE HERE\n","\n","\n","\n","    # Return value from a call to `evalaute`, with `examples`\n","    # as provided by the user and the `prompts` and `gens`\n","    # you built:\n","    return evaluate(examples, prompts, gens)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"beEyy_eOYre3"},"outputs":[],"source":["test_evaluator(evaluate_few_shot_open_qa)"]},{"cell_type":"markdown","metadata":{"id":"oXeQzplkYre3"},"source":["### Answer scoring [2 points]\n","\n","We have so far been assuming that the top-ranked passage retrieved by ColBERT should be used in the prompt and that the single answer returned by the LM is our prediction. It may be possible to improve on this by scoring answers using the ColBERT scores and the probabilities returned by the LM. This question asks you to explore a basic approach to such scoring. The core scoring function:\n","\n","$$\n","\\textbf{score}_{\\text{prompt-func}}(\\textrm{answer}, \\textrm{passage}, \\textrm{question}) = \n","P(\\textrm{passage} \\mid \\textrm{question}) \\cdot \n","P(\\textrm{answer} \\mid \\text{prompt-func}(\\textrm{question}, \\textrm{passage}) ) \n","$$\n","\n","where we estimate the two conditional probabilities as follows:\n","\n","* $P(\\textrm{passage} \\mid \\textrm{question})$ is defined only for the top $k$ passages and defined by the softmax of the top $k$ scores returned by the retriever.\n","\n","* $P(\\textrm{answer} \\mid \\text{prompt-func}(\\textrm{question}, \\textrm{passage}))$ is simply the product of the per-token probabilities of the generated answer given the prompt determined by $\\text{prompt-func}(\\textrm{question}, \\textrm{passage})$. These values can be extracted from the return values of both `run_eleuther` and `run_gpt3` using the key `\"generated_answer_probs\"`. (Your prompt function might of course have other arguments not represented here.)\n","\n","__Your task__: Implement this scoring function for an individual example. The two required pieces are `get_passages_with_scores` and `answer_scoring`. Starter code for each is below, and each has a unit test you can run to check your work.\n","\n","(With this implemented, it is easy to create a new prediction function that uses the $\\textrm{answer}$ from the highest-scoring $\\textrm{answer}/\\textrm{passage}$ pair as the prediction for input $\\textrm{question}$. You are not required to implement such a prediction function, but you might do this as part of [your original system](#Your-original-system-[3-points]).)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"m7PhfMNsYre3"},"outputs":[],"source":["def get_passages_with_scores(question, k=5):\n","    \"\"\"Pseudo-probabilities from the retriever.\n","\n","    Parameters\n","    ----------\n","    question : str\n","    k : int\n","        Number of passages to retrieve.\n","\n","    Returns\n","    -------\n","    passages (list of str), passage_probs (np.array)\n","\n","    \"\"\"\n","    # Use the `searcher` to get `k` passages for `questions`:\n","    ##### YOUR CODE HERE\n","\n","\n","\n","    # Softmax normalize the scores and convert the list to\n","    # a NumPy array:\n","    ##### YOUR CODE HERE\n","\n","\n","\n","    # Get the passages as a list of texts:\n","    ##### YOUR CODE HERE\n","\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"-vqCNOtMYre4"},"outputs":[],"source":["def test_get_passages_with_scores(func):\n","    question = \"What is linguistics?\"        \n","    passages, passage_probs = get_passages_with_scores(question, k=2)    \n","    if len(passages) != len(passage_probs):\n","        print(\"`get_passages_with_scores` should return equal length \"\n","              \"lists of passages and passage probabilities.\")\n","        return\n","    if len(passages) != 2:\n","        print(f\"`get_passages_with_scores` should return `k` passages. Yours returns {len(passages)}\")\n","        return\n","    if not all(isinstance(psg, str) for psg in passages):\n","        print(\"The first return argument should be a list of passage strings.\")\n","        return\n","    if not all(isinstance(p, (float, np.float32, np.float64)) for p in passage_probs): \n","        print(\"The second return argument should be a list of floats.\")\n","        return \n","    print(\"No errors detected in `get_passages_with_scores`\")"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"kfsb4pyHYre4"},"outputs":[],"source":["test_get_passages_with_scores(get_passages_with_scores)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"SwQVdCb6Yre4"},"outputs":[],"source":["def answer_scoring(passages, passage_probs, prompts, gen_func=run_eleuther):\n","    \"\"\"Implements our basic scoring strategy.\n","\n","    Parameters\n","    ----------\n","    passages : list of str\n","    passage_probs : list of float\n","    prompts : list of str\n","    gen_func : either `run_eleuther` or `run_gpt3`\n","\n","    Returns\n","    -------\n","    list of pairs (score, dict), sorted with the largest score first.\n","    `dict` should be the return value of `gen_func` for an example.\n","\n","    \"\"\"\n","    data = []\n","    for passage, passage_prob, prompt in zip(passages, passage_probs, prompts):\n","        # Run `gen_func` on [prompt] (crucially, the singleton list here),\n","        # and get the dictionary `gen` from the singleton list `gen_func`\n","        # returns, and then use the values to score `gen` according to our\n","        # scoring method.\n","        #\n","        # Be sure to use \"generated_answer_probs\" for the scores.\n","        ##### YOUR CODE HERE\n","\n","\n","\n","    # Return `data`, sorted with the highest scoring `(score, gen)`\n","    # pair given first.\n","    ##### YOUR CODE HERE\n","\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"JD7d8ucgYre4"},"outputs":[],"source":["def test_answer_scoring(func):\n","    passages = [\n","        \"Pragmatics is the study of language use.\", \n","        \"Phonology is the study of linguistic sound systems.\"]\n","    passage_probs = [0.75, 0.25]\n","    prompts = passages\n","    \n","    def gen_func(*prompts):\n","        return [{\n","            \"generated_answer\": \"Constant output\", \n","            \"generated_answer_tokens\": [\"Constant\", \"output\"], \n","            \"generated_answer_probs\": [0.1, 0.2]}]\n","    \n","    data = func(passages, passage_probs, prompts, gen_func=gen_func)\n","    \n","    if not all(len(x) == 2 for x in data):\n","        print(\"`answer_scoring` should return a list of pairs (score, gen)\")\n","        return \n","    if not isinstance(data[0][0], (float, np.float32, np.float64)):\n","        print(\"The first member of each pair in `data` should be a score (type `float`).\")\n","        return    \n","    if not isinstance(data[0][1], dict):\n","        print(\"The second member of each pair in `data` should be a dict \" \n","              \"created by running `gen_func` on a single example.\")\n","        return    \n","    if data[0][0] != max([x for x, y in data]):\n","        print(\"`answer_scoring` should sort its data with the highest score first.\")\n","        return \n","    \n","    print(\"No errors detected in `answer_scoring`\")"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"QWwwDzvBYre4"},"outputs":[],"source":["test_answer_scoring(answer_scoring)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"OUcdfU9SYre4"},"outputs":[],"source":["def answer_scoring_demo(question):\n","    \"\"\"Example usage for answer_scoring. Here we extract the top-scoring\n","    results, which can then be used in an evaluation.\"\"\"    \n","    passages, passage_probs = get_passages_with_scores(question)\n","    prompts = [build_zero_shot_openqa_prompt(question, psg) for psg in passages]\n","    data = answer_scoring(passages, passage_probs, prompts)\n","    # Top result:\n","    return data[0][1]"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"y1DGZOuVzTir"},"outputs":[],"source":["answer_scoring_demo(\"How long is Moby Dick?\")"]},{"cell_type":"markdown","metadata":{"id":"PmURWvoJYre4"},"source":["### Your original system [3 points]\n","\n","This question asks you to design your own few-shot OpenQA system. All of the code above can be used and modified for this, and the requirement is just that you try something new that goes beyond what we've done so far. \n","\n","Terms for the bake-off:\n","\n","* You can make free use of SQuAD and other publicly available data.\n","* The LM must be an autoregressive language model. No trained QA components can be used. Our list of preallowed models are those available via the OpenAI API whose names begin with \"text\" and the Eluether models \"gpt-neo-125M\", \"gpt-neo-1.3B\", \"gpt-neo-2.7B\", and \"gpt-j-6B\". If you would like to use a model outside of this set, please check with the teaching team first.\n","\n","Here are some ideas for the original system:\n","\n","* We have so far sampled randomly from the SQuaD train set to create few-shot prompts. One might instead sample passages that have some connection to the target question.\n","\n","* We have used actual SQuAD training examples to build contexts. These might be different in meaningful ways from the passages in our corpus. An alternative is to use the SQuAD question–answer pairs to retrieve passages that contain the answer and use the resulting question–answer–passage triple when building prompts.\n","\n","* There are a lot of parameters to our LMs that we have so far ignored. Exploring different values might lead to better results. The `temperature` parameter is highly impactful for our task.\n","\n","* We have distributed a fixed index of 100K passages. These cover SQuAD plus our bake-off data, but there might still be value in creating a different/expanded index. There is starter code for indexing data with ColBERT [here](https://github.com/stanford-futuredata/ColBERT/blob/new_api/docs/intro.ipynb).\n","\n","* [Khattab et al. (2021a)](https://aclanthology.org/2021.tacl-1.55/) fine-tune the retriever through a handful of successive rounds, using weak supervision from the QA dataset. This is an ambitious direction that could quickly build to an original project, as the role of retriever training is under-explored so far in the context of few-shot OpenQA.\n","\n","* In our \"Answer scoring\" question, we don't normalize scores by answer length. Such normalization might be fairer to long answers and so seems worth adding.\n","\n","* Our \"Answer scoring\" question is inspired by the Retrieval Augmented Generation (RAG) model of [Lewis et al. 2020](https://arxiv.org/abs/2005.11401). Their model fully marginalizes over $k$ retrieved passages to create a proper model of $P(\\textrm{answer} \\mid \\textrm{question})$. Implementing this requires having the probabilities for the prompts. For GPT-3, these can be obtained with `echo=False`, which will lead you to have to make changes to the output processing of `run_gpt3`. For the Eleuther models, one needs to do another call to the model forward function. Here is some starter code that could be used to begin modifying `run_eleuther`:\n","\n","   ```\n","    prompt_logits = eleuther_model(prompt_ids).logits                \n","    prompt_probs = prompt_logits.softmax(-1)                                   \n","    prompt_probs = torch.gather(prompt_probs, 2, prompt_ids[:, :, None]).squeeze(-1)\n","    prompt_probs = [list(prompt_prob.numpy()) for p in prompt_probs]\n","   ```\n","\n","__Original system instructions__:\n","\n","In the cell below, please provide a brief technical description of your original system, so that the teaching team can gain an understanding of what it does. This will help us to understand your code and analyze all the submissions to identify patterns and strategies. \n","\n","We also ask that you report the best macro F1 score your system got during development on `dev_exs` [as defined above](#SQuAD-dev-sample), just to help us understand how systems performed overall.\n","\n","Please review the descriptions in the following comment and follow the instructions."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"D9pfDIrPYre4"},"outputs":[],"source":["# PLEASE MAKE SURE TO INCLUDE THE FOLLOWING BETWEEN THE START AND STOP COMMENTS:\n","#   1) Textual description of your system.\n","#   2) The code for your original system.\n","#   3) The score achieved by your system in place of MY_NUMBER.\n","#        With no other changes to that line.\n","#        You should report your score as a decimal value <=1.0\n","# PLEASE MAKE SURE NOT TO DELETE OR EDIT THE START AND STOP COMMENTS\n","\n","# NOTE: MODULES, CODE AND DATASETS REQUIRED FOR YOUR ORIGINAL SYSTEM\n","# SHOULD BE ADDED BELOW THE 'IS_GRADESCOPE_ENV' CHECK CONDITION. DOING\n","# SO ABOVE THE CHECK MAY CAUSE THE AUTOGRADER TO FAIL.\n","\n","# START COMMENT: Enter your system description in this cell.\n","# My peak score was: MY_NUMBER\n","if 'IS_GRADESCOPE_ENV' not in os.environ:\n","    pass\n","\n","# STOP COMMENT: Please do not remove this comment."]},{"cell_type":"markdown","metadata":{"id":"zS3G0Ss7Yre4"},"source":["## Bake-off [1 point]\n","\n","For the bake-off, you simply need to be able to run your system on the file \n","\n","```data/openqa/cs224u-openqa-test-unlabeled.txt```\n","\n","The following code should download it for you if necessary:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"32yVz-ewzTis"},"outputs":[],"source":["if not os.path.exists(os.path.join(\"data\", \"openqa\", \"cs224u-openqa-test-unlabeled.txt\")):\n","    !mkdir -p data/openqa\n","    !wget https://web.stanford.edu/class/cs224u/data/cs224u-openqa-test-unlabeled.txt -P data/openqa/"]},{"cell_type":"markdown","metadata":{"id":"SDX9uXGfzTis"},"source":["If the above fails, you can just download https://web.stanford.edu/class/cs224u/data/cs224u-openqa-test-unlabeled.txt and place it in `data/openqa`.\n","\n","This file contains only questions. The starter code below will help you structure this. It writes a file \"cs224u-openqa-bakeoff-entry.json\" to the current directory. That file should be uploaded as-is. Please do not change its name."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"GyttYJxoYre4"},"outputs":[],"source":["def create_bakeoff_submission():\n","    filename = os.path.join(\"data\", \"openqa\", \"cs224u-openqa-test-unlabeled.txt\")\n","    \n","    # This should become a mapping from questions (str) to response\n","    # dicts from your system.\n","    gens = {} \n","        \n","    with open(filename) as f:\n","        questions = f.read().splitlines()\n","    \n","    # `questions` is the list of questions you need to evaluate your system on.\n","    # Put whatever code you need to in here to evaluate your system.\n","    # All you need to be sure to do is create a list of dicts with at least\n","    # the keys of the dicts returned by `run_gpt` and `run_eleuther`.\n","    # Add those dicts to `gens`.\n","    #\n","    # Here is an example where we just do \"Open QA with no context\",\n","    # for an \"original system\" that would not earn any credit (since\n","    # it is not original!):\n","    for question in questions:\n","        gens[question] = run_eleuther([question])[0]\n","        \n","    # Quick tests we advise you to run: \n","    # 1. Make sure `gens` is a dict with the questions as the keys:\n","    assert all(q in gens for q in questions)\n","    # 2. Make sure the values are dicts and have the key we will use:\n","    assert all(isinstance(d, dict) and \"generated_answer\" in d for d in gens.values())\n","            \n","    # And finally the output file:\n","    with open(\"cs224u-openqa-bakeoff-entry.json\", \"wt\") as f:\n","        json.dump(gens, f, indent=4)    "]},{"cell_type":"code","execution_count":null,"metadata":{"id":"kYcuKcDXzTis"},"outputs":[],"source":["create_bakeoff_submission()"]}],"metadata":{"accelerator":"GPU","colab":{"provenance":[],"toc_visible":true,"collapsed_sections":["axUPfnNmYrer","slqK_3NEzTil","SgXgw4jezTil","kuVwpDEhzTim","NFYxJPpuYre0","QzRjL61eYre0","dex5KPUTYre1","w9n0_xwdYre1","w2mx3Z4HYre2","PmURWvoJYre4"]},"interpreter":{"hash":"a99ac6d2deb03d0b7ced3594556c328848678d7cea021ae1b9990e15d3ad5c49"},"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.7"}},"nbformat":4,"nbformat_minor":0}