RI_pyterrier_1_etud2022.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/N-nolwenn/RITAL/blob/main/RI_pyterrier_1_etud2022.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sKwxrNgYOZAw"
      },
      "source": [
        "# Recherche d'information : librairie PyTerrier\n",
        "\n",
        "Dans cette partie, nous nous intéressons à la librairie [PyTerrier](https://pyterrier.readthedocs.io/en/latest/#) qui permet de mettre en place diverses briques d'un moteur de recherche.\n",
        "PyTerrier est basée sur [Terrier](http://terrier.org/) qui est un moteur de recherche développé en Java.\n",
        "\n",
        "Nous allons voir : \n",
        "*   l'installation et la configuration\n",
        "*   l'indexation d'une collection\n",
        "*   l'accès à l'index\n",
        "*   l'évaluation d'un moteur de recherche\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hwuWey_dPRa5"
      },
      "source": [
        "## Installation ete configuration\n",
        "\n",
        "Après l'installation de la librairie, il est nécessaire d'initialiser PyTerrier pour importer les fichiers jar et démarrer la machine virtuelle associée; "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6BVCdIzcOCkc",
        "outputId": "bbddcf18-7795-4d4c-aca9-ed0f6105ca18",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "!pip install python-terrier\n",
        "\n",
        "import pyterrier as pt\n",
        "if not pt.started():\n",
        "  pt.init(boot_packages=[\"com.github.terrierteam:terrier-prf:-SNAPSHOT\"])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting python-terrier\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/37/bd/77d14471ff175b648369444715b7be7b49226068683e4b797cc1c0073ffe/python-terrier-0.6.0.tar.gz (86kB)\n",
            "\r\u001b[K     |███▊                            | 10kB 15.6MB/s eta 0:00:01\r\u001b[K     |███████▌                        | 20kB 21.3MB/s eta 0:00:01\r\u001b[K     |███████████▎                    | 30kB 26.0MB/s eta 0:00:01\r\u001b[K     |███████████████                 | 40kB 20.9MB/s eta 0:00:01\r\u001b[K     |██████████████████▉             | 51kB 17.6MB/s eta 0:00:01\r\u001b[K     |██████████████████████▋         | 61kB 13.2MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▍     | 71kB 13.9MB/s eta 0:00:01\r\u001b[K     |██████████████████████████████▏ | 81kB 15.2MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 92kB 5.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from python-terrier) (1.19.5)\n",
            "Requirement already satisfied: pandas>=0.25.0 in /usr/local/lib/python3.7/dist-packages (from python-terrier) (1.1.5)\n",
            "Collecting wget\n",
            "  Downloading https://files.pythonhosted.org/packages/47/6a/62e288da7bcda82b935ff0c6cfe542970f04e29c756b0e147251b2fb251f/wget-3.2.zip\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from python-terrier) (4.41.1)\n",
            "Collecting pyjnius~=1.3.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ea/b1/e33db12a20efe28b20fbcf4efc9b95a934954587cd7aa5998987a22e8885/pyjnius-1.3.0-cp37-cp37m-manylinux2010_x86_64.whl (1.1MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1MB 19.1MB/s \n",
            "\u001b[?25hCollecting matchpy\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/2a/25/6b8fa5846476c2d56856a4926fda859b218656b14571ace76fbcd1d39986/matchpy-0.5.4-py3-none-any.whl (69kB)\n",
            "\u001b[K     |████████████████████████████████| 71kB 7.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: sklearn in /usr/local/lib/python3.7/dist-packages (from python-terrier) (0.0)\n",
            "Collecting deprecation\n",
            "  Downloading https://files.pythonhosted.org/packages/02/c3/253a89ee03fc9b9682f1541728eb66db7db22148cd94f89ab22528cd1e1b/deprecation-2.1.0-py2.py3-none-any.whl\n",
            "Collecting chest\n",
            "  Downloading https://files.pythonhosted.org/packages/18/66/b883b9a26cd2f777dd04b7eedc842d31ea1567b7709b049d46eca418501e/chest-0.2.3.tar.gz\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from python-terrier) (1.4.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from python-terrier) (2.23.0)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from python-terrier) (1.0.1)\n",
            "Collecting nptyping\n",
            "  Downloading https://files.pythonhosted.org/packages/8e/d5/3331d6fe411c0287ef4b028ded3355c4fc10956f4c0fbe5a4415fdd33dc8/nptyping-1.4.2-py3-none-any.whl\n",
            "Requirement already satisfied: more_itertools in /usr/local/lib/python3.7/dist-packages (from python-terrier) (8.8.0)\n",
            "Collecting ir_datasets>=0.3.2\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0a/9c/e35d3fa08b65cf9db740ea04a0b48fde16d94a093368ec6e097f3ccfd00b/ir_datasets-0.4.0-py3-none-any.whl (206kB)\n",
            "\u001b[K     |████████████████████████████████| 215kB 43.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from python-terrier) (2.11.3)\n",
            "Requirement already satisfied: statsmodels in /usr/local/lib/python3.7/dist-packages (from python-terrier) (0.10.2)\n",
            "Collecting ir_measures>=0.1.4\n",
            "  Downloading https://files.pythonhosted.org/packages/74/15/29146d203e05d63407402a8a254992e41352283ec586359de1d320b608a0/ir_measures-0.1.4.tar.gz\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.25.0->python-terrier) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.25.0->python-terrier) (2.8.1)\n",
            "Requirement already satisfied: six>=1.7.0 in /usr/local/lib/python3.7/dist-packages (from pyjnius~=1.3.0->python-terrier) (1.15.0)\n",
            "Requirement already satisfied: cython in /usr/local/lib/python3.7/dist-packages (from pyjnius~=1.3.0->python-terrier) (0.29.23)\n",
            "Collecting multiset<3.0,>=2.0\n",
            "  Downloading https://files.pythonhosted.org/packages/a8/12/813a649f5bc9801865dc6cda95b8f169f784d996322db192907ebe399064/multiset-2.1.1-py2.py3-none-any.whl\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from sklearn->python-terrier) (0.22.2.post1)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from deprecation->python-terrier) (20.9)\n",
            "Requirement already satisfied: heapdict in /usr/local/lib/python3.7/dist-packages (from chest->python-terrier) (1.0.1)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->python-terrier) (2021.5.30)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->python-terrier) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->python-terrier) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->python-terrier) (3.0.4)\n",
            "Collecting typish>=1.7.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/24/a7/83e450157d1613be0725821f8bd8aadab22217fa5dac4795dcfb9408be95/typish-1.9.2-py3-none-any.whl (45kB)\n",
            "\u001b[K     |████████████████████████████████| 51kB 6.0MB/s \n",
            "\u001b[?25hCollecting pyyaml>=5.3.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7a/a5/393c087efdc78091afa2af9f1378762f9821c9c1d7a22c5753fb5ac5f97a/PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636kB)\n",
            "\u001b[K     |████████████████████████████████| 645kB 56.1MB/s \n",
            "\u001b[?25hCollecting zlib-state>=0.1.3\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/69/05/95df36be761a5ff85a39b438d87937f02762c19fea95c0f10b679a259f14/zlib_state-0.1.3-cp37-cp37m-manylinux2010_x86_64.whl (72kB)\n",
            "\u001b[K     |████████████████████████████████| 81kB 8.4MB/s \n",
            "\u001b[?25hCollecting warc3-wet-clueweb09>=0.2.5\n",
            "  Downloading https://files.pythonhosted.org/packages/9f/c1/dd817bf57e0274dacb10e0ac868cb6cd70876950cf361c41879c030a2b8b/warc3-wet-clueweb09-0.2.5.tar.gz\n",
            "Collecting warc3-wet>=0.2.3\n",
            "  Downloading https://files.pythonhosted.org/packages/78/de/017a6bc2e3ba1ad912a08501f58414dd9e8503da1d6239aad548631777ad/warc3_wet-0.2.3-py3-none-any.whl\n",
            "Collecting lz4>=3.1.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e3/52/151c815a486290608e4dc6699a0cfd74141dc5191f8fe928e7d1b28b569e/lz4-3.1.3-cp37-cp37m-manylinux2010_x86_64.whl (1.8MB)\n",
            "\u001b[K     |████████████████████████████████| 1.8MB 52.9MB/s \n",
            "\u001b[?25hCollecting lxml>=4.5.2\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/30/c0/d0526314971fc661b083ab135747dc68446a3022686da8c16d25fcf6ef07/lxml-4.6.3-cp37-cp37m-manylinux2014_x86_64.whl (6.3MB)\n",
            "\u001b[K     |████████████████████████████████| 6.3MB 55.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: beautifulsoup4>=4.4.1 in /usr/local/lib/python3.7/dist-packages (from ir_datasets>=0.3.2->python-terrier) (4.6.3)\n",
            "Collecting trec-car-tools>=2.5.4\n",
            "  Downloading https://files.pythonhosted.org/packages/d1/e2/da4e895e5ad519f9f6aa464530dd482c7132c92a55cf178b8132e84b5c1d/trec_car_tools-2.5.4-py3-none-any.whl\n",
            "Collecting ijson>=3.1.3\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b3/0c/e3b7bf52e23345d5f9a6a3ff6de0cad419c96491893ab60cbbe9161644a8/ijson-3.1.4-cp37-cp37m-manylinux2010_x86_64.whl (126kB)\n",
            "\u001b[K     |████████████████████████████████| 133kB 64.3MB/s \n",
            "\u001b[?25hRequirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->python-terrier) (2.0.1)\n",
            "Requirement already satisfied: patsy>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from statsmodels->python-terrier) (0.5.1)\n",
            "Collecting pytrec-eval-terrier==0.5.1\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d0/f0/582f00ea388bc191d1f1278e5cfabc95be177480220a1af5f1231732f167/pytrec_eval_terrier-0.5.1-cp37-cp37m-manylinux2010_x86_64.whl (291kB)\n",
            "\u001b[K     |████████████████████████████████| 296kB 60.8MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->deprecation->python-terrier) (2.4.7)\n",
            "Collecting cbor>=1.0.0\n",
            "  Downloading https://files.pythonhosted.org/packages/9b/99/01c6a987c920500189eb74a291bd3a388e6c7cf85736bb6b066d9833315e/cbor-1.0.0.tar.gz\n",
            "Building wheels for collected packages: python-terrier, wget, chest, ir-measures, warc3-wet-clueweb09, cbor\n",
            "  Building wheel for python-terrier (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for python-terrier: filename=python_terrier-0.6.0-cp37-none-any.whl size=93214 sha256=ec2eb44fe6c8f44b59ea891ccc6d28a6083034e416f7c91d6e2730d15a1b3d1e\n",
            "  Stored in directory: /root/.cache/pip/wheels/9f/8d/91/58da79fd0a2850843b7e8c6097a52f9ff7aa85953cc9c8d27f\n",
            "  Building wheel for wget (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for wget: filename=wget-3.2-cp37-none-any.whl size=9675 sha256=2ef0465fa55f309da78118dd5a192b08173cf10ab9548e2a3378bf87d6a28a0d\n",
            "  Stored in directory: /root/.cache/pip/wheels/40/15/30/7d8f7cea2902b4db79e3fea550d7d7b85ecb27ef992b618f3f\n",
            "  Building wheel for chest (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for chest: filename=chest-0.2.3-cp37-none-any.whl size=7635 sha256=25924363b22623d36d503fbbae9c7b35cce24c886e9421ae753877660f0e1496\n",
            "  Stored in directory: /root/.cache/pip/wheels/f9/52/97/3f0eb272609dc1b7a20fb678a45003301b51f03cf766f6237f\n",
            "  Building wheel for ir-measures (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ir-measures: filename=ir_measures-0.1.4-cp37-none-any.whl size=40003 sha256=04dc733cc5fb7916cb67880768e568208e9c8fd0f116f42d6096fc94fc6da6ea\n",
            "  Stored in directory: /root/.cache/pip/wheels/71/c1/67/4dfa24e5ea066cc09694d001b2bc9ecbf9a3139d69847150a4\n",
            "  Building wheel for warc3-wet-clueweb09 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for warc3-wet-clueweb09: filename=warc3_wet_clueweb09-0.2.5-cp37-none-any.whl size=18922 sha256=2fdd499f8d7be282385bf740c39d3c200ed70ba9f45a47cfb8061b3f6958dfba\n",
            "  Stored in directory: /root/.cache/pip/wheels/09/da/a9/ec9816edf7f789eab3fea2e57abe37bf7d6ab65f8ef8ee7f31\n",
            "  Building wheel for cbor (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for cbor: filename=cbor-1.0.0-cp37-cp37m-linux_x86_64.whl size=51269 sha256=c2e827c27d2a15b6fe1a1e0648651f8584c2c46a432a7b0b5c1b9985cf73a470\n",
            "  Stored in directory: /root/.cache/pip/wheels/40/5c/a5/e6d629446a6a687ba328c55f1589234c29b99ef35b1a65dbaa\n",
            "Successfully built python-terrier wget chest ir-measures warc3-wet-clueweb09 cbor\n",
            "Installing collected packages: wget, pyjnius, multiset, matchpy, deprecation, chest, typish, nptyping, pyyaml, zlib-state, warc3-wet-clueweb09, warc3-wet, lz4, lxml, cbor, trec-car-tools, ijson, ir-datasets, pytrec-eval-terrier, ir-measures, python-terrier\n",
            "  Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "  Found existing installation: lxml 4.2.6\n",
            "    Uninstalling lxml-4.2.6:\n",
            "      Successfully uninstalled lxml-4.2.6\n",
            "Successfully installed cbor-1.0.0 chest-0.2.3 deprecation-2.1.0 ijson-3.1.4 ir-datasets-0.4.0 ir-measures-0.1.4 lxml-4.6.3 lz4-3.1.3 matchpy-0.5.4 multiset-2.1.1 nptyping-1.4.2 pyjnius-1.3.0 python-terrier-0.6.0 pytrec-eval-terrier-0.5.1 pyyaml-5.4.1 trec-car-tools-2.5.4 typish-1.9.2 warc3-wet-0.2.3 warc3-wet-clueweb09-0.2.5 wget-3.2 zlib-state-0.1.3\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/tqdm/std.py:658: FutureWarning: The Panel class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version\n",
            "  from pandas import Panel\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "terrier-assemblies 5.5  jar-with-dependencies not found, downloading to /root/.pyterrier...\n",
            "Done\n",
            "terrier-python-helper 0.0.5  jar not found, downloading to /root/.pyterrier...\n",
            "Done\n",
            "terrier-prf -SNAPSHOT  jar not found, downloading to /root/.pyterrier...\n",
            "Done\n",
            "PyTerrier 0.6.0 has loaded Terrier 5.5 (built by craigmacdonald on 2021-05-20 13:12)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nOC7UxlTP1IP"
      },
      "source": [
        "## Indexation d'une collection\n",
        "\n",
        "Il est possible d'indexer plusieurs formats de collection : format TREC, fichiers en texte brut ou en PDF, ou encore des Dataframe Pandas ([pour plus de détails](https://pyterrier.readthedocs.io/en/latest/terrier-indexing.html)).\n",
        "\n",
        "Un petit exemple à titre illustratif est foourni dans le code suivant :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sv9pe_KePd3e",
        "outputId": "9176a7c3-7409-4476-b404-7e52840de40d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# configuration de l'affichage\n",
        "pd.set_option('display.max_colwidth', 150)\n",
        "\n",
        "# le jeu de données au format DataFrame\n",
        "docs_df = pd.DataFrame([\n",
        "        [\"d1\", \"this is the first document of many documents\"],\n",
        "        [\"d2\", \"this is another document\"],\n",
        "        [\"d3\", \"the topic of this document is unknown\"]\n",
        "    ], columns=[\"docno\", \"text\"])\n",
        "\n",
        "# création de l'index\n",
        "indexer = pt.DFIndexer(\"./index_3docs\", overwrite=True)         # Définition du format de données (DFIndexer())\n",
        "index_ref = indexer.index(docs_df[\"text\"], docs_df[\"docno\"])    # Indexation des champs text et docno\n",
        "!ls -lh index_3docs/                                            # Affichage de l'index sauvegardé dans \"./index_3docs/\""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "total 36K\n",
            "-rw-r--r-- 1 root root    3 Jun 17 11:51 data.direct.bf\n",
            "-rw-r--r-- 1 root root   51 Jun 17 11:51 data.document.fsarrayfile\n",
            "-rw-r--r-- 1 root root    4 Jun 17 11:51 data.inverted.bf\n",
            "-rw-r--r-- 1 root root  344 Jun 17 11:51 data.lexicon.fsomapfile\n",
            "-rw-r--r-- 1 root root  249 Jun 17 11:51 data.lexicon.fsomaphash\n",
            "-rw-r--r-- 1 root root   24 Jun 17 11:51 data.meta.idx\n",
            "-rw-r--r-- 1 root root   48 Jun 17 11:51 data.meta.zdata\n",
            "-rw-r--r-- 1 root root 4.1K Jun 17 11:51 data.properties\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tajl4sILR3J0"
      },
      "source": [
        "De nombreux fichiers sont créés : index direct, index inverse, méta-données de l'index et de la configuration de l'indexation, etc...\n",
        "\n",
        "\n",
        "Il est également possible de modifier la configuration de l'indexation : [voir ici](https://pyterrier.readthedocs.io/en/latest/terrier-indexing.html#indexing-configuration).\n",
        "\n",
        "Pour chager un index existant en local :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ayENptR-TO4j"
      },
      "source": [
        "index = pt.IndexFactory.of(index_ref)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yd56Q1LgTI2_"
      },
      "source": [
        "Il est aussi possible de voir les statistiques de l'index. \n",
        "pour connaître toutes les fonctions d'interrogation, se référencer à la [javadoc](http://terrier.org/docs/current/javadoc/org/terrier/structures/Index.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FQAm27NnTtRj"
      },
      "source": [
        "# statistiques de la collection\n",
        "print(index.getCollectionStatistics().toString())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uGk9z47PTx3E"
      },
      "source": [
        "# statistiques du vocabulaire.\n",
        "# Nt : document frequency : nombre de documents contenant le terme\n",
        "# TF : term frequency : nombre d'occurences du terme\n",
        "# maxTF : nombre d'occurences maximal pour un document\n",
        "for kv in index.getLexicon():\n",
        "  print(\"%s (%s) -> %s (%s)\" % (kv.getKey(), type(kv.getKey()), kv.getValue().toString(), type(kv.getValue()) ) )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dq-9yKHnUyRZ"
      },
      "source": [
        "# focus sur un terme particulier\n",
        "index.getLexicon()[\"document\"].toString()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MTLZHWKWU78h"
      },
      "source": [
        "# récupère les statistiques de l'index inverse à partir d'un terme particulier\n",
        "pointer = index.getLexicon()[\"document\"]\n",
        "for posting in index.getInvertedIndex().getPostings(pointer):\n",
        "    print(posting.toString() + \" doclen=%d\" % posting.getDocumentLength())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DRXFRasZQ_xT"
      },
      "source": [
        "\n",
        "De plus, PyTerrier met à disposition [une collection de jeux de données pré-traités](https://pyterrier.readthedocs.io/en/latest/datasets.html).\n",
        "Dans ce qui suit, nous allons nous concentrer sur le jeu de données CORD19 qui recense des articles liés à la crise sanitaire Covid-19. Il est"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NqeBM8jBRAdX"
      },
      "source": [
        "import os\n",
        "\n",
        "cord19 = pt.datasets.get_dataset('irds:cord19/trec-covid')\n",
        "pt_index_path = './terrier_cord19'\n",
        "\n",
        "if not os.path.exists(pt_index_path + \"/data.properties\"):\n",
        "  # création de l'index. Utilisation de l'itérateur pour parcourir la collection\n",
        "  indexer = pt.index.IterDictIndexer(pt_index_path)\n",
        "\n",
        "  # on donne à l'index la fonction pour parcourir l'index avec l'itérateur  get_corpus_iter() \n",
        "  # On spécifie les champs à indexer et les meta-données à sauvegarder\n",
        "  index_ref = indexer.index(cord19.get_corpus_iter(), \n",
        "                            fields=('abstract',), \n",
        "                            meta=('docno',))\n",
        "\n",
        "else:\n",
        "  # dans le cas où l'index existe déjà\n",
        "  index_ref = pt.IndexRef.of(pt_index_path + \"/data.properties\")\n",
        "index = pt.IndexFactory.of(index_ref)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MMJ1_O1LV6Uf"
      },
      "source": [
        "**Exercice 1**\n",
        "\n",
        "Afficher les statistiques de l'index Cord19 et analyser statistiques du terme \"tv\" (pas trop fréquent pour question d'affichage)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z3MITxC-V3io"
      },
      "source": [],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CTweZ9B4XD96"
      },
      "source": [
        "## Recherche de documents à partir de l'index\n",
        "\n",
        "Pour effectuer une recherche dans l'index, il faut utiliser la fonction batchRetrieve qui prend en paramètre l'index et le modèle de pondération (tf, tf-idf, etc...). La liste des modèles supportés est disponible [ici](http://terrier.org/docs/current/javadoc/org/terrier/matching/models/package-summary.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0kKhE4BjWV9L"
      },
      "source": [
        "br = pt.BatchRetrieve(index, wmodel=\"Tf\")\n",
        "br.search(\"chemical reactions\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jWGGSR4AXjfq"
      },
      "source": [
        "On récupère alors un DataFrame dont les colonnes sont les suivantes : \n",
        "*   qid : identifiant de la requête. Par défaut, il s'agit de \"1\", puisqu'il s'agit de notre première et unique requête.\n",
        "*   docid : l'identifiant interne de Terrier pour chaque document\n",
        "*   docno : l'identifiant unique externe (chaîne de caractères) pour chaque document\n",
        "*   score : score des documents selon le modèle choisi (ici : fréquence totale des tf des termes de la requête dans chaque document)\n",
        "*   rank : rang du document dénotant l'ordre décroissant par score.\n",
        "*   query : la requête d'entrée"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QtXY5KnRZWe9"
      },
      "source": [
        "# autre exemple de modèle : TF-IDF\n",
        "tfidf = pt.BatchRetrieve(index, wmodel=\"TF_IDF\")\n",
        "tfidf.search(\"chemical reactions\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F8vyslA7Yfa7"
      },
      "source": [
        "On peut aussi fournir plusieurs requêtes grâce à un dataFrame. Pour interroger l'index, on applique la fonction transform() au BatchRetriever (br).\n",
        "pour plus de détails, voir [les propriétés des transformations](https://pyterrier.readthedocs.io/en/latest/transformer.html) ainsi que les [opérations possibles](https://pyterrier.readthedocs.io/en/latest/operators.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WewrczrCXA6X"
      },
      "source": [
        "import pandas as pd\n",
        "queries = pd.DataFrame([[\"q1\", \"document\"], [\"q2\", \"first document\"]], columns=[\"qid\", \"query\"])\n",
        "br.transform(queries)       # ou aussi : br(queries)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SEhSdCsqcycE"
      },
      "source": [
        "**Exercice 2**\n",
        "\n",
        "Ordonnancer les documents pour 3 requêtes : \"covid disease\", \"hospital\" et \"home\".\n",
        "La fonction d'ordonnacement devra être de la forme suivante : \n",
        "\n",
        "\n",
        "```\n",
        "0.4 * score_Bm25 + 0.6 * score_Dirichlet\n",
        "```\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZXSjd9iNeF56"
      },
      "source": [],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WvnkNzOZrC4m"
      },
      "source": [
        "**Excercice 3**\n",
        "\n",
        "Créez un ordonnanceur qui effectue les opérations suivantes :\n",
        "* obtient les 10 documents les mieux notés par fréquence de terme (wmodel=\"Tf\")\n",
        "* obtenir les 10 documents les mieux notés par TF.IDF (wmodel=\"TF_IDF\")\n",
        "* ré-ordonne uniquement les documents trouvés dans les DEUX paramètres de recherche précédents en utilisant BM25.\n",
        "\n",
        "Combien de documents sont récupérés par ce pipeline complet pour la requête \"chemical\"?\n",
        "\n",
        "Vérification : le document avec le docno \"37771\" devrait avoir un score de 12.426309 $ pour la requête \"chemical\"."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a_mSvL3qrkZd"
      },
      "source": [],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "raxeMrjrtSZh"
      },
      "source": [
        "## Reformulation de requêtes \n",
        "\n",
        "Il est également possible de mettre en place des pipelines de [reformulation de requêtes](https://pyterrier.readthedocs.io/en/latest/rewrite.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jNSsCc-btYF2"
      },
      "source": [
        "bo1 = pt.rewrite.Bo1QueryExpansion(index)\n",
        "dph = pt.BatchRetrieve(index, wmodel=\"DPH\")\n",
        "pipeline = dph >> bo1 >> dph\n",
        "pipeline.search(\"chemical reactions\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cC3oUh3qumUc"
      },
      "source": [
        "L'autre solution est de l'intégrer directement dans la fonction d'ordonnancement. Mais la requête reformulée n'est pas visible et la solution précédente fait prendre conscience de la pipeline faite par le système de RI (ranking >> reformulation >> ranking quand on utilise des modèles basés sur la relevance feedback. ou reformulation >> ranking sinon). Plus d'exemples [ici](https://pyterrier.readthedocs.io/en/latest/rewrite.html)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wbBSKWzctufW"
      },
      "source": [
        "# modèle DPH avant reformulation de requête\n",
        "pipelineQE = pt.BatchRetrieve(index, wmodel=\"DPH\", controls={\"qemodel\" : \"Bo1\", \"qe\" : \"off\"})\n",
        "pipelineQE.search(\"chemical reactions\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gjNvZw4GuHIN"
      },
      "source": [
        "# modèle DPH après reformulation de requête\n",
        "pipelineQE = pt.BatchRetrieve(index, wmodel=\"DPH\", controls={\"qemodel\" : \"Bo1\", \"qe\" : \"on\"})\n",
        "pipelineQE.search(\"chemical reactions\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3KUOCsIKZ-tK"
      },
      "source": [
        "### Evaluation d'un système de recherche d'information\n",
        "\n",
        "Pour évaluer un système de RI, il est nécessaire d'avoir un jeu de données constitué de requêtes et de jugements de pertinence. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jAXFBHTIYeP4"
      },
      "source": [
        "# exemple de 5 requêtes pour cord19\n",
        "cord19.get_topics(variant='title').head(5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NJN_uHkYaDea"
      },
      "source": [
        "# exemple de jugements de pertinence pour les 5 premières requêtes\n",
        "cord19.get_qrels().head(5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "em-jlm-Kaxpg"
      },
      "source": [
        "**Exercice 4**\n",
        "\n",
        "A partir des requêtes et des jugements de pertinence du jeu de données CORD19, Ecrire le code qui permet d'afficher les résultats de la première requête de Cord19. L'affichage fusionnera les colonnes retournées par le BatchRetriever et les colonnes des qrels (merge sur qid et docno pour rajouter label et iteration au tableau). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lcf-geD1aGHY"
      },
      "source": [],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qhx3qIFxb0_F"
      },
      "source": [
        "Il existe cependant une fonction qui permet de calculer l'efficacité de ces ordonnancements au travers des métriques d'évaluation (map, précision, rappel, ndcg, ...)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 79
        },
        "id": "l8W698iMavle",
        "outputId": "7b6a4a0e-00cd-4f65-e489-63c9a7e9d79d"
      },
      "source": [
        "pt.Experiment(\n",
        "    [tfidf],\n",
        "    cord19.get_topics(variant='title'),\n",
        "    cord19.get_qrels(),\n",
        "    eval_metrics=[\"map\", \"ndcg\"])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>name</th>\n",
              "      <th>map</th>\n",
              "      <th>ndcg</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>BR(TF_IDF)</td>\n",
              "      <td>0.180008</td>\n",
              "      <td>0.370795</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         name       map      ndcg\n",
              "0  BR(TF_IDF)  0.180008  0.370795"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 99
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JUhh3BHyvaZD"
      },
      "source": [
        "**Exercice 5**\n",
        "\n",
        "Réaliser une expérience comparant l'expansion de requêtes avec le modèle Bo1 et basé sur la KL-divergence; L'expérience est réalisée sur TREC CORD19 avec le modèle de référence BM25. Vous devrez construire des pipelines appropriées (plus de détails sur [l'expansion](https://pyterrier.readthedocs.io/en/latest/rewrite.html) et les [expérimentations](https://pyterrier.readthedocs.io/en/latest/experiments.html)).\n",
        "\n",
        "Quelles approches entraînent des augmentations significatives de NDCG et MAP ou autres métriques ?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cwyl1WBccB2U"
      },
      "source": [],
      "execution_count": null,
      "outputs": []
    }
  ]
}