Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions guides/tasks/adding_tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@ In this tutorial we’ll show you how to add a new task to `jiant`.

We’ll use a real task for this example: Senteval’s Tense task. Senteval’s tense task is a single sentence classification task with labels “PAST” or “PRES”, and it uses accuracy as its evaluation metric.

Adding Senteval’s Tense task will require touching three files:
Adding Senteval’s Tense task will require touching three files:
1. A (new) task file: `jiant/tasks/lib/senteval/tense.py`
2. The task retrieval library: `jiant/tasks/retrieval.py`
3. The evaluation library: `jiant/tasks/evaluate/core.py`

We’ll talk about these files in the following sections.
We’ll talk about these files in the following sections.

## 1. The Task file

Expand All @@ -21,7 +21,7 @@ Tasks files have 5 core components:
4. `DataRow` dataclass
5. `Batch` dataclass

In the following sections we’ll explain these components (and build up a working example in code blocks).
In the following sections we’ll explain these components (and build up a working example in code blocks).

#### 1. `Task` class

Expand Down Expand Up @@ -130,7 +130,7 @@ class Batch(BatchMixin):
segment_ids: torch.LongTensor
label_id: torch.LongTensor
tokens: list
```
```

Now that we've implemented required components of our SentevalTenseTask, we're ready to add it to the task retrieval library.

Expand Down Expand Up @@ -162,6 +162,10 @@ def get_evaluation_scheme_for_task(task) -> BaseEvaluationScheme:
return SimpleAccuracyEvaluationScheme()
```

## 4. Update the supported tasks documentation
Add your task to [`guides/tasks/supported_tasks.md`](../../guides/tasks/supported_tasks.md).

## Congratulations!
And that’s it. You’ve made all the core code changes required to include the `SentevalTenseTask` in your `jiant` experiments.

What's next? To tokenize and cache your `SentevalTenseTask` (which you shortnamed `senteval_tense`) for an experiement, you'll need to provide a task config `json` file pointing to your task's data:
Expand Down
1 change: 1 addition & 0 deletions guides/tasks/supported_tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

| Name | `task_name` | `jiant` | Downloader | `jiant_task_name` | Misc |
|---|---|:---:|:---:|---|---|
| [Argument Reasoning Comprehension](https://arxiv.org/abs/1708.01425) | arct | ✅ | | arct | [Github](https://github.com/UKPLab/argument-reasoning-comprehension-task) |
| Abductive NLI | abductive_nli | ✅ | ✅ | abductive_nli | |
| SuperGLUE Winogender Diagnostic | superglue_axg | ✅ | ✅ | superglue_axg | SuperGLUE |
| Acceptability Definiteness | acceptability_definiteness | ✅ | | acceptability_definiteness | Function Words |
Expand Down