arxiv
Facebook AI
2015.02
主要工作
- 构建数据集 bAbI-tasks 使用 QA 的方式评估阅读理解能力
dataset：babi
source：bAbI-tasks
Tasks
- answer 都是单词或者单词列表
- 无噪声，人类水准可以达到 100% 准确率
- 每个任务有 1000 个 question 用于训练，1000 个 question 用于测试
| # | Task | Class name |
|---|---|---|
| 1 | Basic factoid QA with single supporting fact | WhereIsActor |
| 2 | Factoid QA with two supporting facts | WhereIsObject |
| 3 | Factoid QA with three supporting facts | WhereWasObject |
| 4 | Two argument relations: subject vs. object | IsDir |
| 5 | Three argument relations | WhoWhatGave |
| 6 | Yes/No questions | IsActorThere |
| 7 | Counting | Counting |
| 8 | Lists/Sets | Listing |
| 9 | Simple Negation | Negation |
| 10 | Indefinite Knowledge | Indefinite |
| 11 | Basic coreference | BasicCoreference |
| 12 | Conjunction | Conjunction |
| 13 | Compound coreference | CompoundCoreference |
| 14 | Time manipulation | Time |
| 15 | Basic deduction | Deduction |
| 16 | Basic induction | Induction |
| 17 | Positional reasoning | PositionalReasoning |
| 18 | Reasoning about size | Size |
| 19 | Path finding | PathFinding |
| 20 | Reasoning about agent’s motivation | Motivations |
- Single Supporting Fact (1)：用 1 个 supporting fact 就可以回答的 question
- Two or Three Supporting Facts (2, 3)：用 2 个或者 3 个 supporting facts 可以回答的 question
- Two or Three Argument Relations (4, 5)：回答 question 的关键是能分清楚 subjects 和 objects
- Yes/No Questions (6)：回答 Yes/No 类型的问题的能力
- Counting and Lists/Sets (7, 8)：计数、回答列表物件的能力
- Simple Negation and Indefinite Knowledge (9, 10)：对否定句以及可能性的理解能力
- Basic Coreference, Conjunctions and Compound Coreference (11, 12, 13)：对代指、多 subject 的理解能力
- Time Reasoning (14)：时间推理能力
- Basic Deduction and Induction (15, 16)：推理归纳能力
- Positional and Size Reasoning (17, 18)：空间推理
- Path Finding (19)：根据地理位置进行路径推理
- Agent’s Motivations (20)：主体动机推理
提供了 3 种版本
- 正常英文
- Hindi 文
- 顺序打乱的英文
Example 
训练
各种 baseline 
- AM：adaptive memory
- NG：N-grams
- NL：nonlinear matching function
读后感
就是提出了 babi-tasks 数据集，20 个 tasks 构造的思路很有意思，涵盖了语言理解能力的各种方向