Overview

In the Japanese task, given an input tweet, the task is to retrieve, from a pool of tweets, tweets that are suitable as responses for a dialogue system. Since Twitter contains utterances on a wide variety of topics, if we can extract reasonable tweets as responses, it will lead to the improvement in the language generation capability of dialogue systems.


Note that, in the Japanese task, Twitter is used; however, in the Chinese task, Weibo is used. See http://ntcir12.noahlab.com.hk/stc.htm for the details of the Chinese task.

Evaluation

The extracted tweets will be evaluated by human judges. The evaluation is done by the organizers. In the evaluation process, each extracted tweet is labeled with 0 (inappropriate), 1 (appropriate in some context), and 2 (appropriate) by multiple judges. The rate of appropriate answers and information retrieval (IR) related measures (graded relevance IR measures) will be used as evaluation metrics.

Dataset of Japanese task

The input data will be those randomly sampled from tweets in the year 2015. The pool of tweets (the target for extraction) is the randomly sampled tweet pairs (mention-reply pairs) in the year 2014. The size of the pool is just over one million; that is 500K pairs.


The following data will be provided from the organizers:

(1) Twitter data (by using their IDs) 1M in size

(2) Development data. Input samples and output samples annotated with reference labels. Here, the number of annotators is ten.


The data can be downloaded from https://github.com/mynlp/stc and the test data will be provided at the time of the formal run. Since the Twitter data are provided in the form of IDs, it will be necessary for the participants to crawl the data by themselves. Alternatively, the participants can purchase the data from NTT DATA Corporation. For those who want to purchase the data, please send an email to info<at>nazuki-oto.com, requesting the data for the NTCIR STC Japanese task.

Organizers

Ryuichiro Higashinaka, Nippon Telegraph and Telephone Corporation, Japan

Yusuke Miyao, National Institute of Informatics, Japan

Schedule of Japanese Task

  • Release of the Twitter data and starts accepting registration: Nov 01, 2015
  • Release of the development data: Nov 20, 2015
  • Registration deadline of STC Japanese task: Jan 15, 2016
  • Release of the test data: Feb 15, 2016
  • Formal run deadline: Feb 22, 2016
  • Distribution of evaluation results: Mar 10, 2016
  • Paper draft deadline: Mar 20, 2016
  • (brief review of the draft papers)
  • Camera ready deadline: May 1, 2016
  • NTCIR-12 conference: Jun 7-10 2016

Contact Us

Regarding the Japanese task, please contact the organizers of the Japanese task

  • Ryuichiro Higashinaka (higashinaka.ryuichiro<at>lab.ntt.co.jp)
  • Yusuke Miyao (yusuke<at>nii.ac.jp)