In the Japanese task, given an input tweet, the task is to retrieve, from a pool of tweets, tweets that are suitable as responses for a dialogue system. Since Twitter contains utterances on a wide variety of topics, if we can extract reasonable tweets as responses, it will lead to the improvement in the language generation capability of dialogue systems.
Note that, in the Japanese task, Twitter is used; however, in the Chinese task, Weibo is used. See http://ntcir12.noahlab.com.hk/stc.htm for the details of the Chinese task.
The extracted tweets will be evaluated by human judges. The evaluation is done by the organizers. In the evaluation process, each extracted tweet is labeled with 0 (inappropriate), 1 (appropriate in some context), and 2 (appropriate) by multiple judges. The rate of appropriate answers and information retrieval (IR) related measures (graded relevance IR measures) will be used as evaluation metrics.
The input data will be those randomly sampled from tweets in the year 2015. The pool of tweets (the target for extraction) is the randomly sampled tweet pairs (mention-reply pairs) in the year 2014. The size of the pool is just over one million; that is 500K pairs.
The following data will be provided from the organizers:
(1) Twitter data (by using their IDs) 1M in size
(2) Development data. Input samples and output samples annotated with reference labels. Here, the number of annotators is ten.
The data can be downloaded from https://github.com/mynlp/stc and the test data will be provided at the time of the formal run. Since the Twitter data are provided in the form of IDs, it will be necessary for the participants to crawl the data by themselves. Alternatively, the participants can purchase the data from NTT DATA Corporation. For those who want to purchase the data, please send an email to info<at>nazuki-oto.com, requesting the data for the NTCIR STC Japanese task.
Ryuichiro Higashinaka, Nippon Telegraph and Telephone Corporation, Japan
Yusuke Miyao, National Institute of Informatics, Japan
Regarding the Japanese task, please contact the organizers of the Japanese task