Researchers from Google Brain have presented Meena – the new state-of-the-art chatbot that can chat almost about anything.
In their new paper, named “Towards a Human-like Open-domain Chatbot”, researchers describe their novel neural network model with 2.6 billion parameters that outperforms all existing conversational models. The new model is based on the popular Transformer seq2seq architecture, and its specific architecture was discovered using evolutionary neural architecture search, with the single objective of improving perplexity in the conversations. The evolutionary search algorithm produced an architecture in which there was a single Transformer encoder block and even 13 Transformer decoder blocks. Researchers hypothesize that Meena achieves higher conversational quality due to its powerful decoder module.
The model was trained using conversation threads which were organized as tree threads and samples contained 7 context switches between participants. The data used to train Meena, was coming from social media conversations and in total it was 341 GBs of text, which was previously filtered.
In order to properly evaluate the novel method, researchers decided to build a new metric which they named SSA – Sensibleness and Specificity Average. This metric was designed to capture most of the important features of natural conversations. Using this metric, researchers computed the scores for several other chatbots among Meena, as well as human performance.
The results showed that Meena is superior to other chatbots, achieving 79% of SSA score, 23 score points more than the second-ranked Mitsuku and only 7% less than the human level.
More details about the new model can be found in the paper or in the official blog post. Sample conversations were open-sourced and can be accessed on Github.