AI-Powered Question Generator to Support Automated Chatbot in Japanese

October 7, 2021

Overview

The client is a Japan-based leading global provider of integrated solutions in printing​, communications, security, packaging, décor materials, and electronics. They wanted to develop a question-generating AI in Japanese to support automated chatbots.

Key Features

  • Analyze Japanese instruction manuals
  • Automatically generate questions & answers
  • Power chatbot with pre-generated questions

Technical Stacks

  • CRF
  • LSTM
  • BERT
  • T5
  • GPT2

Project Overview

  • Team: 2 people
  • Market: Japan
gemvietnam

Background

AI chatbots are the latest solution to speed up customer services, marketing, sales, HR, supply chain, and IT help desks. However, developing an AI chatbot involves setting up questions and answers. This task might require lots of effort to analyze thousands of pages of instruction manuals for businesses with complex infrastructure like banking or logistics. Therefore, we want to shorten the process by training this AI model to analyze any Japanese instruction manuals. It will scan the document and automatically generate questions and answers accordingly. Then, businesses can use this output to develop their chatbot in any industry or any business function. 

 

Challenges

We wanted this AI to be able to analyze text from different domains. However, it would be labor-intensive to use a traditional machine learning model for each domain. Therefore, we needed to work on Transfer Learning, which can transfer knowledge from one context to another.  Transfer Learning is more complex when applied in NLP than in Visual Learning. We used some of the latest language models, including T5 (Text-to-Text Transfer Transformer) by Google, BERT, GPT2, etc. There has been little to no application of these models for question generation. Therefore, our team needed to spend more time researching and finding solutions. Meanwhile, Japanese is a sophisticated language, which also makes it more challenging.  

 

Solutions

The AI model needs to identify and select topically important sentences from the document. First, we get it to process the whole paragraph to identify sentences. Next, the AI processes each sentence with a POS Tagger. After that, it performs feature extraction, classifying sentences by their feature, selecting important sentences on which questions can be generated.  The AI will then identify question words with the selected questions and rearrange the sentence to generate questions.  Every week, the client reviews the AI for its usefulness and usability. Then, they give us feedback to improve the model. We are deploying this AI to analyze banking instruction manuals. After months of development, the AI has given more precise output. 

 

Impacts

Developing a chatbot is never simple. But we have strived to work on this AI to make it easier for businesses to build their chatbots in Japanese. Our goal is to make it work for any business function or industry soon.