Learn Japanese (+ other languages) from Kanji writing, Kana, and Image Drawing with AI
This app is designed to help learning languages (mainly Japanese, but also English and Chinese). Kotoba-chan, the AI, will teach you how to write Kanji characters and challenge you by quizzes. There is also a mode (under research) where you can learn Kanji from drawing objects.
The app demonstrates state-of-the-art AI technologies for writing stroke recognition, image recognition, and character expression control.
Education Game for
- Learn Japanese Kanji – Chinese characters from writing strokes
[For Chinese, pinyins are available, but please note that meaning are based on Japanese Kanji] - Learn JP vocabulary from drawing sketch images
- Learn Kana: Hiragana, Katakana, Romanji
- Learn English
- Learn Drawing (?) (not designed for this but also possible)
Game Modes
- Challenge: [to be available]
- Learn Kanji: Write: Draw Kanji for given word(s)
- Learn Kanji: Meaning: Tell the meaning of a given Kanji
- Learn Kanji: Sound Read: Tell the reading sound of a given Kanji
- Draw-Word: Free Draw: [for mobile versions, only Android 8.1+ and iOS] Draw any image, Kotoba will guess what it is.
* This mode is currently unavailable on Windows and macOS. We found it didn’t work on some Windows PCs. May be due to GPU or some requirements by TensorFlow, we are still inspecting and solving this. You can help us voting the problem via “Report Bug” by telling us the details about your device - Draw-Word: What’s this?: A sketch is shown, you guess what it is
- Learn Kana: Kana-Romanji:A Kana is given, you pick a Romanji
- Learn Kana: Romanji-Kana: A Romanji is given, you pick a Kana
- Learn Kana: Kana-Kana: A Kana is given, you pick a Kana that matches
Notes
- This app is optimized for Android and iOS. Accuracy of canvas may drop on desktop/laptop PCs.
- “Free Draw” performance is mainly affect by CPU
- Currently, only N5 Kanji are available. “FreeDraw” supports 130 objects, “What’s This” supports 345 objects
Announcements
- [2021/07/11] Lowering Minimum API in the next version: For Android users, API level has been reduced to 23 (Android 6.0) to support more devices.
However, those with API lower than 27 (Android 8.1) cannot play mode “Free Draw” - To make the game more friendly to Japanese language beginners, Romanji-only and Kana-only will be available on NEXT UPDATE
New Model Announced, with LINE stickers
 |
 |
 |
[MODE] Learn Kanji: Write
In each question, draw Kanji for given word(s). A score will be obtained if you can answer.
Kotoba’s emotion changes depending on on your performance.
* FYI: In standard Kanji writing, strokes are drawn from Top-Bottom, Left-Right; wrong direction does not counted
Detailed calculation of score is not revealed (it depends on the Kanji character), but brief details are as follows:
- Stars: 3 stars for 80+, 2 stars for 60+, 1 star for 30+
- You get 100 when no mistake made and no hint used.
- ‘Clear’ does not affect maximum score, you can still get max score after several tries
- ‘Clear’ reduces hints and allowed mistakes on the next try. Hints and mistakes will affect the score more.
- The initial number of hint and mistakes allowed depends on the question (Kanji).
- No penalty to skip question.
- There is a limit on the number of times to skip question. This limit reset on level completed.
- Kanji is learned only if 3 stars obtained.
Buttons
- Clear: Clear canvas. After clear, you will get less hints and chances to make mistakes. But you can still get a full score.
- GiveUp: End the question. Negatively affect her mood (losing at different progress affect her mood differently).
- Hint: Get a hint for next stroke by scarifying some scores
- NextQuestion: Skip the question. No penalty. You cannot skip if you accept the question by asking for hinting or writing something.
Result Section
- The 1st line, from left to right: Kanji letter in blue, Hiragana with Romanji in () and Katakana with Romanji in (), Chinese PinYin in []
- The 2nd line, from left to right: JLPT level and English translation
[MODE] Learn Kanji: Read
Choose the correct answer. Kotoba’s emotion depends on your performance.
- Win-Lose affects ‘Happiness’
- Consecutiveness of Win-Lose affects ‘Confidence.’
- There is a button to refresh choices. Alternative texts will be used. Refreshing choice will slightly reduce the above consecutiveness
[MODE] Learn Kanji: Sound Read
Similar to the above mode but learn to read sound / spell.
[MODE] Draw-Word: Free Draw
You draw an image, Kotoba will guess what it is.
Please use this mode as a gimmick for fun now, don’t take it serious now.
- 5 best guesses are listed on the drop-down box.
- You can select the correct item, and click ‘Answer’ to tell her the correct answer. She will learn it and improve future guessing (this feature has been not been completed yet).
- ‘Show List’ is for checking what objects she know and may guess.
- There are example images for known objects. You can click the image to see more examples.
- So far, 130 types of objects have been learned.
- In the future, we may let you teach her new objects.
Result / List
- When Show Result: she will tell what the object look like
- When Show List: she will tell how your drawing image looks like the target object, in her perception
Read before use
- This mode is under testing: We are training a model that recognize drawing sketch.
A challenge mode (that you play and get score) based on this model will be later introduced. - This mode may not work on some device: We has not experienced this yet on Android 8.1+. But we heard that TensorFlow (technology behind this mode) did not work on some Android phones [please report us if you found, we will make it work]).
- The accuracy also depends on device: The model use threads to process drawing data within an allocated time. The more powerful your CPU is, and RAM at runtime, the more processing done for prediction.
- You may not like the resolution of drawing image on canvas. Sorry to say this is the limitation of the algorithm in use. A higher resolution canvas will take a lot of time to analyze image. So, we have to limit the image resolution.
- Drawing images as you can see from these examples don’t guarantee that objects will be recognized. For details, 1000 images are used to train the AI to recognize each object. Five images you see are only examples; it is only a small part of data, and the AI learned general patterns from 1000 images.
[MODE] Draw-Word: What’s this?
An image is shown, you guess what it is.
 |
 |
 |
[MODE] Kana: 3 Modes
Learn Hiragana and Katakana
 |
 |
 |
KOTOBA (ことば)
 |
「では、始めましょう！ (Let’s Get Start !) 」
|
In Learn Kanji Modes, her emotions depend on your performance (number of win/lose, consecutive correct/mistake, difficult of question per previous performance).
As on July 1, 2021, she has a total of 642 facial expressions. These expressions, including eye-blinking, are controlled by a novel algorithm that considers a set of emotion parameters, calculated from your game-play performance. Her expression has positive-negative sides, as well as serious-mischievous sides, worrying-confident sides.
Behind the name: “Kotoba (ことば/言葉)” means “word; language; speech”
Subtitle for what she speaks will be added later.
Some expressions ** Her expressions may not be perfect. As we are researching on building algorithm that can automatically choose expression based on parameters describing situations and emotions. Expression are chosen by a trained model instead of manually specifying when each emotion should be expressed.
If you don’t press the button, she may get angry.
However, but her emotion actually won’t changed by this. So please feel free to take time before proceeding.
OTHER INFORMATION
Setting Menu
- Volume Controls: You can mute BGM, voices, sound effects, or adjust their volumes
- JLPT Levels: You can set difficulty of Kanji based on Japanese-Language Proficiency Test Levels. Currently, only Kanji from N5 are available.
- Algorithm: This affect the way how Kanji is selected. Currently, only “Completely Random” is available. Future algorithms will include the one that put more weight on new Kanji. Or help you learning by selecting a small set of Kanji to learn and ensure memorization before moving forward.
- Guide: You can get instructions and details about the mode being played.
Coming Feature
- ore JLPT levels: We are translating. They will sure be added
- More languages: you can also learn Chinese
- Log-in system: to keep your gameplay record
- Challenge Mode: ALL MODES available so far are just for TRAINING, preparing for the real challenges!
Do you know?
- Here are technology used in this system (details provided for those who are interested in AI and programming)
- EfficientNet, a convolutional neural network is used for Sketch Recognition (Drawing Image – Word Conversion)
- Short Straw Algorithm is used for Kanji Stroke Recognition. Length of strokes, positions, and angles are used to classify if your writing is correct
- A novel algorithm is invented for controlling the character emotion and facial expression. Details are not revealed but existing algorithms behind this algorithm are K-Nearest Neighbor and K-mean Clustering.
- Quickdraw by Google is a the dataset in used
Notes
- You’re welcome to share this app (e.g., publish screenshots on social media), use it as teaching material or language skill assessment, or do research on it.
- For language school/institution: this app can be customized/modified for language skill test (it’s possible to collect test score while preventing any kind of cheating). For such purpose, please contact and send a request for such version.
Problems & Feedback: Pujana[dot]P[at]gmail.com
