Augmented Reality Chatbot

7 min readMar 11, 2021

Hello everyone, Dev here. In this blog I will be describing an Augmented Reality enabled chat bot.

Chat bots have started to become a necessity for every company and institute in order to optimize and automate the user/client interaction.

There are many types of chat bots present like e-commerce chat bots, healthcare chat bots, etc.

What we have done here is created a chat bot for the education domain. The chat bot is enabled with Augmented Reality and other basic bot functionalities in order to provide a pleasant experience to the user.

The use-case implemented by us mainly focuses on simplifying the visualization and understanding process in education.

Students sometimes find it difficult to follow and understand topics just by looking at the diagram or attending lectures. This is where the bot comes into picture. The bot provides various functionalities that makes it comparatively easier for the students to get a firm grasp on the subject.

The various functionalities and its implementation in the chat bot for our specific use-case are discussed below.

1. User Interface

A user interface is basically a medium through which the user/human interacts with the machine/software. This medium can be a keyboard, a touch screen enabled device, a mouse, etc.

An easy to use and interactive user interface is what makes a software/application stand out from the rest.

The user interface of chat bot is available as a web page and an android application.

a. Web page

The web page is developed using the following front end technologies:

HTML: It stands for hypertext markup language which is a standard markup language used to write documents which are to be displayed on a web page.
CSS: It stands for Cascading Style Sheets. It describes the style in which the HTML elements will be displayed on a web page. It can control layout of multiple web pages at once and can save a lot of work and thereby optimize the development.
JavaScript: It is a programming used mostly in web development. A JavaScript runs in background and can update and manipulate the web page HTML and CSS dynamically.

>> Integration of Web page with chat bot.

The bot is configured in a jar file, which is package file format typically used to aggregate many Java class files and associated metadata and resources (text, images, etc.) into one file for distribution.
The jar file is executed in a terminal with some application specific parameters. The first parameter is the jar file name.
The second parameter is the mode in which the bot is exposed on the internet. By default it is set to rest API mode.
The third parameter is the name of a specific dialogue definition file(ddf), which is a “xml” file which defines all the tasks that the bot can implement(application/use-case specific).
The fourth parameter is the port number on which the bot will be exposed. It can be set to any available idle port on the host.
Once the bot is exposed, it can accessed using the url and port number of the host. This bot url has to updated in the a javascript file which is used to configure the bot url in order to display the bot on the Web page.
Once the javascript file is configured, accessing the bot url outputs a web page which can be used to interact with the bot instance.

b. Android Application

The android application is developed in Android Studio.
Android Studio is the official integrated development environment for Google’s Android operating system, built on JetBrains’ IntelliJ IDEA software and designed specifically for Android development.
The application provides a user interface to chat with the bot and implement various functionalities of the bot. The android application has Unity support which compliments the Augmented Reality functionalities of the application.

>> Integration of Android Application with chat bot

The jar file mentioned above is executed in a terminal and the bot url is obtained.
In the Android Application’s Login Screen, a user settings button is provided at the top right corner(for development phase). On clicking this button, a pop appears wherein we can configure the “Server Address” i.e. the bot url, “AR Image Reference Url” which is the server address where the content to be used for augmentation is stored and some other configurations.
An option to choose language is also provided in the user settings. After setting up appropriate configuration in the user settings, the user can simply login and use the chat interface to interact with the bot.

2. Augmented Reality

The augmented reality is implemented using Unity and is available in the Android Application of chat bot.

>>Setup for AR functionality

To create augmented reality content for a use-case, four files have to be created and modified accordingly. Along with which, all the images(2D) that are to be augmented must be saved in the content folder in html directory.
The first file to be created/modified is a “csv” file that stores the location/url of target images on the server. Target images are those images that are to be detected to augment content on. In the “Integration of Android Application with chat bot” section, setting up “AR Image Reference Url” was mentioned. The url for this “csv” file on the server is assigned to the “AR Image Reference Url”.
The second file to be created/ modified is a “csv” file that defines all the steps to be executed during AR mode of the application. The coordinates, scale, rotation and server of location of the images to be augmented are defined in this file. A Target Id is also defined which is use-case specific.
The third file to be created/modified is yet another “csv” file that provides a brief description of the use-case and the steps to be executed. It also shows an image of the target image and a tutorial video.
The fourth file to be created/modified is a “yml” file that creates the AR event and executes all the steps stored in the second “csv” file mentioned in the 3rd bullet under “Setup for AR functionality”.

3. Application specific functionalities

The use-case that our team has implemented is targeted towards students. Students can use the chat bot to get guidance on executing lab experiments and practicals.
The first feature is, when the bot loads for the first time, it provides the user with a fact about the use-case, which in our case is Electronics and IOT.
The second feature is, the user can send queries based on the use- case and receive answers in the chat.
The third feature is, a button named “Required Components” is provided in the chat, which asks the user to select from a list of components which are to be used in the experiments/practicals. Post selection, The details of the component, its image and a small video is shown in a card format to the user. This is implemented by defining a “csv” file wherein the details, links for image and video of all components are defined with a unique ID. This file is executed depending on the component selected by user.
The fourth feature is the AR mode. A “Select Expt” button is provided using which the user can select from a list of experiments available. Post selection, the “csv” file mentioned under 4th bullet in “Setup for AR Functionality” is executed and a brief description, target image of the experiment and a small video demo is provided to the user in a card format in chat. After which, the user is asked a query “Do you need DIY assistance?” If the user selects Yes and follows the next step, the user is redirected to the AR mode wherein the user has to focus his camera on the target experiment diagram and the chat bot executes the “yml” file mentioned under 5th bullet in “Setup for AR Functionality” which executes all the steps in “csv” file mentioned under 3rd bullet in “Setup for AR Functionality”, thereby augmenting required images as configured.
After completion of all steps, the chat bot return to the chat window and waits of user’s next action/query.

3. Results