This article is written to educate the user about chatbots, types of chatbots, chatbot testing and chatbot automation testing using Botium. I have used Botium CLI to perform the automation testing of the chatbot.
What is a Chatbot
A chatbot is a piece of software designed to have interactions with humans. Chatbots are often referred to as digital assistants that mimic human capabilities. ChatBots can analyse user intent, handle requests, and provide quick, accurate responses.
Chatbots are powered by Artificial intelligence models. In today’s era, many industries are using chatbots. A few examples are listed below where chatbots are being used.
- Customer Support: Several organizations are using chatbots to assist customers. For example, Tech giants like Amazon, Apple, and Uber are using chatbots to handle customer support.
- E-commerce: Many e-commerce companies are using chatbots to guide customers in finding products and providing status updates.
- Banking and Finance: Chatbots are widely used in the banking and finance sector to handle queries like balance information, credit card-related queries, transaction history etc. We can find these chatbots in almost every banking app like HDFC, ICICI etc.
- Social Media: Several social media platforms are using chatbots to engage with users, answer queries etc. Some examples are Telegram, Instagram, LinkedIn etc.
Types of Chatbots
Chatbots can be mainly categorized as
1)Simple Chatbots
Simple chatbots, often known as rule-based bots, are known for their constrained functionality. They are tailored to predefined rules. Such types of chatbots are keyword-driven. This implies that the consumer can select from the alternatives shown by the bot as it asks questions based on predetermined possibilities until they receive answers to their inquiry.
These rule-based bots are difficult to train and scale, hence the least expensive.
2)Smart Chatbots/AI-based chatbots
AI-powered smart chatbots are designed to replicate customer behaviour that is very similar to actual humans. They can freely converse and understand language, intent, and sentiment. These AI-enabled chatbots learn from user actions and provide individualised conversations.
Training of AI-based bots is comparatively easy as they learn from historical data.
3)Hybrid Bots
They are a combination of simple and smart chatbots. Hybrid chatbots have some rule-based tasks, and they can understand intent and context as well.
How does a Chatbot work?
A chatbot combines several techniques and components to determine the users’ intention to communicate with end users. When the analysis is complete, the user receives the proper response.
Chatbots operate by using three classification methods:
1)Pattern Matching: Bots group similar types of text and classify using pattern-matching techniques and respond to clients appropriately.
2)Natural Language Understanding: It involves transforming language into structured data that a machine can understand.
3)Natural Language Processing: It is the application of computational techniques to do the analysis and synthesis of natural language and speech. Natural Language Processing (NLP) bots are designed to convert the text or speech inputs of the user into structured data.
In general when a chatbot receives a query that can be in any format like text, voice etc then the bot analyses the user’s intention based on the trained data set and tries to provide a more accurate and personalized response.
The Chatbot Testing Checklist
It is crucial that chatbots be trained with a significant amount of data and similarly tested properly with multiple inputs and data to test their effectiveness and accuracy. Here I will try to break down What should be kept in mind while doing chatbot testing.
1)Check ChatBot’s conversational flow: Does the chatbot understand what the user is asking? Are replies prompt and relevant? For example, if you ask “Book a ticket” on a travel company bot then it should ask relevant questions like source, destination, date of travel, etc.
2)Check Accuracy: Test if the answers from the boat are either accurate or nearly accurate.
3)Test Navigation: How easy is navigating through the chatbot conversation?
4)Validate Chatbot error management flow: How well does the chatbot respond in case of meaningless intent or expression? Does it cause crashes/failures or give some meaningful reply like “Sorry, not able to understand” or “Can you rephrase your sentence” or any meaningful sentence trained by the chatbot developer?
5)Test how intelligent your chatbot is: Suppose you are booking a ticket using a chatbot and you provide details like source, destination, date of travel, mode of travel, and no of people then how well does the chatbot remember all the details?
6)Verify Promptness: Check if the reply from the bot is quick or if it takes long pauses before answering.
7)Validate data formats in case of user inputs: A chatbot must be able to detect the proper data formats like email addresses, phone numbers, and zip codes. For example, if a user is providing input as “12345” for an email field it should be able to identify that the expected data format is different from the input data format.
8)UI Validation: Look out for lexical, and grammatical mistakes, and broken links. Chatbot UI should be clean and user-friendly.
Chatbot Testing Tools
There are many tools available for Chatbot testing. Some of them are listed below.
1)Botpress: This is an open-source tool for building, testing and deploying chatbots.
2)Botium: Botium is an open-source and very popular chatbot testing tool. Its API can be integrated and used with other popular test automation tools like web driver IO.
3)Selenium: Selenium is a very popular test automation framework for web-based applications. It can be used to test chatbots.
How To Automate Chatbot Testing
We can perform the automation testing of chatbots to save time and effort. Botium is a tool used for chatbot automation. Botium ecosystems have 4 components:
- Botium Core
- Botium CLI
- Botium Bindings
- Botium Box
We will be using Botium CLI since it is free.
Prerequisites
1)Install node.js and NPM on your machine. Botium needs it.
brew install node
sudo apt update
sudo apt install nodejs npm2)Download and install VS code editor.
How to do Chatbot automation testing?
1)Create a project directory and open it in the VS code terminal. Initialize it and provide the mandatory information.
npm init
npm install -g botium-bindings2)Install the botium connector for Webdriver IO Scripts.
Botium uses the Webdriver IO library to run conversations against a chatbot running on a website.
npm install botium-connector-webdriverio --save-dev3)Install the mocha framework
npm install mocha4)Install selenium bindings
An installed and running Selenium server is required. Below mentioned command downloads and installs the latest web drivers for common browsers.
npm install selenium 5)Install final dependencies
botium-bindings init mocha
npm installAfter successful installation open the package.json file and verify that all the dependencies are listed under dev dependencies.
Under the script section of the package.json add the following code:
"scripts": {
"selenium": "selenium-standalone install"
},Navigate to the open_bot directory and create an open.bot.js file. Write your logic in this js file to interact with the bot. For example, you visit a website and there is a chatbot window that opens with the click of the chatbot icon. Once this window opens then only you can shoot your queries to the chatbot.
module.exports = async (container, browser) => {
const startChat = await browser.$("#chatTogglerImg"); //Unique chatbotlocator
await startChat.waitForClickable({ timeout: 10000 });
await startChat.click();
console.log("Successfully opened chatbot");
}Inside the project directory, we see the convo file. It contains the order of the dialogue you want the chatbot to follow. We write conversation queries in the convo file. For more info read the official doc.
For a greeting scenario create a file named Greetings. convo.txt and add the bot and user conversation samples:
According to Botium official docs convo files can be written in a plain text file as the above screen attached or using Excel, JSON, CSV, or YAML.
Open botium.json and add the below script
"CONTAINERMODE": "webdriverio"
"WEBDRIVERIO_URL": Your_Bot_URL
"WEBDRIVERIO_OPENBOT": "./open_bot/open.bot",
"WEBDRIVERIO_INPUT_ELEMENT": "//input[@id='MyBot']" //Sample Xpath
"WEBDRIVERIO_OUTPUT_ELEMENT": "Sample Xpath"Provide the Bot URL value that you want to automate. Add open. bot.js file and provide valid Xpath locators for input and output elements of your chat window. For more information visit here. Refer Complete sample botium.json file
{
"botium": {
"Capabilities": {
"PROJECTNAME": "Botium",
"CONTAINERMODE": "webdriverio",
"ASSERTERS": [],
"WEBDRIVERIO_OPTIONS": {
"capabilities": {
"browserName": "chrome" ,
"goog:chromeOptions":{
"args":["--disable-notifications"]
}
},
"path": "/wd/hub"
},
"WEBDRIVERIO_URL": "Your Bot URL",
"WEBDRIVERIO_OPENBOT": "./open_bot/open.bot",
"WEBDRIVERIO_INPUT_ELEMENT": "//input[@id='chatInput']",
"WEBDRIVERIO_OUTPUT_ELEMENT": "//div[@class='ui-chatbox-msg-server']",
"WEBDRIVERIO_IGNOREWELCOMEMESSAGES": 0,
"WEBDRIVERIO_START_SELENIUM": true,
"WEBDRIVERIO_START_SELENIUM_OPTS": {
"drivers": {
"chrome": {
"version": "latest"
}
}
}
},
"Sources": {},
"Envs": {}
}
}Botium is free and open source. Once configured it is easy to use. You can add multiple convo & utterances files based on your requirements. Make sure that the spec folder path is correctly mentioned in the package.json file.
"botium": {
"convodirs": [
"spec/convo"
]}Execution of Botium Script
Now convo files are ready which contain our test cases to validate the Chatbot conversational flow, Dependencies installed and configured, spec folder path provided, and selenium is also configured. The script is already in place to launch a browser and navigate to the website in the open.bot.js file. To execute tests run “npm run test” in the terminal.
Once the execution is successful, we can see the results on the terminal window with the number of pass/fail test cases.
Challenges of Chatbot Testing
Chatbot testing has several challenges and limitations. A few of them are listed below:
1)It is sometimes difficult for quality analysts to ensure that a chatbot can handle ambiguous or vague inputs.
2)Chatbots are deployed across platforms and devices so it is very important to ensure consistent behaviour across all devices.
3)It is very difficult to find out the testing data as testing data should be different from Training data to avoid false results.
4)Chatbots may handle sensitive user information, Quality analysts need to look out for security issues and ensure that customer data is safe.