Sentiment Analysis for Customer Experience With Python and Streamlit
Start leveraging on Vonage Messaging API to improve customer service such as collating review feedback from social media platforms such as Facebook.
Sentiment analysis mines insights from customers feedback using natural language processing techniques to determine whether feedback data is positive, negative or neutral. Business leverage on sentiment analysis to monitor product and brand sentiment and also understand what the customer needs.
This tutorial covers the following items:
- Initialise the Messages API Sandbox on Vonage
- Collect data from customers through Facebook
- Create a bot to handle feedback from customers on Facebook by:
Storing this data into a database or text file
Returning relevant responses to the customer
- Create a dashboard through Streamlit to understand the sentiments of customers feedback in real-time
- Measure customers sentiments with a positive, negative, neutral scale
- Analyze customers key pain points with visualization
Vonage API Account
To complete this tutorial, you will need a Vonage API account. If you don’t have one already, you can sign up today and start building with free credit. Once you have an account, you can find your API Key and API Secret at the top of the Vonage API Dashboard.
Creating a Facebook Bot
The Messages API Sandbox allows businesses to send and receive messages through various social channels, such as WhatsApp, Facebook, and Viber. It also allows businesses to send SMS and MMS through this API. In order to send messages through any of the external social channels, you’ll need a business account with each provider and have it connected to your Vonage APIs account.
For this tutorial we’ll be using Facebook, so make sure you have a Facebook account ready to test with.
To start creating our application, in the directory of your project run the following command to initialize the node project, and then install the third party libraries required:
The above commands will create
package.lock files as well as a
node_modules directory. Within the
package.json file you'll see the libraries we've just installed such as
vonage/server-sdk. An example of what you'll see is shown below:
When a message comes in from Facebook, Vonage will send an HTTP request to a preconfigured webhook URL. Your Node application should be accessible to the internet to receive it, so we recommend using Ngrok.
In your Terminal, run the following command to launch Ngrok on port
3000 (Please note, the forwarding URL changes each time ngrok is restarted so keep that in mind if you were to restart ngrok):
We now need to create an application in the dashboard. Make sure to add a name relevant to the purpose of the application, then toggle
Messages on as this application will need to make use of the Messages API. Then click the button "Generate public and private key". This will force your browser to download a
private.key file. Place this within the root directory of your project. Finally, click the
Generate new application button, this will redirect you to the application show page. Make a copy of the
Application ID shown on the page as this will also be needed in the code further below.
While you’re on this page, we need to define the webhooks URLs. A webhook is an extension of an API, but instead of your code requesting data from the API platform, Vonage sends the data to you. The data arrives in a web request to your application, which may be the result of an earlier API call (this type of webhook is also called a “callback”), such as an asynchronous request to the Messaging API. Webhooks are also used to notify your application of events such as an incoming message to be stored in a file or send an outgoing message to the app.
The URLs need to be updated in the dashboard as shown below (Be sure to replace
<ngrok URL> with your ngrok url):
- Inbound Webhooks:
- Status Webhooks:
Navigate back to the Messages Sandbox to whitelist your Facebook account with the Facebook Messenger Sandbox. Click
Add to sandbox and follow the instructions provided on the page.
Store your credentials into a
.env file that can be accessed through
Next, in the
index.js file, include modules that have been installed previously such as express, Vonage server-sdk as shown in the example below:
Then, we need to create a database with the filename
messages.db, add the following line into your
We now need to initialize a new Vonage object with our credentials consisting of the API key, API secret, application ID, along with the path and filename of the private.key. We also need to set the host URL of the API to point at the sandbox URL. In your
index.js file add the following code:
In the example above you’ll see
vonage.channel.send, which sends a message with the parameters of
from and message and in the event of an error, returns this error, and will log the error message.
Next, we’ll need to add the webhook in our application to listen for the path
/webhooks/inbound in the URL, which will receive messages from the Messages API whenever a Facebook message is received. It parses and obtains the messenger id of the sender from the body and also the text that is sent, together with the timestamp and insert in the response messages given either a condition of error if-else will be a successful understanding of the users' feedback.
If an error occurs, the message sent back to the sender will be “Sorry! Could you repeat?”. On the other hand, should the messenger retrieve and send the message by the customers to the app, it will be stored in
messages.db while a response of "Thanks for your feedback and review!" will be returned to the users. So add the following to your
Lastly, we need to add another listener for the
/webhooks/status endpoint, which will return the message "ok". We also need to add the listen functionality, which keeps the application running and listening as a webserver on port
3000. So at the bottom of your
index.js file add the following:
Now it’s time to test your application! First you need to run your web application, in your Terminal, run the following command:
Now, if you go to Facebook Messenger, find the Vonage Sandbox user and send it a message. You’ll receive a response in Messenger.
Analyze User’s Feedback
The data from the customers’ Facebook through Vonage sandbox is stored in the
messages.db file. In a simulated demonstration, a user critiqued the products and services of a bespoke firm and a sample of the data is recorded in the example script below.
We can see that each
message_uuid is unique for each message sent to the Facebook bot. The
to specifies the identity of the customers and bot respectively. The message contains the content which is in a type of text and is simply the feedback provided by the customer. To analyze this feedback, they will be visualized through the python Streamlit framework with libraries such as TextBlob, WordCloud, Matplotlib, and Pandas. Create a new file called
dashboard.py and add the following code into this new file:
Streamlit is an open-source app framework for Machine Learning and Data Science teams. TextBlob is a Python library for processing textual data. It provides a simple API for diving into common natural language processing (NLP) tasks. WordCloud is a technique to show which words are the most frequent among the given text. Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python. Pandas is a fast, powerful, flexible and easy to use open-source data analysis and manipulation tool, built on top of the Python programming language.
With all these tools, first, we open the data stored in messages.db, read and process every record of feedback collected from all users. We look for the messages/content and remove the timestamp because that is not needed for the analysis of sentiment in the feedback. The processed feedback will be stored in a list.
The example below carries out the instructions listed above. Add this to the bottom of your
The example below carries out the instructions listed above. Add this to the bottom of your
Next, we generate a word cloud to visualize the frequency of the words provided in the feedback by the users. We can see that certain keywords are much more prominent such as “terrible”, “long”, “bad” etc. Wordcloud simply counts how many times each word has appeared in all the sentences of feedback provided by customers. A better way would be to analyze the overall sentiment of the customer.
We can use TextBlob basic sentiment polarity analysis on the sentence that gives a relatively reliable estimate of the emotive experiences of the customer from their textual feedback and store in a new list of sentiment values. Now in your
dashboard.py add the following code:
Next, we will show the overall sentiment within a range of -1 to 1. For sentiment polarities above 0, the output will be green whereas, for those negative values which mean unhappiness, the sentiment will be shown as red on the webpage. Copy the following into your
Currently, the overall sentiment is -0.417 which suggests that overall a lot of the feedback is not in support of the products and services of the firm. We can see the feedback in a table with each of the feedback sentiments being shown. There are few sentiments of 0 value. This means that the analysis found this feedback to be neutral and cannot understand the negative connotations of the message. We can see that “This product cannot match up…” and “will only use this product if it is given to me free” would suggest negative sentiments yet, the result is neutral or positive. Hence, a more complex model to learn and train can be developed for sentiment analysis for this use case. Moreover, the overall sentiment analysis may not want to be just a simple average but rather potential more complex calculations or a weighted average.
The Streamlit web dashboard app can be started by initializing the following command. Streamli needs to be installed for the web dashboard to be set up. Other deployment methods in flask or Django work equally as well and can be explored.
In conclusion, this tutorial has shown how to set up and connect a Vonage Messages API to Facebook and collect feedback from customers which is then analyzed using sentiment analysis to do real-time monitoring of the sentiments of customers.