AI Expert Documentation, Download and Quick Start
The following documents how to use the Embody Digital AI Expert app in order to create an AI-driven talking avatar using a spreadsheet (via Google Sheets) or connected to either your existing Google DialogueFlow agent or IBM Watson Assistant.
2.Download and Quick Start
Download the Expert AI app on the Google Play Store:
The app will require a download (approximately 300 mb) in order to deliver all the information that is needed to run properly. We recommend turning on WiFi in order to speed up the download process. A default female avatar will be displayed on startup.
Pess and hold down microphone button, then ask a question like “Who are you?”, “What do you do?”, “How does this work?”, let go, and get the answer back from the avatar.
There are two different looks of the avatars: photorealistic and stylized (cartoony).
3.Configure your avatar
The app utilizes different types of avatars: photorealistic and stylized/cartoony ones.
By default the app uses a stylized/cartoony avatar. There are several ways to change the avatar: (i) downloading a new avatar, (ii) uploading a custom avatar, (iii) building an avatar from a selfie or face scan, or (iv) choosing an existing avatar residing on the device.
i. Downloading avatars
The AI Expert app comes with a set of avatars that can be downloaded (4 female and 4 male). Press the thumbnail of the avatar in order to use it. Once the avatar is downloaded, the app will automatically load it. In addition, the avatar name will appear on your list of avatars on your device. Avatars that need to be downloaded have a faded thumbnail appearance. Note that downloadable avatars cannot be configured with hair and clothing – only custom avatars from (iv) below can be customized with hair and clothing.
ii. Uploading custom avatars from Adobe Fuse
You can create and use any avatar that you make with Adobe Fuse. To do this, create your Adobe Fuse character, choose File -> Animate with Mixamo, then go to Mixamo and download the character using COLLADA(.dae) format in the T-pose with blendshapes. Put that .zip file on your mobile device, then press the UPLOAD AN AVATAR button. You will be asked what gender the avatar is in order to produce the proper text-to-speech voice. Please make sure that the name of the .zip file matches the name of the avatar. For example, if your avatar model name is FemaleAvatar1 make sure your zip file is named FemaleAvatar1.zip.
Details of that process can be found here:
iii. Creating an avatar from a selfie/photo
To create your own avatar, you can generate one based on a selfie or a head shot of someone else. Press the ‘Avatar’ button at the bottom of the main screen, then choose Male or Female and then take a selfie, or press the photo button in the upper left hand to retrieve a picture from the mobile device’s photo gallery.
Your photo will be converted into a photorealistic avatar that resembles your face or input photo. This avatar will be given default hair and clothing options, which can in turn be changed by selecting them from the ‘Hair & Clothes’ button.
You can change the background color or use an image for a background by pressing the ‘Background’ button.
Avatars that you create will appear on the bottom of the Avatar screen and can be used at any time.
iv. Using an existing avatar on the device
There is a default female and male photorealistic avatar (named customAvatar_defaultFemale and customAvatar_defaultMale) as well as a default female stylized/cartoony avatar (named Ella). Press the button with that name to use that avatar. Any avatars that you create with a selfie or any avatars downloaded will appear as buttons and may be selected.
4.Create your AI
You can use a number of different conversational AI systems to connect to your avatar, including:
(i) using a Google Sheets spreadsheet where you enter the questions in one column and the answers in another column,
(ii) by connecting it to an existing Google DialogFlow agent,
(iii) by connecting it to an existing IBM Watson Assistant,
(iv) by connecting to an existing RASA agent,
(v) by connecting you a Pandorabots agent,
(vi) by connecting the Facebook’s Blender chatbot or any other ParlAI agent.
You only need to choose one method. Other interfaces (such as Amazon Lex, Microsoft Bot, and others) will be added as the app is improved.
There is no inherit restriction on the type of conversational AI that can be used; in theory, any system that accepts text as input and returns conversational text as output could be used.
5.AI using a Google Sheets spreadsheet
The AI Expert utilizes the a conversational AI that can be created by entering questions that your avatar will respond to and the answers associated with those questions. For input, we use Google Sheets (an online spreadsheet application by Google). You will need to have a Google account in order to access Google Sheets.
A template that can be used for this purpose is located here:
Step 1. Make a copy of this Google Sheets by going to File -> Make a Copy
Step 2. Rename your Google Sheets copy.
Step 3. Replace the questions and answers with your own. Enter multiple questions for each answer in the same cell by pressing CTRL-ENTER. When the user talks to the app, the AI will find the best match according to both the questions and the matching answers and automatically make the avatar speak and gesture.
The third column is an identifier for the id of the utterance for later use. You can label it numerically (i.e. 1,2,3,4), or with a specific name (with no spaces) to associate with the answer (i.e. greeting, howmany, howold, etc.)
Avoid using any non-alphanumeric characters except for punctuation (period, question mark, exclamation marks.)
Your avatar will automatically speak and gesture based on the responses that you give. The avatar is programmed with a set of gestures and movements that correspond to the way most people communicate with each other, including head movements, facial expressions and arm gestures. When you are finished, turn on link sharing for the Google Sheets which will allow AI Expert to access your Google Sheets. Each Google Sheets has a unique id which can be accessed from the Share button in a browser. Press the green Share button in the upper right from a browser, then click on Get Shareable Link in the dialog box. This will turn on Google Sheets sharing and allow that Google Sheets ID to be accessed by the AI Expert application.
Step 4. Connect your avatar to your Google Sheets AI
In order for the avatar to use the AI that you have programmed, you need link your spreadsheet to the app.
The Google Sheets ID needs to be transferred to the AI Expert app. You can do this by downloading the Google Sheets app from the Google Play Store onto your Android device, open it, then from the three dots on the Google Sheets app select Copy Link which will store the Google Sheets ID into the copy buffer.
In the Expert AI app, press the AI button at the bottom of the screen, then in the Google sheets ID field paste in the link that was copied from the Google Sheets ID. Then press the Train AI button.
Press the back button (arrow) in the upper right to return to the main screen. You can then press and hold the microphone button, ask a question, and the AI will find a match to the proper answer, convert it to synthetic speech and automatically animate your avatar.
To make changes to your AI, simply add more questions or answers to your Google Spreadsheet, press the AI button and touch the Train AI button once again.
6.AI using a Google DialogFlow agent
7.AI using IBM Watson Assistant
The AI Expert can communicate with an existing IBM Watson Assistant agent. To do this, you need to load the IBM Watson credentials onto the mobile device, indicate the Watson Assistant Id on the AI screen, then your Watson Assistant agent will automatically communicate with the device.
Step 1. From the app, press the AI Settings button, and choose IBM Watson from the dropdown at the top of the view.
Step 2. Locate your IBM Watson Assistant Id from the IBM Watson Assistant control panel page. Press the three dots -> Settings to link to the API Details, then copy or record the Assistant Id. Put the Assistant Id into the ‘Watson Assistant Id’ field in the app (shown above.)
Step 3. Download your IBM Watson Service credentials that are associated with your Watson Assistant. Save the file downloaded and store it on your mobile device. From the AI Settings screen on the app, press the ‘Credentials’ button and locate the credentials file that you previously downloads. Once both the credentials are identified and the Watson Assistant Id is entered, the app will automatically connect to your Watson Assistant.
8.AI Using RASA
The AI Expert can communicate with an existing RASA agent via the REST api through a webserver. To do this, you need to run your RASA process using the ‘–enable-api’ option and configure the AI to use that webhook location.
Step 1. Press the AI button from the main screen, then choose ‘RASA’ from the AI Type dropdown menu.
Step 2. Press the button next to the RASA Webhook URL.
Step 3. Enter the location of the server and the webhook URL. Make sure to set either http:// or https:// and the proper IP address or hostname, including port. Press OK and you will either see a message saying that the connection was successful or not.
9.AI Using Pandorabots
You can connect to an existing Pandorabots account.
Step 1. Set up an Pandorabots agent from www.pandorabots.com
Step 2. Select your Pandorabots agent, then go the dashboard for that agent, and under the API option choose ‘Set Up’ or ‘View Settings’ if it has already been set up.
Step 3. Under the ‘Bot Keys’, choose Deploy. The Bot Key needs to be copied to the app.
Step 4. On the app, press the ‘AI’ button, select Pandorabots from the dropdown at the top, then put the public Bot Key into the Pubic Key field, then press ‘Connect’, You will see a message indicating whether the bot was successfully connected or not.
10.Using the avatar
To interact with your avatar, press the microphone button on the app, ask a question, then let go of the button. First, your voice will be transcribed into text via Google’s ASR service. Second, the text will be inputted into our AI, returning an answer, which will then be converted into voice and body language.
When your avatar speaks, the voice will be retrieved from the TTS service. If the exact response has already been spoken, the app will retrieve the voice response from the device locally. You can remove any prerecorded responses from the AI screen by pressing Reset TTS, which will be useful if you want to test out different TTS voices.
You can touch the screen and your avatar will look at the point in the screen that you are touching.
Changing the size and position of the avatar
You can modify the size of the avatar, the camera position, and your agents gaze direction by pressing the sizer button. This can be used to focus on the avatar’s face, or to position the character around backround imagery or A/R backgrounds. The sizer button will turn red, and you can touch and drag to position the avatar, pinch to make the avatar larger or smaller, or move the rotation slider to change the orientation of the character.
Long press on the sizer button (hold the button for a few seconds) in order to return the avatar to its original size.
Modifying your AI
You can iterate over your application by modifying your AI to change how your avatar responds to questions.
If you are using Google Sheets, pressing the AI button, then Train AI, then go back and ask your question again.
If you are using Google DialogFlow or IBM Watson, changes to your agent .or assistant will automatically appear on the avatar.
11.Adding stress and emotion to your avatar's responses
Avatar responses (generated from Google Sheets) can be modified to be more energetic via stress tags, or to be expressed with differing emotion using the emotion tags. In addition, there are a number of special gestures that can be triggered by explicitly indicating those gestures inline in the response utterance. Note that this feature is not available for Google DialogueFlow or IBM Watson responses (yet!)
Each word has a stress value going from 0 to 1 (defaults to 0). To change the stress of a particular word (for emphasis, for example). You can add the stress markup before a word like this: yes, I agree
where the  could be any decimal number between 0 and 1, such as .3 or .7. The stress will continue to affect all the words until changed, so to stress only the first word: yes,  I agree
You can use the following emotional tags:[happy][angry] [surprised] [sad] [agree] [disagree] [sarcasm] [neutral]
To add emotional tone to words, any of the emotional tags can be added inline to the utterance like so:[happy] That’s great![sad] I can’t believe you said that.[surprised] you said what?[angry] I thought I told you to clean up your room![agree] It was amazing.[disagree] no, that doesn’t sound right to me at all.
That’s a great idea [sarcasm] not![happy] Excellent! [neutral] now let’s get back to business.
[gesture:pointleft] Points to the left [gesture:pointright] Points to the right [gesture:pointdown] Points downward [gesture:pointup] Points upward [gesture:middlefinger] Shows the middle finger [gesture:thumbsup] Thumbs upgesture. [gesture:thumbsdown] Thumbs down gesture. [gesture:airquotes] Air quotes with two hands. [gesture:headexplode] Hands to the side of the head in an exploding gesture. [gesture:pray] Hands together in front in a praying gesture. [gesture:number1] Index finger raised, rest of fingers closed. Could be used to specify a count of 1. [gesture:peace] Index and middle finger raised, rest of fingers closed. Could be used to specify a count of 2. [gesture:shakefist] Shaking fist gesture. [gesture:livelongandprosper] Mr. Spock's gesture from Star Trek. [gesture:think] Thinking gesture. [gesture:floss] Floss dance. [gesture:cheer] Cheering gesture.Annotate an utterance like the following:
The bathrooms are over here, [gesture:pointleft] to your left. Here's what I think [gesture:middlefinger] of your suggestion. Great idea! [gesture:thumbsup] Let's do it.
13.Test AI (no voice) mode
You can put the app into an AI testing mode that will show the response as text and will not animate the avatar or produce synthetic speech. This mode is useful if you want to test out your AI quickly, since it will not retrieve any voice or produce any animated behavior. Press the AI button, then turn off the ‘Use text-to-speech?’ toggle.
14.Changing TTS Voices
You can change the TTS voice used. Press the AI button then choose the male or female voice from the voice list.
15.Pregenerate TTS voices
By default, the app will generate and store the synthetic voice that is created for your answer, which will take a few seconds depending on the speed of your Internet connectivity. If an answer has already been given, the synthetic voice response will not be regenerated. If you want to generate the voices for all of your answers in advance, which results in a very fast response time for the character, you can press the GENERATE TTS button. Note that you are limited in the number of voice utterances you can generate. Please contact us at email@example.com to expand this limit.
Also note that this feature is only available when using the Google Sheets AI.
16.Change your background
17.Put your avatar in A/R
You can put your avatar in augmented reality (A/R) by pressing the A/R button, then double clicking on flat surface in the camera view. The avatar will appear on the surface. Press the A/R button again to return to the original view. In A/R mode, the avatar will operate as before, communicating in response to your voice by pressing the microphone button.
Changing the avatar size
You can make the avatar smaller or bigger by pressing the rescale button (upper left, double hoops) then pinch gesturing to make it smaller or bigger, then pressing the rescale button once again. Once you press the rescale button, you can also rotate the avatar left or right.
Change the avatar gaze
By default, the avatar will gaze at the camera, regardless of the positioning in the scene. To turn off the gaze feature, press the gaze button and the avatar will instead look forward
Make the avatar walk
You can make the avatar walk by clicking on the move icon, then clicking on somewhere else in the 3D scene. The avatar will walk on a plane as detected by the camera. Note that the avatar will continue to try to look at the camera unless the gaze button was pressed.
You can press the screen and the avatar will interactively gaze at the location that you are pressing.
18.Set a wake-up word and use it hands-free
Instead of pressing the microphone button to speak, you can set the platform to hands-free mode by choosing the Hands-free voice option in the AI Settings. To speak, first say the wake-up word (by default, it’s ‘hey expert’) then ask your question. You can also change the wake-up word to anything that you like as long as it exists in the dictionary. For example, if you named your avatar ‘Ella’ you could set the wake-up word to ‘ella’ in order to address the assistant by name.
We recommend at least two (but preferrably three) syllables in your wake up word to avoid false positive matches.
Use of a virtual real estate agent in augmented reality:
Use of a virtual doctor to give non-diagnostic medical information:
Use of an interactive newscaster to allow the news viewer to ask more in-depth questions about a news topic:
Virtual concierge for short term rentals that allows you to customize the information for the renters/customers:
Virtual waiter/waitress in augmented reality:
Expert on particular subject, such as cooking, or an information agent about an event
How does your platform know how to animate the avatar?
We have a proprietary, patented technology that allows us to transform a line of text into an animated performance that represents how humans communicate to each other through speech and body movement. You can read more details in our blog that describes this technology applied to a human-like robot: https://medium.com/@ariyshapiro/making-robots-more-human-like-through-nonverbal-behavior-a617e4fe3bcd?source=friends_link&sk=0aa526a81a626226792e7c1c97e7675c
What platforms does it run on?
Currently the user application is running on an Android device. You can, however, acces and change the AI via Google Sheets on any desktop or mobile platform. It is also possible to run the application on a desktop or kiosk. Please contact us at firstname.lastname@example.org if you are interested in this.
Is there an iOS version available?
Not yet, but we plan on releasing one in the near future.
Can this run in VR?
Potentially, yes. Many of the popular VR platforms use Android.Please contact us at email@example.com if you are interested in this. Also, we have a desktop version of the platform that will run on Windows/Linux/OSx.
Do I need Internet connectivity for the app to work?
Yes, the Internet is needed for Google’s speech recognition (ASR) to create a transcription what is spoken by the user of the app. The Internet is also needed to retrieve the synthesized voice files from the text-to-speech (TTS) system.
Can the app be connected to other conversational AI systems such as Google’s DialogueFlow, Amazon’s Lex, Microsoft’s Conversational AI Engine or IBM’s Watson?
Our platform connects to Google DialogFlow as well as IBM Watson Assistant. We will shortly add support for other major conversational AI vendors. The avatar system uses voice as input and outputs a talking, animation based on a response that is given. It is possible to use the other vendor’s AI system instead of our own. Please contact us as firstname.lastname@example.org if you are interested in this.
Can this be used in video games?
Yes, the technology can be used for video games. Please contact us at email@example.com if you are interested.
Is there any cost to use the platform?
No, the platform is currently free.
I’d like to use this capability in my own app/platform/device. Can I do this?
Yes! Please contact us at firstname.lastname@example.org to discuss it!
What kind of applications can I build?
Anything that you could build with a chatbot! The examples we have already built include an:
Real Estate Agent chatbot in A/R https://youtu.be/m8isaEpnj20
Health care information https://youtu.be/QGgpL_XJYXQ
Expert information https://youtu.be/B9Uh6vl6qbs
Virtual waiter/waitress https://youtu.be/pUFVTtZqLW8
Virtual concierge for hospitality https://youtu.be/MioQQZUZJT8
AI-driven news anchor https://youtu.be/oSt-0kmdgOg
Will the A/R function work on any mobile device?
The A/R function will work on any mobile device that has the sensors needed, specifically a gyroscope. Most mid- and high-end mobile devices have the gyroscope. Some low-end Android devices do not, and the app will not allow you to use the A/R functionality on those devices (however, the rest of the app will function normally.)
21.AI Using Facebook's Blender chatbot
You can connect the AI Expert app to a running instance of Facebook’s Blender chatbot or any other ParlAI agent.
Step 1. Run the web_interactive.py script in ParlAI (see https://parl.ai/docs/tutorial_quick.html)
Step 2. Press the ‘AI’ button in the app, then enter the IP address and port number. Press ‘Connect’ to validate that the connection is working.