The following documents how to use the Embody Digital AI Expert app in order to create an AI-driven talking avatar using a spreadsheet (via Google Sheets) or connected to either your existing Google DialogueFlow agent or IBM Watson Assistant.

Download the Expert AI app on the Google Play Store:

https://play.google.com/store/apps/details?id=com.embodydigital.aiexpert

The app will require a download (approximately 300 mb) in order to deliver all the information that is needed to run properly. We recommend turning on WiFi in order to speed up the download process. A default female avatar will be displayed on startup.

Pess and hold down microphone button, then ask a question like “Who are you?”, “What do you do?”, “How does this work?”, let go, and get the answer back from the avatar.

There are two different looks of the avatars: photorealistic and stylized (cartoony).

 

 

 

 

 

The app utilizes different types of avatars: photorealistic and stylized/cartoony ones.

By default the app uses a stylized/cartoony avatar. There are several ways to change the avatar: (i) downloading a new avatar, (ii) uploading a custom avatar,  (iii) building an avatar from a selfie or face scan, or (iv) choosing an existing avatar residing on the device.

i. Downloading avatars

The AI Expert app comes with a set of avatars that can be downloaded (4 female and 4 male). Press the thumbnail of the avatar in order to use it. Once the avatar is downloaded, the app will automatically load it. In addition, the avatar name will appear on your list of avatars on your device. Avatars that need to be downloaded have a faded thumbnail appearance. Note that downloadable avatars cannot be configured with hair and clothing – only custom avatars from (iv) below can be customized with hair and clothing.

 

ii. Uploading custom avatars from Adobe Fuse

You can create and use any avatar that you make with Adobe Fuse. To do this, create your Adobe Fuse character, choose File -> Animate with Mixamo, then go to Mixamo and download the character using COLLADA(.dae) format in the T-pose with blendshapes. Put that .zip file on your mobile device, then press the UPLOAD AN AVATAR button. You will be asked what gender the avatar is in order to produce the proper text-to-speech voice. Please make sure that the name of the .zip file matches the name of the avatar. For example, if your avatar model name is FemaleAvatar1 make sure your zip file is named FemaleAvatar1.zip.

 

Details of that process can be found here:

iii. Creating an avatar from a selfie/photo

To create your own avatar, you can generate one based on a selfie or a head shot of someone else. Press the ‘Avatar’ button at the bottom of the main screen, then choose Male or Female and then take a selfie, or press the photo button in the upper left hand to retrieve a picture from the mobile device’s photo gallery.

Your photo will be converted into a photorealistic avatar that resembles your face or input photo. This avatar will be given default hair and clothing options, which can in turn be changed by selecting them from the ‘Hair & Clothes’ button.

You can change the background color or use an image for a background by pressing the ‘Background’ button.

Avatars that you create will appear on the bottom of the Avatar screen and can be used at any time.

 

iv. Using an existing avatar on the device

There is a default female and male photorealistic avatar (named customAvatar_defaultFemale and customAvatar_defaultMale)  as well as a default female stylized/cartoony avatar (named Ella). Press the button with that name to use that avatar. Any avatars that you create with a selfie or any avatars downloaded will appear as buttons and may be selected.

You can use a number of different conversational AI systems to connect to your avatar, including:

(i) using a Google Sheets spreadsheet where you enter the questions in one column and the answers in another column,

(ii) by connecting it to an existing Google DialogFlow agent,

(iii) by connecting it to an existing IBM Watson Assistant,

(iv) by connecting to an existing RASA agent,

(v) by connecting you a Pandorabots agent,

(vi) by connecting the Facebook’s Blender chatbot or any other ParlAI agent.

You only need to choose one method. Other interfaces (such as Amazon Lex, Microsoft Bot, and others) will be added as the app is improved.

There is no inherit restriction on the type of conversational AI that can be used; in theory, any system that accepts text as input and returns conversational text as output could be used.

The AI Expert utilizes the a conversational AI that can be created by entering questions that your avatar will respond to and the answers associated with those questions. For input, we use Google Sheets (an online spreadsheet application by Google). You will need to have a Google account in order to access Google Sheets.

A template that can be used for this purpose is located here:

https://docs.google.com/spreadsheets/d/1QCOOPyyl5Gbd51Ii9mjOO6h1rVSYj4e4swO3Rx-6WVs/edit?usp=sharing

Step 1. Make a copy of this Google Sheets by going to File -> Make a Copy

Step 2. Rename your Google Sheets copy.

Step 3. Replace the questions and answers with your own. Enter multiple questions for each answer in the same cell by pressing CTRL-ENTER. When the user talks to the app, the AI will find the best match according to both the questions and the matching answers and automatically make the avatar speak and gesture.

The third column is an identifier for the id of the utterance for later use. You can label it numerically (i.e. 1,2,3,4), or with a specific name (with no spaces)  to associate with the answer (i.e. greeting, howmany, howold, etc.)

Avoid using any non-alphanumeric characters except for punctuation (period, question mark, exclamation marks.)

Your avatar will automatically speak and gesture based on the responses that you give. The avatar is programmed with a set of gestures and movements that correspond to the way most people communicate with each other, including head movements, facial expressions and arm gestures. When you are finished, turn on link sharing for the Google Sheets which will allow AI Expert to access your Google Sheets. Each Google Sheets has a unique id which can be accessed from the Share button in a browser. Press the green Share button in the upper right from a browser, then click on Get Shareable Link in the dialog box. This will turn on Google Sheets sharing and allow that Google Sheets ID to be accessed by the AI Expert application.

Step 4. Connect your avatar to your Google Sheets AI

In order for the avatar to use the AI that you have programmed, you need link your spreadsheet to the app.

The Google Sheets ID needs to be transferred to the AI Expert app. You can do this by downloading the Google Sheets app from the Google Play Store onto your Android device, open it, then from the three dots on the Google Sheets app select Copy Link which will store the Google Sheets ID into the copy buffer.

In the Expert AI app, press the AI button at the bottom of the screen, then in the Google sheets ID field paste in the link that was copied from the Google Sheets ID. Then press the Train AI button.

Press the back button (arrow) in the upper right to return to the main screen. You can then press and hold the microphone button, ask a question, and the AI will find a match to the proper answer, convert it to synthetic speech and automatically animate your avatar.

To make changes to your AI, simply add more questions or answers to your Google Spreadsheet, press the AI button and touch the Train AI button once again. 

 

The AI Expert can communicate with an existing Google Dialog agent. To do this, you need to load the Google DialogFlow credentials .json file onto the mobile device (through a USB cable, bluetooth interface, or any other Internet connection to bring the .json file onto your devices filesystem), then load the file into the AI Setting screen. Step 1. Access the Google DialogFlow console at https:// https://dialogflow.cloud.google.com/ and log in to access your DialogFlow agent. Press the gear icon to get to the Project information. Step 2. Click on the Service Account information Step 3. Click on the Service account information for your DialogFlow agent: Step 4. Press the ‘+ CREATE KEY’ button, which will download the needed credentials onto your mobile device as a .json key file. Step 5. In the AI Expert app, press the AI SETTINGS button and choose Google DialogFlow fromt the dropdown at the top. Next, press the CREDENTIALS button to find the .json file on your mobile device. Your DialogFlow agent will be connected automatically, then go back to the main screen and try it out! Your agent name will appear in the ‘DialogFlow Name’ box.

The AI Expert can communicate with an existing IBM Watson Assistant agent. To do this, you need to load the IBM Watson credentials onto the mobile device, indicate the Watson Assistant Id on the AI screen, then your Watson Assistant agent will automatically communicate with the device.

Step 1. From the app, press the AI Settings button, and choose IBM Watson from the dropdown at the top of the view.

Step 2. Locate your IBM Watson Assistant Id from the IBM Watson Assistant control panel page. Press the three dots -> Settings to link to the API Details, then copy or record the Assistant Id. Put the Assistant Id into the ‘Watson Assistant Id’ field in the app (shown above.)

Step 3. Download your IBM Watson Service credentials that are associated with your Watson Assistant. Save the file downloaded and store it on your mobile device. From the AI Settings screen on the app, press the ‘Credentials’ button and locate the credentials file that you previously downloads. Once both the credentials are identified and the Watson Assistant Id is entered, the app will automatically connect to your Watson Assistant.

The AI Expert can communicate with an existing RASA agent via the REST api through a webserver. To do this, you need to run your RASA process using the ‘–enable-api’ option and configure the AI to use that webhook location.

Step 1. Press the AI button from the main screen, then choose ‘RASA’ from the AI Type dropdown menu.

Step 2.  Press the button next to the RASA Webhook URL.

Step 3. Enter the location of the server and the webhook URL. Make sure to set either http:// or https:// and the proper IP address or hostname, including port. Press OK and you will either see a message saying that the connection was successful or not.

You can connect to an existing Pandorabots account. 

Step 1. Set up an Pandorabots agent from www.pandorabots.com

Step 2. Select your Pandorabots agent, then go the dashboard for that agent, and under the API option choose ‘Set Up’ or ‘View Settings’ if it has already been set up.

Step 3. Under the ‘Bot Keys’, choose Deploy. The Bot Key needs to be copied to the app.

Step 4. On the app, press the ‘AI’ button, select Pandorabots from the dropdown at the top, then put the public Bot Key into the Pubic Key field, then press ‘Connect’, You will see a message indicating whether the bot was successfully connected or not.

You can connect the AI Expert app to a running instance of Facebook’s Blender chatbot or any other ParlAI agent.

Step 1. Run the interactive_web.py script in ParlAI (see https://parl.ai/docs/tutorial_quick.html)

Step 2. Press the ‘AI’ button in the app, then enter the IP address and port number. Press ‘Connect’ to validate that the connection is working.

Interaction

To interact with your avatar, press the microphone button on the app, ask a question, then let go of the button. First, your voice will be transcribed into text via Google’s ASR service. Second, the text will be inputted into our AI, returning an answer, which will then be converted into voice and body language.

When your avatar speaks, the voice will be retrieved from the TTS service. If the exact response has already been spoken, the app will retrieve the voice response from the device locally. You can remove any prerecorded responses from the AI screen by pressing Reset TTS, which will be useful if you want to test out different TTS voices.

Gazing

You can touch the screen and your avatar will look at the point in the screen that you are touching.

Changing the size and position of the avatar

You can modify the size of the avatar, the camera position, and your agents gaze direction by pressing the sizer button. This can be used to focus on the avatar’s face, or to position the character around backround imagery or A/R backgrounds. The sizer button will turn red, and you can touch and drag to position the avatar, pinch to make the avatar larger or smaller, or move the rotation slider to change the orientation of the character.

Long press on the sizer button (hold the button for a few seconds) in order to return the avatar to its original size.

Modifying your AI

You can iterate over your application by modifying your AI to change how your avatar responds to questions.

If you are using  Google Sheets, pressing the AI button, then Train AI, then go back and ask your question again.

If you are using  Google DialogFlow or IBM Watson, changes to your agent .or assistant will automatically appear on the avatar.

 

 

Avatar responses (generated from Google Sheets) can be modified to be more energetic via stress tags, or to be expressed with differing emotion using the emotion tags. In addition, there are a number of special gestures that can be triggered by explicitly indicating those gestures inline in the response utterance. Note that this feature is not available for Google DialogueFlow or IBM Watson responses (yet!)

Each word has a stress value going from 0 to 1 (defaults to 0). To change the stress of a particular word (for emphasis, for example). You can add the stress markup before a word like this:

[1] yes, I agree

where the [1] could be any decimal number between 0 and 1, such as .3 or .7.   The stress will continue to affect all the words until changed, so to stress only the first word:

[1] yes, [0] I agree

You can use the following emotional tags:

[happy]

[angry] [surprised] [sad] [agree] [disagree] [sarcasm] [neutral]

To add emotional tone to words, any of the emotional tags can be added inline to the utterance like so:

[happy] That’s great!

[sad] I can’t believe you said that.

[surprised] you said what?

[angry] I thought I told you to clean up your room!

[agree] It was amazing.

[disagree] no, that doesn’t sound right to me at all.

That’s a great idea [sarcasm] not!

[happy] Excellent! [neutral] now let’s get back to business.

 

The avatar will talk, gesture and emote according to the content of the dialog and any emotional markup (such as [happy], [angry] or stress indicators like [1]) . Arm gestures, head movements, eye saccades, lip sync and facial expressions will appear automatically. To change the behavior, you can modify the choice of gestures, and the type of gestures.

You can change the variations of gestures that appear by setting the ‘Gesture Variations’ value to a high number (such as 100). This will randomly pick among the valid gestures that are triggered for the character based on the content of the speech. Setting the value to 0 will enforce a gesture priority scheme where some gesture types (such as metaphoric gestures) will be preferred over others (such as beat gestures). There are two types of gesturing: casual (default) and emotional. The emotional gestures contain a larger set of movements suitable for more emotionally-charged conversations, and can be triggered by using the emotion and stress markup. You can switch between those two gesture types at any time.

  If you use the emotional gesture types, you will have access to a number of other animations that you can tag inside of your Google Sheets AI as follows:

[gesture:pointleft]  Points to the left
[gesture:pointright] Points to the right
[gesture:pointdown] Points downward
[gesture:pointup] Points upward
[gesture:middlefinger] Shows the middle finger
[gesture:thumbsup] Thumbs upgesture.
[gesture:thumbsdown] Thumbs down gesture.
[gesture:airquotes] Air quotes with two hands.
[gesture:headexplode] Hands to the side of the head in an exploding gesture.
[gesture:pray] Hands together in front in a praying gesture.
[gesture:number1] Index finger raised, rest of fingers closed. Could be used to specify a count of 1.
[gesture:peace] Index and middle finger raised, rest of fingers closed.  Could be used to specify a count of 2.
[gesture:shakefist] Shaking fist gesture.
[gesture:livelongandprosper] Mr. Spock's gesture from Star Trek.
[gesture:think] Thinking gesture.
[gesture:floss] Floss dance.
[gesture:cheer] Cheering gesture.

Annotate an utterance like the following:

The bathrooms are over here, [gesture:pointleft] to your left.

Here's what I think [gesture:middlefinger] of your suggestion.

Great idea! [gesture:thumbsup] Let's do it.

 

You can put the app into an AI testing mode that will show the response as text and will not animate the avatar or produce synthetic speech. This mode is useful if you want to test out your AI quickly, since it will not retrieve any voice or produce any animated behavior. Press the AI button, then turn off the ‘Use text-to-speech?’ toggle.

You can change the TTS voice used. Press the AI button then choose the male or female voice from the voice list.

 

By default, the app will generate and store the synthetic voice that is created for your answer, which will take a few seconds depending on the speed of your Internet connectivity. If an answer has already been given, the synthetic voice response will not be regenerated. If you want to generate the voices for all of your answers in advance, which results in a very fast response time for the character, you can press the GENERATE TTS button. Note that you are limited in the number of voice utterances you can generate. Please contact us at info@embodydigital.com to expand this limit.

Also note that this feature is only available when using the Google Sheets AI.

 

You can change the background behind the avatar to a solid color or to use any photo that you have by pressing the Background button. Photos can be cropped or rotated within the app, in order to fill the screen vertically. Photos and backgrounds will not be shown in augmented reality (A/R) mode.    

You can put your avatar in augmented reality (A/R) by pressing the A/R button, then double clicking on flat surface in the camera view. The avatar will appear on the surface. Press the A/R button again to return to the original view. In A/R mode, the avatar will operate as before, communicating in response to your voice by pressing the microphone button.

Changing the avatar size

You can make the avatar smaller or bigger by pressing the rescale button (upper left, double hoops) then pinch gesturing to make it smaller or bigger, then pressing the rescale button once again. Once you press the rescale button, you can also rotate the avatar left or right.

 

Change the avatar gaze

By default, the avatar will gaze at the camera, regardless of the positioning in the scene. To turn off the gaze feature, press the gaze button and the avatar will instead look forward

Make the avatar walk

You can make the avatar walk by clicking on the move icon, then clicking on somewhere else in the 3D scene. The avatar will walk on a plane as detected by the camera. Note that the avatar will continue to try to look at the camera unless the gaze button was pressed.

Interactive gaze

You can press the screen and the avatar will interactively gaze at the location that you are pressing.

 

 

 

Instead of pressing the microphone button to speak, you can set the platform to hands-free mode by choosing the Hands-free voice option in the AI Settings. To speak, first say the wake-up word (by default, it’s ‘hey expert’) then ask your question. You can also change the wake-up word to anything that you like as long as it exists in the dictionary. For example, if you named your avatar ‘Ella’ you could set the wake-up word to ‘ella’ in order to address the assistant by name.

We recommend at least two (but preferrably three) syllables in your wake up word to avoid false positive matches.

Use of a virtual real estate agent in augmented reality:

Use of a virtual doctor to give non-diagnostic medical information:

Use of an interactive newscaster to allow the news viewer to ask more in-depth questions about a news topic:

Virtual concierge for short term rentals that allows you to customize the information for the renters/customers:

Virtual waiter/waitress in augmented reality:

Expert on particular subject, such as cooking, or an information agent about an event

 

 

 

 

How does your platform know how to animate the avatar?

We have a proprietary, patented technology that allows us to transform a line of text into an animated performance that represents how humans communicate to each other through speech and body movement. You can read more details in our blog that describes this technology applied to a human-like robot: https://medium.com/@ariyshapiro/making-robots-more-human-like-through-nonverbal-behavior-a617e4fe3bcd?source=friends_link&sk=0aa526a81a626226792e7c1c97e7675c

What platforms does it run on?

Currently the user application is running on an Android device. You can, however, acces and change the AI via Google Sheets on any desktop or mobile platform. It is also possible to run the application on a desktop or kiosk. Please contact us at info@embodydigital.com if you are interested in this.

Is there an iOS version available?

Not yet, but we plan on releasing one in the near future.

Can this run in VR?

Potentially, yes. Many of the popular VR platforms use Android.Please contact us at info@embodydigital.com if you are interested in this. Also, we have a desktop version of the platform that will run on Windows/Linux/OSx.

Do I need Internet connectivity for the app to work?

Yes, the Internet is needed for Google’s speech recognition (ASR) to create a transcription what is spoken by the user of the app. The Internet is also needed to retrieve the synthesized voice files from the text-to-speech (TTS) system.

Can the app be connected to other conversational AI systems such as Google’s DialogueFlow, Amazon’s Lex, Microsoft’s Conversational AI Engine or IBM’s Watson?

Our platform connects to Google DialogFlow as well as IBM Watson Assistant. We will shortly add support for other major conversational AI vendors. The avatar system uses voice as input and outputs a talking, animation based on a response that is given. It is possible to use the other vendor’s AI system instead of our own. Please contact us as info@embodydigital.com if you are interested in this.

Can this be used in video games?

Yes, the technology can be used for video games. Please contact us at info@embodydigital.com if you are interested.

Is there any cost to use the platform?

No, the platform is currently free.

I’d like to use this capability in my own app/platform/device. Can I do this?

Yes! Please contact us at info@embodydigital.com to discuss it!

What kind of applications can I build?

Anything that you could build with a chatbot! The examples we have already built include an:

Real Estate Agent chatbot in A/R https://youtu.be/m8isaEpnj20

Health care information https://youtu.be/QGgpL_XJYXQ

Expert information https://youtu.be/B9Uh6vl6qbs

Virtual waiter/waitress https://youtu.be/pUFVTtZqLW8

Virtual concierge for hospitality https://youtu.be/MioQQZUZJT8

AI-driven news anchor https://youtu.be/oSt-0kmdgOg

Will the A/R function work on any mobile device?

The A/R function will work on any mobile device that has the sensors needed, specifically a gyroscope. Most mid- and high-end mobile devices have the gyroscope. Some low-end Android devices do not, and the app will not allow you to use the A/R functionality on those devices (however, the rest of the app will function normally.)