Behavior API
Embody your character, avatar or robot. Automatically animate a 3D character, avatar or robot solely from a text- or voice- signal. The animation includes lip syncing, emotional expression, head movements, saccadic eye movements, body gestures and synchronized body movements that reflect how people communicate.
This allows conversational AI systems, such as chatbots, Google DialogFlow, IBM Watson, Amazon Lex, RASA and any other text-based input to be automatically embodied and animated in near real time.
Utterances can be marked up using inline tags in order to change the performance according to emotional tone and content:
[happy] hello, nice to meet you.
[sad] I’m very sorry to hear that. [happy] But don’t worry, I’m here to help!
[sarcasm] That’s a great idea, not!
Usage:
curl -X POST -H "Content-type:application/json" --data "{\"utterance\":\"hi there\"}" https://webapp2.embodydigital.com/behavior/behavior.php
Response will include the behaviors that should be activated at the time related to the words that are spoken. For example, there are two words in the utterance (‘hi there’), the first starts at position 0 and the second at position 2. The from/to indicate the the span of the behavior. The API returns both semantic and syntactic indicators (such as ‘deixis’ which is a pointing behavior, or ‘dir’ which specifies a direction, or specific behavior suggestions, such as ‘gesture_left’, ‘gesture_greeting’, ‘wiggle’ (head movement), ‘gaze_aversion_upright’ (avert your gaze to the upright quadrant then return).
{{“status”:”ok”,”result”:”[
{\”priority\”: \”12\”, \”to\”: \”1\”, \”from\”: \”0\”, \”behavior\”: \”greeting\”},
{\”priority\”: \”14\”, \”to\”: \”2\”, \”from\”: \”1\”, \”behavior\”: \”dir\”},
{\”priority\”: \”13\”, \”to\”: \”2\”, \”from\”: \”1\”, \”behavior\”: \”dir\”},
{\”priority\”: \”12\”, \”to\”: \”2\”, \”from\”: \”1\”, \”behavior\”: \”deixis\”},
{\”priority\”: \”12\”, \”to\”: \”2\”, \”from\”: \”1\”, \”behavior\”: \”deixis\”},
{\”priority\”: \”35\”, \”to\”: \”2\”, \”from\”: \”0\”, \”behavior\”: \”neutral\”},
{\”priority\”: \”3\”, \”to\”: \”2\”, \”from\”: \”0\”, \”behavior\”: \”cogload\”},
{\”priority\”: \”6\”, \”to\”: \”2\”, \”from\”: \”0\”, \”behavior\”: \”first_VP\”},
{\”priority\”: \”6\”, \”to\”: \”2\”, \”from\”: \”0\”, \”behavior\”: \”first_S1\”},
{\”priority\”: \”12\”, \”to\”: \”2\”, \”from\”: \”1\”, \”behavior\”: \”gesture_left\”},
{\”priority\”: \”12\”, \”to\”: \”2\”, \”from\”: \”1\”, \”behavior\”: \”gaze_aversion_left\”},
{\”priority\”: \”12\”, \”to\”: \”1\”, \”from\”: \”0\”, \”behavior\”: \”gesture_greeting\”},
{\”priority\”: \”12\”, \”to\”: \”1\”, \”from\”: \”0\”, \”behavior\”: \”wiggle\”},
{\”priority\”: \”3\”, \”to\”: \”2\”, \”from\”: \”0\”, \”behavior\”: \”gaze_aversion_upright\”},
{“bml”: “…..”}], “reason”: “”}
In addition, the response will contain a Behavior Markup Language (BML) block that can be used in a BML realizer (like SmartBody) to directly create animation.
Emotion tags
Emotion can be added to the tags by specifying one of the following before the words: neutral, happy, sad, surprised, angry, fear, agree, disagree, sarcasm, sad. The neutral tag is the default if no emotion is specified.
curl -X POST -H "Content-type:application/json" --data "{\"utterance\":\"hi there\", \"emotionalPhrase\" : \"happy=0:1\"}" https://webapp2.embodydigital.com/behavior/behavior.php curl -X POST -H "Content-type:application/json" --data "{\"utterance\":\"hi there\", \"emotionalPhrase\" : \"angry=3:6\"}" https://webapp2.embodydigital.com/behavior/behavior.php
Stress tags
The utterance can also be annotated with stress levels from 0 to 1 (0 is the default). This can be used to stress or emphasize a particular word and the associated behaviors with that word:
Emotion can be added to the tags by specifying one of the following before the words: happy, sad, surprised, angry, fear, agree, disagree, sarcasm, sad:
curl -X POST -H "Content-type:application/json" --data "{\"utterance\":\"I saw it happen right over there\", \"stress\" : \"0 0 0 0 0 0 1\"}" https://webapp2.embodydigital.com/behavior/behavior.php
curl -X POST -H "Content-type:application/json" --data "{\"utterance\":\"I saw it happen right over there\", \"stress\" : \"1 1 0 0 0 0 0\"}" https://webapp2.embodydigital.com/behavior/behavior.php
Terms of Use and License
If you are interested in using this API, please contact info@embodydigital.com for the terms of use for this service. In addition, the Behavior API is available as an embeddable library. Please contact us for license details.
Code Sample
Code sample accessing the Behavior API using Unreal and Metahumans can be found in a sample Unreal 4.26 project here:
https://drive.google.com/file/d/1VNYeN2v0NpaT8lFXoXZ4fF62zOlzEZeO/view?usp=sharing
Embody Digital code samples included use the MIT license
Instructions:
- Expand the .zip file into a folder onto a Windows machine that has Unreal 4.26 installed
- A MetaHuman is not included in the project; create and download a MetaHuman using the EPIC Metahuman Creator
- Put the MetaHuman folder into the TestEmbodySDK/Content/ folder
- Open the Unreal project
- Download the UnrealWebServer Plugin
- Drag the MetaHuman into the level
- Place the MetaHuman into the position 0,0,33 with z rotation set to -90
- Run the project
- Use AWS to access the Amazon Polly service, put the service key in TestEmbodySDK/embodysdkdata/ttsapi/ttsapiwithkey.bat
- Run TestEmbodySDK/embodysdkdata/ttsapi/ttsapiwithkey.bat
- Send a command to make the character speak: curl ‘https://localhost:8004/botbatte’ –data ‘{“commandType” : “speak”, “character” : “character”, “value” : “hello, [happy] I am a metahuman”}’
Dependencies (modified SmartBody library, TTS connector to Amazon Polly service) can also be found here (can be built with Visual Studio 2015 or 2017):
https://drive.google.com/file/d/1TaITvlm4S6pF85k-YD8-jMdoa_GPLUbh/view?usp=sharing