Headless Mode

This page is to detail the process of integrating the Conva.AI assistant in headless mode to an app.

Conva.AI supports SDKs for the following platforms

Android
Python
iOS
Flutter
Typescript
Rust - Coming soon

Let's walk through the integration process.

Setting up the build environment

First, you need to add the maven repository URL to your project-level build.gradle file. Open the build.gradle file and add the following code inside the allprojects > repositories block

allprojects {
    repositories {
        maven {
            url "https://pkgs.dev.azure.com/slanglabs-convaai/prod/_packaging/public_prod/maven/v1"
        }
    }
}

Next, open your app-level build.gradle file and add the conva-ai-core dependency inside the dependencies block

dependencies {
    implementation 'in.slanglabs.conva:conva-ai-core:1.0.2-beta'
}

$ pip install conva-ai

$ flutter pub add conva_ai_core

Once done, run the command 'dart pub get' and ensure Conva.AI is added to the dependencies.

dependencies:
  conva_ai_core: ^0.1.3

Check the latest version of SDK here:- https://pub.dev/packages/conva_ai_core

Add the Cocoapod repository path to your Podfile

# Add this to your podfile

source 'https://github.com/CocoaPods/Specs.git'

Next, open your podspec file and add the ConvaAICore dependency.

s.dependency 'ConvaAICore', '1.0.3'

Once done, run the command 'pod install'

npm install conva-ai

Initializing the Assistant

First, initialize Conva.AI using the credentials of the Assistant created via Magic Studio.

ConvaAI.init(
    id = "replace this string with your_assistant_id",
    key = "replace this string with your_api_key",
    version = "LATEST", // this is a special tag to indicate 
                        // the latest version of the Assistant
    application = applicationContext
);

client = AsyncConvaAI(
    assistant_id="replace this string with your_assistant_id", 
    assistant_version="LATEST", # this is a special tag to indicate 
                                # the latest version of the Assistant
    api_key="replace this string with your_api_key"
)

ConvaAI.init(
    id: "your_assistant_id",
    key: "your_api_key", 
    version: "LATEST"
);

ConvaAI.initialize(
    with: "<your_assistant_id>", 
    key: "<your_api_key>", 
    version: "<your_assistant_version>"
)

import ConvaAI from 'conva-ai';

const client = new ConvaAI({
    assistantID:"<YOUR_ASSISTANT_ID>",
    assistantVersion:"<YOUR_ASSISTANT_VERSION>",
    apiKey:"<YOUR_API_KEY>"});

Open the "Integration" tab of your Assistant to get the keys.

Invoking a Agent

Next, pass unstructured text input to Conva.AI and let it invoke the appropriate Agent and return the response.

// Pass an input string to Conva.AI and let it determine the 
// relevant Agent to invoke and return the output params
// after that Agent is executed
val response = ConvaAI.invokeCapability(
    input = "book a bus ticket from bangalore to chennai tomorrow at 2pm",
)

// response = {
//    "capability_name" : "ticket_booking",
//    "message" : "Showing you buses from Bangalore to Chennai for tommorrow around 2pm"
//    "params" : {
//         "source": "BLR",
//         "destination" : "MAS",
//         "date" : "6/7/2024",
//         "time" : "14:00"
//         "mode" : "bus"
//     }
// }

# Pass an input string to Conva.AI and let it determine the 
# relevant Capability to invoke and return the output params
# after that Capability is executed
response = await client.invoke_capability("book a bus ticket from Bangalore to chennai tommorrow at 2pm")
return response

# response = {
#    "capability_name" : "ticket_booking",
#    "message" : "Showing you buses from Bangalore to Chennai for tommorrow around 2pm"
#    "params" : {
#         "source": "BLR",
#         "destination" : "MAS",
#         "date" : "6/7/2024",
#         "time" : "14:00"
#         "mode" : "bus"
#     }
# }

// Pass an input string to Conva.AI and let it determine the 
// relevant Capability to invoke and return the output params
// after that Capability is executed
Response completion = await ConvaAI.invokeCapability(input: "<input_query>");
// completion = {
//    "capability_name" : "ticket_booking",
//    "message" : "Showing you buses from Bangalore to Chennai for tommorrow around 2pm"
//    "params" : {
//         "source": "BLR",
//         "destination" : "MAS",
//         "date" : "6/7/2024",
//         "time" : "14:00"
//         "mode" : "bus"
//     }
// }

Note

Proper error handling should be implemented to manage any potential issues during the invocation.

// Pass an input string to Conva.AI and let it determine the 
// relevant Capability to invoke and return the output params
// after that Capability is executed.

// Invoke a capability (non-streaming)
let response: ConvaAICapability? = try await ConvaAI.invokeCapability(
    with: "<input_query>"
)

Notes

Ensure that this method is called from an async context.
Proper error handling should be implemented to manage any potential issues during the invocation.

// Pass an input string to Conva AI and let it determine the
// capability name and the respective parameters
client.invokeCapability({
    query: 'Hi, how are you doing today?',
    stream: false
  }).then(response => {
    console.log('ConvaAI Response:', response); 
    // TODO: Add application logic here
  }).catch(error => {
    console.error('Error:', error);
  });
  
//Example response:
//{inputQuery: 'Hi, how are you doing today?',
//message: "Hi there! I'm doing well, thank you for asking. How about you? How can I help you today?",
//messageType: 'question',
//parameters: { interaction_type: 'greeting' },
//relatedQueries: [
    //'Ask for a joke',
    //'Get fashion tips',
    //'Inquire about the latest trends'
 // ],
//toolName: 'small_talk'
//}

Response object

The response object contains the following key fields

capability_name: The name of the Agent that was triggered.
message: The string containing either the answer to the original query or a status message. Agent creators can update this via Magic Studio.
related_queries: An array of strings with queries related to the current query and app context. Agent creators can update this field's description if needed.
params: This object contains custom parameters specific to the invoked Agent. For example, if a Agent is created with task_name, date, and time (as shown in the image below), these fields will be in the params object.
history: The current conversation history. This can be used by the app to continue the current conversation further (more on this later).

Developers should check if a specific custom parameters exists before accessing its value, as it might not be marked as required in Magic Studio.

Invoking a specific Agent

To invoke a specific Agent, you can refer to it by name and pass the input string. You can use this in streaming (async) or non-streaming (sync) mode.

// Invoke a specific Agent
val response = ConvaAI.invokeCapabilityWithName(
    input = "book a bus ticket from Bangalore to chennai tommorrow at 2pm", 
    capability = "ticket_booking"
);

# Invoke a specific Capability
response = await client.invoke_capability_with_name("book a bus ticket from Bangalore to chennai tommorrow at 2pm", capability_name="ticket_booking")
return response

// Invoke a specific capability
Response completion = await ConvaAI.invokeCapabilityWithName(
    input: "book a bus ticket from Bangalore to chennai tommorrow at 2pm",
    capability: "ticket_booking"
);

Note

Proper error handling should be implemented to manage any potential issues during the invocation.

// Invoke a capability with name (non-streaming)
let completion: ConvaAICapability? = try await ConvaAI.invokeCapability(
    with: <"input_query">,
    capabilityName: <"capability_name">
)

Notes

Ensure that this method is called from an async context.
Proper error handling should be implemented to manage any potential issues during the invocation.

client.invokeCapabilityName({
    query: 'What are the latest shopping trends?',
    stream: false,
    capabilityName: 'domain_faq'
  }).then(response => {
    console.log('ConvaAI Response:', response); 
    // TODO: Add application logic here
  }).catch(error => {
    console.error('Error:', error);
  });
  
//Example response:
//{inputQuery: 'What are the latest shopping trends?',
//message: 'The latest shopping trends include increased online shopping, personalized shopping experiences, and the use of AI in retail.',
//messageType: 'statement',
//parameters: { query: 'What are the latest shopping trends?' },
//relatedQueries: [
    //'Explore popular products this season',
    //'Learn about AI in retail',
    //'Discover personalized shopping experiences'
//],
  //toolName: 'domain_faq'}

Handling Streaming Response

Conva.AI supports a streaming version of these APIs. The response is sent back incrementally, with the listener invoked multiple times. Use the is_final flag in the Response object to check if all responses are received.

// Invoke a Agent asynchronously with a streaming response
ConvaAI.invokeCapability(
    input = "Hello, how are you?",
    listener = object : ResponseListener {
        override fun onResponse(response: Response, isFinal: Boolean) {
            // Check for is_final
            if (isFinal) {
                // Handle the final response
            }
        }
        override fun onError(error: Throwable) {
            // Handle the error
        }
    }
);

# streaming version
response = await client.invoke_capability_stream(query)
out = ""
async for res in response:
    out = res
    
# At this point the "is_final" flag in the out object will be set
return out

// Invoke a capability asynchronously with a streaming response
Stream<Response> completionStream =
            ConvaAI.invokeCapabilityStream(input: "Hello, how are you?");
completionStream.listen(
      (event) {
      // Use the response object
  },
  onError: (error) {
    // Handle error
  },
);

// Invoke a capability (streaming)
for try await response in ConvaAI.invokeCapabilityStream(
    with: "<input_query>"
) {
   // Handle the streaming response here
}

Notes

Ensure that this method is called from an async context.
Proper error handling should be implemented to manage any potential issues during the invocation.

Invoking a Agent Group

When creating Agents via Magic Studio, it places all Agents into the "default" group. But users can create their own groups and put specific Agents into it. If you want to limit the Agent invocation to specific groups, you can pass the group name to the APIs.

// Invoke a Agent group synchronously (non-streaming)
val response = ConvaAI.invokeCapability(
    input = "Hello, how are you?",
    capabilityGroup = "default"
);

// Invoke a Agent group asynchronously to get a streaming response
ConvaAI.invokeCapability(
    input = "Hello, how are you?",
    capabilityGroup = "general_conversation",
    listener = object : ResponseListener {
        override fun onResponse(response: Response, isFinal: Boolean) {
            // Handle the response
        }
        override fun onError(error: Throwable) {
            // Handle the error
        }
    }
);

# 'client' is the initialized ConvaAI object
# Streaming
response = await client.invoke_capability_stream(query, capability_group="default")
out = ""
async for res in response:
    out = res
return out

# Non-streaming
response = await client.invoke_capability(query, capability_group="default")
return response

// Invoke a capability group synchronously (non-streaming)
Response completion = await ConvaAI.invokeCapability(
    input: "Hello, how are you?",
    capabilityGroup: "default"
);

// Invoke a capability group asynchronously to get a streaming response
Stream<Response> completionStream =
ConvaAI.invokeCapabilityStream(
  input: "Hello, how are you?",
  capabilityGroup: "general_conversation"
);
completionStream.listen(
      (event) {
      // Use the response object
  },
  onError: (error) {
    // Handle error
  },
);

Note

Proper error handling should be implemented to manage any potential issues during the invocation.

// Invoke a capability with name (non-streaming)
let response: ConvaAICapability? = try await ConvaAI.invokeCapability(
    with: <"input_query">,
    capabilityGroup: <"capability_group">
)

Notes

Ensure that this method is called from an async context.
Proper error handling should be implemented to manage any potential issues during the invocation.

Handling Conversational Context

Conva.AI supports maintaining context across different conversations, allowing the AI to build upon prior interactions for more relevant responses.

Parameters

history (Optional): Available after the first response. Pass this from the previous response if you want to maintain conversational context in subsequent calls.
capabilityContext (Optional): Context related to a specific Agent, refining AI responses for that particular Agent.

Usage

On the first invocation, call the API without history.
On subsequent invocations, pass the history from the previous response if you want to maintain the conversation's context.
You can also pass the capabilityGroup or capabilityName along with the context to refine the AI’s responses further.

// First Response
val response1 = ConvaAI.invokeCapability(
    input = "Blue jeans",
);

// Second response. Pass the history from the first response
// to the second one to maintain conversational context
val reponse2 = ConvaAI.invokeCapability(
    input = "show me in red",
    context = ConvaAIContext(
        history = response1.history,
        capabilityContext = <"Map of String and Any">
    )
);

(Any): The values can only include the following types:
String: For textual data.
Int: For integer numbers.
Double: For floating-point numbers.
Boolean: For true/false values.
Array: For lists of items (e.g., Array<String>, Array<Int>, etc.).
Map: For nested maps, allowing for hierarchical data structures.

val items = listOf("green tea", "freshly squeezed juice", "herbal tea", "smoothies", "coconut water", "coffee")

// Example for passing capability context
val capabilityContext = mapOf(
    "product_search" to mapOf(
        "best beverages" to items
    )
)

// Note: Here <product_search> is the Agent name.
// Use your own Agent name and describe Agent data according to it

// Optional Context: CapabilityContext is optional.
// If omitted, the AI will function based on the default 
// or previously provided context.

# First Response
response1 = await client.invoke_capability("blue jeans")
return response

# Second response. Pass the history from the first response
# to the second one to maintain conversational context
response2 = await client.invoke_capability("show me in red", 
 history=response1.conversation_history,
 capability_context={ <capability_name>: <knowledge> })
return response

(Knowledge): The values can only include the following types:
String: For textual data.
Int: For integer numbers.
Double: For floating-point numbers.
Boolean: For true/false values.
Array: For lists of items (e.g., Array<String>, Array<Int>, etc.).
Map: For nested maps, allowing for hierarchical data structures.

items = ["green tea", "freshly squeezed juice", "herbal tea", "smoothies", "coconut water", "coffee"]

# Example for passing capability context
capabilityContext = {
    "product_search": {
        "best beverages": items
    }
}

# Note: Here <product_search> is the capability name. 
# Use your own capability name and describe capability data according to it

// First Response
var firstResponse = ConvaAI.invokeCapability(
  input: "Blue jeans",
);

// Second response. Pass the history from the first response
// to the second one to maintain conversational context
var response = ConvaAI.invokeCapability(
  input: "show me in red",
  context: ConvaAIContext(
    history: firstResponse.history,
    capabilityContext: <"Map of String and Any">
  ),
);

(Any): The values can only include the following types:
String: For textual data.
Int: For integer numbers.
Double: For floating-point numbers.
Boolean: For true/false values.
Array: For lists of items (e.g., Array<String>, Array<Int>, etc.).
Map: For nested maps, allowing for hierarchical data structures.

var items = ["green tea", "freshly squeezed juice", "herbal tea", "smoothies", "coconut water", "coffee"];

// Example for passing capability context
var capabilityContext = {
  "product_search": {
    "best beverages": items
  }
};

// Note: Here <product_search> is the capability name.
// Use your own capability name and describe capability data according to it

// Optional Context: CapabilityContext is optional.
// If omitted, the AI will function based on the default 
// or previously provided context.

// First Response
let firstResponse: ConvaAICapability? = try await ConvaAI.invokeCapability(
    with: <"input_query">
)

let response: ConvaAICapability? = try await ConvaAI.invokeCapability(
    with: <"input_query">,
    context: ConvaAIContext(
        history: firstResponse.history,
        capabilityContext: <"Map of String and Any">
    )
)

(Any): The values can only include the following types:
String: For textual data.
Int: For integer numbers.
Double: For floating-point numbers.
Boolean: For true/false values.
Array: For lists of items (e.g., Array<String>, Array<Int>, etc.).
Map: For nested maps, allowing for hierarchical data structures.

let items = ["green tea", "freshly squeezed juice", "herbal tea", "smoothies", "coconut water", "coffee"]

// Example for passing capability context
capabilityContext: [
    "product_search": [
        "best beverages": items
    ]
]
// Note :- Here <product_search> is the capability name. 
// Use your own capability name and describe capability data according to it

// Optional Context: CapabilityContext is optional.
// If omitted, the AI will function based on the default 
// or previously provided context.

Notes

Ensure that this method is called from an async context.
Proper error handling should be implemented to manage any potential issues during the invocation.

client.invokeCapability({
    query: 'What can I wear for my graduation party?',
    stream: false,
    history: conversationHistory
}).then(response => {
    console.log('ConvaAI Response (First Query):', response);
    
    if (response && 'conversationHistory' in response) {
        conversationHistory = JSON.stringify(response.conversationHistory);
    }

    return client.invokeCapability({
        query: 'Can you tell me what you do?',
        stream: false,
        history: conversationHistory 
    });
}).then(response => {
    console.log('ConvaAI Response (Second Query):', response);
});

//ConvaAI Response (First Query): ConvaAIResponse {
//inputQuery: ‘What can I wear for my graduation party?’,
//message: ‘Searching for graduation party outfit ideas’,
//parameters: {
//search_term: ‘graduation party outfit’,
//category: ‘clothing’,
//filter_price_range: ‘’
//},
//relatedQueries: [
// ‘Explore formal dresses’, ‘Check out party accessories’, ‘View trending outfits for events’, ‘Browse new arrivals in clothing’, ‘Filter by color or style’ ],

// “conversationHistory”: { “items”: [ { “user_input”: “What can I wear for my graduation party?”, “assistant_response”: { “thought”: “Let’s think about this step by step. The user is looking for outfit ideas for a graduation party”, “category”: “clothing”
// … } }
// … ] }
//toolName: ‘product_search’ }
//ConvaAI Response (Second Query): ConvaAIResponse {
//inputQuery: ‘Can you tell me what you do?’,
//message: “I am here to help you explore the latest fashion trends and find stylish clothing and beauty products on Myntra! How can I assist you today?”,
//parameters: { interaction_type: ‘greeting’ },
//relatedQueries: [
//‘Ask about the latest fashion trends’, ‘Inquire about beauty products’, ‘Get style advice’ ],

// “conversationHistory”: { “items”: [ { “user_input”: “Can you tell me what you do?”, “assistant_response”: { “thought”: “Let’s think about this step by step. The user is asking about my role and what I do. I should provide a friendly and informative response without going into task-oriented details.”,
// “message”: “I’m here to help you explore the latest fashion trends and find stylish clothing and beauty products on Myntra! How can I assist you today?”
// … } }
// … ] }
//toolName: ‘small_talk’ }

Using the Copilot UI

The Copilot experience includes a well designed bottom sheet that appears on top of the app and provides the following elements:

Text box: For users to type their queries.
Mic icon: To trigger Conva.AI's highly accurate speech recognition feature
Message Area: Displays the message (the message field in the Response object)
Voice Feedback: Speak back the message, can be optionally muted by the user
Mute icon: To allow end-users to enable or disable the voice feedback
Feedback Icons: Automatically appear after each response.
Dynamic Suggestions: Buttons showing related queries (the related_queries field in the Response object) or custom suggestions provided by the app via an API.

Here are the high level steps to use the Copilot:

Setup the Copilot.
- Theme the Copilot
- Register handlers for Agent handling and Suggestion handling
  - Agent Handling: Handle responses generated by the Copilot
  - Suggestion Handling: Handle user interactions with suggestion buttons directed to the app
Attach the Copilot to an Activity
Start the Copilot
Handle the Agents and Suggestion clicks

PreviousIntegrating the Assistant NextCopilot Mode

Last updated 1 month ago