Headless Mode

This page is to detail the process of integrating the Conva.AI assistant in headless mode to an app.

Conva.AI supports SDKs for the following platforms

• Android

• Python

• iOS

• Flutter

• Rust - Coming soon

Let's walk through the integration process.

Setting up the build environment

Android

First, you need to add the maven repository URL to your project-level build.gradle file. Open the build.gradle file and add the following code inside the allprojects > repositories block

allprojects {
    repositories {
        maven {
            url "https://gitlab.com/api/v4/projects/25706948/packages/maven"
        }
    }
}

Next, open your app-level build.gradle file and add the conva-ai-core dependency inside the dependencies block

dependencies {
    implementation 'in.slanglabs.conva:conva-ai-core:1.0.1-beta'
}

Python

$ pip install conva-ai

Flutter

$ flutter pub add conva_ai_core

Once done, run the command 'dart pub get' and ensure Conva.AI is added to the dependencies.

dependencies:
  conva_ai_core: ^0.1.3

Check the latest version of SDK here:- https://pub.dev/packages/conva_ai_core

iOS

Add the Cocoapod repository path to your Podfile

# Add this to your podfile

source 'https://github.com/CocoaPods/Specs.git'

Next, open your podspec file and add the ConvaAICore dependency.

s.dependency 'ConvaAICore', '1.0.3'

Once done, run the command 'pod install'

Initializing the Assistant

First, initialize Conva.AI using the credentials of the Assistant created via Magic Studio.

Android

ConvaAI.init(
    id = "replace this string with your_assistant_id",
    key = "replace this string with your_api_key",
    version = "LATEST", // this is a special tag to indicate 
                        // the latest version of the Assistant
    application = applicationContext
);

Python

client = AsyncConvaAI(
    assistant_id="replace this string with your_assistant_id", 
    assistant_version="LATEST", # this is a special tag to indicate 
                                # the latest version of the Assistant
    api_key="replace this string with your_api_key"
)

Flutter

ConvaAI.init(
    id: "your_assistant_id",
    key: "your_api_key", 
    version: "LATEST"
);

iOS

ConvaAI.initialize(
    with: "<your_assistant_id>", 
    key: "<your_api_key>", 
    version: "<your_assistant_version>"
)

Open the "Integration" tab of your Assistant to get the keys.

Invoking a Capability

Next, pass unstructured text input to Conva.AI and let it invoke the appropriate Capability and return the response.

Android

// Pass an input string to Conva.AI and let it determine the 
// relevant Capability to invoke and return the output params
// after that Capability is executed
val response = ConvaAI.invokeCapability(
    input = "book a bus ticket from bangalore to chennai tomorrow at 2pm",
)

// response = {
//    "capability_name" : "ticket_booking",
//    "message" : "Showing you buses from Bangalore to Chennai for tommorrow around 2pm"
//    "params" : {
//         "source": "BLR",
//         "destination" : "MAS",
//         "date" : "6/7/2024",
//         "time" : "14:00"
//         "mode" : "bus"
//     }
// }

Python

# Pass an input string to Conva.AI and let it determine the 
# relevant Capability to invoke and return the output params
# after that Capability is executed
response = await client.invoke_capability("book a bus ticket from Bangalore to chennai tommorrow at 2pm")
return response

# response = {
#    "capability_name" : "ticket_booking",
#    "message" : "Showing you buses from Bangalore to Chennai for tommorrow around 2pm"
#    "params" : {
#         "source": "BLR",
#         "destination" : "MAS",
#         "date" : "6/7/2024",
#         "time" : "14:00"
#         "mode" : "bus"
#     }
# }

Flutter

// Pass an input string to Conva.AI and let it determine the 
// relevant Capability to invoke and return the output params
// after that Capability is executed
Response completion = await ConvaAI.invokeCapability(input: "<input_query>");
// completion = {
//    "capability_name" : "ticket_booking",
//    "message" : "Showing you buses from Bangalore to Chennai for tommorrow around 2pm"
//    "params" : {
//         "source": "BLR",
//         "destination" : "MAS",
//         "date" : "6/7/2024",
//         "time" : "14:00"
//         "mode" : "bus"
//     }
// }

Note

• Proper error handling should be implemented to manage any potential issues during the invocation.

iOS

// Pass an input string to Conva.AI and let it determine the 
// relevant Capability to invoke and return the output params
// after that Capability is executed.

// Invoke a capability (non-streaming)
let response: ConvaAICapability? = try await ConvaAI.invokeCapability(
    with: "<input_query>"
)

Notes

• Ensure that this method is called from an async context.

• Proper error handling should be implemented to manage any potential issues during the invocation.

Response object

The response object contains the following key fields

• capability_name: The name of the Capabiity that was triggered.

• message: The string containing either the answer to the original query or a status message. Capability creators can update this via Magic Studio.

• related_queries: An array of strings with queries related to the current query and app context. Capability creators can update this field's description if needed.

• params: This object contains custom parameters specific to the invoked Capability. For example, if a Capability is created with task_name, date, and time (as shown in the image below), these fields will be in the params object.

• history: The current conversation history. This can be used by the app to continue the current conversation further (more on this later).

Capability with custom parameters

Developers should check if a specific custom param exists before accessing its value, as it might not be marked as required in Magic Studio.

Invoking a specific Capability

To invoke a specific Capability, you can refer to it by name and pass the input string. You can use this in streaming (async) or non-streaming (sync) mode.

Android

// Invoke a specific capability
val response = ConvaAI.invokeCapabilityWithName(
    input = "book a bus ticket from Bangalore to chennai tommorrow at 2pm", 
    capability = "ticket_booking"
);

Python

# Invoke a specific Capability
response = await client.invoke_capability_with_name("book a bus ticket from Bangalore to chennai tommorrow at 2pm", capability_name="ticket_booking")
return response

Flutter

// Invoke a specific capability
Response completion = await ConvaAI.invokeCapabilityWithName(
    input: "book a bus ticket from Bangalore to chennai tommorrow at 2pm",
    capability: "ticket_booking"
);

Note

• Proper error handling should be implemented to manage any potential issues during the invocation.

iOS

// Invoke a capability with name (non-streaming)
let completion: ConvaAICapability? = try await ConvaAI.invokeCapability(
    with: <"input_query">,
    capabilityName: <"capability_name">
)

Notes

• Ensure that this method is called from an async context.

• Proper error handling should be implemented to manage any potential issues during the invocation.

Handling Streaming Response

Conva.AI supports a streaming version of these APIs. The response is sent back incrementally, with the listener invoked multiple times. Use the is_final flag in the Response object to check if all responses are received.

Android

// Invoke a capability asynchronously with a streaming response
ConvaAI.invokeCapability(
    input = "Hello, how are you?",
    listener = object : ResponseListener {
        override fun onResponse(response: Response, isFinal: Boolean) {
            // Check for is_final
            if (isFinal) {
                // Handle the final response
            }
        }
        override fun onError(error: Throwable) {
            // Handle the error
        }
    }
);

Python

# streaming version
response = await client.invoke_capability_stream(query)
out = ""
async for res in response:
    out = res
    
# At this point the "is_final" flag in the out object will be set
return out

Flutter

// Invoke a capability asynchronously with a streaming response
Stream<Response> completionStream =
            ConvaAI.invokeCapabilityStream(input: "Hello, how are you?");
completionStream.listen(
      (event) {
      // Use the response object
  },
  onError: (error) {
    // Handle error
  },
);

iOS

// Invoke a capability (streaming)
for try await response in ConvaAI.invokeCapabilityStream(
    with: "<input_query>"
) {
   // Handle the streaming response here
}

Notes

• Ensure that this method is called from an async context.

• Proper error handling should be implemented to manage any potential issues during the invocation.

Invoking a Capability Group

When creating Capabilities via Magic Studio, it places all Capabilities into the "default" group. But users can create their own groups and put specific Capabilities into it. If you want to limit the Capability invocation to specific groups, you can pass the group name to the APIs.

Android

// Invoke a capability group synchronously (non-streaming)
val response = ConvaAI.invokeCapability(
    input = "Hello, how are you?",
    capabilityGroup = "default"
);

// Invoke a capability group asynchronously to get a streaming response
ConvaAI.invokeCapability(
    input = "Hello, how are you?",
    capabilityGroup = "general_conversation",
    listener = object : ResponseListener {
        override fun onResponse(response: Response, isFinal: Boolean) {
            // Handle the response
        }
        override fun onError(error: Throwable) {
            // Handle the error
        }
    }
);

Python

# 'client' is the initialized ConvaAI object
# Streaming
response = await client.invoke_capability_stream(query, capability_group="default")
out = ""
async for res in response:
    out = res
return out

# Non-streaming
response = await client.invoke_capability(query, capability_group="default")
return response

Flutter

// Invoke a capability group synchronously (non-streaming)
Response completion = await ConvaAI.invokeCapability(
    input: "Hello, how are you?",
    capabilityGroup: "default"
);

// Invoke a capability group asynchronously to get a streaming response
Stream<Response> completionStream =
ConvaAI.invokeCapabilityStream(
  input: "Hello, how are you?",
  capabilityGroup: "general_conversation"
);
completionStream.listen(
      (event) {
      // Use the response object
  },
  onError: (error) {
    // Handle error
  },
);

Note

• Proper error handling should be implemented to manage any potential issues during the invocation.

iOS

// Invoke a capability with name (non-streaming)
let response: ConvaAICapability? = try await ConvaAI.invokeCapability(
    with: <"input_query">,
    capabilityGroup: <"capability_group">
)

Notes

• Ensure that this method is called from an async context.

• Proper error handling should be implemented to manage any potential issues during the invocation.

Handling Conversational Context

Conva.AI supports maintaining context across different conversations, allowing the AI to build upon prior interactions for more relevant responses.

Parameters

• history (Optional): Available after the first response. Pass this from the previous response if you want to maintain conversational context in subsequent calls.

• capabilityContext (Optional): Context related to a specific capability, refining AI responses for that particular capability.

Usage

• On the first invocation, call the API without history.

• On subsequent invocations, pass the history from the previous response if you want to maintain the conversation's context.

• You can also pass the capabilityGroup or capabilityName along with the context to refine the AI’s responses further.

Android

// First Response
val response1 = ConvaAI.invokeCapability(
    input = "Blue jeans",
);

// Second response. Pass the history from the first response
// to the second one to maintain conversational context
val reponse2 = ConvaAI.invokeCapability(
    input = "show me in red",
    context = ConvaAIContext(
        history = response1.history,
        capabilityContext = <"Map of String and Any">
    )
);

(Any): The values can only include the following types:
String: For textual data.
Int: For integer numbers.
Double: For floating-point numbers.
Boolean: For true/false values.
Array: For lists of items (e.g., Array<String>, Array<Int>, etc.).
Map: For nested maps, allowing for hierarchical data structures.

val items = listOf("green tea", "freshly squeezed juice", "herbal tea", "smoothies", "coconut water", "coffee")

// Example for passing capability context
val capabilityContext = mapOf(
    "product_search" to mapOf(
        "best beverages" to items
    )
)

// Note: Here <product_search> is the capability name.
// Use your own capability name and describe capability data according to it

// Optional Context: CapabilityContext is optional.
// If omitted, the AI will function based on the default 
// or previously provided context.

Python

# First Response
response1 = await client.invoke_capability("blue jeans")
return response

# Second response. Pass the history from the first response
# to the second one to maintain conversational context
response2 = await client.invoke_capability("show me in red", 
 history=response1.conversation_history,
 capability_context={ <capability_name>: <knowledge> })
return response

(Knowledge): The values can only include the following types:
String: For textual data.
Int: For integer numbers.
Double: For floating-point numbers.
Boolean: For true/false values.
Array: For lists of items (e.g., Array<String>, Array<Int>, etc.).
Map: For nested maps, allowing for hierarchical data structures.

items = ["green tea", "freshly squeezed juice", "herbal tea", "smoothies", "coconut water", "coffee"]

# Example for passing capability context
capabilityContext = {
    "product_search": {
        "best beverages": items
    }
}

# Note: Here <product_search> is the capability name. 
# Use your own capability name and describe capability data according to it

Flutter

// First Response
var firstResponse = ConvaAI.invokeCapability(
  input: "Blue jeans",
);

// Second response. Pass the history from the first response
// to the second one to maintain conversational context
var response = ConvaAI.invokeCapability(
  input: "show me in red",
  context: ConvaAIContext(
    history: firstResponse.history,
    capabilityContext: <"Map of String and Any">
  ),
);

(Any): The values can only include the following types:
String: For textual data.
Int: For integer numbers.
Double: For floating-point numbers.
Boolean: For true/false values.
Array: For lists of items (e.g., Array<String>, Array<Int>, etc.).
Map: For nested maps, allowing for hierarchical data structures.

var items = ["green tea", "freshly squeezed juice", "herbal tea", "smoothies", "coconut water", "coffee"];

// Example for passing capability context
var capabilityContext = {
  "product_search": {
    "best beverages": items
  }
};

// Note: Here <product_search> is the capability name.
// Use your own capability name and describe capability data according to it

// Optional Context: CapabilityContext is optional.
// If omitted, the AI will function based on the default 
// or previously provided context.

iOS

// First Response
let firstResponse: ConvaAICapability? = try await ConvaAI.invokeCapability(
    with: <"input_query">
)

let response: ConvaAICapability? = try await ConvaAI.invokeCapability(
    with: <"input_query">,
    context: ConvaAIContext(
        history: firstResponse.history,
        capabilityContext: <"Map of String and Any">
    )
)

(Any): The values can only include the following types:
String: For textual data.
Int: For integer numbers.
Double: For floating-point numbers.
Boolean: For true/false values.
Array: For lists of items (e.g., Array<String>, Array<Int>, etc.).
Map: For nested maps, allowing for hierarchical data structures.

let items = ["green tea", "freshly squeezed juice", "herbal tea", "smoothies", "coconut water", "coffee"]

// Example for passing capability context
capabilityContext: [
    "product_search": [
        "best beverages": items
    ]
]
// Note :- Here <product_search> is the capability name. 
// Use your own capability name and describe capability data according to it

// Optional Context: CapabilityContext is optional.
// If omitted, the AI will function based on the default 
// or previously provided context.

Notes

• Ensure that this method is called from an async context.

• Proper error handling should be implemented to manage any potential issues during the invocation.

Using the Copilot UI

The Copilot experience includes a well designed bottom sheet that appears on top of the app and provides the following elements:

• Text box: For users to type their queries.

• Mic icon: To trigger Conva.AI's highly accurate speech recognition feature

• Message Area: Displays the message (the message field in the Response object)

• Voice Feedback: Speak back the message, can be optionally muted by the user

• Mute icon: To allow end-users to enable or disable the voice feedback

• Feedback Icons: Automatically appear after each response.

• Dynamic Suggestions: Buttons showing related queries (the related_queries field in the Response object) or custom suggestions provided by the app via an API.

Here are the high level steps to use the Copilot:

• Setup the Copilot.

  • Theme the Copilot

  • Register handlers for Capability handling and Suggestion handling

    • Capability Handling: Handle responses generated by the Copilot

    • Suggestion Handling: Handle user interactions with suggestion buttons directed to the app

• Attach the Copilot to an Activity

• Start the Copilot

• Handle the Capabilities and Suggestion clicks