Config file

This guide covers usage of configuration file.

Idea

This document outlines how to configure the Maia Test Framework using the maia_test_config.yaml file and the MaiaConfig class.

The framework can be configured via a YAML file. By default, it looks for a file named maia_test_config.yaml in the current working directory.

Custom Configuration Path

You can specify a different path for the configuration file by setting the MAIA_TEST_CONFIG environment variable:

export MAIA_TEST_CONFIG=/path/to/your/config.yaml

Configuration Structure

The configuration file allows you to define different sections to manage your test settings. A common use case is to define different providers.

You can use the framework/tests/maia_test_config.template.yaml as a starting point.

Example maia_test_config.yaml:

providers:
  ollama:
    class: OllamaProvider
    config:
      model: mistral
      host: http://localhost:11434
  another_provider:
    class: AnotherProvider
    config:
      api_key: "some_key"

Note:

Class name shall be equal to name of the provider class in your implementation. For instance, LiteLLM has:

  class GenericLiteLLMProvider(LiteLLMBaseProvider):

so GenericLiteLLMProvider needs to be passed in class property. :

providers:
  ollama:
    class: GenericLiteLLMProvider
    config:
      model: mistral
      api_base: http://localhost:11434

Thanks to that, Maia knows how to map configuration to provider. Please note that config may differ between providers.

Using Environment Variables

You can embed environment variables directly into your YAML configuration file using the ${VAR_NAME} syntax. The MaiaConfig class will automatically expand these variables when it loads the file.

This is useful for keeping sensitive information like API keys out of your configuration files.

Example with Environment Variables:

providers:
  secret_provider:
    class: SecretProvider
    config:
      api_key: "${SECRET_API_KEY}"
      host: "${API_HOST}"

If an environment variable is not found, it will be replaced with the string <<MISSING_ENV:VAR_NAME>>.

Accessing Configuration in Code

The MaiaConfig class is a singleton that provides a global point of access to the configuration.

Getting the Singleton Instance

To access the configuration, first get the singleton instance of MaiaConfig:

from maia_test_framework.testing.maia_config import MaiaConfig

config_manager = MaiaConfig.get_instance()

Retrieving Configuration Sections

You can retrieve a specific top-level section of the configuration using the get_section(key) method:

# Get the 'providers' section
providers_config = config_manager.get_section("providers")

# Get the configuration for a specific provider
ollama_config = providers_config.get("ollama")
if ollama_config:
    print(f"Ollama Model: {ollama_config['config']['model']}")

Getting the Full Configuration

To get the entire configuration as a dictionary, use the get_full_config() method:

full_config = config_manager.get_full_config()
print(full_config)

Usage in test

Usage in test is simple, here is the example:

class TestConversationSessions(MaiaTest):
    def setup_agents(self):
        self.create_agent(
            name="Alice",
            provider=self.get_provider("ollama"),
            system_message="You are a weather assistant. Only describe the weather.",
            ignore_trigger_prompt="You MUST NOT answer questions about any other topic, including what to wear. If the user asks about anything other than the weather, you MUST respond with ONLY the exact text: IGNORE_MESSAGE. No explanation, no additional words, just the: IGNORE_MESSAGE",
        )

        self.create_agent(
            name="Bob",
            provider=GenericLiteLLMProvider(config={
                "model": "ollama/mistral",
                "api_base": "http://localhost:11434"
            }),
            system_message="You are a pirate assistant who only suggests clothing.",
            ignore_trigger_prompt="If the question is not about what to wear, you MUST respond with only the exact text: IGNORE_MESSAGE. No explanation, no additional words, just the: IGNORE_MESSAGE",
        )

    @pytest.mark.asyncio
    async def test_conversation_broadcast_no_policy(self):
        session = self.create_session(["Alice", "Bob"])

        await session.user_says_and_broadcast("Please describe the usual weather in London in July, including temperature and conditions.")
        
        assert_agent_participated(session, 'Alice')
        assert_agent_participated(session, "Bob")

You may notice that Alice agent has been created by using configuration for ollama provider. The configuration file for above test looks like this:

providers:
  ollama:
    class: OllamaProvider
    config:
      model: mistral
      host: http://localhost:11434

Orchestration

Tool