Assertion

Idea

In Maia Framework, we can differentiate two kinds of assertions - Message Assertion and Test Assertion.

Message Assertion

A Message Assertion in the Maia Framework is used to check messages produced by agents. When attached to a session, the passed function checks every message, and if a message does not match, an error is raised.

Here is an example of assertion usage:

session = self.create_session(
    ["Alice", "Bob"],
    assertions=[assert_professional_tone],
)

A Message Assertion is just a function that takes a Message as an argument and is executed against every message in a session. You can easily implement your own assertion and pass it to the session.

Please be aware that to have assertion properly working, you need to use our decorator called @as_assertion_factory. This guarantees rich debugging and visualization in our dashboard.

Example of implementation of assert_professional_tone:

from maia_test_framework.core.message import Message
from maia_test_framework.testing.assertions.base import as_assertion_factory

@as_assertion_factory
def assert_professional_tone(response: Message):
    """Assert response maintains professional tone"""
    unprofessional_indicators = [r"\blol\b", r"\bwtf\b", r"\bomg\b", r"\bur\b", r"\bu r\b"]
    found = [word for word in unprofessional_indicators if re.search(word, response.content.lower())]
    assert not found, f"Unprofessional language detected: {found}"

Test assertion

There are also special assertions that can be used outside of the session. At the moment, Maia provides one assertion (more are on the way), assert_agent_participated, which is used to check if an agent has participated in the conversation. It is especially useful when testing message broadcasting.

Here is an example of such an assertion:

from maia_test_framework.core.session import Session
from maia_test_framework.testing.assertions.base import as_assertion_factory

@as_assertion_factory
def assert_agent_participated(session: Session, agent_name: str):
    """Assert that specific agent participated in conversation"""
    agent_messages = [msg for msg in session.message_history if msg.sender == agent_name]
    assert len(agent_messages) > 0, f"Agent '{agent_name}' did not participate in conversation"
    return f"Agent '{agent_name}' participated in conversation"

Optionally, Test assertion can return str for richer debugging which gives status of assertion when assertion error is not raised.

To run Test assertion in Maia Framework you need to use self.run_assertion which guarantees rich debugging and visualization on our dashboard.

Example:

await session.user_says("Please describe the usual weather in London in July, including temperature and conditions.")
await session.agent_responds('Alice')
self.run_assertion(assert_agent_participated(session, 'Alice'))
await session.agent_responds('Bob')
self.run_assertion(assert_agent_participated(session, 'Bob'))

Using @as_assertion_factory gives you great visualizing effects on dashboard: