Validator

This chapter describes the concept of a validator.

Idea

A Validator in the Maia Framework is used to check the whole session for specific scenarios.

Here is an example of validator usage:

  @pytest.mark.asyncio
  async def test_multi_turn_agent_conversation_fails_on_etiquette(self):
      session = self.create_session(["Alice", "Bob"])
      
      await session.agent_says("Alice", "Bob", "What's the weather?")
      await session.agent_responds("Alice")

      with pytest.raises(AssertionError, match=r"Agent Alice sent two messages in a row."):
          # This will be called in teardown, but we can call it manually for testing
          self.run_validator(ConversationValidator(session))

And here is the validator implementation:

class ConversationValidator(BaseValidator):
    def __init__(self, session: Session):
        super().__init__(session)
        
    def assert_turn_taking_etiquette(self):
        """Asserts that agents take turns in the conversation."""
        messages = self.session.message_history
        for i in range(1, len(messages)):
            if messages[i].sender == messages[i-1].sender:
                raise AssertionError(f"Agent {messages[i].sender} sent two messages in a row.")

    def validate(self):
        self.assert_turn_taking_etiquette()

In the example above, ConversationValidator is responsible for checking if every agent responded only once to a message.

You can create your own validator using the BaseValidator class.

Validator vs Assertion

Validator works similarly to an Assertion, but it is executed at teardown, so after the whole session, it has access to the entire history. This means you can have more holistic checks. On the other hand, you can often achieve a similar thing with an Assertion, but it would require more checks in tests. You are free to choose your own way of testing.

Assertion

Writing tests