class Riffer::Evals::ScenarioResult

Represents the result of evaluating a single scenario.

Attributes

ground_truth [R]

The ground truth used during evaluation.

input [R]

The input that was evaluated.

messages [R]

The full message history from the agent conversation.

output [R]

The agent output for this scenario.

results [R]

Individual evaluation results.

token_usage [R]

Token usage the agent under test spent generating this scenario’s output.

Public Class Methods

new (input:, output:, ground_truth:, results:, messages: [], token_usage: nil)

Source

# File lib/riffer/evals/scenario_result.rb, line 26
def initialize(input:, output:, ground_truth:, results:, messages: [], token_usage: nil)
  @input = input
  @output = output
  @ground_truth = ground_truth
  @results = results
  @messages = messages
  @token_usage = token_usage
end

Public Instance Methods

evaluator_token_usage ()

Source

# File lib/riffer/evals/scenario_result.rb, line 51
def evaluator_token_usage
  results.filter_map(&:token_usage).reduce(:+)
end

Returns the summed token usage across this scenario’s LLM-as-judge evaluators, or nil when none reported usage.

scores ()

Source

# File lib/riffer/evals/scenario_result.rb, line 39
def scores
  acc = {} #: Hash[singleton(Riffer::Evals::Evaluator), Float]
  results.each_with_object(acc) do |result, hash|
    hash[result.evaluator] = result.score
  end
end

Returns scores keyed by evaluator class.

to_h ()

Source

# File lib/riffer/evals/scenario_result.rb, line 59
def to_h
  {
    input: input,
    output: output,
    ground_truth: ground_truth,
    scores: scores.transform_keys(&:name),
    results: results.map(&:to_h),
    messages: messages.map(&:to_h),
    token_usage: token_usage&.to_h,
    evaluator_token_usage: evaluator_token_usage&.to_h,
  }
end

Returns a hash representation of the scenario result.