Model Ranking Insights Advanced#
Overview#
To unlock the full potential of Model Ranking Insights (MRI), you can use the advanced features. These include sophisticated configuration options for benchmarks, leaderboards, and evaluation settings that give you fine-grained control over your model evaluation process.
Benchmark Configuration#
Using Identifiers#
In the MRI quickstart we used the prompts to identify the media and create the appropriate matchups. However, more generally you might not have an exact 1-to-1 relationship between prompts and media (e.g., you may have different settings or inputs for the same prompt - for example input images for image-to-video models. More about this below). To handle this case, we allow you to supply your own identifiers, which will then be used when creating the matchups.
# Example 1: Explicit identifiers
benchmark = client.mri.create_new_benchmark(
name="Preference Benchmark",
identifiers=["scene_1", "scene_2", "scene_3"],
prompts=[
"A serene mountain landscape at sunset",
"A futuristic city with flying cars",
"A portrait of a wise old wizard"
],
prompt_assets=[
["https://assets.rapidata.ai/mountain_sunset.png"],
["https://assets.rapidata.ai/futuristic_city.png"],
["https://assets.rapidata.ai/wizard_portrait.png"]
]
)
# Example 2: Identifiers used for the same prompts but different seeding
benchmark = client.mri.create_new_benchmark(
name="Preference Benchmark",
identifiers=["seed_1", "seed_2", "seed_3"],
prompts=["prompt_1", "prompt_1", "prompt_1"],
prompt_assets=[["https://example.com/asset1.jpg"], ["https://example.com/asset1.jpg"], ["https://example.com/asset1.jpg"]]
)
# Example 3: Using only prompt assets
benchmark = client.mri.create_new_benchmark(
name="Preference Benchmark",
identifiers=["image_1", "image_2", "image_3"],
prompt_assets=[["https://example.com/asset1.jpg"], ["https://example.com/asset2.jpg"], ["https://example.com/asset3.jpg"]]
)
# Example 4: Several assets for one prompt, e.g. a reference clip plus a start frame
benchmark = client.mri.create_new_benchmark(
name="Camera Motion Benchmark",
identifiers=["move_forward", "pan_left"],
prompts=["The camera moves forward", "The camera pans left"],
prompt_assets=[
["https://example.com/move_forward.gif", "https://example.com/interior.jpg"],
["https://example.com/pan_left.gif", "https://example.com/street.jpg"]
]
)
Note
Media assets are images, videos, or audio files that provide visual or auditory context for your evaluation prompts. For example when evaluating image to video models. prompt_assets takes one list per prompt, the same shape as media_contexts on job definitions: one entry for a single asset, several to show them together. benchmark.prompt_assets reads them back in that same shape.
Tagging System#
Tags provide metadata for filtering and organizing benchmark results without showing them to evaluators. These tags can also be set and used in the frontend. To view the frontend, you can use the view method of the benchmark or leaderboard.
Pass tags as plain strings — one list per prompt:
tags = [
["beach", "outdoor"],
["mountain", "outdoor"],
["city"],
["vehicle", "indoor"],
]
benchmark = client.mri.create_new_benchmark(
name="Tagged Benchmark",
identifiers=["scene_1", "scene_2", "scene_3", "scene_4"],
prompts=["A sunny beach", "A mountain landscape", "A city skyline", "A car in a garage"],
tags=tags,
)
# Filter leaderboard results by tag value
standings = leaderboard.get_standings(tags=["outdoor"])
Grouping tags by category#
To group related tags, swap any string for a Tag with an optional category. Strings and Tags can be mixed freely in the same list — strings become tags with no category. You can also record where a prompt came from with an origin.
from rapidata import Tag
benchmark = client.mri.create_new_benchmark(
name="Tagged Benchmark",
identifiers=["scene_1", "scene_2"],
prompts=["A sunny beach", "A car in a garage"],
tags=[
[Tag("beach", category="scene"), "outdoor"],
[Tag("vehicle", category="object"), "indoor"],
],
origins=["coco", "coco"],
)
Note
Categories are additive — existing scripts that pass list[list[str]] keep
working unchanged. Read tags back with the values-only benchmark.tags
(list[list[str]]), or benchmark.structured_tags (list[list[Tag]]) to see
categories, and prompt origins with benchmark.origins.
Adding prompts and assets after benchmark creation#
If you have already created a benchmark and want to add new prompts and assets after the fact. Note however that these will only take effect for new models.
# Adding prompts with assets (one or many, matched up by index)
benchmark.add_prompts(
identifiers=["new_style"],
prompts=["Generate artwork in this new style"],
prompt_assets=[["https://assets.rapidata.ai/new_style_ref.jpg"]],
tags=[["abstract", "modern"]],
)
Editing a prompt's tags and origin#
Update the tags and/or origin of an already-registered prompt. Only the fields you pass are changed; omit one to leave it as-is.
Leaderboard Configuration#
Inverse Ranking#
For evaluation questions where lower scores are better (e.g., "Which image is worse?"), use inverse ranking.
leaderboard = benchmark.create_leaderboard(
name="Quality Assessment",
instruction="Which image has lower quality?",
inverse_ranking=True, # Lower scores = better performance
show_prompt=True,
show_prompt_asset=True
)
Vote Aggregation#
Each matchup — one comparison of two models on one prompt — collects several responses. By default the matchup's winner is the side the majority of those responses picked, with ties split 0.5/0.5.
Pass vote_aggregation=VoteAggregation.ALL_VOTES to count every individual
response as its own matchup instead:
from rapidata import VoteAggregation
leaderboard = benchmark.create_leaderboard(
name="Realism",
instruction="Which image is more realistic?",
vote_aggregation=VoteAggregation.ALL_VOTES,
)
Standings are derived from the raw responses on every read, so switching the aggregation on an existing leaderboard also changes how its already-collected responses are counted — no re-evaluation needed.
print(leaderboard.vote_aggregation) # VoteAggregation.ALL_VOTES
leaderboard.update(vote_aggregation=VoteAggregation.MAJORITY_VOTE)
Level of Detail (response budget)#
level_of_detail sets the leaderboard's response budget — the total number of
comparison responses collected per model evaluation. A larger budget buys more
matchups, which makes the standings more precise but makes each evaluation slower
and more expensive. The named levels map to concrete budgets:
level_of_detail |
Responses per model evaluation |
|---|---|
"debug" |
20 |
"low" |
2,000 |
"medium" |
4,000 |
"high" |
8,000 |
"very high" |
16,000 |
Omitting level_of_detail lets the server pick a default.
Need a budget between (or beyond) the named levels? Pass a positive integer instead of a name to set a custom response budget directly:
leaderboard = benchmark.create_leaderboard(
name="Custom Budget",
instruction="Which image do you prefer?",
level_of_detail=5000, # exactly 5,000 responses per model evaluation
)
# Different detail levels
leaderboard_fast = benchmark.create_leaderboard(
name="Quick Evaluation",
instruction="Which image do you prefer?",
level_of_detail="low" # Fewer comparisons, faster results
)
leaderboard_precise = benchmark.create_leaderboard(
name="Precise Evaluation",
instruction="Which image do you prefer?",
level_of_detail="very high" # More comparisons, higher accuracy
)
You can read the budget on an existing leaderboard through the level_of_detail
property and change it with update(), which likewise accepts a named level or a
custom integer. Changes apply to future evaluations; standings that have already
been computed are not recomputed.
print(leaderboard.level_of_detail) # e.g. "low"
leaderboard.update(level_of_detail="high") # named level
leaderboard.update(level_of_detail=5000) # custom budget
A custom budget reads back as "custom"; use the response_budget property to
get the exact number.
leaderboard.update(level_of_detail=5000)
print(leaderboard.level_of_detail) # "custom"
print(leaderboard.response_budget) # 5000
Changing a leaderboard's configuration#
update() is the single entry point for every mutable leaderboard setting. Only
the arguments you pass are changed; anything you omit keeps its stored value, and
all of them go out in one request.
leaderboard.update(
name="Realism v2",
level_of_detail="high",
min_responses_per_matchup=5,
vote_aggregation=VoteAggregation.MAJORITY_VOTE,
)
Restricting which prompts a leaderboard uses#
A leaderboard normally builds matchups from every prompt in its benchmark. Pass
included_tags and/or excluded_tags at creation to narrow that down to a slice
of the tagged prompts — useful for running a focused
leaderboard (say, outdoor scenes only) against a broad benchmark.
leaderboard = benchmark.create_leaderboard(
name="Realism (outdoor)",
instruction="Which image is more realistic?",
included_tags=["outdoor"], # only prompts tagged "outdoor"
excluded_tags=["nsfw"], # never these, even if also tagged "outdoor"
)
A prompt is used when it carries at least one included_tags value and no
excluded_tags value. excluded_tags always wins. An empty or omitted
included_tags means no restriction; a non-empty one drops prompts that have no
tags at all. Matching is on the tag value — a tag's category is irrelevant.
The rules are applied when a run starts rather than snapshotted at creation, so re-tagging a prompt changes which future runs pick it up. They only affect what gets collected; votes already recorded stay in the standings.
Note
This is a different thing from get_standings(tags=...). These arguments decide
which prompts get matchups collected for them, while get_standings(tags=...)
filters the standings you read back out of what was already collected.
Both are set at creation and read back as read-only properties:
They cannot be changed afterwards — a leaderboard whose history was collected under changing rules is hard to interpret. To re-scope, create a new leaderboard.
Prompt and Asset Display#
Control what evaluators see during comparison.
leaderboard = benchmark.create_leaderboard(
name="Context-Aware Evaluation",
instruction="Which generated image better matches the prompt?",
show_prompt=True, # Show the original text prompt
show_prompt_asset=True, # Show reference images/videos
level_of_detail="medium"
)
Participant Management#
Listing Participants#
You can list all participants in a benchmark using the participants property:
for participant in benchmark.participants:
print(f"{participant.name} - {participant.status} - {participant.price} per {participant.price_unit}")
price and price_unit are None for models without a price. See
Update participant metadata for how to set them.
Submitting Participants#
When using add_model, participants are created in the CREATED state and are not yet submitted for evaluation. You can submit them individually or in bulk:
# Submit a single participant
participant = benchmark.add_model(
name="ModelA",
media=["https://example.com/img1.png"],
identifiers=["scene_1"]
)
participant.run()
# Or add multiple models and submit them all at once
benchmark.add_model(name="ModelB", media=["https://example.com/img2.png"], identifiers=["scene_1"])
benchmark.add_model(name="ModelC", media=["https://example.com/img3.png"], identifiers=["scene_1"])
benchmark.run() # Submits all CREATED participants in a single batch request
Submitting in one batch evaluates the participants symmetrically as a single run — each model compared against every other and against the benchmark's already-submitted field — rather than as separate per-participant runs.
Inspecting a Participant's Elo#
Each participant has an Elo score aggregated across all of the benchmark's leaderboards. Read it directly from the participant:
participant = benchmark.participants[0]
elo = participant.get_elo() # None if not computed yet
print(f"{participant.name}: {elo}")
Renaming a Participant#
Disabling Participants#
A disabled participant is excluded from evaluation and the computed standings. Unlike deleting, this is reversible:
participant = benchmark.participants[0]
participant.disable()
# Bring it back into the evaluation later
participant.enable()
Deleting Participants#
You can remove a participant — and its uploaded media — from the benchmark. This cannot be undone:
Voter Demographics#
Every vote on a benchmark carries the voter's demographics, so you can see who
evaluated your models and how different groups ranked them. Both methods live on
the benchmark and return a pandas DataFrame, like get_overall_standings.
Note
ageBucket, gender and occupation are estimated (inferred from
behaviour), not self-declared. country and language are observed. Each
dimension includes an "unknown" bucket for votes whose attribute could not
be determined.
Voter composition#
get_demographics returns one row per (dimension, value) with the raw vote
count and its share of the dimension (shares within a dimension sum to 1). The
dimension column holds BenchmarkDemographicDimension values (AgeBucket,
Gender, Occupation, Country, Language):
from rapidata import BenchmarkDemographicDimension
demographics = benchmark.get_demographics()
# columns: dimension, value, votes, share
countries = demographics[
demographics["dimension"] == BenchmarkDemographicDimension.COUNTRY
]
print(countries)
Standings per demographic segment#
get_standings_breakdown returns one row per (segment, model) — how each
demographic segment of voters ranks the models, with that segment's vote count.
Useful for spotting where a model is ranked differently by different groups. For
the overall standings across all voters, use get_overall_standings. The
dimension is a BenchmarkDemographicDimension.
from rapidata import BenchmarkDemographicDimension
breakdown = benchmark.get_standings_breakdown(
dimension=BenchmarkDemographicDimension.AGEBUCKET,
)
# columns: segment, segment_votes, name, wins, total_matches, score
Segments are raw vote counts.
Filtering standings and matrices by demographics#
The demographic dimensions are also filters. get_overall_standings,
get_win_loss_matrix, get_demographics and get_standings_breakdown on the
benchmark — and get_standings / get_win_loss_matrix on a leaderboard — all
accept country, language, gender, age_bucket, occupation and run_id
(on top of tags / leaderboard_ids). This computes the result from only the
votes cast by matching voters, e.g. the standings among young women:
from rapidata import BenchmarkDemographicDimension, Gender, AgeGroup
standings = benchmark.get_overall_standings(
gender=[Gender.FEMALE],
age_bucket=[AgeGroup.BETWEEN_18_29],
country=["US", "GB"],
)
# and any of the other read methods, e.g.
breakdown = benchmark.get_standings_breakdown(
dimension=BenchmarkDemographicDimension.COUNTRY,
tags=["landscape"],
gender=[Gender.FEMALE],
)
gender and age_bucket take the SDK's Gender / AgeGroup enums; country,
language and occupation take plain values.
Win/Loss Matrix#
get_standings collapses every matchup into one Elo score per model. When you want
the head-to-head breakdown instead, use get_win_loss_matrix. It returns a square
pandas DataFrame with participant names on both axes, where cell [i, j] is how
often the row model i beat the column model j. Reading a single row tells you
how one model fared against each opponent; the diagonal is always 0.
# Per-leaderboard: matchups within one leaderboard
matrix = leaderboard.get_win_loss_matrix()
# Benchmark-wide: aggregated across all leaderboards
matrix = benchmark.get_win_loss_matrix()
print(matrix)
# ModelA ModelB ModelC
# ModelA 0 12 9
# ModelB 4 0 7
# ModelC 6 8 0
Both methods accept tags to restrict the count to matchups with specific prompt
tags. The benchmark-level method additionally accepts participant_ids and
leaderboard_ids to narrow the participants and leaderboards included.
By default the cells are raw win counts. Pass use_weighted_scoring=True to weight
each matchup by the responding annotators' reliability (userScore) instead — the
cells then hold weighted sums (floats) rather than integer counts.
Accessing the Underlying Jobs#
The standings and win/loss matrix are aggregates. If you need the raw responses
behind them, use the leaderboard's jobs property: every model evaluation on a
leaderboard runs as a job, and this returns all of them, most recent first.
leaderboard = benchmark.leaderboards[0]
jobs = leaderboard.jobs # list[RapidataJob], newest first
print(f"{len(jobs)} evaluations have run for this leaderboard")
Each item is a full RapidataJob, so you can pull the detailed comparison
results, check status, or open a job in the dashboard:
for job in leaderboard.jobs:
if job.get_status() != "Completed":
continue
results = job.get_results() # RapidataResults — the individual matchups
df = results.to_pandas() # or work with the raw dict directly
Runs that don't yet have an associated job (for example, an evaluation still being set up) are skipped, so the list only contains jobs you can act on.