Templates

We run model comparisons for our own benchmarks and for our customers, and we've found a few setups that work for the most common annotation use cases. To use a template, you only need the image or video pairs, and for alignment also the prompts. Everything else uses our field-tested setup.

Template	Annotators answer	Use it to
Alignment	Which image matches the description better?	check how well a model follows prompts
Preference	Which image do you prefer?	find the model people like more
Coherence	Which image is more likely AI-generated?	see which model makes fewer AI mistakes

Alignment: does it follow the prompt?

Annotators trained for this read the prompt, look at both images and pick the one that matches it better. Send the prompt each pair was made from.

Refresh

from rapidata import RapidataClient

client = RapidataClient()

job = client.job.template.run_alignment(
    name="my-model vs flux-2",
    datapoints=[["mine/cat.png", "flux/cat.png"],
                ["mine/dog.png", "flux/dog.png"]],
    contexts=["A cat wearing a red scarf",
              "A dog surfing at sunset"],
)
results = job.get_results()

We recommend keeping prompts shorter than 200 characters for optimal results.

Preference: which one do people like?

A broad crowd picks the image they like more. No prompt needed.

Refresh

job = client.job.template.run_preference(
    name="my-model vs flux-2",
    datapoints=[["mine/cat.png", "flux/cat.png"],
                ["mine/dog.png", "flux/dog.png"]],
)
results = job.get_results()

Coherence: which one looks AI-generated?

Annotators trained to spot AI mistakes pick the image that looks more AI-generated.

Refresh

job = client.job.template.run_coherence(
    name="my-model vs flux-2",
    datapoints=[["mine/cat.png", "flux/cat.png"],
                ["mine/dog.png", "flux/dog.png"]],
)
results = job.get_results()

In Coherence, fewer wins is better

The winning side is the one that looks more AI-generated, so the better model wins fewer pairs.

See who won

Name the two sides, and the results count each side's wins:

job = client.job.template.run_preference(
    name="my-model vs flux-2",
    datapoints=[["mine/cat.png", "flux/cat.png"],
                ["mine/dog.png", "flux/dog.png"]],
    a_b_names=["my-model", "flux-2"],
)
job.get_results()["summary"]
# {'my-model_wins_total': 2, 'flux-2_wins_total': 0}

The results also hold all 11 votes for every pair.

Good to know

Images and videos both work as assets for templates. Send one kind per call.

Copy for AI