Rapidata leaderboard

RapidataLeaderboard

RapidataLeaderboard(
    name: str,
    instruction: str,
    show_prompt: bool,
    show_prompt_asset: bool,
    inverse_ranking: bool,
    response_budget: int,
    min_responses_per_matchup: int,
    benchmark_id: str,
    id: str,
    openapi_service: OpenAPIService,
    included_tags: list[str] | None = None,
    excluded_tags: list[str] | None = None,
    vote_aggregation: VoteAggregation | None = None,
)

An instance of a Rapidata leaderboard.

Used to interact with a specific leaderboard in the Rapidata system, such as retrieving prompts and evaluating models.

Parameters:

Name	Type	Description	Default
name	str	The name that will be used to identify the leaderboard on the overview.	required
instruction	str	The instruction that will determine what how the models will be evaluated.	required
show_prompt	bool	Whether to show the prompt to the users.	required
id	str	The ID of the leaderboard.	required
openapi_service	OpenAPIService	The OpenAPIService instance for API interaction.	required
included_tags	list[str] | None	The prompt tag values the leaderboard restricts its matchups to.	None
excluded_tags	list[str] | None	The prompt tag values the leaderboard skips when building matchups.	None
vote_aggregation	VoteAggregation | None	How the responses on a single matchup are aggregated into that matchup's result. None resolves it on first read.	None

Source code in src/rapidata/rapidata_client/benchmark/leaderboard/rapidata_leaderboard.py

def __init__(
    self,
    name: str,
    instruction: str,
    show_prompt: bool,
    show_prompt_asset: bool,
    inverse_ranking: bool,
    response_budget: int,
    min_responses_per_matchup: int,
    benchmark_id: str,
    id: str,
    openapi_service: OpenAPIService,
    included_tags: list[str] | None = None,
    excluded_tags: list[str] | None = None,
    vote_aggregation: VoteAggregation | None = None,
):
    self.__openapi_service = openapi_service
    self.__name = name
    self.__instruction = instruction
    self.__show_prompt = show_prompt
    self.__show_prompt_asset = show_prompt_asset
    self.__inverse_ranking = inverse_ranking
    self.__response_budget = response_budget
    self.__min_responses_per_matchup = min_responses_per_matchup
    self.__benchmark_id = benchmark_id
    self.__included_tags = list(included_tags) if included_tags else []
    self.__excluded_tags = list(excluded_tags) if excluded_tags else []
    self.__vote_aggregation = vote_aggregation
    self.id = id
    self.__leaderboard_page = f"https://app.{self.__openapi_service.environment}/mri/benchmarks/{self.__benchmark_id}/leaderboard/{self.id}"

response_budget property

response_budget: int

The leaderboard's response budget — the total number of comparison responses collected per model evaluation.

This is the raw number behind :attr:level_of_detail; read it when you need the exact budget rather than the named level (which is "custom" for budgets that don't match a named level).

level_of_detail property

level_of_detail: ResolvedLevelOfDetail

The level of detail of the leaderboard.

This is a friendly name for the leaderboard's :attr:response_budget — the total number of comparison responses collected per model evaluation. A larger budget buys more matchups and therefore more precise standings, at the cost of a slower and more expensive evaluation. The named levels map to these budgets: 'debug' (20), 'low' (2,000), 'medium' (4,000), 'high' (8,000), 'very high' (16,000). Any other budget reads back as 'custom' — use :attr:response_budget to get the exact number.

min_responses_per_matchup property

min_responses_per_matchup: int

Returns the minimum number of responses required to be considered for the leaderboard.

vote_aggregation property

vote_aggregation: VoteAggregation

How the responses on a single matchup are aggregated into that matchup's result.

:attr:VoteAggregation.MAJORITY_VOTE collapses each matchup to one win for the side the majority of responses picked (ties split 0.5/0.5), so every matchup carries the same weight regardless of how many responses it collected. :attr:VoteAggregation.ALL_VOTES instead counts every individual response as its own matchup.

show_prompt_asset property

show_prompt_asset: bool

Returns whether the prompt asset is shown to the users.

inverse_ranking property

inverse_ranking: bool

Returns whether the ranking is inverse.

show_prompt property

show_prompt: bool

Returns whether the prompt is shown to the users.

instruction property

instruction: str

Returns the instruction of the leaderboard.

included_tags property

included_tags: list[str]

The prompt tag values this leaderboard restricts its matchups to.

Only benchmark prompts carrying at least one of these tags are used to build matchups; an empty list (the default) means no restriction. Fixed at creation — to re-scope a leaderboard, create a new one.

Not to be confused with the tags argument of :meth:get_standings, which filters the standings you read back rather than what gets collected.

excluded_tags property

excluded_tags: list[str]

The prompt tag values this leaderboard skips when building matchups.

A prompt carrying any of these tags is never used, even if it also carries one of :attr:included_tags. Fixed at creation — to re-scope a leaderboard, create a new one.

name property

name: str

Returns the name of the leaderboard.

jobs property

jobs: list[RapidataJob]

Returns all Rapidata jobs that have run for this leaderboard.

Every model evaluation on the leaderboard is carried out by a job. This collects the jobs across all runs of the leaderboard, most recent first.

Returns:

Type	Description
list[RapidataJob]	A list of RapidataJob instances, one per run that has an associated job.

update

update(
    name: str | None = None,
    level_of_detail: LevelOfDetail | int | None = None,
    min_responses_per_matchup: int | None = None,
    vote_aggregation: VoteAggregation | None = None,
) -> None

Updates the leaderboard's configuration.

Only the arguments you pass are changed; anything omitted keeps its stored value.

Parameters:

Name	Type	Description	Default
name	str | None	The new name of the leaderboard. (not shown to the users)	None
level_of_detail	LevelOfDetail | int | None	The new response budget — either one of the named levels ('debug', 'low', 'medium', 'high', 'very high') or a positive integer for a custom budget. Takes effect for future evaluations; already-computed standings are not recomputed.	None
min_responses_per_matchup	int | None	The new minimum number of responses collected per matchup. Must be at least 3.	None
vote_aggregation	VoteAggregation | None	How the responses on a single matchup are aggregated into that matchup's result. Standings are derived from the raw responses on every read, so this also changes how already-collected responses are counted — no re-evaluation needed.	None

Source code in src/rapidata/rapidata_client/benchmark/leaderboard/rapidata_leaderboard.py

def update(
    self,
    name: str | None = None,
    level_of_detail: LevelOfDetail | int | None = None,
    min_responses_per_matchup: int | None = None,
    vote_aggregation: VoteAggregation | None = None,
) -> None:
    """
    Updates the leaderboard's configuration.

    Only the arguments you pass are changed; anything omitted keeps its stored
    value.

    Args:
        name: The new name of the leaderboard. (not shown to the users)
        level_of_detail: The new response budget — either one of the named levels ('debug', 'low', 'medium', 'high', 'very high') or a positive integer for a custom budget. Takes effect for future evaluations; already-computed standings are not recomputed.
        min_responses_per_matchup: The new minimum number of responses collected per matchup. Must be at least 3.
        vote_aggregation: How the responses on a single matchup are aggregated into that matchup's result. Standings are derived from the raw responses on every read, so this also changes how already-collected responses are counted — no re-evaluation needed.
    """
    with tracer.start_as_current_span("RapidataLeaderboard.update"):
        if name is not None and (not isinstance(name, str) or len(name) < 1):
            raise ValueError("Name must be a string of at least 1 character")

        response_budget = (
            DetailMapper.resolve_budget(level_of_detail)
            if level_of_detail is not None
            else None
        )

        if min_responses_per_matchup is not None:
            # bool is an int subclass — reject it so `True` isn't read as 1.
            if isinstance(min_responses_per_matchup, bool) or not isinstance(
                min_responses_per_matchup, int
            ):
                raise ValueError("Min responses per matchup must be an integer")

            if min_responses_per_matchup < 3:
                raise ValueError("Min responses per matchup must be at least 3")

        if vote_aggregation is not None and not isinstance(
            vote_aggregation, VoteAggregation
        ):
            raise ValueError(
                "Vote aggregation must be one of: "
                + ", ".join(
                    f"VoteAggregation.{member.name}" for member in VoteAggregation
                )
            )

        logger.info(
            "Updating leaderboard %s with name %s, response_budget %s, min_responses_per_matchup %s, vote_aggregation %s",
            self.id,
            name,
            response_budget,
            min_responses_per_matchup,
            vote_aggregation.name if vote_aggregation is not None else None,
        )

        self.__openapi_service.leaderboard.leaderboard_api.leaderboard_leaderboard_id_patch(
            leaderboard_id=self.id,
            update_leaderboard_endpoint_input=UpdateLeaderboardEndpointInput(
                name=name,
                responseBudget=response_budget,
                minResponses=min_responses_per_matchup,
                voteAggregation=(
                    vote_aggregation._to_backend_model()
                    if vote_aggregation is not None
                    else None
                ),
            ),
        )

        if name is not None:
            self.__name = name
        if response_budget is not None:
            self.__response_budget = response_budget
        if min_responses_per_matchup is not None:
            self.__min_responses_per_matchup = min_responses_per_matchup
        if vote_aggregation is not None:
            self.__vote_aggregation = vote_aggregation

get_standings

get_standings(
    tags: Optional[list[str]] = None,
    country: Optional[list[str]] = None,
    language: Optional[list[str]] = None,
    gender: Optional[list[Gender]] = None,
    age_bucket: Optional[list[AgeGroup]] = None,
    occupation: Optional[list[str]] = None,
    run_id: Optional[str] = None,
) -> "pd.DataFrame"

Returns the standings of the leaderboard.

The demographic filters compute the standings from only the votes cast by matching voters — e.g. the standings among women, or among US voters. gender and age_bucket are estimated (inferred); country and language are observed.

Parameters:

Name	Type	Description	Default
tags	Optional[list[str]]	The matchups with these tags should be used to create the standings. If tags are None, all matchups will be considered. If tags are empty, no matchups will be considered.	None
country	Optional[list[str]]	Only count votes from these countries (ISO-2 codes).	None
language	Optional[list[str]]	Only count votes from these languages.	None
gender	Optional[list[Gender]]	Only count votes from voters of these (estimated) genders.	None
age_bucket	Optional[list[AgeGroup]]	Only count votes from voters in these (estimated) age buckets.	None
occupation	Optional[list[str]]	Only count votes from voters of these (estimated) occupations.	None
run_id	Optional[str]	Only count votes from this evaluation run.	None

Returns:

Type	Description
'pd.DataFrame'	A pandas DataFrame containing the standings of the leaderboard.

Source code in src/rapidata/rapidata_client/benchmark/leaderboard/rapidata_leaderboard.py

def get_standings(
    self,
    tags: Optional[list[str]] = None,
    country: Optional[list[str]] = None,
    language: Optional[list[str]] = None,
    gender: Optional[list[Gender]] = None,
    age_bucket: Optional[list[AgeGroup]] = None,
    occupation: Optional[list[str]] = None,
    run_id: Optional[str] = None,
) -> "pd.DataFrame":
    """
    Returns the standings of the leaderboard.

    The demographic filters compute the standings from only the votes cast by
    matching voters — e.g. the standings among women, or among US voters.
    ``gender`` and ``age_bucket`` are estimated (inferred); ``country`` and
    ``language`` are observed.

    Args:
        tags: The matchups with these tags should be used to create the standings.
            If tags are None, all matchups will be considered.
            If tags are empty, no matchups will be considered.
        country: Only count votes from these countries (ISO-2 codes).
        language: Only count votes from these languages.
        gender: Only count votes from voters of these (estimated) genders.
        age_bucket: Only count votes from voters in these (estimated) age buckets.
        occupation: Only count votes from voters of these (estimated) occupations.
        run_id: Only count votes from this evaluation run.

    Returns:
        A pandas DataFrame containing the standings of the leaderboard.
    """
    with tracer.start_as_current_span("RapidataLeaderboard.get_standings"):
        votes = demographic_filters(
            country, language, gender, age_bucket, occupation, run_id
        )
        participants = self.__openapi_service.leaderboard.leaderboard_api.leaderboard_leaderboard_id_standings_query_get(
            leaderboard_id=self.id,
            tags=in_filter(tags),
            country=votes.country,
            language=votes.language,
            gender=votes.gender,
            age_bucket=votes.age_bucket,
            occupation=votes.occupation,
            run_id=votes.run_id,
        )

        import pandas as pd

        standings = []
        for participant in participants.items:
            standings.append(
                {
                    "name": participant.name,
                    "wins": participant.wins,
                    "total_matches": participant.total_matches,
                    "score": (
                        round(participant.score, 2)
                        if participant.score is not None
                        else None
                    ),
                }
            )

        return pd.DataFrame(standings)

get_win_loss_matrix

get_win_loss_matrix(
    tags: Optional[list[str]] = None,
    use_weighted_scoring: Optional[bool] = None,
    country: Optional[list[str]] = None,
    language: Optional[list[str]] = None,
    gender: Optional[list[Gender]] = None,
    age_bucket: Optional[list[AgeGroup]] = None,
    occupation: Optional[list[str]] = None,
    run_id: Optional[str] = None,
) -> DataFrame

Returns the pairwise win/loss matrix for the participants in this leaderboard.

The returned DataFrame is square, with participant names on both the index (rows) and columns. Cell [i, j] is how often participant i (row) beat participant j (column) in their direct matchups. Read a row to see how a model did against every opponent; the diagonal (a model against itself) is always 0. This is the head-to-head breakdown behind :meth:get_standings, which collapses the same matchups into a single Elo score per model.

The demographic filters restrict the matrix to matchups decided by matching voters. gender and age_bucket are estimated (inferred); country and language are observed.

Parameters:

Name	Type	Description	Default
tags	Optional[list[str]]	Only count matchups carrying one of these prompt tags. If None, every matchup on the leaderboard is included; if an empty list, none are.	None
use_weighted_scoring	Optional[bool]	If True, each matchup is weighted by the responding annotators' reliability (userScore) instead of being counted as a plain win, so cells hold weighted sums (floats) rather than raw counts. If False, cells are raw win counts. When None (default), the server applies the leaderboard's configured default.	None
country	Optional[list[str]]	Only count votes from these countries (ISO-2 codes).	None
language	Optional[list[str]]	Only count votes from these languages.	None
gender	Optional[list[Gender]]	Only count votes from voters of these (estimated) genders.	None
age_bucket	Optional[list[AgeGroup]]	Only count votes from voters in these (estimated) age buckets.	None
occupation	Optional[list[str]]	Only count votes from voters of these (estimated) occupations.	None
run_id	Optional[str]	Only count votes from this evaluation run.	None

Returns:

Type	Description
DataFrame	A pandas DataFrame indexed by participant name on both axes, where cell
DataFrame	[i, j] holds the (optionally weighted) number of wins of the row
DataFrame	participant over the column participant.

Source code in src/rapidata/rapidata_client/benchmark/leaderboard/rapidata_leaderboard.py

def get_win_loss_matrix(
    self,
    tags: Optional[list[str]] = None,
    use_weighted_scoring: Optional[bool] = None,
    country: Optional[list[str]] = None,
    language: Optional[list[str]] = None,
    gender: Optional[list[Gender]] = None,
    age_bucket: Optional[list[AgeGroup]] = None,
    occupation: Optional[list[str]] = None,
    run_id: Optional[str] = None,
) -> pd.DataFrame:
    """
    Returns the pairwise win/loss matrix for the participants in this leaderboard.

    The returned DataFrame is square, with participant names on both the index
    (rows) and columns. Cell ``[i, j]`` is how often participant ``i`` (row) beat
    participant ``j`` (column) in their direct matchups. Read a row to see how a
    model did against every opponent; the diagonal (a model against itself) is
    always 0. This is the head-to-head breakdown behind :meth:`get_standings`,
    which collapses the same matchups into a single Elo score per model.

    The demographic filters restrict the matrix to matchups decided by matching
    voters. ``gender`` and ``age_bucket`` are estimated (inferred); ``country``
    and ``language`` are observed.

    Args:
        tags: Only count matchups carrying one of these prompt tags. If None,
            every matchup on the leaderboard is included; if an empty list, none are.
        use_weighted_scoring: If True, each matchup is weighted by the responding
            annotators' reliability (``userScore``) instead of being counted as a
            plain win, so cells hold weighted sums (floats) rather than raw counts.
            If False, cells are raw win counts. When None (default), the server
            applies the leaderboard's configured default.
        country: Only count votes from these countries (ISO-2 codes).
        language: Only count votes from these languages.
        gender: Only count votes from voters of these (estimated) genders.
        age_bucket: Only count votes from voters in these (estimated) age buckets.
        occupation: Only count votes from voters of these (estimated) occupations.
        run_id: Only count votes from this evaluation run.

    Returns:
        A pandas DataFrame indexed by participant name on both axes, where cell
        ``[i, j]`` holds the (optionally weighted) number of wins of the row
        participant over the column participant.
    """
    with tracer.start_as_current_span("RapidataLeaderboard.get_win_loss_matrix"):
        votes = demographic_filters(
            country, language, gender, age_bucket, occupation, run_id
        )
        result = self.__openapi_service.leaderboard.leaderboard_api.leaderboard_leaderboard_id_matrix_query_get(
            leaderboard_id=self.id,
            tags=in_filter(tags),
            use_weighted_scoring=use_weighted_scoring,
            country=votes.country,
            language=votes.language,
            gender=votes.gender,
            age_bucket=votes.age_bucket,
            occupation=votes.occupation,
            run_id=votes.run_id,
        )

        import pandas as pd

        return pd.DataFrame(
            data=result.data,
            index=pd.Index(result.index),
            columns=pd.Index(result.columns),
        )

view

view() -> None

Views the leaderboard.

Source code in src/rapidata/rapidata_client/benchmark/leaderboard/rapidata_leaderboard.py

def view(self) -> None:
    """
    Views the leaderboard.
    """
    logger.info("Opening leaderboard page in browser...")
    could_open_browser = webbrowser.open(self.__leaderboard_page)
    if not could_open_browser:
        encoded_url = urllib.parse.quote(
            self.__leaderboard_page, safe="%/:=&?~#+!$,;'@()*[]"
        )
        managed_print(
            Fore.RED
            + f"Please open this URL in your browser: '{encoded_url}'"
            + Fore.RESET
        )