Rapidata leaderboard

RapidataLeaderboard #

RapidataLeaderboard(
    name: str,
    instruction: str,
    show_prompt: bool,
    show_prompt_asset: bool,
    inverse_ranking: bool,
    response_budget: int,
    min_responses_per_matchup: int,
    benchmark_id: str,
    id: str,
    openapi_service: OpenAPIService,
    included_tags: list[str] | None = None,
    excluded_tags: list[str] | None = None,
)

An instance of a Rapidata leaderboard.

Used to interact with a specific leaderboard in the Rapidata system, such as retrieving prompts and evaluating models.

Parameters:

Name	Type	Description	Default
`name`	`str`	The name that will be used to identify the leaderboard on the overview.	required
`instruction`	`str`	The instruction that will determine what how the models will be evaluated.	required
`show_prompt`	`bool`	Whether to show the prompt to the users.	required
`id`	`str`	The ID of the leaderboard.	required
`openapi_service`	`OpenAPIService`	The OpenAPIService instance for API interaction.	required
`included_tags`	`list[str] \| None`	The prompt tag values the leaderboard restricts its matchups to.	`None`
`excluded_tags`	`list[str] \| None`	The prompt tag values the leaderboard skips when building matchups.	`None`

Source code in src/rapidata/rapidata_client/benchmark/leaderboard/rapidata_leaderboard.py

def __init__(
    self,
    name: str,
    instruction: str,
    show_prompt: bool,
    show_prompt_asset: bool,
    inverse_ranking: bool,
    response_budget: int,
    min_responses_per_matchup: int,
    benchmark_id: str,
    id: str,
    openapi_service: OpenAPIService,
    included_tags: list[str] | None = None,
    excluded_tags: list[str] | None = None,
):
    self.__openapi_service = openapi_service
    self.__name = name
    self.__instruction = instruction
    self.__show_prompt = show_prompt
    self.__show_prompt_asset = show_prompt_asset
    self.__inverse_ranking = inverse_ranking
    self.__response_budget = response_budget
    self.__min_responses_per_matchup = min_responses_per_matchup
    self.__benchmark_id = benchmark_id
    self.__included_tags = list(included_tags) if included_tags else []
    self.__excluded_tags = list(excluded_tags) if excluded_tags else []
    self.id = id
    self.__leaderboard_page = f"https://app.{self.__openapi_service.environment}/mri/benchmarks/{self.__benchmark_id}/leaderboard/{self.id}"

response_budget `property` #

response_budget: int

The leaderboard's response budget — the total number of comparison responses collected per model evaluation.

This is the raw number behind :attr:level_of_detail; read it when you need the exact budget rather than the named level (which is "custom" for budgets that don't match a named level).

level_of_detail `property` `writable` #

level_of_detail: ResolvedLevelOfDetail

The level of detail of the leaderboard.

This is a friendly name for the leaderboard's :attr:response_budget — the total number of comparison responses collected per model evaluation. A larger budget buys more matchups and therefore more precise standings, at the cost of a slower and more expensive evaluation. The named levels map to these budgets: 'debug' (20), 'low' (2,000), 'medium' (4,000), 'high' (8,000), 'very high' (16,000). Any other budget reads back as 'custom' — use :attr:response_budget to get the exact number.

min_responses_per_matchup `property` `writable` #

min_responses_per_matchup: int

Returns the minimum number of responses required to be considered for the leaderboard.

show_prompt_asset `property` #

show_prompt_asset: bool

Returns whether the prompt asset is shown to the users.

inverse_ranking `property` #

inverse_ranking: bool

Returns whether the ranking is inverse.

show_prompt `property` #

show_prompt: bool

Returns whether the prompt is shown to the users.

instruction `property` #

instruction: str

Returns the instruction of the leaderboard.

included_tags `property` #

included_tags: list[str]

The prompt tag values this leaderboard restricts its matchups to.

Only benchmark prompts carrying at least one of these tags are used to build matchups; an empty list (the default) means no restriction. Fixed at creation — to re-scope a leaderboard, create a new one.

Not to be confused with the tags argument of :meth:get_standings, which filters the standings you read back rather than what gets collected.

excluded_tags `property` #

excluded_tags: list[str]

The prompt tag values this leaderboard skips when building matchups.

A prompt carrying any of these tags is never used, even if it also carries one of :attr:included_tags. Fixed at creation — to re-scope a leaderboard, create a new one.

name `property` `writable` #

name: str

Returns the name of the leaderboard.

jobs `property` #

jobs: list[RapidataJob]

Returns all Rapidata jobs that have run for this leaderboard.

Every model evaluation on the leaderboard is carried out by a job. This collects the jobs across all runs of the leaderboard, most recent first.

Returns:

Type	Description
`list[RapidataJob]`	A list of RapidataJob instances, one per run that has an associated job.

get_standings #

get_standings(
    tags: Optional[list[str]] = None,
) -> "pd.DataFrame"

Returns the standings of the leaderboard.

Parameters:

Name	Type	Description	Default
`tags`	`Optional[list[str]]`	The matchups with these tags should be used to create the standings. If tags are None, all matchups will be considered. If tags are empty, no matchups will be considered.	`None`

Returns:

Type	Description
`'pd.DataFrame'`	A pandas DataFrame containing the standings of the leaderboard.

Source code in src/rapidata/rapidata_client/benchmark/leaderboard/rapidata_leaderboard.py

def get_standings(self, tags: Optional[list[str]] = None) -> "pd.DataFrame":
    """
    Returns the standings of the leaderboard.

    Args:
        tags: The matchups with these tags should be used to create the standings.
            If tags are None, all matchups will be considered.
            If tags are empty, no matchups will be considered.

    Returns:
        A pandas DataFrame containing the standings of the leaderboard.
    """
    with tracer.start_as_current_span("RapidataLeaderboard.get_standings"):
        tags_filter = AudienceAudienceIdJobsGetJobIdParameter()
        tags_filter.var_in = tags
        participants = self.__openapi_service.leaderboard.leaderboard_api.leaderboard_leaderboard_id_standings_query_get(
            leaderboard_id=self.id, tags=tags_filter
        )

        import pandas as pd

        standings = []
        for participant in participants.items:
            standings.append(
                {
                    "name": participant.name,
                    "wins": participant.wins,
                    "total_matches": participant.total_matches,
                    "score": (
                        round(participant.score, 2)
                        if participant.score is not None
                        else None
                    ),
                }
            )

        return pd.DataFrame(standings)

get_win_loss_matrix #

get_win_loss_matrix(
    tags: Optional[list[str]] = None,
    use_weighted_scoring: Optional[bool] = None,
) -> DataFrame

Returns the pairwise win/loss matrix for the participants in this leaderboard.

The returned DataFrame is square, with participant names on both the index (rows) and columns. Cell [i, j] is how often participant i (row) beat participant j (column) in their direct matchups. Read a row to see how a model did against every opponent; the diagonal (a model against itself) is always 0. This is the head-to-head breakdown behind :meth:get_standings, which collapses the same matchups into a single Elo score per model.

Parameters:

Name	Type	Description	Default
`tags`	`Optional[list[str]]`	Only count matchups carrying one of these prompt tags. If None, every matchup on the leaderboard is included; if an empty list, none are.	`None`
`use_weighted_scoring`	`Optional[bool]`	If True, each matchup is weighted by the responding annotators' reliability (`userScore`) instead of being counted as a plain win, so cells hold weighted sums (floats) rather than raw counts. If False, cells are raw win counts. When None (default), the server applies the leaderboard's configured default.	`None`

Returns:

Type	Description
`DataFrame`	A pandas DataFrame indexed by participant name on both axes, where cell
`DataFrame`	`[i, j]` holds the (optionally weighted) number of wins of the row
`DataFrame`	participant over the column participant.

Source code in src/rapidata/rapidata_client/benchmark/leaderboard/rapidata_leaderboard.py

def get_win_loss_matrix(
    self,
    tags: Optional[list[str]] = None,
    use_weighted_scoring: Optional[bool] = None,
) -> pd.DataFrame:
    """
    Returns the pairwise win/loss matrix for the participants in this leaderboard.

    The returned DataFrame is square, with participant names on both the index
    (rows) and columns. Cell ``[i, j]`` is how often participant ``i`` (row) beat
    participant ``j`` (column) in their direct matchups. Read a row to see how a
    model did against every opponent; the diagonal (a model against itself) is
    always 0. This is the head-to-head breakdown behind :meth:`get_standings`,
    which collapses the same matchups into a single Elo score per model.

    Args:
        tags: Only count matchups carrying one of these prompt tags. If None,
            every matchup on the leaderboard is included; if an empty list, none are.
        use_weighted_scoring: If True, each matchup is weighted by the responding
            annotators' reliability (``userScore``) instead of being counted as a
            plain win, so cells hold weighted sums (floats) rather than raw counts.
            If False, cells are raw win counts. When None (default), the server
            applies the leaderboard's configured default.

    Returns:
        A pandas DataFrame indexed by participant name on both axes, where cell
        ``[i, j]`` holds the (optionally weighted) number of wins of the row
        participant over the column participant.
    """
    with tracer.start_as_current_span("RapidataLeaderboard.get_win_loss_matrix"):
        tags_filter = AudienceAudienceIdJobsGetJobIdParameter()
        tags_filter.var_in = tags
        result = self.__openapi_service.leaderboard.leaderboard_api.leaderboard_leaderboard_id_matrix_query_get(
            leaderboard_id=self.id,
            tags=tags_filter,
            use_weighted_scoring=use_weighted_scoring,
        )

        import pandas as pd

        return pd.DataFrame(
            data=result.data,
            index=pd.Index(result.index),
            columns=pd.Index(result.columns),
        )

view #

view() -> None

Views the leaderboard.

Source code in src/rapidata/rapidata_client/benchmark/leaderboard/rapidata_leaderboard.py

def view(self) -> None:
    """
    Views the leaderboard.
    """
    logger.info("Opening leaderboard page in browser...")
    could_open_browser = webbrowser.open(self.__leaderboard_page)
    if not could_open_browser:
        encoded_url = urllib.parse.quote(
            self.__leaderboard_page, safe="%/:=&?~#+!$,;'@()*[]"
        )
        managed_print(
            Fore.RED
            + f"Please open this URL in your browser: '{encoded_url}'"
            + Fore.RESET
        )

Rapidata leaderboard