cappr.utils.classify#

Transform completion token log-probabilites into a probability distribution over completions

cappr.utils.classify.agg_log_probs(log_probs: Sequence[Sequence[float]] | Sequence[Sequence[Sequence[float]]], func: Callable = <function _avg_then_exp>) → npt.NDArray[np.floating] | list[list[float]][source]#

Aggregate token log-probabilities along the last dimension.

Parameters:

  • log_probs (Sequence[Sequence[float]] | Sequence[Sequence[Sequence[float]]]) – nested sequences where token log-probabilities are in the last dimension. A 2-D sequence corresponds to inputting a single prompt string or cappr.Example object with completions. A 3-D sequence corresponds to inputting multiple prompt strings or cappr.Example objects with completions

  • func (Callable, optional) – a function which aggregates a sequence of token log-probabilities into a single number, by default a probability. If the function is vectorized, it must take an axis argument, e.g., np.mean will efficiently average the token log-probabilities. By default, token log-probabilities are averaged and then exponentiated.

Returns:

agg –

If log_probs is 2-D, then agg is a numpy array or a list where:

agg[j] = func(log_probs[j])

If log_probs is 3-D, then agg is a numpy array or a list of list of probabilities where:

agg[i][j] = func(log_probs[i][j])

agg is a numpy array when there are constant number of completions, else it’s a list.

Return type:

npt.NDArray[np.floating] | list[list[float]]

Raises:

ValueError – if log_probs is not 2-D or 3-D

cappr.utils.classify.posterior_prob(likelihoods: npt.ArrayLike[float], axis: int = -1, prior: Sequence[float] | None = None, normalize: bool | Sequence[bool] = True, check_prior: bool = True) → npt.NDArray[np.floating][source]#

Compute posterior probabilities from likelihoods and a prior.

Parameters:

  • likelihoods (npt.ArrayLike[float]) – 1-D or 2-D array of probabilities of data given a hypothesis

  • axis (int, optional) – the axis/dimension along which the probability distribution is defined. By default, the last axis

  • prior (Sequence[float] | None, optional) – a probability distribution over the axis of likelihoods. By default, a uniform prior is used

  • normalize (bool | Sequence[bool], optional) – whether or not to return a normalized probability distribtution for each row, by default True (normalize all rows)

  • check_prior (bool, optional) – whether or not to check that the prior is indeed a probability distribution, by default True

Returns:

posterior_probs – array of probabilities of a hypothesis given data. Its shape is the same as likelihood.shape

Return type:

npt.NDArray[np.floating]

Raises:

ValueError – if normalize is a sequence whose length is different than that of likelihoods