Skip to main content
`Ctrl`+`K`

# A Design Specification for `nan_policy`

Many functions in `scipy.stats` have a parameter called `nan_policy` that determines how the function handles data that contains `nan`. In this section, we provide SciPy developer guidelines for how `nan_policy` is intended to be used, to ensure that as this parameter is added to new functions, we maintain a consistent API.

## The basic API

The parameter `nan_policy` accepts three possible strings: `'omit'`, `'raise'` and `'propagate'`. The meanings are:

- `nan_policy='omit'`: Ignore occurrences of `nan` in the input. Do not generate a warning if the input contains `nan` (unless the equivalent input with the `nan` values removed would generate a warning). For example, for the simple case of a function that accepts a single array and returns a scalar (and ignoring the possible use of `axis` for the moment):

      func([1.0, 3.0, np.nan, 5.0], nan_policy='omit')

  should behave the same as:

      func([1.0, 3.0, 5.0])

  More generally, for functions that return a scalar, `func(a, nan_policy='omit')` should behave the same as `func(a[~np.isnan(a)])`.

  For functions that transform a vector to a new vector of the same size and for which each entry in the output array depends on more than just the corresponding value in the input array [1] (e.g. `scipy.stats.zscore`, `scipy.stats.boxcox` when `lmbda` is None),:

      y = func(a, nan_policy='omit')

  should behave the same as:

      nan_mask = np.isnan(a)
      y = np.empty(a.shape, dtype=np.float64)
      y[~nan_mask] = func(a[~nan_mask])
      y[nan_mask] = np.nan

  (In general, the dtype of `y` might depend on `a` and on the expected behavior of `func`). In other words, a nan in the input gives a corresponding nan in the output, but the presence of that nan does not affect the calculation of the non-nan values.

  Unit tests for this property should be used to test functions that handle `nan_policy`.

  For functions that return a scalar and that accept two or more arguments but whose values are not related (e.g. `scipy.stats.ansari`, `scipy.stats.f_oneway`), the same idea applies to each input array. So:

      func(a, b, nan_policy='omit')

  should behave the same as:

      func(a[~np.isnan(a)], b[~np.isnan(b)])

  For inputs with related or paired values (e.g. `scipy.stats.pearsonr`, `scipy.stats.ttest_rel`) the recommended behavior is to omit all the values for which any of the related values are `nan`. For a function with two related array inputs, this means:

      y = func(a, b, nan_policy='omit')

  should behave the same as:

      hasnan = np.isnan(a) | np.isnan(b)  # Union of the isnan masks.
      y = func(a[~hasnan], b[~hasnan])

  The docstring for such a function should clearly state this behavior.

- `nan_policy='raise'`: Raise a `ValueError`.

- `nan_policy='propagate'`: Propagate the `nan` value to the output. Typically, this means just execute the function without checking for `nan`, but see

  > scipy/scipy#7818

  for an example where that might lead to unexpected output.

## `nan_policy` combined with an `axis` parameter

There is nothing surprising here–the principle mentioned above still applies when the function has an `axis` parameter. Suppose, for example, `func` reduces a 1-d array to a scalar, and handles n-d arrays as a collection of 1-d arrays, with the `axis` parameter specifying the axis along which the reduction is to be applied. If, say:

    func([1, 3, 4])     -> 10.0
    func([2, -3, 8, 2]) ->  4.2
    func([7, 8])        ->  9.5
    func([])            -> -inf

then:

    func([[  1, nan,   3,   4],
          [  2,  -3,   8,   2],
          [nan,   7, nan,   8],
          [nan, nan, nan, nan]], nan_policy='omit', axis=-1)

must give the result:

    np.array([10.0, 4.2, 9.5, -inf])

## Edge cases

A function that implements the `nan_policy` parameter should gracefully handle the case where all the values in the input array(s) are `nan`. The basic principle described above still applies:

    func([nan, nan, nan], nan_policy='omit')

should behave the same as:

    func([])

In practice, when adding `nan_policy` to an existing function, it is not unusual to find that the function doesn’t already handle this case in a well-defined manner, and some thought and design may have to be applied to ensure that it works. The correct behavior (whether that be to return `nan`, return some other value, raise an exception, or something else) will be determined on a case-by-case basis.

## Why doesn’t `nan_policy` also apply to `inf`?

Although we learn in grade school that “infinity is not a number”, the floating point values `nan` and `inf` are qualitatively different. The values `inf` and `-inf` act much more like regular floating point values than `nan`.

- One can compare `inf` to other floating point values and it behaves as expected, e.g. `3 < inf` is True.

- For the most part, arithmetic works “as expected” with `inf`, e.g. `inf + inf = inf`, `-2*inf = -inf`, `1/inf = 0`, etc.

- Many existing functions work “as expected” with `inf`: `np.log(inf) = inf`, `np.exp(-inf) = 0`, `np.array([1.0, -1.0, np.inf]).min() = -1.0`, etc.

So while `nan` almost always means “something went wrong” or “something is missing”, `inf` can in many cases be treated as a useful floating point value.

It is also consistent with the NumPy `nan` functions to not ignore `inf`:

    >>> np.nanmax([1, 2, 3, np.inf, np.nan])
    inf
    >>> np.nansum([1, 2, 3, np.inf, np.nan])
    inf
    >>> np.nanmean([8, -np.inf, 9, 1, np.nan])
    -inf

## How not to implement `nan_policy`

In the past (and possibly currently), some `stats` functions handled `nan_policy` by using a masked array to mask the `nan` values, and then computing the result using the functions in the `mstats` subpackage. The problem with this approach is that the masked array code might convert `inf` to a masked value, which we don’t want to do (see above). It also means that, if care is not taken, the return value will be a masked array, which will likely be a surprise to the user if they passed in regular arrays.

Footnotes

[1]

If an element of the output depends only on the corresponding element of the input (e.g. `numpy.sin`, `scipy.special.gamma`), then there is no need for a `nan_policy` parameter.