Nearly isotonic regression

Suppose that we have $n$ independent normal observations $X_i \sim \mathrm{N}(\mu_i,1)$ for $i=1,\dots,n$, where $\mu_1,\mu_2,\dots,\mu_n$ is a piecewize monotone sequence of normal means. The nearly isotonic regression problem is formulated as

\[\begin{aligned} \hat{\mu} &=\argmin_{\mu_1, \dots, \mu_n}\left(\sum_{i=1}^n -\log p(x_i\mid \mu_i) +\lambda\sum_{i=1}^{n-1} (\mu_i-\mu_{i+1})_+\right)\\ &= \argmin_{\mu_1,\dots , \mu_n}\left(\sum_{i=1}^n\frac{1}{2} (x_i-\mu_i)^2+\lambda\sum_{i=1}^{n-1} (\mu_i-\mu_{i+1})_+\right), \end{aligned}\]

where $(a)_+=\mathrm{max}(a,0)$ and $\lambda>0$ is a regularization parameter.

The weighted formulation reads

\[\hat{\mu} =\argmin_{\mu_1, \dots, \mu_n} \left( \sum_{i=1}^n \frac{1}{2} w_i (x_i-\mu_i)^2+\lambda\sum_{i=1}^{n-1} (\mu_i-\mu_{i+1})_+\right)\]

where $w_i > 0$ $(i=1,\dots,n)$.

The function neariso solves these problems. Note that the result $\hat{\mu} = (\hat{\mu}_1,\dots, \hat{\mu}_n)$ consists of several clusters. This function outputs $\hat{\mu}$ and the number of clusters.

IsoFuns.neariso — Function

neariso(x::Vector, λ, w::Vector) -> y, K

Perform weighted nearly isotonic regression

Arguments

x: input vector
λ: parameter
w: weights, default to ones if not provided

Keywords

var: variances of the Gaussian distributions

Outputs

y: output vector, which is piecewise monotone
K: the number of clusters

Algorithm

modified PAVA

The number of clusters decreases if one chooses a bigger $\lambda$. The function neariso_path outputs the $\lambda$ values at which several clusters merge, and the corresponding number of clusters.

IsoFuns.neariso_path — Function

neariso_path(x::Vector, w::Vector) -> knot, K

λ-path of weighted nearly isotonic regression

Arguments

x: input vector
w: weights, default to ones if not provided

Outputs

knot: checkpoints of the relaxation parameter
K : the number of clusters at each checkpoint

Algorithm

modified PAVA

Generalized nearly isotonic regression

The above formulation can be generalized to one parameter exponential families, as is done for the isotonic regression. Such a formulation is called the generalized nearly isotonic regression.

When $X_i\sim p_i(x_i\mid \theta_i)$ for $i=1,\dots,n$, where $p_i(x_i\mid\theta_i )$ is a one parameter exponential family defined by

\[p_i(x_i\mid \theta_i) = h_i(x_i) \exp (\theta_i x_i -w_i\psi(\theta_i)),\]

the problem

\[\hat{\theta} = \argmin_{\theta_1, \dots , \theta_n} \left(\sum_{i=1}^n -\log p_i(x_i\mid \theta_i)+\lambda\sum_{i=1}^{n-1} (\theta_i-\theta_{i+1})_+\right)\]

can also be solved by the function neariso, as the formulation is equivalent to

\[\hat{\eta} = \argmin_{\eta_1,\dots , \eta_n}\left(\sum_{i=1}^n\frac{1}{2} w_i (x_i-\eta_i)^2+\lambda\sum_{i=1}^{n-1} (\eta_i-\eta_{i+1})_+\right) \quad \text{with} \quad \eta = \mathrm{E}_\theta [x] = \psi ^\prime (\theta).\]

Further, apart from neariso_DistibutionName and neariso_path_DistributionName, the function neariso_AIC_value_DistributionName provides the AIC value at $\lambda$. Additionally, neariso_AIC_DistributionName outouts the collection of the AIC values at each checkpoint of the relaxation parameter. Here, the Akaike Information Criterion (AIC) is defined by

\[\mathrm{AIC}(\lambda) = -2 \sum_{i=1}^n \log p_i(x_i\mid (\hat{\theta}_\lambda)_i) + 2K_\lambda,\]

where $\hat{\theta}_\lambda = (\hat{\theta}_1,\dots,\hat{\theta}_n)$ is the solution for $\lambda$, and $K_\lambda$ signifies the corresponding number of clusters.

IsoFuns.neariso_Normal — Function

neariso_Normal(x::Vector,λ,variance::Vector) -> y, K

Perform weighted nearly isotonic regression (Normal)

Arguments

x: input vector
λ: parameter
variance: weights, default to ones if not provided

Outputs

y: output vector, which is piecewise monotone
K: the number of clusters

IsoFuns.neariso_path_Normal — Function

neariso_path_Normal(x::Vector, variance::Vector) -> knot, K

λ-path of weighted nearly isotonic regression (Normal)

Arguments

x: input vector
variance a vector consisting of variances, default to ones if not provided

Outputs

knot: checkpoints of the relaxation parameter
K : the number of clusters at each checkpoint

IsoFuns.neariso_AIC_Normal — Function

neariso_AIC_Normal(x::Vector, variance::Vector) -> knot, AIC

λ-path and AIC of weighted nearly isotonic regression (Normal)

Arguments

x: input vector
w: weights, default to ones if not provided

Outputs

knot: checkpoints of the relaxation parameter
AIC : AIC at each checkpoint

IsoFuns.neariso_AIC_value_Normal — Function

neariso_AIC_value_Normal(x::Vector, λ, variance::Vector) -> AIC

AIC value of weighted nearly isotonic regression (Normal)

Arguments

x: input vector
w: weights, default to ones if not provided

Outputs

AIC : AIC at λ

IsoFuns.neariso_Binomial — Function

nieariso_Binomial(success::Vector, λ, trial::Vector) -> y, K

Perform weighted nearly isotonic regression (Binomial)

Arguments

success: input vector consisting of the number of success
λ: parameter
trials: a vector consisting of the number of trials

Outputs

y: output vector, which is piecewise monotone
K: the number of clusters

IsoFuns.neariso_path_Binomial — Function

neariso_path_Binomial(success::Vector, trial::Vector) -> knot, K

λ-path of weighted nearly isotonic regression (Binomial)

Arguments

success: input vector consisting of the number of success
trials: a vector consisting of the number of trials

Outputs

knot: checkpoints of the relaxation parameter
K : the number of clusters at each checkpoint

IsoFuns.neariso_AIC_Binomial — Function

neariso_AIC_Binomial(success::Vector, trial::Vector) -> knot, AIC

λ-path and AIC of weighted nearly isotonic regression (Binomial)

Arguments

success: input vector consisting of the number of success
trials: a vector consisting of the number of trials

Outputs

knot: checkpoints of the relaxation parameter
AIC : AIC at each checkpoint

IsoFuns.neariso_AIC_value_Binomial — Function

neariso_AIC_value_Binomial(success::Vector, λ, trial::Vector) -> AIC

AIC value of weighted nearly isotonic regression (Binomial)

Arguments

success: input vector consisting of the number of success
trials: a vector consisting of the number of trials

Outputs

AIC : AIC at λ

IsoFuns.neariso_Poisson — Function

nieariso_Poisson(x::Vector, λ) -> y, K

Perform weighted nearly isotonic regression (Poisson)

Arguments

x: input vector consisting of the number of events
λ: parameter

Outputs

y: output vector, which is piecewise monotone
K: the number of clusters

IsoFuns.neariso_path_Poisson — Function

neariso_path_Poisson(x::Vector) -> knot, K

λ-path of weighted nearly isotonic regression (Poisson)

Arguments

x: input vector consisting of the number of events

Outputs

knot: checkpoints of the relaxation parameter
K : the number of clusters at each checkpoint

IsoFuns.neariso_AIC_Poisson — Function

neariso_AIC_Poisson(x::Vector) -> knot, AIC

λ-path and AIC of weighted nearly isotonic regression (Poisson)

Arguments

x: input vector consisting of the number of events

Outputs

knot: checkpoints of the relaxation parameter
AIC : AIC at each checkpoint

IsoFuns.neariso_AIC_value_Poisson — Function

neariso_AIC_value_Poisson(x::Vector, λ) -> AIC

AIC value of weighted nearly isotonic regression (poisson)

Arguments

x: input vector consisting of the number of events

Outputs

AIC : AIC at λ

IsoFuns.neariso_Chisq — Function

neariso_Chisq(x::Vector, λ, d::Vector) -> x, K

Perform weighted nearly isotonic regression (Chisq)

Arguments

x: input vector
λ: parameter
d: a vector consisting of the degrees of freedom

Outputs

y: output vector, which is piecewise monotone
K: the number of clusters

IsoFuns.neariso_path_Chisq — Function

neariso_path_Chisq(x::Vector, d::Vector) -> knot, K

λ-path of weighted nearly isotonic regression (Chisq)

Arguments

x: input vector
default: a vector consisting of the degrees of freedom

Outputs

knot: checkpoints of the relaxation parameter
K : the number of clusters at each checkpoint

IsoFuns.neariso_AIC_Chisq — Function

neariso_AIC_Chisq(x::Vector, d::Vector) -> knot, AIC

λ-path and AIC of weighted nearly isotonic regression (Chisq)

Arguments

x: input vector
d: a vector consisting of the degrees of freedom

Outputs

knot: checkpoints of the relaxation parameter
AIC : AIC at each checkpoint

IsoFuns.neariso_AIC_value_Chisq — Function

neariso_AIC_value_Chisq(x::Vector, λ, d::Vector) -> AIC

AIC value of weighted nearly isotonic regression (Chisq)

Arguments

x: input vector
d: a vector consisting of the degrees of freedom

Outputs

AIC : AIC at λ