Total Nuclear Variation and Jacobian Extensions of Total Variation for Vector Fields

We explore a class of vectorial total variation (VTV) measures formed as the spatial sum of a pixel-wise matrix norm of the Jacobian of a vector field. We give a theoretical treatment that indicates that, while color smearing and affine-coupling bias (often reported as gray-scale bias) are typically cited as drawbacks for VTV, these are actually fundamental to smoothing vector direction (i.e., smoothing hue and saturation in color images). In addition, we show that encouraging different vector channels to share a common gradient direction is equivalent to minimizing Jacobian rank. We thus propose total nuclear variation (TNV), and since nuclear norm is the convex envelope of matrix rank, we argue that TNV is the optimal convex regularizer for enforcing shared directions. We also propose extended Jacobians, which use larger neighborhoods than the conventional finite difference operator, and we discuss efficient VTV optimization algorithms. In simple color image denoising experiments, TNV outperformed other common VTV regularizers, and was further improved by using extended Jacobians. TNV was also competitive with the method of nonlocal means, often outperforming it by 0.25-2 dB when using extended Jacobians.