Why Sharpening a Cutout Draws a Halo Around Fur

Take a cutout of a long-haired cat, run it through an upscaler in photo mode, and look at the fur. There is a good chance you will find a bright rim tracing the silhouette — brightest exactly where the hair is wispiest, fading out a couple of pixels in. It looks like the AI hallucinated a glow.

It did not. The super-resolution model never saw the transparency, and the halo is not in its output. It is added afterwards, by a sharpening pass, and the reason is a single assumption that holds for every ordinary photograph and fails for every cutout.

Unsharp masking, and the assumption underneath it

The standard way to sharpen an image is the unsharp mask, which is less exotic than the name suggests. Blur a copy of the image. Subtract the blurred version from the original. What remains is the fine detail — the parts that changed most under the blur. Add a multiple of that difference back onto the original and the detail is exaggerated:

result = original + amount × (original − blurred)

Where the image is flat, original − blurred is zero and nothing happens. Where there is an edge, the blurred copy has smeared it, the difference is large, and the edge gets steeper. This is the whole algorithm, and it works well.

It also quietly assumes that every pixel inside the blur window is part of the picture. For a photograph, that is true by definition. There is no such thing as a pixel that is not part of a photograph.

A cutout is full of pixels that are not part of the picture

An image with transparency carries colour and an alpha channel, and the colour that sits underneath a fully transparent pixel is not meaningful — nothing is showing there. The trouble is that it is not absent either. It is some specific number, and once the image has been through a canvas, that number is black.

Canvases store colour premultiplied by alpha. The image data you read back out is unpremultiplied again, so the subject keeps its colour right up to the edge, but the round trip is lossy in a way that depends on alpha: dividing by a small alpha to undo the multiply amplifies the rounding, so the damage grows as a pixel gets more transparent. At alpha 0 there is nothing left to divide back out and the recovery is total loss. The colour comes back as black, whatever the subject was.

So a cutout arriving at the sharpener looks like this: solid subject, a band of partial-alpha fur, then a cliff to black.

The cliff is what makes the rim

Put the blur window on a pixel just inside the silhouette. With a five-tap kernel, that window reaches two pixels out — far enough to include some of the black. The blur averages it in and reads far darker than the pixel really is.

Now run the arithmetic. original − blurred is a large positive number, not because there was any detail there, but because the blur was contaminated. The pixel gets pushed brighter. Do that along the whole silhouette and you get a rim of brightening, and on the partial-alpha pixels that make up fur and hair, that rim is the reported halo.

Two details of the symptom fall straight out of the mechanism. The halo is local, because with a five-tap kernel only pixels within two of the transparent region can see the black at all — which is why it reads as a glow around the fur rather than a general brightening of the animal. And it is worst on fur specifically, because wispy edges have the most partial-alpha pixels sitting next to the cliff.

Why it reached the final image

Two things made it stick.

The sharpening ran after the joint-bilateral upsample that produces the final alpha, so the silhouette was already frozen by the time the damage was applied. Nothing downstream had any opportunity to notice or undo it.

And photo mode does not sharpen once. Every AI pass ends with an unsharp, the 4× cascade adds one more between its two passes, and the default finishing chain adds a last one on the final pixels. Each pass reads the output of the one before it. An error that reappears every pass does not stay a faint rim.

The pass itself was also duplicated — one copy in the Real-ESRGAN wrapper, another in the upscale worker — so there were two places to fix and two places for a fix to be missed. Both now call one shared implementation.

What alpha-weighted sharpening does instead

Two changes, and the whole fix is in them.

The blur is weighted by alpha. Each neighbour contributes to the blur in proportion to how opaque it is, and the total is divided by the sum of those weights rather than the sum of the kernel taps. A fully transparent neighbour contributes exactly nothing and adds nothing to the divisor. The black behind the cutout cannot cross the silhouette, because it is never in the average in the first place. There is no contamination, so there is nothing to overshoot against.

The sharpening amount is scaled by the pixel's own alpha. A pixel that is thirty percent opaque is mostly background, and cranking its contrast manufactures detail that was never in the source. Scaling the amount by alpha means the correction fades out exactly as the pixel does.

Two smaller decisions matter more than they look. The blur is separable — a horizontal pass then a vertical one — and the alpha-weighted colour sums and the weights are both carried through both passes and normalised only at the very end. Normalising after the horizontal pass would turn the intermediate back into a plain colour image and re-admit precisely the bleed the function exists to prevent. And where a window contains no opaque pixel anywhere, the divisor is approximately zero; rather than divide by it, the pixel is left exactly as it arrived. Alpha itself is never modified, and fully transparent pixels come back byte for byte unchanged.

The property that made it safe to ship

Nearly every upscale is an ordinary opaque photograph, and a fix for cutouts that perturbed those would be a bad trade.

It does not perturb them, and not by being careful — by arithmetic. On a fully opaque image every alpha weight is 1, so the weighted sum is the plain sum, the normalisation divides by the same total the plain version divides by, and the amount is scaled by 1. Every term collapses. The alpha-weighted unsharp is the plain unsharp on opaque input, exactly.

That is pinned by a regression test rather than asserted in a comment: the new implementation is compared byte for byte against a preserved copy of the old alpha-blind code, across three kernels and four sharpening amounts, and required to match exactly. A companion test requires both of them to actually change the image, so the comparison cannot pass by doing nothing.

What the numbers are, and what they were measured on

This is the part worth stating precisely, because it is easy to overclaim.

The measurements come from a synthetic fixture, not from photographs: a 24×24 image with a flat opaque subject, a linear partial-alpha band standing in for fur, and then transparency, with the premultiply-and-back round trip reproduced exactly so the black cliff is real rather than assumed. The subject is deliberately flat, which is what makes the measurement meaningful — a flat field contains no detail to sharpen, so any brightening of a visible pixel is manufactured contrast, and the halo can be read off directly as the largest such brightening.

On that fixture:

• The alpha-blind version overshoots by more than 10 levels at the silhouette.
• The alpha-weighted version stays at 1 level or below, which is the quantisation noise floor of the round trip itself.
• The reduction is over 90%, pinned as an assertion that the new peak is less than a tenth of the old one.
• Stacked four passes deep, the alpha-blind overshoot keeps growing pass over pass, while the alpha-weighted version stays at the noise floor no matter how many run.

What those numbers support is that the mechanism is understood and the fix removes it. They are not a measurement on real photographs, and they should not be read as one. A synthetic silhouette reproduces the cause cleanly, which is what a regression lock needs; it does not tell you how the fur on your particular cat looks at 4×.

What this does not fix

Worth being explicit, because "halo" gets used loosely for several different edge artefacts.

Two tile-seam problems on the Swin2SR path are untouched by this work: tiles butted together with no feathering between them, and a joint-bilateral window that gets truncated at every core-tile edge. Those produce edge damage too, they are a genuinely separate mechanism, and attributing them properly needs a real fixture that reproduces them. They remain open.

If your cutout upscales have carried a bright fringe along hair or fur, the upscaler is worth another run. And if what you are seeing is a coloured fringe rather than a bright one, that is a different mechanism with its own fix — edge decontamination covers it.