Cleaner Edges Without the Pasted-On Look: What Happens After the Model

A cutout looks pasted on for two reasons that have nothing to do with where the subject is. The edge sits a few pixels off the real outline, so a sliver of the old background rides along with it, or the edge carries a thin rim of that background's colour. Either one is enough for the eye to see a sticker instead of an object.

Neither problem is decided by the neural network alone. The model produces a first draft of the mask, and a chain of cleanup stages that run afterwards, on your device, decides what the delivered edge looks like. This post walks through that chain in the order it runs in the image worker, says what each stage changes, and says where each one deliberately does nothing. None of it is a new feature; it is the pipeline that runs every time you remove a background in NSS, written out so you can predict what it will do to your photo.

The mask arrives smaller than your photo

The Best Quality model runs on a fixed-size graph. Its shipped export works at 512 pixels, and the Fast model has a fixed input size of its own. Your photograph is usually far larger: a 4000-pixel product shot is roughly eight times wider than the mask describing it.

So the first thing that happens is an upscale. The mask is stretched to the photograph's size, and a plain upscale never looks at the photograph while it does so. Each model pixel becomes a run of about eight output pixels, and interpolation alone decides where inside that run the edge lands. Anything thinner than one model pixel, which is most hair, fur and wire, arrives as a soft ramp in roughly the right place instead of an edge in the right place.

Every stage after this one inherits that ramp. That is why the next stage matters more than any of the later ones.

Stage one: put the edge back on the photograph

The guided upsample re-derives the pixels in the edge band from the same model-resolution samples, but weights each sample by how closely the photograph's brightness at that sample matches the brightness at the output pixel. A pixel that looks like the subject leans on the subject's samples; a pixel that looks like the background leans on the background's. The alpha edge moves onto the picture's own edge instead of sitting near it.

Two limits keep it honest:

• Interiors and empty background are not touched. The pass skips any pixel whose model-resolution neighbourhood is flat, so those values stay exactly what the plain upscale produced.
• It declines where the photograph is too flat to place an edge. Without that rule a low-contrast subject came out worse than leaving the interpolation alone, which was measured rather than assumed. A grey jumper on a grey wall therefore gets little help here.

It also costs no extra full-resolution memory: it writes into the buffer the upscale already allocated. On the synthetic edge shapes in the test suite it rewrites roughly one per cent of the frame, but because every later stage reads neighbourhoods, the delivered mask ends up differing on between 2.65 and 10 per cent of it. A small, local change at the start moves a lot downstream, which is exactly why the edge is corrected first.

Stage two: decide the uncertain pixels

1. Upscaled model mask

A soft ramp in roughly the right place. The strand is lost in it.

Photo

Cutout

Alpha

1111.90.75.60.45.30.20.100

PreviousPlayStep 1 of 5Next

After the upsample, many edge pixels are still undecided, sitting at 0.4 or 0.6 where the true answer is nearer 0 or 1. A guided filter pushes them toward one side using local colour evidence from the photograph. It pulls a little harder for the Fast model, whose raw edges are softer, than for Best Quality.

It also has an ambiguity gate. Where a pixel's colour is about equally close to the subject and to the background, the filter skips the blend instead of guessing. A forced guess on those pixels is how a soft shoulder turns into a jagged one.

Stage three: keep hair soft on purpose

Pushing every edge pixel to 0 or 1 is exactly wrong for hair. A strand in front of the sky genuinely covers part of a pixel, and the correct alpha is partial. Hair preservation keeps soft alpha on high-frequency detail, but only near a genuine mask edge: a pixel has to have alpha below 0.5 somewhere within four pixels of it.

That gate exists because an earlier version had no gate. Texture inside the subject, such as the weave of a jumper or the grain of wood, also looks high-frequency, and it was dragging the body of the subject toward half transparency. Gating the pass to the outline fixed that without taking anything away from real strands.

Stage four: clear a studio background, then tidy the interior

A background-kill pass looks at the corners of the photograph. If they show a uniform light or white background, it zeroes low-alpha pixels whose colour is close to that background. On a dark or busy background it does nothing, and it is skipped entirely in screen-capture mode, where white browser chrome in the corners would otherwise make it delete white interface elements. A second, lighter guided filter then runs over the result.

Next, a confident-interior snap. Nothing later in the chain can raise alpha, so a pixel deep inside the subject that the model scored at 0.9 would otherwise export slightly see-through forever. Pixels that are confidently foreground and have no sub-0.5 alpha anywhere nearby snap to exactly 1. Real soft edges always have a sub-0.5 neighbour, so they are left alone.

Then a trim of the ambiguous fringe, and only the fringe. A conventional erosion of four to seven pixels used to delete thin chair legs and cables, because it replaced solid foreground with neighbouring background values. The current trim tightens only the semi-transparent band, uses a one-pixel radius on near-white backgrounds and none otherwise, and protects thin structures.

Stage five: remove the colour that is not the subject's

This is where the pasted-on rim is dealt with. It comes in two stages because it has two causes.

The smart edge pass samples the background colour from pixels that are already fully transparent. For each edge pixel with alpha between 0.1 and 0.8, it compares the pixel's colour with that background in Lab colour space, where distance tracks what the eye sees better than raw RGB does. A pixel that matches the background is spill and is pushed to transparent. A pixel that does not is a genuine subject edge and keeps its alpha.

Fringe decontamination handles the white halo that can survive on hard-edged subjects such as logos, packaging and products. It samples the background from the image border and drops alpha by 60 per cent on fringe pixels that both match that colour and are mostly surrounded by transparent pixels. When the subject has hard, binary edges, it adds a one-pixel erosion.

The earlier post on why cutouts get a coloured fringe explains where that rim comes from in the first place.

When the chain stands aside

The most important behaviour in this pipeline is the stages that switch themselves off:

• Glass and clear plastic. In transparent-materials mode the interior snap and fringe decontamination are skipped, because partial alpha inside a bottle is the point, not an error. Instead, the worker estimates see-through interiors the model marked solid and then runs an edge-band preservation pass. Removing backgrounds from glass covers how to shoot for it.
• Screen captures. The background kill, the fringe trim, the smart edge pass and fringe decontamination are all skipped, because interface screenshots break their assumptions about corners and borders.
• Flat, low-contrast edges. The guided upsample declines, as described above.
• Memory pressure. If a very large photo runs out of memory, the worker retries at half size and says so on the result, rather than failing silently or pretending the output is full resolution.

What none of these stages do

Every stage above changes alpha. None of them changes colour. The pixels of your subject leave the pipeline with the RGB values they arrived with, and the export keeps straight alpha, which the export verification post explains.

That means one kind of edge problem is outside what this chain can fix today: an edge pixel that belongs to the subject but has picked up the background's colour through the lens, such as a blond strand tinted green by a green wall. Removing that tint needs the foreground colour to be re-estimated, which is a different operation. Our own roadmap lists it, along with a bounded second pass over hard regions, as work that has not started. We would rather say that plainly than let this post imply otherwise.

When the automatic edge is still not right

The editor exposes the same building blocks as Edge Refine, with two presets:

1. Soft, for hair, fur and fuzzy edges. A gentle guided filter plus hair preservation. It skips fringe decontamination and erosion on purpose, because both read stray wisps as halo and delete them.
2. Crisp, for products, logos and hard edges. A stronger guided filter, hair preservation, fringe decontamination and a light erosion.

Both run on the CPU against the existing mask, typically in half a second to three seconds, with no model download. If the problem is missing subject or leftover background rather than edge quality, the Erase and Restore brushes and the Magic Wand are the right tools; fixing an imperfect AI cutout walks through them.

A quick way to judge an edge

Put the cutout on a background that is the opposite of the one it was shot on: dark if it was shot on white, light if it was shot on dark. A halo that was invisible on the checkerboard becomes obvious at once. If the rim is the old background's colour, try Crisp. If hair has gone stringy or chopped, try Soft. If neither helps, the edge was probably too flat in the photograph for any of these stages to find, and a reshoot with more separation between subject and background will do more than another pass.

All of it runs in your browser. The photograph is never uploaded, and the stages above cost nothing beyond the model you already downloaded.