Advertisement

Technical Deep Dives9 min read

Which Video Edits Keep Your Audio Untouched, and Which Re-encode It

Rotating, resizing and padding a clip stream-copy the audio; compressing and converting re-encode it; metadata removal copies both. Knowing which is which tells you the one order that spends a lossy audio pass exactly once.

By · Part of the Image processing and on-device AI topic cluster

Run a clip through four different operations and the picture is obviously affected by all of them. The soundtrack is a quieter story. Some of those operations hand your audio through completely untouched, byte for byte. Others decode it and encode it again, and that second kind costs you a small amount of quality every single time.

Nobody tells you which is which, so here it is, with the reason behind each one.

Two things a tool can do with an audio stream

When a video file is rebuilt, its audio can be handled in one of two ways.

A stream copy lifts the compressed audio out of the old container and drops it into the new one without decoding it. The bytes are identical. Whatever the audio sounded like before, it sounds exactly the same after, because it is the same data. This is fast, because no encoding happens, and it is free of loss by definition.

A re-encode decodes the audio back to samples and compresses it again. Every lossy audio codec discards information it judges inaudible, and doing that a second time means the encoder is now making those judgements about audio that has already had a pass of them applied. The result is not usually dramatic. It is also not nothing, and it compounds if you do it repeatedly.

The same distinction applies to video, and the same logic follows.

Which operations copy the audio

These rebuild the picture and pass the sound through untouched:

  • Video Rotate. The frames go through a transpose filter and are re-encoded; the audio is stream-copied.
  • Video Resizer. Scaling is a video-only operation. Audio is stream-copied.
  • Video Canvas Extender. Padding changes the frame, not the soundtrack. Audio is stream-copied.

In all three the picture is re-encoded, because it genuinely has to be: you cannot change pixel dimensions or rotate a frame without producing new frames. But there is no reason to touch the audio, so it is not touched.

The one that copies both

Video Metadata Remover is the only operation here that is lossless end to end. It remuxes: both streams are copied into a fresh container with the metadata mapping discarded, and no decoder runs on either of them.

This is worth internalising because it inverts the usual intuition. Stripping GPS coordinates, device details, and encoder strings out of a video sounds like the sort of thing that must involve rebuilding the file. It does not. It is the cheapest and safest operation available, and it is the one you can run last without paying for it.

Which operations re-encode the audio

  • Video Compressor re-encodes the audio to AAC alongside the video.
  • Video Format Converter re-encodes both, and which audio codec you get depends on the container: WebM output pairs VP9 video with Opus audio, and MP4 output pairs H.264 with AAC.
  • Video Format Comparison produces its candidates the same way, which is the point, since comparing file sizes only means anything if each candidate really is the codec it claims. VP8 is paired with Vorbis and VP9 with Opus, because those are the pairings those containers actually use.

None of these is doing anything wrong. A format conversion that did not re-encode the audio would be a container change and not a conversion, and a compressor that left a large uncompressed audio stream alone would be missing a chunk of the file size it was asked to reduce.

What this means for the order you work in

It leads to one straightforward rule: do the audio re-encode once, and do it last.

If you are going to rotate, resize, pad, and then compress a clip, that final compress is the only step that touches your soundtrack, and it touches it exactly once. Do it in a different order, compressing in the middle and converting at the end, and your audio has been through two lossy passes for no benefit.

The same rule you would apply to image formats applies here. Keep the intermediate steps in the cheap category, spend the loss deliberately, and spend it at the end.

The video side of the same question

For completeness, because the copy-versus-re-encode distinction does not stop at the soundtrack.

Rotate, resize, and canvas extension re-encode the picture at a high quality setting, which is close to but not identical to the source. If you chain several of them you are stacking video generations, and the honest fix is the same one: fewer passes.

Metadata removal, again, is the exception that copies everything.

And one genuinely surprising consequence of an MP4 rebuild: padding a video with a transparent background does not produce transparency. H.264 inside an MP4 has no alpha channel, so there is nothing transparent to pad with, and the pad is filled with black instead. That substitution is deliberate and it is the correct behaviour, but it will catch you out if you were expecting a transparent letterbox. If you need transparency to survive, the output has to be a format that can carry it.

Where the audio survives that you might not expect

Several of the model-backed operations go out of their way to preserve sound, and it is worth naming them because "the AI rebuilt my video" is a reasonable thing to fear.

Background removal on video keeps the audio, whether the output is a transparent WebM or a composited MP4. Reviewed privacy blur applied across a tracked video keeps it. A highlights commit that cuts a long clip down to reviewed ranges is audio-preserving. Burning captions into a clip renders new frames and then muxes the original audio back into the result rather than re-recording it.

In each case the design decision is the same: the operation is about the picture, so the soundtrack should come out the other side as close to untouched as the container allows.

The summary, in one list

Copy the audio: rotate, resize, canvas extend. Copy everything: metadata removal. Re-encode the audio: compress, format convert, format comparison.

If you remember only one line of it, remember that metadata stripping is free and belongs at the end, and that the format decision is the one that costs you something, so make it once.

Applies to NSS Background Remover v2.4.0, the release that was current when this article was published.

Was this article helpful?

Your answer is saved in this browser only. This control makes no network request and does not send the answer to an NSS feedback endpoint.

Found this useful?