In the previous article, we covered how to load images, canvases, and video into a texture. This article will cover a more efficient way to use video in WebGPU.
In the previous article we loaded video data into a webgpu texture
by calling copyExternalImageToTexture. This function copies
the current frame of video from the video itself into a pre-existing
texture that we created.
WebGPU has another method for using video. It’s called importExternalTexture
and, like the name suggests, it provides a GPUExternalTexture. This external
texture represents the data in the video directly. No copy is made. [1]
You pass importExternalTexture a video and it returns a texture, ready to use.
There are a few big caveats to using an texture from importExternalTexture.
For most WebGPU apps that means the texture only exists until your
requestAnimationCallback function ends. Or whatever event you’re rendering on; requestVideoFrameCallback, setTimeout,
mouseMove, etc… When your function exits the texture is expired.
To use the video again you must call importExternalTexture again.
An implication of this is that you must make a new bindgroup each
time you call importExternalTexture [2] so that
you can pass the new texture into your shader.
texture_external in your shadersWe’ve been using texture_2d<f32> in all previous texture examples
but textures from importExternalTexture can only be bound to
binding points using texture_external.
textureSampleBaseClampToEdge in your shadersWe’ve been using textureSample in all previous texture examples
but textures from importExternalTexture can only use
textureSampleBaseClampToEdge. [3] Like the name suggests,
textureSampleBaseClampToEdge will only sample the base texture mip level
(level 0). In other words, external textures can not have a mipmap.
Further, the function clamps to the edge, meaning, setting a sampler
to addressModeU: 'repeat' will be ignored.
Note that you can do your own repeating by using fract as in:
let color = textureSAmpleBaseClampToEdge( someExternalTexture, someSampler, fract(texcoord) );`
If these restrictions are not okay for your needs then you’ll need to use
copyExternalImageToTexture like we covered in the previous article.
Let’s make a working example using importExternalTexture. Here’s a video
Here are the changes need from our previous example.
First we need to update our shader.
struct OurVertexShaderOutput {
@builtin(position) position: vec4f,
@location(0) texcoord: vec2f,
};
struct Uniforms {
matrix: mat4x4f,
};
@group(0) @binding(2) var<uniform> uni: Uniforms;
@vertex fn vs(
@builtin(vertex_index) vertexIndex : u32
) -> OurVertexShaderOutput {
let pos = array(
vec2f( 0.0, 0.0), // center
vec2f( 1.0, 0.0), // right, center
vec2f( 0.0, 1.0), // center, top
// 2st triangle
vec2f( 0.0, 1.0), // center, top
vec2f( 1.0, 0.0), // right, center
vec2f( 1.0, 1.0), // right, top
);
var vsOutput: OurVertexShaderOutput;
let xy = pos[vertexIndex];
vsOutput.position = uni.matrix * vec4f(xy, 0.0, 1.0);
- vsOutput.texcoord = xy * vec2f(1, 50);
+ vsOutput.texcoord = xy;
return vsOutput;
}
@group(0) @binding(0) var ourSampler: sampler;
-@group(0) @binding(1) var ourTexture: texture_2d<f32>;
+@group(0) @binding(1) var ourTexture: texture_external;
@fragment fn fs(fsInput: OurVertexShaderOutput) -> @location(0) vec4f {
- return textureSample(ourTexture, ourSampler, fsInput.texcoord);
+ return textureSampleBaseClampToEdge(
+ ourTexture,
+ ourSampler,
+ fsInput.texcoord,
+ );
}
Above we stopped multiplying the texture coordinates by 50 since that was only there to show repeating and external textures do not repeat.
We also made the required changes as mentioned above. texture_2d<f32>
becomes texture_external and textureSample becomes textureSampleBaseClampToEdge.
We removed all of the code related to creating a texture and to generating mips.
We of course need to point to our video
- video.src = 'resources/videos/Golden_retriever_swimming_the_doggy_paddle-360-no-audio.webm'; + video.src = 'resources/videos/pexels-anna-bondarenko-5534310 (540p).mp4';
Since we can’t have mip levels there’s no need to create samplers that would use them.
const objectInfos = [];
- for (let i = 0; i < 8; ++i) {
+ for (let i = 0; i < 4; ++i) {
const sampler = device.createSampler({
addressModeU: 'repeat',
addressModeV: 'repeat',
magFilter: (i & 1) ? 'linear' : 'nearest',
minFilter: (i & 2) ? 'linear' : 'nearest',
- mipmapFilter: (i & 4) ? 'linear' : 'nearest',
});
...
Since we don’t get a texture until we call importExternalTexture we can’t
create our bindgroups in advance so we’ll save off the information needed to
create them later . [4]
const objectInfos = [];
for (let i = 0; i < 4; ++i) {
...
- const bindGroups = textures.map(texture =>
- device.createBindGroup({
- layout: pipeline.getBindGroupLayout(0),
- entries: [
- { binding: 0, resource: sampler },
- { binding: 1, resource: texture.createView() },
- { binding: 2, resource: { buffer: uniformBuffer }},
- ],
- }));
// Save the data we need to render this object.
objectInfos.push({
- bindGroups,
+ sampler,
matrix,
uniformValues,
uniformBuffer,
});
At render time we’ll call importExternalTexture and create
the bindgroups
function render() {
- copySourceToTexture(device, texture, video);
...
const encoder = device.createCommandEncoder({
label: 'render quad encoder',
});
const pass = encoder.beginRenderPass(renderPassDescriptor);
pass.setPipeline(pipeline);
+ const texture = device.importExternalTexture({source: video});
objectInfos.forEach(({sampler, matrix, uniformBuffer, uniformValues}, i) => {
+ const bindGroup = device.createBindGroup({
+ layout: pipeline.getBindGroupLayout(0),
+ entries: [
+ { binding: 0, resource: sampler },
+ { binding: 1, resource: texture },
+ { binding: 2, resource: { buffer: uniformBuffer }},
+ ],
+ });
...
pass.setBindGroup(0, bindGroup);
pass.draw(6); // call our vertex shader 6 times
});
Also, given we can’t repeat the texture, let’s adjust the matrix math to make the quads we are drawing more visible and not stretch them out 50 to 1 like we had before.
function render() {
...
objectInfos.forEach(({bindGroups, matrix, uniformBuffer, uniformValues}, i) => {
const bindGroup = bindGroups[texNdx];
const xSpacing = 1.2;
- const ySpacing = 0.7;
- const zDepth = 50;
+ const ySpacing = 0.5;
+ const zDepth = 1;
- const x = i % 4 - 1.5;
- const y = i < 4 ? 1 : -1;
+ const x = i % 2 - .5;
+ const y = i < 2 ? 1 : -1;
mat4.translate(viewProjectionMatrix, [x * xSpacing, y * ySpacing, -zDepth * 0.5], matrix);
- mat4.rotateX(matrix, 0.5 * Math.PI, matrix);
- mat4.scale(matrix, [1, zDepth * 2, 1], matrix);
+ mat4.rotateX(matrix, 0.25 * Math.PI * Math.sign(y), matrix);
+ mat4.scale(matrix, [1, -1, 1], matrix);
mat4.translate(matrix, [-0.5, -0.5, 0], matrix);
// copy the values from JavaScript to the GPU
device.queue.writeBuffer(uniformBuffer, 0, uniformValues);
pass.setBindGroup(0, bindGroup);
pass.draw(6); // call our vertex shader 6 times
});
And with that we get a zero copy video texture in WebGPU
What actually happens is up to the browser implementation. The WebGPU spec was designed in the hope that browser would not need to make a copy. ↩︎
The spec actually says the implementation can return the same texture but it is not required to. If you want to check if you got the same texture, compare it to the previous texture as in
const newTexture = device.importExternalTexture(…);If it is the same texture then you can reuse your existing bindgroup and referenced
const same = oldTexture === newTexture;
oldTexture. ↩︎
You can also use textureLoad with external textures. ↩︎
We could split the bind groups so there’s one that holds the sampler and uniformBuffer which we could create in advance and another that just references the external texture that we create at render time. Whether that’s worth it is up to your particular needs. ↩︎