We covered some basics about using textures in the previous article.
In this article we’ll cover loading an image into a texture
as well as generating mipmaps on the GPU.
In the previous article we’d created a texture by calling device.createTexture and then
put data in the texture by calling device.queue.writeTexture. There’s another function
on device.queue called device.queue.copyExternalImageToTexture that let’s us copy
an image into a texture.
The code above calls fetch with the url of an image. This returns a Response. We then
use that to load a Blob which opaquely represents the data of the image file. We then pass
that to createImageBitmap which is a standard browser function to create an ImageBitmap.
We pass { colorSpaceConversion: 'none' } to tell the browser not to apply any color space. It’s up to you if
you want the browser to apply a color space or not. Often in WebGPU we might load
an image that is a normal map or a height map or something that is not color data.
In those cases we definitely don’t want the browser to muck with the data in the image.
Now that we have code to create an ImageBitmap let’s load one and create a texture of the same size.
We’ll load this image
I was taught once that a texture with an F in it is a good example texture because we can instantly
see its orientation.
const texture = device.createTexture({
label:'yellow F on red',
size:[kTextureWidth, kTextureHeight],
format:'rgba8unorm',
usage:
GPUTextureUsage.TEXTURE_BINDING |
GPUTextureUsage.COPY_DST,
});
const url ='resources/images/f-texture.png';
const source = await loadImageBitmap(url);
const texture = device.createTexture({
label: url,
format:'rgba8unorm',
size:[source.width, source.height],
usage:GPUTextureUsage.TEXTURE_BINDING |
GPUTextureUsage.COPY_DST |
GPUTextureUsage.RENDER_ATTACHMENT,
});
Note that copyExternalImageToTexture requires that we include to
GPUTextureUsage.COPY_DST and GPUTextureUsage.RENDER_ATTACHMENT
usage flags.
So then we can copy the ImageBitmap to the texture
device.queue.writeTexture(
{ texture },
textureData,
{ bytesPerRow: kTextureWidth *4},
{ width: kTextureWidth, height: kTextureHeight },
);
device.queue.copyExternalImageToTexture(
{ source, flipY:true},
{ texture },
{ width: source.width, height: source.height },
);
The parameters to copyExternalImageToTexture are
The source, the destination, the size. For the source
we can specify flipY: true if we want the texture flipped on load.
In the previous article we also generated a mipmap
but in that case we had easy access to the image data. When loading an image, we
could draw that image into a 2D canvas, the call getImageData to get the data, and
finally generate mips and upload. That would be pretty slow. It would also potentially
be lossy since how canvas 2D renders is intentionally implementation dependant.
When we generated mip levels we did a bilinear interpolation which is exactly what
the GPU does with minFilter: linear. We can use that feature to generate mip levels
on the GPU
First, let’s change the code that creates the texture to create mip levels. We need to know how many
to create which we can calculate like this
const numMipLevels =(...sizes)=>{
const maxSize =Math.max(...sizes);
return1+Math.log2(maxSize)|0;
};
We can call that with 1 or more numbers and it will return the number of mips needed, so for example
numMipLevels(123, 456) returns 9.
level 0: 123, 456
level 1: 61, 228
level 2: 30, 114
level 3: 15, 57
level 4: 7, 28
level 5: 3, 14
level 6: 1, 7
level 7: 1, 3
level 8: 1, 1
9 mip levels
Math.log2 tells us the power of 2 we need to make our number.
In other words, Math.log2(8) = 3 because 23 = 8. Another way to say the same thing is, Math.log2 tells us how
many times can we divide this number by 2.
Math.log2(8)
8/2=4
4/2=2
2/2=1
So we can divide 8 by 2 three times. That’s exactly what we need to compute how many mip levels to make.
It’s Math.log2(largestSize) + 1. 1 for the original size mip level 0
So, we can now create the right number of mip levels
To generate the next mip level, we’ll draw a textured quad, just like we’ve been doing, from the
existing mip level, to the next level, with minFilter: linear.
Here’s the code
const generateMips =(()=>{
let sampler;
let module;
const pipelineByFormat ={};
returnfunction generateMips(device, texture){
if(!module){
module = device.createShaderModule({
label:'textured quad shaders for mip level generation',
The code above looks long but it’s almost the exact same code we’ve been using in our examples with textures so far.
What’s changed
We make a closure to hold on to 3 variables. module, sampler, pipelineByFormat.
For module and sampler we check if they have not be set and if not, we create a GPUSShaderModule
and GPUSampler which we can hold on to and use in the future.
We have a pair of shaders that are almost exactly the same as all the examples. The only difference
is this part
The hard coded quad position data we have in shader goes from 0.0 to 1.0 and so, as is, would only
cover the top right quarter texture we’re drawing to, just as it does in the examples. We need it to cover the entire
area so by multiplying by 2 and subtracting 1 we get a quad that goes from -1,-1 to +1,+1.
We also flip the Y texture coordinate. This is because when drawing to the texture +1, +1 is at the top right
but we want the top right of the texture we are sampling to be there. The top right of the sampled texture is +1, 0
We have an object, pipelineByFormat which we use as a map of pipelines to texture formats.
This is because a pipeline needs to know the format to use.
We check if we already have a pipeline for a particular format and if not create one
The only major difference here is targets is set from the texture’s format,
not from the presentationFormat we use when rendering to the canvas
We finally use some parameters to texture.createView
We loop over each mip level that we need to generate.
We create a bind group for the last mip with data in it
and we set the renderPassDescriptor to draw to the current mip level. Then we encode
a renderPass for that specific mip level. When we’re done. All the mips will have
been filled out.
Here’s a function that given a source (in this case an ImageBitmap) will
create a texture of the matching size and then call the previous function
to fill it in with the data
function createTextureFromSource(device, source, options ={}){
and here’s a function that given a url will load the url as an ImageBitmap call
call the previous function to create a texture and fill it with the contents of the image.
async function createTextureFromImage(device, url, options){
copyExternalImageToTexture takes other sources. Another is an HTMLCanvasElement.
We can use this to draw things in a 2d canvas, and then get the result in a texture in WebGPU.
Of course you can use WebGPU to draw to a texture and use that texture you just drew to
in something else you render. In fact we just did that, rendering to a mip level and then
using that mip level a texture attachment to render to the next mip level.
But, sometimes using 2d canvas can make certain things easy. The 2d canvas has relatively
high level API.
So, first let’s make some kind of canvas animation.
Loading video this way is no different. We can create a <video> element and pass
it to the same functions we passed the canvas to in the previous example and it should
just work with minor adjustments
ImageBitmap and HTMLCanvasElement have their width and height as width and height properties but HTMLVideoElement has its width and height
on videoWidth and videoHeight. So, let’s update the code to handle that difference
function getSourceSize(source){
return[
source.videoWidth || source.width,
source.videoHeight || source.height,
];
}
function copySourceToTexture(device, texture, source,{flipY}={}){
device.queue.copyExternalImageToTexture(
{ source, flipY,},
{ texture },
{ width: source.width, height: source.height },
getSourceSize(source),
);
if(texture.mipLevelCount >1){
generateMips(device, texture);
}
}
function createTextureFromSource(device, source, options ={}){
One complication of videos is we need to wait for them to have started
playing before we pass them to WebGPU. In modern browsers we can do
this by calling video.requestVideoFrameCallback. It calls us each time
a new frame is available so we can use it to find out when at least
one frame is available.
For a fallback, we can wait for the time to advance and pray 🙏 because
sadly, old browsers made it hard to know when it’s safe to use a video 😅
Another complication is we need to wait for the user to interact with the
page before we can start the video [1]. Let’s add some HTML with
a play button.
<body>
<canvas></canvas>
<divid="start">
<div>▶️</div>
</div>
</body>
And some CSS to center it
#start {
position:fixed;
left:0;
top:0;
width:100%;
height:100%;
display: flex;
justify-content: center;
align-items: center;
}
#start>div {
font-size:200px;
cursor: pointer;
}
Then let’s write a function to wait for it to be clicked and hide it.
With this change we’d only update the video for each new frame. So, for example, on a device
with a display rate of 120 frames per second we’d draw at 120 frames per second so animations,
camera movements, etc would be smooth. But, the video texture itself would only update at its own frame
rate (for example 30fps).
BUT! WebGPU has special support for using video efficiently
We’ll cover that in another article.
The way above, using device.query.copyExternalImageToTexture is actually
making a copy. Making a copy takes time. For example a 4k video’s resolution
is generally 3840 × 2160 which for rgba8unorm is 31meg of data that needs to be
copied, per frame. External textures
let you use the video’s data directly (no copy) but require different methods
and have some restrictions.
From the examples above, we can see that to draw something with a texture
we have to create the texture, put data it in, bind it to bindGroup with
a sampler,
and reference it from a shader. So what would we do if we wanted
to draw multiple different textures on an object? Say we had a chair where the legs and back
are made of wood but the cushion is made of cloth.
If we did nothing else you might think we’d have to draw 2 times for
the chair, once for the wood with a wood texture, and once for the
cushion with a cloth texture. For the car we’d have several draws, one for
the tires, one for the body, one for the bumpers, etc…
That would end up being slow as every object would require multiple
draw calls. We could try to fix that by adding more inputs to our
shader (2, 3, 4 textures) with texture coordinates for each one
but that would not be very flexible and would be slow as well
as we’d need to read all 4 textures and add code to chose between them.
The most common way to cover this case is to use what’s called a
Texture Atlas.
A Texture Atlas is a fancy name for a texture with
multiple images it in. We then use texture coordinates to select
which parts go where.
Let’s wrap a cube with these 6 images
Using some image editing software like Photoshop or Photopea we could put all 6 images into a single image
We’d then make a cube and provide texture coordinates that select each
portion of the image onto a specific face of the cube. To keep it simple I put
all 6 images in the texture above in squares, 4x2. So it should be pretty
easy to compute the texture coordinates for each square.
The diagram above might be confusing because it is often suggested that texture coordinates
have 0,0 as the bottom left corner. Really though there is no “bottom”. There is just the idea
that texture coordinate 0,0 references the first pixel in the texture’s data. The first
pixel in the texture’s data is the top left corner of the image.
If you subscribe to the idea that 0,0 = bottom left then our texture coordinates
would be visualized like this. They’re still the same coordinates.
0,0 at bottom left
Here’s the position vertices for a cube and the texture coordinates
to go with them
function createCubeVertices(){
const vertexData =newFloat32Array([
// position | texture coordinate
//-------------+----------------------
// front face select the top left image
-1,1,1,0,0,
-1,-1,1,0,0.5,
1,1,1,0.25,0,
1,-1,1,0.25,0.5,
// right face select the top middle image
1,1,-1,0.25,0,
1,1,1,0.5,0,
1,-1,-1,0.25,0.5,
1,-1,1,0.5,0.5,
// back face select to top right image
1,1,-1,0.5,0,
1,-1,-1,0.5,0.5,
-1,1,-1,0.75,0,
-1,-1,-1,0.75,0.5,
// left face select the bottom left image
-1,1,1,0,0.5,
-1,1,-1,0.25,0.5,
-1,-1,1,0,1,
-1,-1,-1,0.25,1,
// bottom face select the bottom middle image
1,-1,1,0.25,0.5,
-1,-1,1,0.5,0.5,
1,-1,-1,0.25,1,
-1,-1,-1,0.5,1,
// top face select the bottom right image
-1,1,1,0.5,0.5,
1,1,1,0.75,0.5,
-1,1,-1,0.5,1,
1,1,-1,0.75,1,
]);
const indexData =newUint16Array([
0,1,2,2,1,3,// front
4,5,6,6,5,7,// right
8,9,10,10,9,11,// back
12,13,14,14,13,15,// left
16,17,18,18,17,19,// bottom
20,21,22,22,21,23,// top
]);
return{
vertexData,
indexData,
numVertices: indexData.length,
};
}
To make this example we’re going to have start with an example from the article on cameras.
If you haven’t read the article yet you can read it and the series it’s a part of the learn how do 3D.
For now, the important part is, like we did above, we output positions and texture coordinates from our
vertex shader and use them to look up values from a texture in our fragment shader. So, here’s
the changes needed from the shader in the camera example, applying what we have above.
All we did was switch from taking a color per vertex to a texture coordinate per vertex
and passing that texture coordinate to the fragment shader, like we did above. We then
use it, in the fragment shader, like we did above.
In JavaScript we need to change that example’s pipeline from taking a color to taking
texture coordinates
const pipeline = device.createRenderPipeline({
label:'2 attributes',
layout:'auto',
vertex:{
module,
buffers:[
{
arrayStride:(4)*4,// (3) floats 4 bytes each + one 4 byte color
arrayStride:(3+2)*4,// (3+2) floats 4 bytes each
attributes:[
{shaderLocation:0, offset:0, format:'float32x3'},// position
{shaderLocation:1, offset:12, format:'unorm8x4'},// color
We need to copy all of the texture loading and mip generation code into this example
and then use it to load the texture atlas image. We also need to make a sampler
and add them our bindGroup
Using a texture atlas is good because there’s just 1 texture to load, the shader stays simple as it only has to reference 1 texture, and it only
requires 1 draw call to draw the shape instead of 1 draw call per texture as it might if we keep the images separate.
There are various ways to get a video, usually without audio,
to autoplay without having to wait for the user to interact with the page.
They seem to change over time so we won’t go into solutions here. ↩︎
ErdtR
