Material: Added onBeforeCompile()

[mrdoob]

Over the years, a common feature request has been to be able to modify the built-in materials. Today I realised that it could be implemented in a similar way to Object3D's onBeforeRender().

e55898c

WebGLPrograms adds onBeforeCompile.toString() to the program hash so this shouldn't affect other built-in materials used in the scene.

Here's an example of the feature in action:

http://rawgit.com/mrdoob/three.js/dev/examples/webgl_materials_modified.html

material.onBeforeCompile = function ( shader ) {

	// console.log( shader )

	shader.uniforms.time = { value: 0 };

	shader.vertexShader = 'uniform float time;\n' + shader.vertexShader;
	shader.vertexShader = shader.vertexShader.replace(
		'#include <begin_vertex>',
		'vec3 transformed = vec3( position.x + sin( time + position.y ) / 2.0, position.y, position.z );'
	);

	materialShader = shader;

};

Not only we can mess with the shader code, but we can add custom uniforms.

if ( materialShader ) {

	materialShader.uniforms.time.value = performance.now() / 1000;

}

Is it too hacky?

/cc @WestLangley @bhouston @tschw

[unconed]

It does seem a bit hacky to require custom string manipulation when WebGLProgram.js already does a lot of processing. There is a related use case, which is to add new #includes rather than overriding existing ones. You can do this today by modifying THREE.ShaderLib at runtime, but it feels equally dirty.

Perhaps a nicer solution for both is to allow you to provide a custom ShaderLib for a material, that would override/augment the built-in chunks without manual string manipulation and without permanently clobbering them. That doesn't preclude an onBeforeCompile hook for other uses, but it seems more in line with what your example is trying to do.

That said, shader composition based on raw includes is always going to be very fragile from one version to the next. If you don't mind including all the skeleton code (like meshphong_vert.glsl), you can already extend the built-in materials with something like this:

export class MyPhongMaterial extends ShaderMaterial {
  constructor({ color, radius, map, normalMap, emissiveMap }) {
    super();
    this.vertexShader = "...."
    this.fragmentShader = "....";
    this.uniforms = UniformsUtils.clone(ShaderLib.phong.uniforms);

    this.isMeshPhongMaterial = true;
    this.lights = true;
   
    this.uniforms.time = { value: 1 };
    //...
  }
}

In all cases, you have to rely on the internal organization of built-in shaders not to change too much from one release to the next. Each include is already a function in disguise, with an ill-specified signature. I suspect it is going to be much cleaner and maintainable on both sides to provide function-level rather than include-level overrides.

[pailhead]

Today I thought..

Nice! I thought so too on february 10th when i made this pull request:

#10791

It seems the only difference is that you're doing it through a callback? I tapped into the unused argument in the WebGLRenderer.

I like using this for example to get threejs to work with normal maps, but I've linked some issues that seem like they would be trivial to solve with this.

You can look at my branch and take this for a spin or check out this super simple example (just tests a custom deformation with lighting/shadows etc, and adds a few uniforms).

http://dusanbosnjak.com/test/webGL/three-material-shader-override/webgl_materials_shader_override.html

Perhaps a nicer solution for both is to allow you to provide a custom ShaderLib for a material, that would override/augment the built-in chunks without manual string manipulation and without permanently clobbering them.

Exactly sums up what happens in: #10791 I wish i could write better descriptions :)

---

I must admit that i don't know wth is going on here. I think i know someone who gave up contributing to three, didn't understand it then, but now i find it kinda frustrating.

Frustrating mostly because i'm having a deja vu when i read the first paragraph:

...similar way to Object3D's onBeforeRender().

This exact same thing happened with the onBeforeRender() #9738, when it took a year of convincing to get the feature in.

Jeez, twice, with the same dev? I'm obviously doing something wrong here, but what? The first time around was the build files being checked in, i didn't know what to do, and it just got forgotten. But this time around i was on top of this pull request, there's a conflict right now but it's easily solved, were no conflicts for months.

Don't get me wrong, i don't think it's vanity in the works here, i think i just want to run ideas by people and get feedback. @unconed it's obvious that you're familiar with the problem and you've probably tried different things. What may have caused an user like you to miss the other PR?

I felt pretty good about the per material shaderlib, but i did find one function in the renderer that looked/compared entire shaders as strings to figure out caching, wasn't feeling all that great about that. Wish there was some feedback given...

Was the example flawed? The head may make it much more obvious what is going on than my abstract spike ball, but the whole example works with shadows which covers more ground. It's heavy for some reason, but i think it's because sin/cos on many verts.

[pailhead]

I guess if you're starting over... Even though scene kit is absolutely horrid, this api is really nice:

https://developer.apple.com/documentation/scenekit/scnshadable

Albeit horribly documented, i can't find the part that's of more interest, but basically they have abstracted hooks at many different stages of the shader.

[mrdoob]

Earlier this year i made a pull request that seems to do something similar if not the exact same thing:

#10791

It seems the only difference is that you're doing it through a callback? I tapped into the unused argument in the WebGLRenderer.

Sorry for not being able to take care of that PR yet @pailhead. I guess the main advantage of my approach is that it only required 3 new lines.

Having said that, I'll now spend some time studying yours and @unconed suggestions.

[pailhead]

This one definitely feels more elegant, only three lines. I was following a different pattern in the other one. Was going to say that it may be a bit more verbose, but not so sure. I guess the only advantage of the other one is that it was good to go a few months ago :)

[mrdoob]

Jeez, twice, with the same dev? I'm obviously doing something wrong here, but what?

Sorry about that. The only thing I can think of is that probably I got overwhelmed. Maybe a long discussion or a complicated PR that would consume too much of my energies, so I decide to leave it for later and move to simpler PRs.

It's not only you @pailhead. @bhouston has had a lot of PRs like this, even @WestLangley and @Mugen87.

Again, is not that I don't like the PR, is that the PR requires some attention I can't offer at the time. A good example is the Instancing PR. I managed to find time to read through it, did my own experimentation and suggested simplifications. Hopefully I'll be able to revisit it soon.

It's difficult to manage all this and give every PR the attention they deserve.

Was the example flawed? The head may make it much more obvious what is going on than my abstract spike ball, but the whole example works with shadows which covers more ground. It's heavy for some reason, but i think it's because sin/cos on many verts.

Now you mention it... Yeah, runs at 10fps on a new MacBook Pro when zooming in 😮

[pailhead]

I think it's the shadows and the rand, sin cos and stuff, but yeah it looks like it takes too much of a hit.

[pailhead]

Either way, i'm surprised you pulled this off in three lines, i was focused too much on how material params get turned into uniforms and followed that pattern. The difference is that I provide an alternative ShaderLib like dictionary, so #include <> brings in different code. You remove the include before this happens, and replace it with glsl. I think if you return some kind of a wrapper around the shader, perhaps this syntax could be cleaner (just provide the chunk name + glsl, instead of replace(). But it may end up looking much more like the other one.

It would be really nice to have this, i haven't worked with that many 3d engines, but unity and scene kit seem to have something like this.

[mrdoob]

@unconed

I guess one benefit from the onBeforeCompile() is that the material properties remain unchanged. So it feels more like extending the built-in materials.

export class MyMeshPhongMaterial extends MeshPhongMaterial {
  constructor( parameters ) {
    super( parameters );
    this.onBeforeCompile = function ( shader ) {
      shader.vertexShader = shader.vertexShader.replace(
        '#include <begin_vertex>',
        'vec3 transformed = vec3( position.x + sin( position.y ) / 2.0, position.y, position.z );'
      );
    };
  }
}

var material = new MyMeshPhongMaterial();
material.color.setRGB( 1, 0, 0 ); // this still works

Messing with ShaderLib is tricky indeed. Adding forever-unchanged hooks should be doable though:

	#include <begin_vertex>
	% vertex %
	#include <morphtarget_vertex>
	#include <skinning_vertex>
	% transformed_vertex %
	#include <project_vertex>
	% projected_vertex %

The replace code will become this:

this.onBeforeCompile = function ( shader ) {
  shader.vertexShader = shader.vertexShader.replace(
    '% vertex %',
    'transformed.x += sin( position.y ) / 2.0;'
  );
);

We'll then remove any hook that hasn't been used before compiling, of course.

[pailhead]

here's what it looks like compared to the per instance ShaderChunks

//given some material
var material = new THREE.MeshNormalMaterial();

//and some shader snippet
var myShader = [
	'float theta = sin( time + position.y ) / 2.0;',
	'float c = cos( theta );',
	'float s = sin( theta );',
	'mat3 m = mat3( c, 0, s, 0, 1, 0, -s, 0, c );',
	'vec3 transformed = vec3( position ) * m;', //and making assumptions about THREE's shader framework
	'vNormal = vNormal * m;'
].join( '\n' );

#10791 same as Material.defines approach:

material.shaderIncludes = {
	
        begin_vertex: myShader,

	//uv_pars_vertex: [
	//	THREE.ShaderChunk['uv_pars_vertex'], //this doesn't have to be
	//	"uniform float time;",
	//].join('\n')
};

material.shaderUniforms = { time: { value: 0, type: 'f' || 'float' } }; //because this could just inject it in the right place (but needs type)

It's only a dictionary of chunk names, and the uniforms object. The string manipulation here is more along the lines of "i want to reuse this chunk, and do something on top of it"

this PR:

material.onBeforeCompile = function ( shader ) {

	shader.uniforms.time = { value: 0 };

	shader.vertexShader = 'uniform float time;\n' + shader.vertexShader; //this feels hacky
	
	shader.vertexShader = shader.vertexShader.replace( //this is more verbose
		'#include <begin_vertex>',
		myShader
	);

};

I guess one benefit from the onBeforeCompile() is that the material properties remain unchanged.

It works the same in the other PR, so I think it doesn't matter how it's done.

---

One probably irrelevant benefit that i can think of is that you can keep the logic for manipulating the material in the same block and on the same level. Even if the code doesn't look much different, conceptually, you're manipulating .color on Material while manipulating a .uniforms object on a "shader". If you had a piece of code somewhere changing color, and you later add custom GLSL to that material, you can add the code to change the uniform with the color. (skip wall of text)

---

If the uv_pars_vertex is confusing, i think a good question to ask is:

"where do I inject my custom GLSL function, like float rand( vec2)?"

Then this one might make sense #11050

---

Another argument could be - assuming you know GLSL, assuming you know THREE's shader framework, you still have to be creative and think about where you inject what. I had to think a bit and look through most of the shaders to figure out that uv_pars_vertex is a convenient place to do my material extension.

It would be better to go from something like this towards an abstraction, have a lightingPhase, objectTransformation , perspectiveTransformation etc hooks. Or at least a guaranteedToBeAboveMainButBelowCommonsAndExtensionCalls dummy chunk that's included in every shader program. It looks like you're going for that with % vertex %? But i'm unfamiliar with that syntax.

This PR, as is, seems like it's going in the opposite direction. On top of knowing where you need to tap in, you also need to be explicit and manipulate strings, repeating some of the work that the renderer already does for you. Also, if you ask another question in addition to the first one

How do i use a function, declared in the commons chunk, inside the function that i'm injecting?

you might find yourself doing more string manipulation than just prepending the uniforms to the entire shader.

[mrdoob]

I had to think a bit and look through most of the shaders to figure out that uv_pars_vertex is a convenient place to do my material extension.

I don't think there is a way around that unless we over-engineer. The idea with % vertex % is to try helping with that a bit by naming the hooks and more or less tell you at what state you're injecting code, but it would be hard to document what variables are available at what point.

We're opening a door for allowing built-in materials hacking, but I don't think we can provide proper "support". The user should be aware that chances are things may break.

[pailhead]

I tried to tackle it in the other PR. I added a dummy hook at least for the functions, varyings, attributes and uniforms. Lots of the GLSL logic already happens on structs, scenekit documented each variable (cant find the same documentation anymore though).

It would probably need to be refactored as we go along but something along these lines:

#ifdef PHASE_FOO

#include <shader_foo> 
//someGlobalStruct.mvPosition = modelViewMatrix * myLogic( transformed );
//if there is glsl provided for phase "foo" document that it should operate on "transformed" 
//and that it should return "mvPosition" 
#end including <shader_foo>

#else 

  //default
  #ifdef USE_SKINNING

	vec4 mvPosition = modelViewMatrix * skinned;

  #else

	vec4 mvPosition = modelViewMatrix * vec4( transformed, 1.0 );

  #endif

#ifdef PHASE_BAR

#include <something_like_this>
//mvPosition = myPostDefaultTransformationLogic( mvPosition );

#endif

gl_Position = projectionMatrix * mvPosition;

#endif

Of course ifdefs and includes would proably not work like this. But i can see it being relatively straight forward with the other approach. We could add #include <some_phase> always, with empty strings. Something could help manage what gets triggered by using #ifdefs. So for example, if you provide logic for "vertex_transformation" phase, the shader will define whatever you provide. Documenting what each chunk does with GLSL would be overall helpful. Without going further into abstraction, i think it would be great to have this map to how the chunks are structured now.

[pailhead]

We're opening a door for allowing built-in materials hacking, but I don't think we can provide proper "support". The user should be aware that chances are things may break.

We already have the defines stuff available on every material. #10764 suggested to modify THREE.ShaderLib and then define a .defines property on an instance of a Material. Granted, the property was undocumented until i wrote the entry (but you approved it :)), and is still undefined.

But the way this currently works, and is inherited by every material, it is supported - if you put stuff in it, it will affect the shader for that material. Furthermore, if you happen to define something that is already part of the shader library, it may break things.

Not trying to be a smart-ass but trying to come up with an example. I mostly used defines with ShaderMaterial but it does affect all materials, just because how WebGLRenderer works. There is no logic that says:

if you're a much smarter, surface like abstraction of a material like Phong then don't take defines into account. Because three is not designed to have it's shader chunks overriden, and the Phong template is the final authority on what GLSL needs to do, and serves to hide this from the use, it makes sense to not have GLSL defines interfere with this.

I'm not entirely sure if it can be broken though, i'd have to try.

With that being said, i'd like the option to break things, if it provides a lot of gain. With a pretty encapsulated and tiny GLSL snippet, i could easily change my version of three to render normal maps differently. Taking a normal, and transforming it to get another normal is super straight forward, unless we start doing computer graphics in a completely different way, i can't see this ever breaking :)

Instancing was another example:

oh i'd like to use ANGLE_INSTANCED_ARRAYS, three doesn't seem to support it with it's shadows or PBR materials, let me just add these two lines, and make it work for me"