Seams with normal map and mirrored uvs

[njarraud]

Description of the problem

I guess it may not be just a threejs issue but I ran into a problem when using a normal map with a mesh having mirrored uvs.

With the current code, my understanding is that the RGB value (128, 128, 255) which is supposed to represent a non perturbed normal by convention is read as (0.50196, 0.50196, 1.0) - in linear color space. When converted to [-1, 1] using the function map * 2.0 - 1.0 the vector encoded in the texture becomes (0.00392, 0.00392, 1.0) when it shall be (0.0, 0.0, 1.0). This is the reason why when using a mesh with mirrored uvs and a normal texture, the two normals at a seam have different directions creating the artifact.

It can be easily proven by modifying the perturbNormal2Arb function.
Replace vec3 mapN = map * 2.0 - 1.0; by vec3 mapN = map * 2.0 - 1.00392; and the seams disappear. I attached two screenshots made using a fully metallic material and no roughness to make it very obvious:

Not too sure how this problem is generally solved as this is not my area of expertise.

Three.js version

• Dev
• r113

Browser

• All of them

OS

• All of them

[Mugen87]

I'm not sure but I think you hit here a design limitation. AFAICS, Babylon.js implements similar code in their perturbNormal() method:

https://github.com/BabylonJS/Babylon.js/blob/8bdf14354f66d12d10d61114489e1de13a9c66f8/src/Shaders/ShadersInclude/bumpFragmentFunctions.fx#L12

BTW: It seems the code in normalmap_pars_fragment and NormalMapNode is a bit out of sync. The node shader code still has a workaround that was removed from the core here: #17958

[njarraud]

Blender may also use the same type of calculation as it requires the same hack to make it work. It is maybe more understandable as asset size is maybe less of an issue and 32bit solves it.

That is really weird though because you would think that the neutral vector is kind of the base line. Anyway, I'll continue to use my hack as it does the trick perfectly. Mirroring allows to use much smaller normal textures which means smaller loading time and better GPU performance.

[makc]

so the root of the problem is, the 128 used for 0 where 127.5 should be?

map * 2.0 - 1.00392

btw for me the 1st approximation that gives 0 for 128 in the console is 1.003921568627451:

[makc]

it seems you need something like this to be on the safe side:

[donmccurdy]

Can you share the model?

Some issues like this can be avoided by computing vertex tangents and setting material.vertexTangents = true. You can add tangents with an option in the Blender glTF exporter, or with BufferGeometryUtils.

[makc]

@donmccurdy I actually get 'vTangent' : undeclared identifier shader error trying to do what you say 😊 here is the fiddle reproducing @njarraud issue, if you uncomment lines 32-34 it explodes.

[donmccurdy]

@makc use MeshStandardMaterial and there are no errors:

material = new THREE.MeshStandardMaterial({
  metalness: 1,
  roughness: 0,
  normalMap: normalMap,
  envMap: texture
});

I would've expected it to work for Phong, but not Basic or Lambert materials. We may need to update docs in that case. I'm also surprised you don't need to clone the material before setting .vertexTangents=true, but apparently it works without that.

[makc]

@donmccurdy ok it does not break with MeshStandardMaterial, that is something. but the seam issue is still there even after computing tangents - https://jsfiddle.net/jz64ow0m/

[donmccurdy]

Hm, ok — my earlier suggestion is in the right direction, but isn't going to work. On the bright side I think I've learned enough to explain the issue here. From PolyCount • Normal Map Technical Details:

When you look at a tangent-space normal map for a character, you typically see different colors along the UV seams. This is because the UV shells are often oriented at different angles on the mesh, a necessary evil when translating the 3D mesh into 2D textures. The body might be mapped with a vertical shell, and the arm mapped with a horizontal one. This requires the normals in the normal map to be twisted for the different orientations of those UV shells. The UVs are twisted, so the normals must be twisted in order to compensate. The tangent basis helps reorient (twist) the lighting as it comes into the surface's local space, so the lighting will then look uniform across the normal mapped mesh.

To eliminate seams and shading artifacts, the game engine and the normal map baking tool should use the same tangent basis. ... When you export a model from your baking tool, it is best to choose a format like FBX which can store the tangents. This insures the same tangents which were used for baking, will be used for rendering.

In other words, my suggestion of computing tangents using BufferGeometryUtils is no good — the normal map needs to be created for those tangents, because the whole thing is relative to a tangent basis.

I tried to reproduce your JSFiddle in Blender, by creating a plane, cutting it into two pieces, then UV unwrapping it with a very obvious seam: the two UV islands are rotated and separated from one another. I then baked the normal map in Blender, and you can see that even though it's a flat plane, the normal map has a different color for each UV island because of the different UV orientation:

Unfortunately I've hit a bug while trying to export the tangents from that model: the seam is reduced, but still visible, and all reflections are rotated by 90º. I think that rotation is due to Blender's different +Z=up convention, and needs to be fixed in the glTF exporter.

In short: normal maps crossing UV seams are hard, and I don't expect any change to perturbNormal2Arb is going to solve it. If you baked your normal map using software based on MikktSpace, which many programs use, then computing tangents with that same algorithm would work. But for an arbitrary normal map, if you don't have tangents and don't know what convention it was baked with, that UV probably seam isn't going away... here's the model from above, without tangents, and with a flat normal map instead of the baked one. I don't expect any tweak to perturbNormal2Arb can fix this seam:

NormalTest_naive.glb.zip

[donmccurdy]

/cc KhronosGroup/glTF-Blender-IO#920

[donmccurdy]

A note about this line:

it is best to choose a format like FBX which can store the tangents...

I'm only aware of two formats that store tangents: glTF and FBX. THREE.FBXLoader doesn't load tangents currently (@looeee I can try to create an example if we want to add this?), so GLTFLoader is the only option for now.

[donmccurdy]

And... here's one much, much simpler option if you want to avoid all that: use an object-space normal map instead of a tangent-space one. https://jsfiddle.net/donmccurdy/4zx1aumq/

	screenshot
before
after

[makc]

@donmccurdy I dont know why you need to write the research paper above when topic starter already identified both the problem and the solution :)

In short: normal maps crossing UV seams are hard, and I don't expect any change to perturbNormal2Arb is going to solve it.

No matter if you expect it to solve the problem or not, it does solve it - https://jsfiddle.net/cuotmdfL/

if you want to avoid all that: use an object-space normal map instead of a tangent-space

or do not use normal map
or do not use geometry mirroring

it is like saying, just do not run into a bug. it works, butnot really helping it :)

[donmccurdy]

THREE.ShaderChunk.normal_fragment_maps = THREE.ShaderChunk.normal_fragment_maps
  .split("texture2D( normalMap, vUv ).xyz * 2.0 - 1.0;")
  .join("texture2D( normalMap, vUv ).xyz * 2.0 - 1.003921568627451;")

I see your point, it fixes the specific issue reported here, but ... this patch is not a general solution to UV seams in normal maps. Please feel free to try the same patch on the model attached above. 😬

Do you feel it should be merged to normal_fragment_maps.glsl?