Tiling Rock Tutorial

Tiling Rock Tutorial

This tutorial is a walk-through of creating a tiled rock surface normal map with accompanying parallax and diffuse maps. This asset was chosen in particular for the tutorial because its creation required some advanced techniques in Max, ZBrush, and Photoshop, as well as close integration between the three programs. Many of the techniques shown here can be used for a far greater range of assets than simply rock. This is just one example.

We start by finding good rock photo. This is just for visual reference, so it is not necessary to find an image of the exact rock we’re aiming for, just something that will give a good idea of surface texture, flow, and structure.

For a tiled texture, a certain amount of homogeneity is necessary, or else the tiling will become too apparent. The above photo shows too much variation, but nevertheless a great deal of information can be gleaned from it. Of particular note are the lines and cracks of where the rock has broken, which have been highlighted below. Notice the angles. While it is certainly not a grid pattern, there are distinctive horizontal lines intersecting with downward ones, almost never at a perfect 90 degrees, but often between 60-90 degrees. The effect is of layers that have broken and shifted, cracking and shearing off as things have moved. While the rock that we are making will ultimately be very different from this one, those lines will be a good starting place.

Here is another reference photo. Notice where the layers have broken, leaving a jagged surface that conforms to the relatively orthogonal original lines of the rock, but with a great deal of faceted noise at the break points.

We will begin in Max with a plane. Cut the base lines into the plane at angles that approximate the reference images. Keep enough variation to make it feel natural, but not so much that it feels completely random or that that there are easily identifiable features which would interfere with tiling. For the moment do not worry about matching the tile seams, we will deal with that at the end in Photoshop.

When the lines are in place, as shown below to the left, extrude shapes outward. Cut more lines as necessary and tweak the surface so that different facets face in different directions, as show below to the right. Make sure to keep reasonable smoothing groups, so large swaths seem contiguous, but the cracks and major facets are separated by hard edges.

It is necessary to create these hard polygonal lines in Max, because the tools in ZBrush are poorly suited to such things. In ZBrush we will be able to make these look far more natural and weathered, but it is necessary to have hard sharp polygonal edges to begin with, or else things would come out looking too smooth and claylike.

The next step is to UV map the sheet. This will be necessary in order to take advantage of certain tools in ZBrush. Since the geometry will eventually be reduced to a simple normal map eventually anyhow, it is not necessary to do anything elaborate with the UVs. Apply a helper image to see the UVs, and use a UVW Mapping modifier with the Planar mapping type, then apply a UVW Unwrap modifier. Open the Edit dialog, select all the vertices, and use the Edit UVs > Tools > Relax Dialog. A little tweaking of the numbers as shown below will sufficiently relax the UVs at the edges where the worst distortion occurs, which is all that is really necessary.

Select all the faces and extrude them upward to create an airtight model. While ZBrush can import an object with open edges, its generally not a good idea, because the brushes behave strangely when they hit the edge.

When that is complete, MeshSmooth the object several iterations with the Separate By Smoothing Groups option checked. This is necessary to preserve the hard edges of the cracks, because if the geometry were subdivided in ZBrush, it would not take into account the smoothing information as Max does, and the hard edges would become very soft. This does not mean that we can’t do further subdivision with ZBrush, but this step in Max ensures that the edges don’t suffer much from it.

Now import the model to ZBrush, over a canvas that is the size that the normal map should eventually be. Subdivide further as hardware allows. We have subdivided to slightly under 1.8 million polys.

We will now apply a global displacement using an alpha to create the small scale details. It is best to do that at this early stage so that we can smooth back out the appropriate bits as we’re working on the mid-level detail. The global displacement will eliminate the need to laboriously sculpt in all the small scale noise and cracks by hand. This was the primary reason for UV mapping the sheet in Max, because the displacement feature relies on the UVs for placement of the alpha. Find a high resolution photo of rock that has the appropriate amount of noise, cracks, and other variations, then in Photoshop alter and crop it until it is appropriate for a displacement map. Before converting it to a grayscale, switch to 16 bits per channel. RGB images generally have a far greater luminance range than an 8 bit grayscale, because the ultimate luminance of each pixel is an average of the values for all three channels. Save the 16 bit grayscale as a PSD and import it into the alpha palette of ZBrush. Save the model now before applying the displacement.

Make sure you are at the highest subdivision level. Now use the slider under Tools > Displacement > Intensity to adjust how much displacement will occur, then hit Apply DispMap. It may be necessary to undo and retry with different intensity values, which unfortunately triggers a minor bug in the current version of ZBrush. Every time you undo a displacement and redo it, ZBrush will take more and more memory. After a few times, it might be necessary to restart ZBrush and load the file that was just saved.

Now drop back down a couple subdivision levels. Notice how at this level the fine grains from the displacement disappear, but larger ones still remain. We can sculpt at this lower level without significantly altering the fine grain at the highest levels unless we chose to. It is useful to step between the levels often while you work, using the D and Shift-D keys.

Use the nudge and pinch tools to make the lines more natural looking. Paint in extra cracks by inverting the Inflate and Standard tools, then squeeze them together using the pinch tool.

Don’t underestimate the usefulness of the pinch tool. It is the best way to make surfaces press against each other, and to create hard edges that weren’t modeled in Max. A particularly useful technique is to paint in ridges with the Std or Inflate tool, then pinch along the peak to create an edge. Then invert the Std or Inflat tool, and paint downward again on the sides with a smaller brush size to make them concave. This will result in surfaces and edges that better approximate rock that has chipped off.

Continue adding more details and rock protrusions. Make larger features at lower subdivision levels, and smaller ones at high levels.

Don’t worry about stretch marks that appear when you nudge or pinch the surface. On the untextured model in ZBrush these might look unsightly, but the stretch marks will actually help the surface look more natural and weathered once there is a diffuse applied (which of course won’t stretch as the geometry does). Rock has a flow, and so long as these stretch marks flow in a natural way, they should be kept.

It may become necessary to work at a lower subdivision level than what was originally imported. For example, here some of the shapes are looking too much like bricks mortared together, but using the nudge tool to make them more natural becomes very cumbersome with this many polys. The original Max geometry had been deliberately MeshSmoothed several times to preserve the edges, so there is no low poly level to return to. Fortunately ZBrush allows you to reconstruct lower subdivision levels with the Tools > Geometry > Reconstruct Subdiv button. It should be noted however that this step burns a bridge. Normally it is possibly to export a mesh at any subdivision level from ZBrush out to Max, alter it in Max, then reimport it back to that subdivision level in ZBrush, while still keeping all the data in the other levels. Once you use the Reconstruct Subdiv feature however, ZBrush changes the point order, which creates incompatibilities. It will still be possible to export out to Max, but not to bring that model back into a subdivision level while preserving the other levels. That is not an issue here however, so we use the Reconstruct Subdiv button.

Here we have created several lower levels and used the nudge tool at the lowest to move the contours of the rock into a more natural looking flow, as well as changing the elevations of certain features to create more variation.

Once the surface looks decent from a distance, isolate sections using Ctl-Shift-Drag, and zoom in on the details. Smooth down sections that look too rough with the Smooth tool, and continue pinching thing to sharpen them up. It is important that the rock look just a good close up as it does as a whole.

The section below here looked fine until it was zoomed up close, which revealed all sorts of details that are too noisy, soft, muddy, or otherwise un-rock-like.

You may notice that many parts of the surface actually look better at subdivision levels that are a couple steps lower than the fine grain detail, because the midrange does not get drowned out in the noise. In the image below, the left side shows the highest subdivision level, while the right side shows two levels below it. The right side ends up looking far more jagged and rocklike, but certain parts are obviously too low res, showing the rectangular polys much like an image shows pixelation when zoomed too far. Thankfully we can capture the normal map from various subdivision levels and states of the model, then blend them later in Photoshop. When youre ready to bring the normal maps into Photoshop, position the sheet over the ZBrush canvas pointing straight up. Unfortunately ZBrush does not make precise rotation easy. You must leave edit mode entirely and select the rotate button, before the Transform > Info sliders will allow you to explicitly enter angles of 0, 90, or whatever values are necessary. Alternately, there are several user-created ZScripts that offer one-click buttons for easily rotating a model to its orthogonal positions. MoldyTool is particularly useful for all sorts of modeling tasks, including this.

Apply the NormalRGBMat Material, which can be found in the ZBrush2/ZMaterials directory. Before dropping the model to the canvas, create a marker of it so that you can easily recover the placement, then save the model. Now leave edit mode to drop it. Select the MRGBZGrabber tool, and under Tool > Modifiers turn off Auto Crop. Grab the entire canvas. This will put the color data to the Texture palette, and the depth data to the Alpha palette. Save each of the images out to PSDs. Clear the canvas and use the Marker to recover the positioning of the model. Change the subdivision level and repeat the steps to export the normal map (though it is not necessary to export multiple height maps). It is good to have normal maps for the highest level (for the fine grain), one that is a couple levels below it (for the midrange), and one that is very low (for very sharp edges). Another variation that can be useful is to use the Tool > Deformation > Inflate slider at the highest subdivision level to expand out the grains into a more pebbly surface and to squeeze the cracks into sharper angles. We will blend together all of these maps later in Photoshop.

The image below shows a close-up progression between the normal maps, starting with a very low subdivision level on the left, a medium one, the highest, then the highest with an Inflate deformation applied to it.

There is one more variation that we will need. Go to a medium subdivision level, select the Nudge tool, set it to a very high Z Intensity (between 50 and 100), and begin pushing the geometry of the rock facets around. The goal here is to push it around the same general area, so for example if you nudge it left, then while the brush is still pushed down nudge back to the right again, and continue the back and forth motions. This will stretch the geometry into shapes that look very much like shattered rock, where pieces have broken off. The key is that we see the polygons themselves which are stretched along the flow of the break. This is why we do it at a medium subdivision level rather than the highest, because the jaggedness of the stretched polys is more evident. Pinch the edges and corners to make them less rounded. Unfortunately there is a side effect to this nudge-pinch method, which is that it sometimes bunches up the edges around the break in unrealistic ways. But don’t worry about that, we have all those other renders that can be blended over it in Photoshop.

When you’re ready, drop this version to the canvas and export the normal map with the MRGBZGrabber tool the same way as the others.

Import all of the normal maps into Photoshop and layer them atop each other, giving each a layer mask. Paint into the masks to fade in the different normal maps where they are appropriate. The fine grain (either in the high level or the inflated high level renders) should be present everywhere to some degree, but very faint in some of the sheared off areas and the jagged edges. Make sure to use it heavily where the midrange surface has a pixelated look from being too low a subdivision level. Paint in the mid level at places where the nudged version bunches up unrealistically, and the low level where an edge needs to be particularly hard and straight. Feel free to import the image into Max or UnrealEd at any time to check how it looks.

The image below shows the final results of this process.

Now we need to make it tile seamlessly. Copy the merged image, paste it as another layer, and run Filter > Other > Offset with Wrap Around selected. Offset it by half of the image resolution, so the seam ends up in the middle of the image. Now add a layer mask, and mask out the parts with the seam, making sure to paint forms that seem appropriate for the flow of the rock, and that blend well with the area around them.

Now you have the final normal map.

Open the depth image now. It will be a 16-bit grayscale. Do a Levels adjust to maximize the range of the image. Then use Image > Mode to convert it to an 8-bit grayscale. Use the same offset technique described above to make it seamless. It should be possible to simply copy and paste the mask from the previous offset to maintain consistency.

Now we move on to creating the diffuse. Find a source image (or several) that have large swaths of open rock without much cracks or other drastic changes in luminance. It is important to have quality image source that is high enough resolution. Shown below are several images from a rock climbing site (hence the yellow marks showing the route), which are perfect for texture source. Use the stamp tool to create an even swath of rock without any cracks, like the image below to the right.

We need to add some more variation, with bits of lichen and sections of rock that are a different color. Find an image with a good amount of noise with the type of variation we’re looking for. Don’t worry if it’s the wrong colors or luminance values. The image below to the left was intended for rusty metal, but has all the correct elements. Place the image in a Layer Set on top and start adding adjustment layers. The goal is to get something which can be Multiplied on top of the original. A great deal of variable tweaking might be necessary. The result to the right was accomplished with a Levels, a Color Balance, and a Hue/Sat.

The image below shows the result of Multiplying the layer set on the rock swath. Tile it seamlessly with the same method used for the other maps.

All that we need now are some global illumination shadows so that when the normal map is lit head-on, the form does not disappear completely.

In Max create a plane, then duplicate it and raise the duplicate by a small amount. In a new material, apply the normal map to the bump channel with a Normal Bump component type, and apply it to the top plane. Go into the vertex sub-object mode, click the Preserve UVs checkbox, select all the vertices, and scale them out a bit so that it tiles off the edge to give it a buffer zone to bleed over. Select the bottom plane and set it up to accept projection from the top one, making sure to turn off the cage. Choose a Lighting Map to project instead of the usual normal map. Create a white Skylight with shadows. Render the Lighting Map.

Bring the resulting light map into Photoshop and multiply it over the diffuse. Since it was generated from a seamlessly tiled normal map that extends past the seam, the light map is already seamless and can be laid right over the image without any worries. Some Levels adjustment might be necessary, but don’t make it too dark, or else it will invalidate the actual real-time lighting that the normal map is for.

The image below shows the final result.

If you haven’t already, bring the textures you’ve made into 3D and preview them on geometry. Make alterations if they seem necessary. Here it is on a plane in Max with a DirectX9 shader.

And here it is on curved geometry in the Unreal 3 Engine. Perhaps not perfect, but pretty good looking rock.