Category Archives: Graphics

Creative Victoria, Game design, Graphics, Procedural generation, Really long posts, Sun Shy

Animating on uneven surfaces

One of the looming technical challenges in Sun Shy is getting a character to look right as they run and jump around. For most of our game’s development, our character has looked like this:

We’ve known for a while what we want our characters to look like – the player character of Sun Shy is to be a long-legged satyr-looking biped creature. Here’s some very old concept art from Halley that gives a bit of the vibe:

As a programmer with no artistic skills, I consider artists to be wizards of some kind, so I feel a lot of trust has been placed in me when I’m trying to recreate something like the above feel with code. There are various stylistic choices that can be made to make this kind of thing easier – give your characters short legs and take very fast, short steps, for instance, or giving them feet with no legs (like Rayman). These are/were all certainly options, but thanks to the support of Creative Victoria I have time to try doing it the hard way.

This post is a long one, and I don’t even really get to the bottom of it here. I’m sure I’ll be talking about animation-related things again in the future.

A note about game design philosophy

One part of this process is something we consider non-negotiable – visuals are in service to gameplay, not the other way around. We tuned the player controls so that they feel right before we started thinking about animation, and if it turns out that the way the player controls is impossible to animate without it looking goofy, we’re going to have a goofy-looking game and that’s just the way it is. At no point will we allow any of the systems mentioned in this blog post feed values back into player movement. This is mostly a game design thing – if the player wants their character to do something, but the character can’t do it because their feet aren’t in the right place, and the positioning of their feet is controlled not by the player but by our animation code, that’s not fair to the player. Also, what feels right and what looks right aren’t necessarily the same, and given the kind of game we’re making, the first one wins. I’m not trying to impose this rule on all game developers everywhere, and there have been some pretty notable games that go the other way (the movement in the old Prince of Persia games was animation-driven and worked pretty well, for instance), but it’s how we’re doing Sun Shy, and probably how all Snake Hill games will be done in future.

There’s also a technical reason for this. The systems described below can be expensive, and we want to be able to turn them down or off if necessary. Not just in a ‘turn down the graphics settings’ kind of way – if there are a hundred entities walking around, and only three of them are on screen, we want to only do the complex footfall calculations for the three that are visible, and we don’t want that change to affect movement. It also means in networked situations, the client can do their own local footfall calculations rather than needing to sync it over the network.

Inverse kinematics and foot placement

There are two parts to this problem – where to put the feet at any given time, and having done that, how to draw the leg. The second problem is solved with something called Inverse Kinematics, often abbreviated to IK. Today I’m just going to be talking about foot placement, because the IK part of this took place outside the Creative Victoria research. It does have a few interesting things to be said about it, though, especially because the creatures we’re animating here have two knees per leg – I’ll be writing a blog post about this later on.

A straightforward attempt at foot placement

The first attempt we made to have feet get placed coherently was vaguely inspired by David Rosen’s GDC talk about procedural animation in Overgrowth. (I say vaguely, because they’re really not terribly similar, and I don’t want to implicate David in the fact that this technique didn’t work for us.) Our plan was essentially to create an ellipse, roughly where the player’s feet should be, and move two feet around that ellipse. We would then do a ray cast straight from the player’s hip to the current foot location, and if it hit something, place the foot there.

This method was initially quite promising. The foot isn’t really planted, so this system doesn’t guarantee a lack of foot skate, but if you base the rotation rate of the ellipse on the horizontal velocity of the character you can make it look pretty convincing. It even looked pretty good while the player was airborne without any further work – they’d kind of flail around in the air, but while it looked a bit comical, it was okay, and mostly not broken.

Endearing, but kind of dodgy. Any initial promise this showed didn’t really materialise when we tried to polish it, though.

The problem proved to be the ‘mostly’. The last 10% of quality here was elusive – sometimes foot skate reared its head, so we made a system to actually lock the feet to world space when they hit something, then lerp them back to the ellipse when they stopped hitting something. That ended up having feet sometimes get located weirdly up on ledges when the player was running towards a step and then stopped at the last minute. We also needed to do some raycasts to determine where the centre of the foot ellipse should be, and then some smoothing of those values so it wouldn’t pop when the player walked over a ledge.

Ultimately the system became incomprehensible. I ended up with variable names like footGroundedHeightSmoothedFudgeMultiplier, and trying to fix a given problem involved way too much trial and error. Worse, fixing one problem would often break something else. In other words, all the problems that usually come up with badly written code. This is what inspired the next attempt – a system that, whether or not it works, might at least make sense to me rather than being a bunch of hacks hanging together in strange equilibrium.

The Analytical Approach

So, what’s the opposite of the above ‘make something up and hope it looks kind of good’ approach? Simulate everything! You can probably guess from the number of subheadings that this is the approach that eventually worked, or at least currently shows the most promise. Mentally, I broke this part of the task up into three sections:

  • When a given foot should start a step (timing)
  • Where that foot should be stepping to (foot placement)
  • How it’s going to get there (stride shape)

I won’t be going through these in turn, because it’s not the order I actually solved things in the end.

Stride shape

The shape a footstep takes is probably the least analytical part of the process. Basically, it’s a spline – the foot moves along it. Formally, it’s a cubic Bézier spline – the same things that you draw with in vector art programs. This isn’t even remotely based on a physical system. This kind of spline has four control points – the first and last will be the location the foot is coming from and the location the foot is targeting, and the second and third will determine the shape it follows to get there. So far we have adjusted this basically by eye, and it’s been okay, but it’s a future target for tweaking to make things look nicer after other parts of the rendering (such as bringing in actual skinned character meshes) have been polished.

Using splines here has a few major advantages. Splines are very predictable – we can decide exactly how long we’re going to take to traverse the spline, and then ensure that it happens, by simply ticking our T value appropriately. This means we can look ahead and say “We need this foot to be at this location exactly this many frames in the future”, and it will definitely happen. Somewhat more complicatedly, it also allows for retargeting of a step – we can get the parametric form of a spline and take its derivative to get its velocity at any point. This means that if the player is half way through a step and they need to put their foot somewhere else instead, we can create a new spline from the current position to the new target without any discontinuities in velocity or position (which looks unnatural)

Foot Placement

This turned out to be one of the biggest, trickiest parts, but it also has some of the most actionable advice for anyone out there attempting a similar system. That advice is: Prediction is king.

Our initial plan involved predicting where the player would be by using simple dead reckoning – that is, we just take the character’s position and velocity, and if we want to know where they’ll be in n seconds, we assume it will be currentPosition + currentVelocity * n. This will be accurate assuming our velocity stays constant

Because we’re using splines for our stride shape, we can dictate exactly how long a given stride is going to take – so when a foot determines that it needs to initiate a step, we know exactly how far in the future it’s going to land. So we simply predict where we’re going to be at that time, and do some raytraces down from there to find an appropriate target for the foot to plant. This is going to be great.

Turns out, it isn’t great, because dead reckoning sucks. Behold:

Essentially, any time our character would have a path that wasn’t totally linear, such as running over a bump, the system might fail badly. Any time it happened to initiate a foot step while on the upward step of a bump, it would say to itself “Well, where am I going to be when this foot needs to land… IN THE SKY APPARENTLY” and then start flailing as though it had just run off a cliff (I regret that I didn’t save any gifs from this period, and I’m too lazy to go back through version control history for them, but it was mostly less hilarious and more just broken-looking).

Biting the prediction bullet

It became apparent that we were going to have to do this properly. We were going to have to simulate ahead of time where our player’s character is going.

At this point, I realised there wasn’t anything in between the half-arsed ‘project velocity forward’ method, and a complete ‘predict by simulating the player’s movement for a little while into the future’ solution, so that’s what needed to happen. This seems like it would be straightforward – or rather, it seems like something your engine would do for you, so you wouldn’t have to worry about it. Unfortunately, that’s probably not true.

The reason it’s not always true is that physics engines don’t always let you say “Hey, this one single rigid body, could you simulate it in isolation for a few seconds, then throw all that information away and go back to the current frame”. For instance, I believe Unity has a function Physics.Simulate() that allows you to look ahead, but you can only do this for the entire world, not a single object. I believe Box2D could be persuaded to do something similar, but I think this would only be acceptable for a relatively simple game (in terms of physical complexity) – Sun Shy can easily have dozens or hundreds of rigid bodies present, and thousands of pieces of static collision geometry, so I don’t think this would be very feasible.

This leaves us with the not-terribly-fun option of recreating a simplified version of our own physics engine so we can run our own look-ahead simulation each frame. Remember that this doesn’t have to be perfect – and indeed, it can’t be because the look-ahead can’t know what the player will input. In our case, we have to reproduce simple physics, as well as the scripted game entity behaviour described above (the circle cast down and the foot spring and all that). Simple physics is done something like this:

position += velocity * deltaTime;

velocity += force * deltaTime / mass;

The force comes from gravity and whatever controller forces we have cooked up. Proper physics people call the above technique ‘Euler integration’, aka the least accurate kind of integration, but for our purposes it’s acceptable.

We also have to attempt to reproduce collision response. This is a bit hairy, but bear in mind that we can still use our physics engine’s collision query, we just can’t simulate – so we can check if a collision will occur in the future, and just cancel the velocity in the direction of the collision normal by using vector projection. This isn’t perfectly identical to what the physics would do for collision response, but in practice it turned out to be okay.

It turned out to be invaluable for tweaking this mini-simulation to draw the line of predicted positions, as well as kind of reverse-breadcrumbs every ten frames or so to show where the player is predicted to be at frame 10, 20, etc. In theory, if you get things perfect, the circles should be stationary except when something unexpected happens (like the player jumps or changes input direction).

Note that the blue bits of the line correspond to the points in the prediction when the player’s ‘foot’ is touching the ground, and the red parts are parts when the player is predicted to be airborne. This turns out to be helpful later on.

Also, note that this prediction isn’t cached in some fancy way – we’re calculating about sixty frames into the future, then throwing it away and redoing it, every frame.

Footfall placement for ‘free’

Now that we’re basically running a mini local version of the player update step to predict the player position, it turns out we get something else of value here. Our player update step involves circle casts downwards to look for a base of support – so in addition to our array of future positions, we can record an array of future support locations. We can use these as potential places to put our character’s feet! Even better, these positions have the following advantages:

  • Already calculated
  • Definitely correspond to where the player is standing and has a sensible place to put their feet – it won’t report a footfall location at the bottom of a narrow gap or something like that.
  • Tends to favour high or upward-facing surfaces, as feet usually would.
  • Comes complete with surface normals, so we can look for more suitable options based on surface orientation.

Visualised, here’s what our footfall predictions end up looking like:

Not actually for free, though

Not so surprisingly, there’s a little bit more work to do here.

Look at that last gif. If we took the next two yellow circles as the timing for our next steps, and the corresponding furry yellow normals as the place to place the foot, we’d be in reasonably good shape, right? But look closely. Unfortunately, this isn’t possible – and worse, it’s impossible for a somewhat unfixable reason. The yellow circles happen at evenly spaced intervals in the future, and sometimes, those intervals happen when they player is airborne. Sometimes we’re properly airborne – like, we’ve run off a cliff or jumped or something – but other times, it’s just a slight hiccup in the shape of the ground we’re running across, so there’s nowhere to place your foot for a bit. So, what do we do?

The way I’ve gone here is to take the yellow circles as a place to start searching in our prediction array. Searching back from that point suggests taking a short step – like how you might take a shuffling step or two if you’re running up to an edge you have to jump off, just so you have your strong foot land just on the edge. Searching forward in the array corresponds to delaying the step. This is more ‘dangerous’ because if we delay a step too long, it can look unrealistic – anyone can take arbitrarily short steps, but as steps get longer they start to look physically impossible. Ultimately I ended up searching backwards about five frames and forwards up to about eight, but these values will vary wildly for different implementations.

I did a lot of experimenting here to get things to look right, and if I tried to recount every detail I’d never finish this article. However, there are two pieces of information I came away with that I can share to save some time for anyone attempting something similar:

  • If you have to shorten or lengthen one step, you shouldn’t have that affect the timing of the steps after it. It seems logical to do this (ie the next step happens three frames early, so subsequent steps will all happen three frames early to keep the timing right), and not doing it will mean a short step tends to get followed by a long step. I think that’s just how bipeds work. I’m no animator, so I can’t fully justify this, but delaying subsequent steps when you take a long step just made things look off to me, in a way that was immediately fixed when I removed that feature.
  • When you’re searching for an ‘ideal’ foot placement, it can be good to have a scoring system, where you award a footstep candidate ‘points’ for things. The system I used awarded points for being close to the ideal timing (the yellow circle), for the normal was facing more upwards, and for being physically higher up. This effectively caused footfalls to favour the tops of rocks and other obstacles. If you character runs up and down smooth slopes, though, you might want to de-emphasise the favouring of high ground.

With that all put together, here’s a visualisation of the strides themselves. The actual control points can be kind of made up. There are a few tweaks I gave mine, the subject of another post perhaps, but it’s mostly just taking a pre-built roughly square shape and shearing/scaling it to fit the vector from the foot origin to the foot target.

So now, we can add actual foot steps! This is where our previously mentioned inverse kinematics setup can finally show itself.

Suddenly it looks like a creature! This was the first time I really thought this might be doable, and I can’t fully describe what a relief that was.

Getting airborne

This is, for a nice change, relatively straightforward. When the player is in flight, I simply positioned their feet on the outside of a circle below their centre of mass, with the angle determined by their direction of velocity. We then squash the circle to make it an ellipse, and that’s it! The maths looks like this:

And the results look like this:

We want to scale and cap the magnitude of the velocity here so that our feet don’t end up in crazy locations when the player is falling very fast, too. There are lots of things that require you to basically mess with numbers ‘to taste’ when you’re doing what is effectively procedural art, and this is one of them.

This brings us to the next logical step, which is transitioning between running and being airborne (and vice versa). This is roughly the current stage of my research. By predicting future footfalls, we can actually get most of the way there fairly easily – detecting being airborne is simply a matter of noticing when the system can’t find any appropriate foot placements for a given foot. If that happens, I simply interpolate from the current foot position to the elliptical airborne foot positions, as described above. In the other direction, as soon as the system does detect a foot placement again, that means we’re about to land, so we start stepping towards that point. So long as we set up our stride spline so its initial velocity matches our character’s flight velocity, the transition is smooth – again, more on that in a different post.

So that brings us to the current ‘state of the art’, as it were – our best bipedal character, running a little obstacle course I put together. Note that this entire animation is the result of just running to the left – we’re not jumping or anything here.

My personal verdict? I think this is getting there. It’s not perfect, but for situations that are basically describable as ‘running along flat-ish ground and occasionally going over a drop off’, I think it’s pretty solid. At this point, while there are a few features still to add, I’m going to come back to tweaking after we have character models in and I can get a good look at a more final version to judge what needs to be done.

Lies, Damn Lies, and Tech Demos

It pays to be careful of articles like this. If you’ve ever had the experience of trying to implement a game-related algorithm based on a whitepaper or similar thing, you’ve probably had the experience of learning that actually there are huge, gaping flaws in the technique that the author conveniently neglected to mention. Even as someone who plays games and follows early press for games with cool new tech in them, you’ve probably felt that sting of noticing that things were a little on the optimistic side when they were presented at E3 (or whatever). With that in mind, there are a few things I should mention about the above techniques.

This isn’t a perfectly complete guide

I’m not confident that many people really read all the way through this kind of blog post, and I’m even less confident that anyone is going to sit down and actually implement this from start to finish. But if I’m wrong about this, and that enthusiastic person is you, be aware: Putting this technique in your game will be an adventure. This post is already way too long, and I’m not a good enough writer to fully elucidate every little detail that went into this experimentation without writing a lot more (which I hopefully eventually will). Getting this right – or at least, as right as it currently is – has been one of the more challenging pieces of development in Sun Shy so far, and I can’t really recommending attempting it if you’re not up for a bit of experimentation.

The algorithm isn’t perfectly complete yet either

There are a few situations that this doesn’t yet handle, which I’ve kind of glossed over. Our character doesn’t handle disappearing or moving foot supports yet (if, for instance, they dig out from under their own feet). Our character sometimes looks a bit weird when they turn around and change direction. We haven’t handled jumping yet. We sometimes get some weird leg shenanigans when the character lands from a high jump, or runs up a steep slope.

I’m about 80% confident that, when all is said and done, and we have a much more tweaked version of this, it will look good about 98% of the time. That last gif is an honest one – that’s how it looks when it runs situations that it has already been designed to handle, and I think it looks pretty good – when we have real character art, I think it will be pretty sweet. However, I don’t want to pretend that the algorithm never does something that looks a bit weird, nor am I certain that all those situations can be stamped out by the final release. I’ll definitely be posting more about this as I keep working on it.

Coming up next…

These last four posts have been so tightly packed because I’m terrible about procrastinating about blogging, and these represent the bulk of the research worth writing about for (my part of) the Creative Victoria project. So the next post probably won’t come in the next couple of days, but I do intend for there to be another one.

I still haven’t talked about how three-section inverse kinematics works, and I glossed over a lot of the details of using splines for stride shape – especially the bits about deciding the shape, and calculating exact velocities to get continuous motion on the feet when you have to interrupt a step. I also didn’t really talk about retargeting steps, because figuring out how to get that to look good is an ongoing process – it’s definitely necessary, though, because the best motion-prediction algorithm still can’t always guess when the player is going to change direction. I’m also going to have to talk a bit about standing still, changing direction, and eventually, dealing with situations like fighting or taking damage. So, file this under ‘ongoing research’. But it’s been a lot of fun to work on, and I’m mostly optimistic about the tasks ahead.

Graphics, Programming, Sprite Lamp

Multiple sprite sheets, Sprite Lamp, and Unity

So I mentioned yesterday that I’ve been working on a small tech demo in Unity, to help make sure I notice all the difficulties that can come up with Sprite Lamp, and the first one that has presented itself is the issue of animating with multiple (sets of) sprite sheets. A few people asked me about this already but I didn’t quite have my head in the game enough to give good answers. Hopefully now that I’ve played with it a bit more directly, I can do better.

Here’s the script referenced in this post.

Oh and before I go on, a quick annoying reminder that Sprite Lamp is at 30% off on Steam as I write this!

Animations with a single sprite sheet

The simple situation for frame animation in Unity is one big sprite sheet. This is, I suspect, reasonably common – especially with people working with pixel art, and textures going up to 4096×4096. Someone in this situation who wants to use Sprite Lamp is in a pretty easy situation in terms of programming – essentially, everything just works easily. You’ll want to make a sprite sheet of the diffuse channel, cut it up with Unity’s sprite editor, create animations, and apply them to a game object with an animation controller. So far, everything is normal. Then, you’ll want to create a corresponding normal map sprite sheet with Sprite Lamp (either using whatever texture packer tool you want, though be wary of rotating normal maps, or by drawing the lighting profiles in sprite sheet positions then processing them all at once). You don’t need to cut this up into sprites in Unity after you’ve imported it. Then, apply a Sprite Lamp (or other) material to your game object, drag the normal map sprite sheet into the NormalDepth slot, and everything should be fine.

The reason this works smoothly is because if you have one big sprite sheet, all the animation system is changing is the UVs on the sprite, and because the Sprite Lamp shader doesn’t do anything fancy with UVs, it’ll just look up into each sheet in the same spot.

Multiple sprite sheets

This is where things get tricky. The issue is that Unity’s animation controller automatically switches textures as necessary, but it only provides for switching the main texture, because usually you only have one texture when you’re doing this kind of animation. By way of example, say you have a character who can walk or run. If you’re making a normal 2D game without lighting, you might have two textures: WalkSheet, and RunSheet. Ordinarily, when your character is walking around, Unity will be switching between sprites in one sheet, and thus changing the UVs – but when your character switches to running, Unity has to switch to the RunSheet texture, which it does automatically.

Suppose instead though, you are using Sprite Lamp, so you have six textures. WalkSheet, WalkSheet_Normal, WalkSheet_Emissive, RunSheet, RunSheet_Normal, and RunSheet_Emissive. Your character starts walking around fine, with WalkSheet, WalkSheet_Normal, and WalkSheet_Emissive applied – all is well. Then they break into a run though, and we’re screwed. The animation system switches the diffuse texture to RunSheet, but the normal and emissive channels keep the Walk versions. This will look somewhere between pretty broken and horribly broken, depending on whether or not the walk and run sheets have similar layouts.

How to get around this? Well, essentially, I’ve written this script. It’s still not tested all that much, so I haven’t added it to the official Unity pack yet, but it has worked in my use case and will hopefully work in yours (if not, let me know). The script works by creating a dictionary at load time, which uses a texture as its key, to look up the corresponding textures. Then in the update, it will check the object it’s attached to to see if the texture it’s using has changed – if it has, it automatically sets all the other textures.

To use it, the first thing to do is attach it to your character (or whatever). Second, you have to populate some lists – this is currently done manually, but I’m looking into nicer ways to handle this.  The lists are ‘primaryTextures’ and ‘otherTextures’. Into ‘primaryTextures’, drag all the sprite sheets that your AnimationController knows about – so in our above example, that would be WalkSheet and RunSheet. In the ‘otherTextures’ list, you’ll want to drag all the auxiliary stuff Sprite Lamp uses – here, that would be ‘WalkSheet_Normal’, ‘WalkSheet_Emissive’, ‘RunSheet_Normal’, and ‘RunSheet_Emissive’. For each element of ‘primaryTextures’, the script will search for any textures in ‘otherTextures’ that are suitably named, and associate them so that they get set at runtime.

That should be just about it! As always with Unity things, if I’ve missed some obvious smarter way to do this, please let me know. Also, I’ll be along soon with a more concrete example of this method that you can download, in case my description here doesn’t make things clear.

Coming up next…

The other problem I rapidly ran into here is the problem of ‘light sources’ that exist in the emissive texture map. They tend to move around within the sprite, and you’ll likely want an actual light source to move with it, but the engine doesn’t know how to move it appropriately. I’m working on a script that will automatically parse an emissive map and approximate clusters of bright pixels by creating a light of the appropriate intensity/colour/position. It’s at proof-of-concept-ish stage now and isn’t ready to distribute just yet, but so far it looks pretty cool, particularly by matching the light’s movement to the animation’s framerate.

Graphics, Sprite Lamp

Unity shader update/info (minor)

This isn’t quite the update I wanted to make, but it’s something. I’ve just uploaded a new example project and Unity package stuff to the engine integration page.

Working with Unity’s shader system has been a big learning experience, and I’m slowly coming to understand how it all works (and how my initial attempts have failed to do things The Unity Way, as it were). I’m in the process of reworking the shaders to make use of the (startlingly deep) Unity shader macros, and when that’s done I’ll post another update, and it will cause attenuation and light cookies to all work in the nice simple way you’d expect Unity stuff to work. In the meantime, this smaller release fixes the problem a few people have reported involving weird specular lighting appearing where the alpha channel is set to zero and should thus not be visible (this turned out to be a schrödinbug relating to multipass lighting and blend modes), and adds the experimental new implementation of per texel lighting (both the palette and regular shaders have new versions with this feature).

I should also mention something that came up on the forums, which is an unfortunate development regarding performance of Unity on mobile platforms. It seems that Unity2D relies heavily on dynamic batching to keep its performance up to acceptable levels on mobile. However, dynamic batching gets broken by multipass lighting, which is how Unity handles all per-pixel lighting as far as I can see, and so this is causing some performance issues. I currently don’t have a good solution to this – for a while it looked like this would be solvable with a hackish application of Unity’s per-vertex lights, and making use of them in a special fragment shader, but I’m not really comfortable claiming an official solution to a problem that involves such unofficial methods (and I definitely shouldn’t promise they’re going to work on all present and future platforms Unity supports!). One possible solution is static batching, but that’s a Unity Pro feature. Another possible solution is a script that automatically groups tiles together into bigger tiles, which is essentially a custom static batching solution – I’d happily write such a script and distribute it if it would help people out. I’ll keep looking for solutions. In the meantime, you need not buy Sprite Lamp to test such performance issues on your game/hardware – any normal map will do the job.

Graphics, Programming, Sprite Lamp

Unity Palette Shader first pass

2 Comments

Hi all,

You know how sometimes, after you’ve been working on some important thing, you get sick after you finish it? Like your body was holding off on getting sick until it could get away with it? Maybe it’s just me, but either way, I got sick right after putting the new version of Sprite Lamp out there. Anyway, sorry it took me a few days, but here’s my first crack at the palette shader from Sprite Lamp, in Unity form.

ZombiePaletteUnity
Come for the palette zombie, stay for the flawless framing of the screengrab.

 

A couple of important things to note:

  • It basically works as you’d expect, I think. Put the textures from Sprite Lamp in the appropriate texture slots on the material and you should be good.
  • Currently it makes use of the diffuse map, just to get the opacity value from it. The obvious thing to do is put the opacity in the index map’s alpha channel, but since Sprite Lamp doesn’t output them that way automatically yet, I made the shader so it works readily with what Sprite Lamp exports.
  • You can get some very crappy results if you don’t load your textures in as ‘uncompressed’. Compressing the palette map in particular will pretty much make everything awful (or at least, that’s what happened when I tested it).
  • The palette system is designed to work without coloured lighting. I might hack something in later that just multiplies the output by the light colour, but that kind of misses the point of having close artist control over what colours are rendered to the screen. Ultimately, if you’re making use of the palette system, the key is to set the lighting mood via the palettes.
  • Since this is a straightforward shader-based implementation, it makes use of simple additive lighting to handle multiple lights. Technically, this isn’t quite correct, but I think for most cases it should look fine. I hope to work on a more complete and correct approach to this in the future, but it might get a little hairy (might require something resembling a deferred rendering pass). I’ll go into some detail later as to how this might work.

Anyway, that’s it! As you can probably guess, this shader is a work in progress, and will change in the future – both in terms of how it works under the hood and how you use it as a developer. Let me know if it’s giving you trouble, failing to compile, or if it’s not clear how to make use of it, and I’ll try to straighten things out.

Graphics, Sprite Lamp

Sprite Lamp’s palette system

Yesterday I released the latest update to Sprite Lamp, and with it, a properly implemented and documented palette system. Still pending are shaders for various engines to make it usable in your games, but in the mean time, I’m going to give a quick rundown of what it is, how it works, and how the (really pretty simple) shader works.

Why palettes?

This is something that came up due to repeated requests from artists. As a graphics programmer, my first thought was to calculate the light based on the angles of the light and the surface normal using standard Lambertian and basically call it a day. However, as several artists pointed out to me, this means that when a part of an image gets darker it approaches black, and artists very rarely use real black in their paintings. Shadows are more likely to have some blue to them, and highlights are likely to be slightly yellow rather than white. There are various ways to get around this – tweaking the ambient and direct light colour, for instance – and while those options remain, I thought it would be good to give artists more direct control. Hence the palette system.

This works by taking the diffuse map (that the artist creates), taking all the unique colours from it, and creating a sort of template palette image. This palette is then saved out and modified by the artist to get the effects they want. A palette image has a vertical column of pixels for every unique colour in the depth map. The vertical position in this column represents the colours that those pixels will be at different lighting levels. The midpoint of each column is the colour when that pixel is lit with full diffuse lighting but no specular lighting. Below that it fades as the diffuse lighting drops away to nothing, and above that it represents the colour when an increasingly strong specular highlight is added. Sprite Lamp can be used to generate either an ’empty’ palette that will create flat lighting (the columns are the same colour all the way up), or a palette that will produce simple calculated lighting, fading to black at the bottom and white at the top.

As an example, here is the diffuse channel of the Sprite Lamp zombie:

Diffuse image of a zombie

And here are the autogenerated palettes for said zombie (with and without default lighting built in):

EmptyAndDefaultPalettes

From here, you save one of these images out (whichever would work best for you as a guide) and then change the colours as you see fit. The only really important thing is that you generate the index map (more on that later) and then make sure you keep the horizontal positions of the pixel columns in the same place/order in the palette.

Here are some examples of effects that are possible using this system, with the zombie’s palette image visible on the inset.

ZombiePalette2
This is a simple case of adding some blue to the shadows and some yellow to the highlights to add some depth to the result.

ZombiePalette3
In this image, we’ve done something like a traditional palette swap, to make a different colour scheme entirely.

ZombiePalette4
This one makes use of the Dawnbringer palette – it uses fewer colours and gives a more retro look.

The Shader

So, as promised, I’m going to give a quick outline of how the shader works. It’s actually not very complicated, but it hinges around something that Sprite Lamp will generate for you, called an index map. You’ll need a normal map, as usual, and then a diffuse map, and from the diffuse map Sprite Lamp will generate the palette map template (which you then modify) and the index map (which you don’t). The index map for the zombie looks like this:

zombie_Index_Bigger

The grey values in this image are actually a value from 0-1, representing the horizontal position in the palette map where that column of colours resides. Black represents the leftmost column of the palette map, then the second darkest grey (the hair) will be the second column across, and so forth. This is why it’s important to keep the columns of the palette map in the same place.

As the shader programmer, all you need to do is calculate some lighting value between zero and one (the way Sprite Lamp’s palette shader does this is by averaging the diffuse and specular components and then clamping to the correct range). Then, a look up into the index map will get you a luminosity value between zero and one. From there, you do a look up into the palette map, using the value read from the index map as your U coordinate, and the calculated level of illumination from your lighting algorithm of choice for the V coordinate, and you have your final colour. Because the whole point of the palette system is to give the artist precise control over the colours that end up on the screen, nothing more is done – the colour is outputted onto the screen. The pseudocode for the shader might look something like this:

float colourLevel = (diffuseLevel + specularLevel) * 0.5;
float indexPosition = tex2D(indexMap, textureCoords.uv).r;
//Note that we use 1.0 - colourLevel because usually positive V
//goes down the texture map.
vec3 finalColour = 
         tex2D(paletteMap, vec2(indexPosition, 1.0 - colourLevel);

The generation of the index map

For the most part this isn’t something you’ll be worried about, but there are certain circumstances where it’s important to know the details. Sprite Lamp generates a palette map from the diffuse map to use as a template, but then it generates the index map from the diffuse and the palette maps. It does this by going through every pixel of the diffuse map, and then searching for that pixel’s colour in the middle row of the palette map (that is, it looks through the row of pixels half way down the palette map). When it finds the colour it’s looking for, it will use the horizontal position of that to determine the greyness of that pixel in the index map.

This probably isn’t terribly relevant to most use cases, but it’s possible that you will want to use a palette map for more than one piece of art in your game (to render a whole scene coherently, for instance). In that case, you’ll want to make your own palette map from scratch, making sure the important colours are present all the way along the centre. Having done that, paint all your diffuse maps using only colours from that centre row of pixels. Then, load up a given diffuse map, and rather than generating a palette from it, open up the palette you made into the palette map, then click the ‘generate index’ button. If the diffuse colours are all present in the palette’s central row, Sprite Lamp should generate a fully functional index map without issue – it doesn’t matter that there are colours in the palette not present in the diffuse.

Conclusion

I would imagine that this problem has been tackled by other developers at various points through gaming history – this approach is just what came to mind to answer artists’ complains about black shadows. Given that, this explanation is straight from my brain to the page, as it were. Please, let me know if there’s anything I’ve been unclear on, so I can fix my explanation.

More importantly, if you do anything cool and unexpected with this system, I’d be keen to see your work. I’ve already been surprised/impressed with Halley’s implementation of the Drawbringer palette from above, and I’m confident that various cel shading and other techniques are possible using this system too.

Graphics, Programming, Sprite Lamp

Per-texel lighting

6 Comments

So, I’ve been pretty negligent about documenting this one last feature in the Sprite Lamp shader. It’s rather obscure and difficult to explain, but I’m going to try to overcome that with pictures.

Pixels, texels, and fragments

These three things can get pretty confusing during the coming article, so I wanted to clear a few things up.

You’ve probably heard of pixel and fragment shaders, and observed that they’re basically the same thing. Well, they are. Technically, I believe, the correct term is ‘fragment shader’. A pixel (‘picture element’) is a single tiny rectangle of colour on a computer screen, whereas a fragment is also depth information and a few other things. A fragment might never become a pixel, because it might fail a depth test or whatever. The wikipedia page on fragments is a bit helpful. Either way, it’s good to know the difference, but this isn’t the most important distinction.

The distinction between a pixel and a texel is more important, especially if you’re a pixel artist. In a sense, I suppose, you could be described as a texel artist – texel means ‘texture element’ – basically, it’s one of the coloured rectangles that make up a texture. This is relevant because sometimes you draw a texture to the screen at a size other than its native size – in fact, in 3D games this is almost always what happens. That’s what filtering is for – bilinear filtering, for example, is a maxification filter – a way of zooming in on a texture – without it appearing blocky.

Per texel lighting

This is probably nothing new to artists who are used to working with computer graphics. The point is, when you’re viewing pixel art such that it is deliberately blocky (using nearest neighbour filtering (aka point filtering)), each texel is made up of many pixels. That is, each pixel in the source art is blown out to be many pixels on the monitor. This is the important thing to understand, because when a fragment shader operates on this artwork, it’s doing a lighting calculation once per pixel, NOT once per texel.

The result of this is that, even with the diffuse/normal/everything maps set to nearest neighbour filtering, you end up with colour variation within a texel. Almost always, this is imperceptible and doesn’t matter. However, when combined with cel shading, you can run into trouble. Here’s an example of what I’m talking about:

TexelVersusPixelBecause I have been staring at this kind of thing for a while now, it’s hard for me to tell how obvious the difference between these two images is. If it’s not immediately apparent to you, the point is that the image on the right has diagonal lines – discontinuities in the lighting level – running across texels. The image on the left could just be pixel art with static lighting – the image on the right definitely isn’t. This problem becomes less clear if the lighting is changing rapidly, but more clear if the light source is moving, but slowly. To me, and I daresay to various pixel purists out there, this is a problem worth fixing.

As you can see from the screenshot above, this has been solved in the Sprite Lamp shader. However, Sprite Lamp’s preview window makes this problem easier to solve than it is in the general case. I’m still working out the details of solving the general case, although I’ll document them carefully in the Unity shader so it can be reproduced elsewhere.

Until then, though, I have a question for anyone reading – would this feature be important to you? As mentioned, to me the visual problems are obvious, but I can understand that for many they wouldn’t be. Of course, it’s not relevant for high res art, and it’s mostly not relevant for low res art either unless there’s cel shading involved. Even then, arguably the problem being solved is fairly subtle. I’m curious to hear your thoughts.

 

Graphics, Programming, Sprite Lamp

Engine integration so far

8 Comments

Hi folks! I’ve been spending some time lately investigating a few of the most commonly requested engines for Sprite Lamp integration – namely Unity, Game Maker, and Construct2. The results so far have been pretty positive – I’m hoping to get at least some support for these three engines made public before I head to GDC. With this article, hopefully people who have been wondering about these three engines will come away with a better idea of what will be possible.

Unity

So, Unity takes the lead as most commonly requested (even though I think mostly people assume it goes without saying). Fortunately, despite it being used for 2D games very often, it is a 3D engine, and that means it has a lighting engine, which means this will be both a smooth integration and a feature-complete one.

As some may have seen, this article by Steve Karolewics of Indreams Studios details an initial attempt at getting Sprite Lamp lighting working in Unity. Steve (along with everyone else who has ever used Unity for anything) knows more about Unity than I do, and I’ve taken his work as a base for the full shader (currently a work in progress). So far, I’ve been working on the hardest part – self-shadowing. Here’s progress so far (hint: it’s working).

The light on the left is closer to the stone wall than the one on the right, which affects shadow length and attenuation.
The light on the left is closer to the stone wall than the one on the right, which affects shadow length and attenuation.

The big part that remains to be done here is that the Sprite Lamp shader has to know the resolution of the textures (for per-texel lighting and for shadowing). At present (for the above screenshot) it’s just hardcoded, but obviously that’s not a long term solution – I’m going to write a little script that runs at load time that sets up the relevant shader variables automatically, so the user doesn’t have to worry about it.

The important thing to note with the Unity integration is that, as mentioned, it already has a lighting system. Light entities already exist in the game, and they will work with Sprite Lamp as you’d expect. Multiple lights work as expected, and Unity will automatically handle stuff like calculating which lights affect a given object. You can also place lights in 3D space (that is, vary their depth position) and have everything work as expected, because as mentioned, Unity is at its heart a 3D engine.

Basically, everything that can be done in the Sprite Lamp preview window is going to be achievable in Unity. Other features that I will most likely add at some point will be support for tessellation (making use of the depth map) which might be cool for the sake of parallax or 3D, and (pending further investigation) support for the deferred rendering pipeline.

Game Maker

I often say that I don’t have a clue about Unity, and that’s pretty much true, but I have at least used it before. That is not the case with Game Maker – I literally opened it for the first time the other day to investigate stuff. It’s therefore possible that I’m missing obvious things – if so, please let me know!

That said, the results of this investigation have been positive. The short answer is, lighting is go:
GameMakerSL

This might not look like much, and it’s not as far along as the Unity integration, but the important thing is that I can load my own custom shader – from here on in, being able to get cel shading and self shadowing working is kind of just details.

At present, there’s some boilerplate scripts that go along with this too – one that you’ll attach to the Create event of your dynamically lit objects and one that goes with the Draw event – which sets appropriate shader variables and that kind of stuff. My understanding is that Game Maker is actually usable without any custom scripts at all, so for the benefit of users who aren’t scripters, I’ll keep the interface with that script as small as possible and document thoroughly any interactions you have to do with it. They’ll almost certainly be very simple – stuff like setting light levels, etc.

That brings me to the next part of this. Unlike Unity, Game Maker doesn’t have a lighting engine built in – this means that there aren’t, for instance, native entity types for stuff like light sources. At this point, I don’t understand the engine well enough to know what the best solution to this is, but it should at least be possible to have a dummy object that acts as a light source, and then tell the scripts on the lit objects to update their shader variables based on the position of the dummy object.

Slightly more complicated is the question of multiple light sources. At the very least, it is simple to have an hemispheric ambient light and a single point light at any given time. I can also add a few more lights by just making the shader more complex (basically, have multiple shader variables for multiple light positions, and add the results together in the shader). However, having an arbitrary number of light sources becomes more complicated – this will involve building a system that automatically tracks lights and what objects they’re close enough to have an effect on. For the moment, my first release of the integration with Game Maker will be the shader and code snippets necessary to get basic lighting going. I’d rather get a basic implementation going for multiple engines, then return to more in depth work like that later if it’s in high demand.

Construct2

This is the least advanced and most recent investigation I’ve done, but I know a few important things so far, and I wanted to pass them on. I’ve also been in touch with one of the developers of the engine, which helped a lot.

Basically, the way Construct2 works doesn’t naturally lend itself to this kind of thing, because of course a 2D engine designed for ease of use isn’t designed with 3D lighting in mind. But, it can be done, with a few caveats.

The rendering engine of Construct2 works, simply enough, by drawing sprites. In addition to regular sprites, you can draw effect sprites – these have shaders attached to them, and are often used for things like colour shifts and distortion effects. They also have a texture attached to them, and can sample the colour of what they’re drawn over. We can leverage this to make some lighting effects.

My first attempt is based on a shader by a developer named Pode. Here goes:
ConstructLighting

So far so good. The folks from Scirra tell me there’s a built in bump map shader that would be a good basis for me to work from too, which works in a slightly different way (the above example multiplies the effect result with the underlying colour, but you can also directly sample the underlying colour and render the result over the top).

From here, given access to the shader, I can get a lot of Sprite Lamp’s lighting effects going. The things I said in the last two paragraphs on Game Maker apply to Construct2 as well, regarding multiple lights and dummy objects and whatnot.

However, there’s another issue with Construct2 that is a limiting factor – namely, an effect sprite can only have one texture attached to it. Now, the Sprite Lamp shader makes use of a lot of textures, some more necessary than others. The diffuse texture is taken care of – the diffuse channel will be contained in the layer underneath, rendered before the effect sprite. Obviously normal maps are required. If shadows are required, the depth map can probably be squished into the alpha channel of the normal map, except insofar as you will also need that alpha channel for opacity (since it’s a separate sprite) – possibly this can be worked around, however. Emissive maps can simply be rendered as another additive layer for objects that have them – you won’t even need a special shader for that. However, ambient occlusion and specularity/gloss maps might have to be sacrificed.

There is a possible workaround here that involves putting multiple textures into one (that is, literally just the diffuse and normal maps next to each other in one image, for instance) and then extracting them with uv maths in the shader. If this turns out to be worthwhile, I’ll add an option to Sprite Lamp to automate exporting maps in this form. However, I foresee visual issues if this is used on tiling textures if your game has a camera that zooms in and out (bilinear filtering and mipmaps will not be kind in this situation).

The other possible issue is performance. Shader complexity not withstanding, this system requires you to draw at least two sprites for every game entity that needs dynamic lighting. I don’t have much of a feel for how performance gets limited in Construct2, but as an educated guess, if your game already has (or almost has) performance issues relating to the number of sprites on screen at a time, they will probably get worse if you add dynamic lighting to them all – my guess is that this will be a load on the CPU and the GPU.

One last thing to note about Construct2 that the devs warned me about is the renderer. Everything I’ve said here is completely dependent on shader effects, which require a WebGL renderer. Unfortunately, this means that environments that use canvas2d instead won’t have access to any dynamic lighting effects (notably, Internet Explorer and Safari, apparently). Sorry about that – that’s out of my (and Scirra’s) hands.

Other engines

If you’re reading all this wondering when I’ll get to your engine – don’t worry, this isn’t a complete list. It just made the most sense to approach the most commonly requested engines first, but a minimal implementation doesn’t take a terribly long time (assuming it’s possible) for a given engine. Even if it’s an obscure engine that I’m not going to do an official engine for, I’m entirely happy to help you out with getting it up and running yourself. Feel free to give a shout regarding the engine you want looked at (besides these three) in the comments.

Graphics, Programming, Sprite Lamp

Sprite Lamp’s basic shader

So, now that I’m back from Christmas/New Year’s festivities, the first order of the day is to do a proper write up of the basic shader from Sprite Lamp’s preview window. I thought it would be good to get this out there early on, because people who are keen to get cracking with a version of it for an engine of their choice can do so. I’ll eventually be doing an implementation for major engines, but of course I can’t cover all engines myself, and I’m betting some people won’t want to wait for me anyway. With the exception of the self-shadowing, none of this is very complicated – if you’re advanced enough to get a normal mapping shader going, this should be well within reach.

Someone suggested that I make a page to collect links to people who have implemented these shaders in various game engines, and I think that’s a pretty great idea, so if you’re working on such an implementation, get in touch! I’ll be putting a page up here soon with that information, and eventually I’ll add my own integrations there when they get done.

First off, if you backed Sprite Lamp (which you can still do via PayPal, by the way), you already have the shaders in plain text. They might have some dodgy commented out code, and they’re not well documented, but they’re there in the ‘Shaders’ directory. The one I’m going to talk about for the moment is StandardPhong.fsd. As the name suggests, fundamentally this is based on the Phong illumination model (not to be confused with Phong shading).

I won’t talk too much about Phong illumination because it’s pretty common and well-documented elsewhere (including at that wikipedia link). Fundamentally, it consists of ambient lighting (pretty much just a constant value), diffuse lighting (which is calculated as the dot product of the light direction and the surface normal), and specular lighting (which is slightly too complicated to sum up in a sentence, but there’s more detail at the wikipedia page). I’m going to go over the modifications I’ve made in the Sprite Lamp version.

Cel shading

This is a fairly simple trick – the more complex palette-driven cel shading will be a subject of another blog, once the tech is finalised. Simply put, this applies a step pattern to the illumination level of the pixel. The relevant piece of code is this:

diffuseLevel *= cellShadingLevel;
diffuseLevel = floor(diffuseLevel);
diffuseLevel /= cellShadingLevel - 0.5;

By this point in the shader, ‘diffuseLevel’ should have been calculated to contain a value between zero and one representing the diffuse illumination level. ‘cellShadingLevel’ contains an integer value representing the number of levels of illumination (at least two). After this calculation, ‘diffuseLevel’ should be multiplied with the value in the diffuse map. Note that other things that affect the diffuse level, such as shadows (discussed below) or attenuation, should be calculated before this stage.

CelLevels

Wraparound lighting

I’m sure I’m not the first person to think of this feature, because it’s very simple, but I haven’t heard it given a name (perhaps for the same reason). Normal dot lighting is of this form:

 float diffuseLevel= clamp(dot(normal, lightVec), 0.0, 1.0);

In this scenario, the surface normal and the unit vector pointing in the direction of the light source are dotted together. This gives a result of 1.0 if the surface is facing directly at the light source, and a result of 0.0 if the surface normal is at right angles to the light rays. However, it also gives a result of -1.0 if the surface is facing directly away from the light source. Physically speaking, surfaces facing away from the light source should all be equally dark, and as they start facing the light source more and more, they become lighter – this is why the value is clamped to be between zero and one.

I don’t have any software handy for generating pretty graphs, but to visualise this, draw a graph of ‘y = cos (x)’ with x between 0 and 180 (degrees). If X is the angle between the surface normal and the light ray, Y is the light level. You’ll notice that from x=90 all the way up to x=180, the lighting value is negative (and thus gets clamped to zero).

However, if you don’t care about your lighting being a bit physically unsound, you can make use of the entire range of angles between 0 and 180 degrees. This makes the lighting wrap around the sides of the object, which can be useful for faking larger light sources (such as the sky while the sun is behind a cloud). For Sprite Lamp, I’ve created a special value called ‘lightWrap’ that varies between zero and one. Zero means normal diffuse lighting, and one means that every surface gets at least some light unless it is facing in the exact opposite direction to the light source. To visualise that second scenario, draw a graph of ‘y = (cos (x) + 1) / 2’. The implementation of this in the Sprite Lamp shader looks like this:

float diffuseLevel = 
     clamp(dot(normal, lightVec) + lightWrap, 0.0, lightWrap + 1.0)
            / (lightWrap + 1.0);

I find that light wrapping can look pretty weird if it gets higher than about 0.5, but of course the easiest way to play with the value is with the slider labelled ‘Wrap-around lighting’ in the Sprite Lamp preview window.

LightWrapping

Hemispheric ambience

This is a ludicrously simple technique that can get you much better value out of your ambient lighting levels. Perhaps everyone out there in industry is doing this already, but for those unfamiliar, I’ll go through this quickly. Instead of specifying a single ambient light colour, with hemispheric ambience you specify two – an above light colour and a below light colour. Usually the above light colour would be a bit lighter, and perhaps a slightly different colour, but it depends on the environment. Then, you simply mix between them based on the y component of your world space normal. In Sprite Lamp’s shader code it looks like this:

float upFactor = normal.y * 0.5 + 0.5;
vec3 ambientResult = ambientLightColour * upFactor + 
                     ambientLightColour2 * (1.0 - upFactor);

In this scenario ‘ambientLightcolour’ is the above light, and ‘ambientLightColour2’ is the below light colour. ‘upFactor’ is basically the extent to which the normal is facing up. Note that if you aren’t rotating the object in question, ‘upFactor’ can simply be replaced with the green channel of your normal map, which makes things easier still.

Incidentally, if you’re also making use of ambient occlusion maps, you should get the colour of the AO map, mix it with some amount of white (depending on how intense you want the effect), and multiply the result with ‘ambientResult’ for the final ambient light colour.

Self-shadowing

This is the only effect here that is at all hard on the graphics card, because there’s a loop and a lot of texture lookups involved. It’s also mildly hacky in the way it softens the shadows, and in the current version of the shader included with Sprite Lamp, it includes a few magic numbers. Hopefully I can explain it in a way that makes sense.

float thisHeight = fragPos.z;
vec3 tapPos = vec3(centredTexCoords, fragPos.z + 0.01);
vec3 moveVec = lightVec.xyz * vec3(1.0, -1.0, 1.0) * 0.006;
moveVec.xy *= rotationMatrix;
moveVec.x *= textureResolution.y / textureResolution.x;
for (int i = 0; i < 20; i++)
{
   tapPos += moveVec;
   float tapDepth = 
             texture2D(depthMap, tapPos.xy).x * amplifyDepth;
   if (tapDepth > tapPos.z)
   {
      shadowMult -= 0.125;
   }
}
shadowMult = clamp(shadowMult, 0.0, 1.0);

Essentially what this is doing is tracing a dotted line from the fragment position towards the light source, and for each dot, checking if that point is inside the object (that is, beneath the depth map). Traditionally, if any point is inside the depth map that would mean the ray has hit an object and thus this fragment is in shadow. However, to fake soft shadows, I darken the fragment more as more points are inside the objects. This is completely nonphysical, but I found that it looks pretty alright. At the end, you have a value called ‘shadowMult’ that you multiply against other lighting values to darken them appropriately.

I’ll try to break this up so it makes a bit more sense. Note that fragPos has already been initialised to have value in it – it represents the world position of this fragment. The x and y values here are calculated from the actual position, but the z value is taken from the depth map.

We initialise the value ‘thisHeight’ to the z value of fragPos. ‘thisHeight’ is going to record the position of the current tap (dots along the dotted line) that we’re up to. ‘tapPos’ refers to the position in the texture that we’re looking at, and ‘moveVec’ is the vector that we increment ‘tapPos’ by to get to the next dot in the line. We need to rotate the x and y values of ‘moveVec’, and multiply the y value by the texel aspect ratio too. Finally, I’ll note that ‘moveVec’ is multiplied by a magic number, in this case 0.006, to determine the step size of the ray trace. Setting this to a higher value will make it possible to cast longer shadows, but at the cost of sometimes missing shadows cast by narrower obstacles.

We then go through the loop an arbitrary number of times (in this case, 20). The more iterations, the more expensive the shader, but also the longer the cast shadows can be.

Each step through the loop starts by moving ‘tapPos’ along the ray by adding ‘moveVec’ to it. We then obtain a value, ‘tapDepth’, by looking up into the depth texture at this position. Finally, we compare ‘tapDepth’ against the z component of ‘tapPos’ – this is checking whether this tap is inside the object or not. If it is, we subtract a value from ‘shadowMult’. The value that gets subtracted is another magic number – in this case 0.125. This number controls the softness of the shadows. At 0.125, it takes 8 or more taps inside the object to completely darken a fragment. If we set this value to 1.0, it would instead make the shadows completely hard (black or white). Lowering this value would make the shadows fuzzier still. At some point I’ll make these magic numbers tweakable from the Sprite Lamp UI.

Conclusion

Hopefully this will serve as a launching point for some of the more enthusiastic and experienced Sprite Lamp users who are wanting to implement the Sprite Lamp shaders in the engine they’re using. I’m not quite satisfied that the self-shadowing system here makes a lot of sense – revisiting it now has reminded me that it was somewhat hacked together when I first made it, and I never got around to cleaning it up. However, it’s hard to know which parts of an article like this need more detail and which parts don’t, so please feel free to get in touch and ask for clarification, and I’ll do my best to update the article.

Graphics, Programming, Sprite Lamp

Tech demo: Complete hand drawn scene with shadows

3 Comments

Today I’m going to talk a bit about a tech demo I was working on recently. It’s Sprite Lamp-related (at time of writing, Sprite Lamp has three days left to reach its next stretch goal, by the way). This is part of a series of blog posts about more advanced techniques that fall into the category of “hand drawing and dynamic lighting” – some make heavy use of Sprite Lamp, others are merely suggestions for things to use in conjunction with Sprite Lamp. Just to be clear, the following examples make use of Sprite Lamp, but they are not possible using only Sprite Lamp. This is an example of something you can achieve with a dedicated graphics programmer on board. So yeah, basically, don’t support/buy Sprite Lamp because you want this effect in your game, unless you’re willing to do a lot of the programming tasks described here.

With that out of the way, the project was Project Lonsdale, and it was an attempt to create a full hand drawn scene with full dynamic lighting and shadowing. The plan was to put it to use in a point-and-click adventure game, and perhaps one day that’s what will happen. I’m showing this off now as a demonstration of some of the deeper uses of Sprite Lamp – this isn’t currently planned as a product launch, it was just a thing I played around with a few months ago.

[youtube=http://youtu.be/uV0wGBylkXs]

So there you have it. Explaining how this was all done will take ages and I don’t want to get too distracted from working on Sprite Lamp, so I’m going to do it in parts. Today I’m going to give an overview of what’s involved in creating the scene, both in terms of art and in terms of processing. Next time, I’ll give an overview of what goes into rendering a frame. Then over time, I’ll write more posts that flesh out bits of these overviews. Hopefully in the end, I’ll have a detailed technical write up.

Note that this tech is currently abandoned (though I may one day pick it up again), and that’s largely because the pipeline was just too involved for Halley and me to realistically create a whole game with. It’s really convoluted, seriously. Fully developing Sprite Lamp has made me realise a few ways in which it could be improved – what I’m describing here is the process, as far as we got.

Creation of a scene

I mentioned in passing that this involves not just Sprite Lamp, but also a bit of 3D modelling and quite a lot of programming. Here’s a quick rundown of what was involved.

1. Make a 3D model of the scene. This was actually a really easy step, because the 3D model didn’t have to be super sophisticated. In fact, for the first scene in the video, I (Finn, the programmer, and not a skilled artist at all) did the modelling in about a half hour. We really just couldn’t think of a good way to do proper shadow casting without some kind of mesh. Alas, the dream of not having to use modelling software at all dies. Here is what our mesh looked like at that point:

Viewed from a completely different angle as the rest of the scenes, of course.
Viewed from a completely different angle as the rest of the scenes, of course.

2. Load the mesh into the Lonsdale tool, and manually position the camera where you want it. This is for framing the image. Then, you export a bunch of stuff. Lonsdale at this point rendered out a 16-bit depth map, as well as a very basic preview render to be used as a guide for the artist.

The red channel is the first eight bits of a sixteen bit integer representing depth - the green channel is the other eight bits.
The red channel is the first eight bits of a sixteen bit integer representing depth – the green channel is the other eight bits.

3. Now it’s the artist’s turn. Halley would draw a whole bunch of stuff at this point. She’d draw the lighting images, as per Sprite Lamp’s requirements – the whole scene, approximately matched up to the render of the mesh, lit from all directions. Details that don’t need to cast shadows (such as textures on a surface, embossed lettering, etc) get added here. These would go into (the early version of the thing that eventually became) Sprite Lamp to create a normal map. Meanwhile, she would also paint a diffuse and a specularity map. Finally, she would draw something called a silhouette map. This is important, because it is simply a greyscale image where distinct edges define the outlines of the objects she’s drawn in the scene. There is always going to be a disparity between the rendered depth map and the hand-drawn everything else maps – the silhouette map helps resolve this disparity. More on that later.

The normal map of the scene - the most important part, probably.
The normal map of the scene – the most important part, probably.

Different shades represent different objects - designed to be easy to do an edge detection on.
Different shades represent different objects – designed to be easy to do an edge detection on.

As though the scene was lit perfectly evenly.
As though the scene was lit perfectly evenly.

4. And, back to the Lonsdale tool. This would do an edge detection on the silhouette map to figure out the shapes of Halley’s hand-drawn objects, and then jigger around with the rendered depth map to create a ‘fixed’ depth map. It would then combine the fixed depth map (red and green channels) with the silhouette map (blue channel), because the silhouette map is used in the shadowing algorithm. Now we have the depth/silhouette map, the normal map, the diffuse map, and the specularity/glossiness map. Not to mention the mesh. Everything we need to start rendering.

The final result.
The final result.

Coming up next time… rendering the scene

This is almost as involved as preparing the assets, but it does contain a cool screen-space shadow blur that I think might have some applications in more traditional rendering, too. Stay tuned.

Graphics, Programming, Sprite Lamp

Extended depth map features – goal met!

Over at  Sprite Lamp’s Kickstarter campaign, we have just hit our next stretch goal at 20k! This is pretty awesome. This means extended depth map features, so it’s time to give a big rundown of how this will work. I mentioned two aspects of this – they are depth map editing from within Sprite Lamp, and exporting meshes that are based around the depth map.

Depth map editing

Currently, Sprite Lamp is able to generate a depth map from a normal map. This feature is most visible in the examples I’ve posted in the form of self-shadowing, particularly this image of a stone wall, which I’m going to post again because I love it so much:

Stonewall_TwoProfiles

If you look closely, Sprite Lamp has caught the depth quite accurately, right down to the little notches in the stone, and the bits where there are weird mortar shapes left on. Feel free to inspect the detail:

stonewall2_DepthNew

While it’s important to remember that there’s no such thing as a ‘correct’ depth map based on a drawing, I think the above image is pretty good. In general, I’m pretty pleased with Sprite Lamp’s algorithm/system for generating a depth map from a normal map. This is a pretty difficult problem, made worse by the fact that you get a physically imperfect normal map when it’s drawn by hand. Textures like the one above are, fortunately, a pretty good case for this application. Alas, not everything can be a best case scenario. I won’t go into too much detail here, but the worst case scenarios for Sprite Lamp’s depth generation are all about discontinuities in the depth map. They happen when what you draw has one object passing in front of another – it’s hard for Sprite Lamp to guess how far in front. This doesn’t happen much with textures (like the stone wall, above) but it does happen with character art. Sprite Lamp gets pretty good results here too – good enough for some nice self-shadowing effects, as I think is evidenced by the sample art on the main page and in the Kickstarter.

However! With algorithms like these that involve guessing what the artist intended, there are always times where you guess wrong. For that reason, I’d like to give the user the ability to mess with Sprite Lamp’s ‘interpretation’ manually a little bit. Now, naturally, I’m not going to offload all the dirty work onto the user – if I was going to do that you might as well just draw it by hand. The goal is small and intelligent tweaks.

With that in mind, I have a few features that I’m going to experiment with to make working with depth maps better. I’m not saying these will all make it into the final program – these are what I’ll try, but I don’t yet know what will be worth including and what will end up being a waste of time.

  • Edge detection: Sprite Lamp will go through the lighting profiles and try to figure out where depth discontinuities might be.
  • Silhouette maps: This is a plan I have to have the artist draw a simple map that will be deliberately easy for Sprite Lamp to detect the edges of. May or may not end up being necessary, but will mostly be trivial to create because it comes from mask layers that come from the drawing process anyway.
  • Edge-aware soft selection: The first two points here are ways of figuring out where depth discontinuities are – this is useful information to have, because it allows the user to easily select areas of the depth map without going ‘outside the lines’. If you have a foreground object and a background object, this will allow you to grab the depth values of the foreground object and move it back and forth without interfering with adjacent pixels (which are the depth values of another object entirely).
  • Spring systems representing depth values: This will allow the user to drag a single depth pixel forward and back, and have adjacent pixels come with to a greater or lesser extent. There won’t be springs connecting pixels along discontinuity edges.
  • Curve editor to do an image-wide readjustment of values: This is something similar to how colour readjustment works in programs like Photoshop, and it could indeed be done in those programs – however, I think it might be worth including it in Sprite Lamp because getting immediate feedback on how it looks will make things a lot easier to deal with.

So, that’s that. I’m looking forward to playing around with these things, and of course I’ll keep you posted as to how it all goes. As for the other part of this stretch goal…

Mesh Exporting

The other part of this stretch goal is to add the ability to export meshes. This doesn’t take quite as much explaining as the other feature, though it will take a bit of work on my part. The goal here is to export an intelligently-created mesh that takes the shape of the object according to the depth map. Naturally, the easy approach to this is to simply subdivide a quad into many quads, then displace the vertices according to the depth map. I’m hoping I can do quite a bit better than that, by placing vertices in important places along ridges and points, and making the geometry more sparse in flatter areas. The user will have a slider that allows them to adjust the target vert count of the generated mesh. This has a couple of concrete uses:

  • Stereographic rendering without relying on complex shaders: Naturally, I quite enjoy complex shaders, but there are times when it’s easier to just use geometry. This can save you from various annoying consequences of displacement shaders, such as correctly handling extreme angles.
  • Correct rendering into depth buffers for ‘true’ shadowing: I’m going to be writing a lot about the various sneaky tricks you can use to fake shadows using depth maps – however, there are certain shadowing algorithms where it’s just nicer to do things with geometry. Having geometry at your disposal gives you more options for plugging directly into existing ‘general’ shadowing options, such as shadow mapping and CSMs, which are well-documented already and amply implemented in existing engines.
  • As a base for further tessellation/displacement: The user will be able to determine how strong the vertex displacement will be, and that will include zero – that means that you can use the flat mesh as a basis for more effective GPU-based tessellation in the GPU, and displace generated verts based on the depth map in a shader.

In addition to these, though, I’m looking forward to see what else people come up with using this feature. I’m certainly planning on making some 3D prints of artwork once I get it done!

One last thing I’ll say on this subject is that I haven’t settled on formats for exporting of meshes. I’ll probably make use of the dubiously-named AssImp (short for ‘Asset Importer’, obviously), which can export to a handful of formats, in accordance with this list. However, if anyone knows of any straightforward mesh exporting libraries for C# you think would be better, I’m all ears.