Friday, May 31, 2013

Motion Detection and Three.js

This post was inspired by and based upon the work of Romuald Quantin: he has written an excellent demo (play a xylophone using your webcam!) at http://www.soundstep.com/blog/2012/03/22/javascript-motion-detection/ and an in-depth article at http://www.adobe.com/devnet/html5/articles/javascript-motion-detection.html. Basically, I re-implemented his code with a few minor tweaks:

  • The jQuery code was removed.
  • There are two canvases: one containing the video image (flipped horizontally for easier interaction by the viewer), the second canvas containing the superimposed graphics
  • For code readability, there is an array (called "buttons") that contains x/y/width/height information for all the superimposed graphics, used for both the drawImage calls and the getImageData calls when checking regions for sufficient motion to trigger an action.


The first demo is a minimal example which just displays a message when motion is detected in certain regions; no Three.js code appears here.

The second demo incorporates motion detection in a Three.js scene.  When motion is detected over one of three displayed textures, the texture of the spinning cube in the Three.js scene changes to match.

Here is a YouTube video of these demos in action:


Happy coding!

Monday, May 27, 2013

Animated Shaders in Three.js (Part 2)


After my previous post on animated shaders, I decided to build on the shader code to create a more visually sophisticated "fireball effect". This time, my inspiration came from Jaume S├ínchez Elias' experiments with perlin noise, particularly his great article on vertex displacement.

In the previous post, we have shader code for animating a "noisy" random distortion of a texture (the "base texture").  The noise is generated from the RGB values of a second texture (the "noise texture").  This time around, we add the following features:

  • allow repeating the textures in each direction (repeatS and repeatT)
  • a second texture (the "blend texture") that is randomly distorted (at a different rate) and then additively blended with the base texture; we also pass in a float that controls the speed of distortion (blendSpeed) and a float value (blendOffset) that is subtracted from the pixel data so that darker parts of the blend texture may result in decreasing values in the base texture
  • randomly distort the positions of the vertices of the sphere using another texture (the "bump texture"); the rate at which these distortions change is controlled by a float value (bumpSpeed) and the magnitude is controlled by a second float value (bumpScale).


Causing the texture to repeat and blending in the color values takes place in the fragment shader, as follows:

<script id="fragmentShader" type="x-shader/x-vertex">
uniform sampler2D baseTexture;
uniform float baseSpeed;
uniform float repeatS;
uniform float repeatT;

uniform sampler2D noiseTexture;
uniform float noiseScale;

uniform sampler2D blendTexture;
uniform float blendSpeed;
uniform float blendOffset;

uniform float time;
uniform float alpha;

varying vec2 vUv;

void main()
{
    vec2 uvTimeShift = vUv + vec2( -0.7, 1.5 ) * time * baseSpeed;
    vec4 noise = texture2D( noiseTexture, uvTimeShift );
    vec2 uvNoiseTimeShift = vUv + noiseScale * vec2( noise.r, noise.b );
    vec4 baseColor = texture2D( baseTexture, uvNoiseTimeShift * vec2(repeatS, repeatT) );

    vec2 uvTimeShift2 = vUv + vec2( 1.3, -1.7 ) * time * blendSpeed;
    vec4 noise2 = texture2D( noiseTexture, uvTimeShift2 );
    vec2 uvNoiseTimeShift2 = vUv + noiseScale * vec2( noise2.g, noise2.b );
    vec4 blendColor = texture2D( blendTexture, uvNoiseTimeShift2 * vec2(repeatS, repeatT) ) - blendOffset * vec4(1.0, 1.0, 1.0, 1.0);

    vec4 theColor = baseColor + blendColor;
    theColor.a = alpha;
    gl_FragColor = theColor;
}
</script>

The bump mapping occurs in the vertex shader.  Bump mapping is fairly straightforward -- see the article above for an explanation; the key code snippet is:
vec4 newPosition = position + normal * displacement
However, there are two particularly interesting points in the code below:

  1. Using time-shifted UV coordinates seems to create a "rippling" effect in the bump heights, while using the noisy-time-shifted UV coordinates seems to create more of a "shivering" effect.
  2. There is a problem at the poles of the sphere -- the displacement needs to be the same for all vertices at the north pole and south pole, otherwise a tearing effect (a "jagged hole") appears, as illustrated at StackOverflow (and solved by WestLangley).  This is the reason for the conditional operator  that appears in the assignment of the distortion (and introduces not-so-random fluctuations at the poles).

Without further ado, The code for the new and improved vertex shader is:

<script id="vertexShader" type="x-shader/x-vertex">
uniform sampler2D noiseTexture;
uniform float noiseScale;

uniform sampler2D bumpTexture;
uniform float bumpSpeed;
uniform float bumpScale;

uniform float time;

varying vec2 vUv;

void main()
{
    vUv = uv;

    vec2 uvTimeShift = vUv + vec2( 1.1, 1.9 ) * time * bumpSpeed;
    vec4 noise = texture2D( noiseTexture, uvTimeShift );
    vec2 uvNoiseTimeShift = vUv + noiseScale * vec2( noise.r, noise.g );
    // below, using uvTimeShift seems to result in more of a "rippling" effect
    //   while uvNoiseTimeShift seems to result in more of a "shivering" effect
    vec4 bumpData = texture2D( bumpTexture, uvTimeShift );

    // move the position along the normal
    //  but displace the vertices at the poles by the same amount
    float displacement = ( vUv.y > 0.999 || vUv.y < 0.001 ) ?
    bumpScale * (0.3 + 0.02 * sin(time)) :
    bumpScale * bumpData.r;
    vec3 newPosition = position + normal * displacement;

    gl_Position = projectionMatrix * modelViewMatrix * vec4( newPosition, 1.0 );
}
</script>

The code for integrating these shaders into Three.js:

// base image texture for mesh
var lavaTexture = new THREE.ImageUtils.loadTexture( 'images/lava.jpg');
lavaTexture.wrapS = lavaTexture.wrapT = THREE.RepeatWrapping;
// multiplier for distortion speed 
var baseSpeed = 0.02;
// number of times to repeat texture in each direction
var repeatS = repeatT = 4.0;

// texture used to generate "randomness", distort all other textures
var noiseTexture = new THREE.ImageUtils.loadTexture( 'images/cloud.png' );
noiseTexture.wrapS = noiseTexture.wrapT = THREE.RepeatWrapping;
// magnitude of noise effect
var noiseScale = 0.5;

// texture to additively blend with base image texture
var blendTexture = new THREE.ImageUtils.loadTexture( 'images/lava.jpg' );
blendTexture.wrapS = blendTexture.wrapT = THREE.RepeatWrapping;
// multiplier for distortion speed
var blendSpeed = 0.01;
// adjust lightness/darkness of blended texture
var blendOffset = 0.25;

// texture to determine normal displacement
var bumpTexture = noiseTexture;
bumpTexture.wrapS = bumpTexture.wrapT = THREE.RepeatWrapping;
// multiplier for distortion speed 
var bumpSpeed   = 0.15;
// magnitude of normal displacement
var bumpScale   = 40.0;

// use "this." to create global object
this.customUniforms = {
    baseTexture:  { type: "t", value: lavaTexture },
    baseSpeed:    { type: "f", value: baseSpeed },
    repeatS:      { type: "f", value: repeatS },
    repeatT:      { type: "f", value: repeatT },
    noiseTexture: { type: "t", value: noiseTexture },
    noiseScale:   { type: "f", value: noiseScale },
    blendTexture: { type: "t", value: blendTexture },
    blendSpeed:   { type: "f", value: blendSpeed },
    blendOffset:  { type: "f", value: blendOffset },
    bumpTexture:  { type: "t", value: bumpTexture },
    bumpSpeed:    { type: "f", value: bumpSpeed },
    bumpScale:    { type: "f", value: bumpScale },
    alpha:        { type: "f", value: 1.0 },
    time:         { type: "f", value: 1.0 }
};

// create custom material from the shader code above
//   that is within specially labeled script tags
var customMaterial = new THREE.ShaderMaterial(
{
    uniforms: customUniforms,
    vertexShader:   document.getElementById( 'vertexShader'   ).textContent,
    fragmentShader: document.getElementById( 'fragmentShader' ).textContent
}   );

var ballGeometry = new THREE.SphereGeometry( 60, 64, 64 );
var ball = new THREE.Mesh( ballGeometry, customMaterial );
scene.add( ball );

For a live example, check out the demo in my GitHub collection, which uses this shader to create an animated fireball.


Happy coding!

Saturday, May 25, 2013

Animated Shaders in Three.js

Once again I have been investigating Shaders in Three.js.  My last post on this topic was inspired by glow effects; this time, my inspiration comes from the Three.js lava shader and Altered Qualia's WebGL shader fireball.


For these examples, I'm really only interested in changing the pixel colors, so I'll just worry about the fragment shader.

The Three.js (v.56) code  for the a simple fragment shader that just displays a texture would be:
<script id="fragmentShader" type="x-shader/x-vertex">
uniform sampler2D texture1;
varying vec2 vUv;
void main() 
{
    vec4 baseColor  = texture2D( texture1, vUv );
    gl_FragColor = baseColor;
} 
</script>

(note that this code is contained within its own script tags.)

Then to create the material in the initialization of the Three.js code:
var lavaTexture = new THREE.ImageUtils.loadTexture( 'images/lava.jpg' );
lavaTexture.wrapS = lavaTexture.wrapT = THREE.RepeatWrapping; 

var customUniforms = { texture1: { type: "t", value: lavaTexture }   };

// create custom material from the shader code above within specially labeled script tags
var customMaterial = new THREE.ShaderMaterial( 
{
    uniforms: customUniforms,
    vertexShader:   document.getElementById( 'vertexShader'   ).textContent,
    fragmentShader: document.getElementById( 'fragmentShader' ).textContent
}   );

What we would like to create a more realistic effect is to:
  • add a bit of "random noise" to the texture coordinates vector (vUv) to cause distortion
  • change the values of the random noise so that the distortions in the image change fluidly
  • (optional) add support for transparency (custom alpha values)
To accomplish this, we can add some additional parameters to the shader, namely:
  • a second texture (noiseTexture) to generate "noise values"; for instance, we can add the red/green values at a given pixel to the x and y coordinates of vUv, causing an offset
  • a float (noiseScale) to scale the effects of the "noise values"  
  • a float (time) to pass a "time" value to the shader to continuously shift the texture used to generate noise values
  • a float (baseSpeed) to scale the effects of the time to either speed up or slow down rate of distortions in the base texture
  • a float (alpha) to set the transparency amount
The new version of the fragment shader code looks like this:
<script id="fragmentShader" type="x-shader/x-vertex"> 
uniform sampler2D baseTexture;
uniform float baseSpeed;
uniform sampler2D noiseTexture;
uniform float noiseScale;
uniform float alpha;
uniform float time;
varying vec2 vUv;

void main() 
{
    vec2 uvTimeShift = vUv + vec2( -0.7, 1.5 ) * time * baseSpeed;
    vec4 noise = texture2D( noiseTexture, uvTimeShift );
    vec2 uvNoisyTimeShift = vUv + noiseScale * vec2( noise.r, noise.g );
    vec4 baseColor = texture2D( baseTexture, uvNoisyTimeShift );
    baseColor.a = alpha;
    gl_FragColor = baseColor;
}
</script>

The new code to create this shader in Three.js:
// initialize a global clock to keep track of time
this.clock = new THREE.Clock();

var noiseTexture = new THREE.ImageUtils.loadTexture( 'images/noise.jpg' );
noiseTexture.wrapS = noiseTexture.wrapT = THREE.RepeatWrapping;

var lavaTexture = new THREE.ImageUtils.loadTexture( 'images/lava.jpg' );
lavaTexture.wrapS = lavaTexture.wrapT = THREE.RepeatWrapping; 

// create global uniforms object so its accessible in update method
this.customUniforms = {
    baseTexture:  { type: "t", value: lavaTexture },
    baseSpeed:    { type: "f", value: 0.05 },
    noiseTexture: { type: "t", value: noiseTexture },
    noiseScale:   { type: "f", value: 0.5337 },
    alpha:        { type: "f", value: 1.0 },
    time:         { type: "f", value: 1.0 }
}

// create custom material from the shader code above within specially labeled script tags
var customMaterial = new THREE.ShaderMaterial( 
{
    uniforms: customUniforms,
    vertexShader:   document.getElementById( 'vertexShader'   ).textContent,
    fragmentShader: document.getElementById( 'fragmentShader' ).textContent
}   );

and in the update or render method, don't forget to update the time using something like:
var delta = clock.getDelta();
customUniforms.time.value += delta;

That's about it -- for a live example, check out the demo in my GitHub collection, which uses this shader to create an animated lava-style texture and animated water-style texture.


Happy coding!