Stylized Edge Detection Effects

Introduction

Welcome back to Advanced Rendering and Visual Effects! In our previous lesson, we built a powerful multi-perspective system using framebuffers . In this third lesson, we'll take that foundation to the next level by exploring stylized post-processing effects, transforming our rendered scene with cartoon-style edge detection and realistic screen glare. This marks an exciting step in our course as we move from realistic rendering to creating unique, artistic visual styles. Building upon the render-to-texture techniques we've established, we will enhance our framebuffer system with sophisticated visual effects that turn ordinary rendered content into stylized imagery. These techniques are fundamental to modern non-photorealistic rendering (NPR) and form the backbone of many popular visual styles in games and digital media. By combining edge detection algorithms with specular glare calculations, we will create a compelling visual experience that demonstrates the creative potential of shader-based post-processing.

Understanding Post-Processing

Post-processing is a powerful technique in which we apply visual transformations to an image after it has been rendered. Instead of working with 3D geometry, post-processing operates directly on the final 2D image, treating it as a grid of pixels that can be analyzed and modified. This approach allows us to implement complex effects like bloom, motion blur, color grading, and the stylized effects we will build today. The beauty of post-processing lies in its flexibility. We can achieve stunning visual styles without altering our core 3D rendering pipeline. Since our framebuffer system already captures our 3D scene into a texture, we have the perfect starting point for applying these image-based transformations in a second rendering pass.

Edge Detection Fundamentals

G_x

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

glsl

// Apply cartoony edge detection effect
vec2 texelSize = 1.0 / textureSize(uTexture, 0);

// Sample the center and surrounding pixels
vec3 center = texture(uTexture, scaledUV).rgb;
vec3 left   = texture(uTexture, scaledUV + vec2(-texelSize.x, 0.0)).rgb;
vec3 right  = texture(uTexture, scaledUV + vec2( texelSize.x, 0.0)).rgb;
vec3 up     = texture(uTexture, scaledUV + vec2(0.0,  texelSize.y)).rgb;
vec3 down   = texture(uTexture, scaledUV + vec2(0.0, -texelSize.y)).rgb;

// Calculate horizontal and vertical gradients (Sobel-like)
vec3 horizontal = abs(left - right);
vec3 vertical = abs(up - down);

glsl

// Combine gradients and convert to grayscale
float edge = length(horizontal + vertical);

// Threshold for edge detection
float edgeThreshold = 0.3;

vec3 finalColor;
if (edge > edgeThreshold) {
    // Draw outline in black
    finalColor = vec3(0.0, 0.0, 0.0);
} else {
    // Normal texture color
    finalColor = center;
}

glsl

// Add specular glare effect
vec3 normal = vec3(0.0, 0.0, 1.0); // Plane normal facing towards camera
vec3 lightDir = normalize(lightPos - vWorldPos);
vec3 viewDir = normalize(viewPos - vWorldPos);
vec3 reflectDir = reflect(-lightDir, normal);

float specularStrength = 0.8;
float shininess = 64.0;
float spec = pow(max(dot(viewDir, reflectDir), 0.0), shininess);
vec3 specular = specularStrength * spec * lightColor;

glsl

// Apply cartoony edge detection effect
vec2 texelSize = 1.0 / textureSize(uScreenTexture, 0);

// Sample the center and surrounding pixels  
vec3 center = texture(uScreenTexture, vUV).rgb;
// ... (sampling left, right, up, down is identical to the plane shader)

if (edge > edgeThreshold) {
    FragColor = vec4(0.0, 0.0, 0.0, 1.0);
} else {
    FragColor = vec4(center, 1.0);
}

glsl

// Apply glare more prominently in certain areas
float glareIntensity = smoothstep(0.3, 0.8, spec);
finalColor = mix(finalColor, finalColor + specular, glareIntensity);

C++

// FIRST PASS: Render cube from virtual camera to framebuffer
framebuffer.bind();
glClearColor(0.2f, 0.0f, 0.3f, 1.0f);  // Purple background
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

C++

// SECOND PASS: Render final scene from real world camera  
framebuffer.unbind();
glClearColor(0.05f, 0.05f, 0.1f, 1.0f);  // Dark blue-gray background
glViewport(0, 0, gFBW, gFBH);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

Introduction

Understanding Post-Processing

Edge Detection Fundamentals

Join the 1M+ learners on CodeSignal

Implementing Sobel-Based Edge Detection

Edge Detection Decision Logic

Specular Glare Implementation

Screen-Space Post-Processing

Glare Integration and Enhancement

Clear Color Differentiation

Visual Results

Conclusion and Next Steps