This skill provides comprehensive guidance for developing virtual reality applications, covering the full stack from rendering pipelines to input handling, performance optimization, and cross-platform deployment. It is designed for developers building VR experiences on modern headsets including Meta Quest, Apple Vision Pro, PlayStation VR2, and PC VR platforms.

## Key Points

- **Frame rate targets**: 72 Hz minimum, 90 Hz recommended, 120 Hz ideal
- **Stereoscopic rendering**: Two viewpoints separated by interpupillary distance (IPD), typically 60-68mm
- **Low latency**: Motion-to-photon latency must stay below 20ms
- **Foveated rendering**: Reduce peripheral detail to save GPU cycles
- **3DoF (3 Degrees of Freedom)**: Rotation only (pitch, yaw, roll)
- **6DoF (6 Degrees of Freedom)**: Rotation + position (x, y, z)
- **Inside-out tracking**: Cameras on the headset track the environment
- **Outside-in tracking**: External sensors track the headset
1. **Choose your runtime**: OpenXR is the industry standard cross-platform API
2. **Select a rendering approach**: Forward rendering preferred over deferred for mobile VR
3. **Configure quality tiers**: Different settings for standalone vs. PC VR
4. **Set up performance profiling from day one**

VR Development Fundamentals

Purpose

This skill provides comprehensive guidance for developing virtual reality applications, covering the full stack from rendering pipelines to input handling, performance optimization, and cross-platform deployment. It is designed for developers building VR experiences on modern headsets including Meta Quest, Apple Vision Pro, PlayStation VR2, and PC VR platforms.

Core Concepts

The VR Rendering Pipeline

VR rendering differs fundamentally from traditional 3D rendering because it must produce two slightly offset views (one per eye) at consistently high frame rates to prevent motion sickness.

Key constraints:

• Frame rate targets: 72 Hz minimum, 90 Hz recommended, 120 Hz ideal
• Stereoscopic rendering: Two viewpoints separated by interpupillary distance (IPD), typically 60-68mm
• Low latency: Motion-to-photon latency must stay below 20ms
• Foveated rendering: Reduce peripheral detail to save GPU cycles

Rendering Budget per Frame (90 Hz):
Total frame time:    11.1ms
CPU work:            ~4-5ms
GPU work:            ~6-7ms
Compositor overhead: ~1-2ms

Per eye resolution (Quest 3): 2064 x 2208
Total pixels per frame:       ~9.1 million (both eyes)

Coordinate Systems and Tracking

VR applications must handle multiple coordinate spaces:

World Space
  └── Stage Space (play area origin)
       └── Head Space (HMD position/rotation)
            ├── Left Eye Space (offset by -IPD/2)
            ├── Right Eye Space (offset by +IPD/2)
            └── Controller Spaces
                 ├── Left Controller (grip + aim poses)
                 └── Right Controller (grip + aim poses)

Tracking types:

• 3DoF (3 Degrees of Freedom): Rotation only (pitch, yaw, roll)
• 6DoF (6 Degrees of Freedom): Rotation + position (x, y, z)
• Inside-out tracking: Cameras on the headset track the environment
• Outside-in tracking: External sensors track the headset

Input Systems

VR input goes far beyond mouse and keyboard:

Standard Controller Inputs:
├── Thumbstick (2-axis analog)
├── Trigger (analog, 0.0-1.0)
├── Grip (analog, 0.0-1.0)
├── Primary Button (A/X)
├── Secondary Button (B/Y)
├── Menu Button
└── Thumbstick Press

Hand Tracking Inputs:
├── 26 joints per hand
├── Pinch gestures (thumb + each finger)
├── Grab gestures
├── Point gestures
└── Custom pose detection

Eye Tracking Inputs:
├── Gaze direction (ray)
├── Fixation point (world position)
├── Pupil dilation
└── Blink detection

Development Workflow

Project Setup

When starting a VR project, establish these foundations:

1. Choose your runtime: OpenXR is the industry standard cross-platform API
2. Select a rendering approach: Forward rendering preferred over deferred for mobile VR
3. Configure quality tiers: Different settings for standalone vs. PC VR
4. Set up performance profiling from day one

Performance Optimization Strategies

VR performance is non-negotiable. Here are the primary optimization techniques:

Draw Call Reduction:

Techniques (ordered by impact):
1. Static batching      - Combine non-moving meshes
2. GPU instancing       - Render multiple copies efficiently
3. Texture atlasing     - Reduce material switches
4. LOD groups           - Simplify distant objects
5. Occlusion culling    - Skip hidden geometry

Shader Optimization for Mobile VR:

// BAD: Complex PBR shader on mobile VR
float3 specular = GGX_Specular(N, V, L, roughness, F0);
float3 diffuse = Lambert_Diffuse(albedo);
float ao = SampleAO(uv);
float shadow = SampleShadowMap(worldPos);

// GOOD: Simplified mobile VR shader
float3 diffuse = albedo * saturate(dot(N, L)) * lightColor;
float3 ambient = albedo * ambientColor;
float3 color = diffuse + ambient;
// Use baked lighting instead of real-time shadows

Memory Management:

Quest 3 Memory Budget:
├── Application RAM:     ~6 GB available
├── Texture memory:      Target < 1.5 GB
├── Mesh memory:         Target < 500 MB
├── Audio memory:        Target < 200 MB
└── Runtime overhead:    ~500 MB reserved

Optimization tactics:
- ASTC texture compression (4x4 or 6x6 blocks)
- Mesh LOD with aggressive distance thresholds
- Stream audio instead of loading full clips
- Pool and recycle objects instead of instantiating

Comfort and Accessibility

Preventing motion sickness is a core VR design responsibility:

Locomotion comfort levels:

Level 1 (Most comfortable):
  - Teleportation
  - Room-scale movement only
  - Stationary experiences

Level 2 (Moderate):
  - Snap turning (30-45 degree increments)
  - Vignette during movement
  - Slow smooth locomotion

Level 3 (Least comfortable):
  - Smooth locomotion + smooth turning
  - Vehicle movement
  - Rapid acceleration/deceleration

Always provide:

• Comfort settings menu accessible at any time
• Multiple locomotion options
• Stable visual reference points (horizon, cockpit, nose)
• Gradual introduction to movement mechanics

Cross-Platform Considerations

Platform Matrix:
┌──────────────────┬──────────┬────────┬───────────┬──────────┐
│ Platform         │ GPU Tier │ Input  │ Tracking  │ Runtime  │
├──────────────────┼──────────┼────────┼───────────┼──────────┤
│ Quest 3          │ Mobile   │ 6DoF+HT│ Inside-out│ OpenXR   │
│ Quest Pro        │ Mobile   │ 6DoF+HT│ Inside-out│ OpenXR   │
│ Vision Pro       │ Desktop  │ HT+Eye │ Inside-out│ ARKit    │
│ PSVR2            │ Console  │ 6DoF+ET│ Inside-out│ Sony SDK │
│ Valve Index      │ Desktop  │ 6DoF   │ Outside-in│ OpenXR   │
│ Pico 4           │ Mobile   │ 6DoF+HT│ Inside-out│ OpenXR   │
└──────────────────┴──────────┴────────┴───────────┴──────────┘

HT = Hand Tracking, ET = Eye Tracking

Abstraction strategy:

Application Layer
    ↓
Input Abstraction Layer (map actions, not buttons)
    ↓
Platform Adapter Layer (per-platform implementation)
    ↓
Runtime API (OpenXR / ARKit / Sony SDK)

Interaction Design Patterns

Grab and Manipulate

The most fundamental VR interaction:

Grab System Architecture:
1. Detect overlap (hand/controller enters trigger volume)
2. Highlight object (outline, glow, haptic pulse)
3. On grab input:
   a. Calculate grab offset (hand-to-object transform)
   b. Attach object to hand (maintain offset)
   c. Disable physics on object
4. While held:
   a. Update object transform each frame
   b. Track velocity for throw calculation
5. On release:
   a. Detach object
   b. Re-enable physics
   c. Apply tracked velocity as throw force

UI in VR

VR UI requires fundamentally different approaches than flat-screen UI:

VR UI Principles:
├── World-space canvases (not screen-space)
├── Minimum text size: 0.5m at 2m distance
├── Curved panels for peripheral readability
├── Laser pointer or direct touch interaction
├── Haptic feedback on button press
├── Audio feedback for all interactions
└── Avoid head-locked UI (causes discomfort)

Recommended distances:
├── Near UI (menus): 0.5 - 1.0m from user
├── Mid UI (HUD): 1.0 - 2.0m from user
└── Far UI (signage): 3.0 - 10.0m from user

Spatial Audio

Sound design in VR must account for 3D space:

Audio Architecture:
├── HRTF spatialization (head-related transfer function)
├── Distance attenuation curves
├── Occlusion (sounds blocked by walls)
├── Reverb zones (room acoustics)
├── Ambisonics for environmental audio
└── Direct audio for UI/narration

Implementation priorities:
1. Spatialize all diegetic sounds
2. Keep UI sounds non-spatialized
3. Use audio occlusion for immersion
4. Match reverb to room size
5. Attach sounds to interactable objects

Testing VR Applications

Automated Testing

Test Categories:
├── Unit tests (logic, no VR hardware needed)
├── Integration tests (mock VR runtime)
├── Performance tests (frame timing assertions)
├── Comfort tests (acceleration limits)
└── Platform compliance tests (store requirements)

Frame Timing Assertions:
- 95th percentile frame time < 11.1ms (90 Hz target)
- No frame drops > 2 consecutive frames
- Motion-to-photon latency < 20ms
- GPU utilization < 85% sustained

Manual Testing Checklist

Pre-submission testing:
□ Test on target hardware (not just simulator)
□ Test at room-scale boundaries
□ Test with different IPD settings
□ Test all locomotion modes
□ Test for 30+ minutes continuously (comfort)
□ Test with hand tracking if supported
□ Test guardian/boundary interactions
□ Test app suspend/resume cycle
□ Test with different lighting conditions (tracking)
□ Test accessibility features

Common Pitfalls

1. Ignoring frame budget: Every dropped frame is felt physically in VR
2. Moving the camera without user input: This is the primary cause of VR sickness
3. Scaling objects incorrectly: Objects at wrong scale feel deeply uncomfortable
4. Neglecting audio spatialization: Flat audio breaks immersion instantly
5. Hard-coding IPD or play space assumptions: Users have diverse physical setups
6. Skipping comfort options: Not everyone has the same VR tolerance
7. Testing only in editor: The editor is not representative of headset performance
8. Using screen-space effects carelessly: Bloom, motion blur, and depth of field can cause nausea

Resources and Standards

• OpenXR Specification: The cross-platform VR/AR API standard
• WebXR Device API: Browser-based XR standard (W3C)
• Meta Quest Developer Guidelines: Platform-specific requirements
• Apple visionOS HIG: Human Interface Guidelines for spatial computing
• Khronos Group glTF: Standard 3D asset format for interoperability

When to Apply This Skill

Use this skill when:

• Starting a new VR project from scratch
• Porting an existing application to VR
• Debugging performance issues in a VR application
• Designing interaction systems for VR
• Preparing a VR application for store submission
• Evaluating VR SDKs and frameworks for a project