The Vrui Manifesto

Design Goals

Goals for Developers

• Applications must not be developed for a particular type of environment (CAVE, 3D TV, desktop, ...).
  • Applications never meant to run in VR must still run in all VR environments.
    It is expected that applications will primarily be developed on the desktop; Vrui must ensure that no unintentional desktop dependencies creep in. This should be enforced directly via API if possible, or via clearly stated development rules where not. API enforcement works by providing value-added abstractions that are voluntarily used by developers, and that automatically follow portability rules internally.
    Example value-added abstractions with built-in portability:
    • High-performance cluster-transparent file and network access
    • Cluster-transparent file selection dialog with ability to traverse into and return files from ZIP archives
    • Simple multipipe-transparent rendering state management
    • Simple scene graph modeled after the VRML specification
    Example explicit development rules:
    • Don't call srand(...) anywhere in your code. Vrui initializes the random number generator automatically.
    • Don't use any operating system-supplied time functions. Vrui supplies its own cluster-safe time functions, such as Vrui::getTimeOfDay, Vrui::getApplicationTime, or Vrui::getFrameTime.
  • Applications must not assume anything about the layout or size of the environment in which they are running. Vrui provides functions to query any important values, such as coordinate system orientations (forward and up directions), environment sizes, suggested sizes for user interface elements, interactive selection radii, etc.
    • Particularly, applications must make no assumptions about an environment's number of viewers, screens, display windows, OpenGL contexts, or stereo rendering modes. These things vary significantly between environments, and can even change dynamically during an application's runtime.
  • Applications must not directly reference input devices.
    • The wide variety of types and numbers of input devices is the distinguishing feature between desktop and VR environments. There is no equivalent to the desktop world's keyboard/video/mouse paradigm in VR. Some environments may even allow adding or removing input devices dynamically during an application's runtime.
    • Applications should automatically work better if run in an environment with more expressive input device types (such as tracked 6-DOF devices vs 2D mice), or multiple input devices for bimanual interaction.
      ... ideally even if any of those devices are "virtual."
    • User interactions are communicated to applications via high-level events, not (real or virtual) input devices.
      "High-level events" means higher than "user pressed button b on device d at position (x, y, z)."
  • Cluster-unaware applications must work on clusters, without loss of functionality.
    Cluster-aware applications may explicitly reference clustering to improve performance, but must contain fallbacks to run on single-system environments.
  • Vrui should be competitive with dedicated desktop development toolkits such as Qt or GTK+, or, at a lower level, glut, in desktop environments.
    This means competitiveness in terms of ease of development, not in terms of ease of use (which is a separate goal).
• Simple applications must be simple.
  • Vrui must require only the least possible amount of boilerplate code.
    Vrui "Hello world" should not be significantly longer than regular "Hello world" (7 non-empty lines of code).
    The shortest possible Vrui application (it draws a sphere) is 22 non-empty lines of code. It still offers multiple navigation metaphors, interactive measurement and annotation, interactive clipping planes, interactive light sources, and more.
  • Vrui must not require any references to features unused by a particular application.
    "If you don't use it, you don't need to know it even exists." This implies that all toolkit features must have reasonable default settings.
  • It must be possible to port a simple existing 3D graphics application such as glxgears to Vrui without reading documentation.
    Vrui must contain code template(s) saying "create display list here," "update application state here," "insert rendering code here," etc.
  • Vrui should provide enough free added value such that developers will use it even for one-off, throw-away, non-VR graphics programs, including 2D graphics programs.
    Vrui should take over the niche currently inhabited by glut and its ilk. It should be the toolkit of choice to:
    • ... look at result of a couple of lines of OpenGL code.
    • ... figure out why a GLSL shader produces all-black output.
    • ... have a quick look at how two 3D point clouds align, and interactively measure point pair offsets.
    • ... create an interactive driver for some new 2D or 3D computational geometry algorithm, including recording a session that breaks it and measuring the positions of problem spots to set breakpoints in a debugger.
  • Building a one-off, single-source program must be simple.
    • g++ -o MyApp `pkgconfig --cflags --libs Vrui` MyApp.cpp
    • make MyApp
      Vrui must provide a template makefile that can be copied to the current working directory and be used for simple applications without changing it (or even knowing how make works).
• Vrui must not force a fixed parallelism model.
  • Making effective use of multi-CPU or multi-core systems requires application-specific and dynamic fine-grain parallelism. A fixed model forced by a toolkit (such as one foreground rendering thread and one background simulation thread) causes unnecessary overhead for simple applications that do not benefit from parallelism (violating the simplicity principle), and is too limited for more complex applications that do.
  • Instead, Vrui should leave application-side parallelism entirely up to applications, and use parallelism internally wherever beneficial (such as when rendering to multiple windows controlled by separate graphics cards).
• Vrui must foster code reuse.
  • Vrui must provide powerful lower-level abstractions so that application code can be built upon a common base.
    This includes abstractions that are not typically thought of as part of a VR toolkit, such as:
    • Threading abstraction:
      • RAII for threads, mutexes and mutex locks, or condition variables
      • Higher-level data types for interthread communication such as triple buffers or queues
    • File access abstraction:
      • High-performance access to local and remote files using a common interface
      • Transparent handling of compressed files
      • Common interface for access to directory-like objects, such as local or remote file system directories or directories inside package files such as ZIP archives
      • File adapter classes for on-the-fly (de-)compression, background read-ahead, etc.
    • Cluster abstraction:
      • High-performance intra-cluster communication
      • Cluster-transparent access to local or remote files or external network services
    • Geometry abstraction:
      Geometry data types (points, vectors, transformations, ...) must be first-class data types and must be used throughout the toolkit and across its API. Representing points, vectors, or transformations as raw arrays of floats/doubles at the API is bad because:
      • It causes severe loss of functionality or performance.
        • Developers must take care to treat points and vectors differently, even though the API treats them as the same type.
        • 4x4 matrices do not represent semantic distinctions between different classes of transformations (pure rotation, rigid body transformation, ...).
        • Inverting a matrix transformation requires a 4x4 matrix inversion instead of a few sign flips and multiplications.
      • It forces application developers to reinvent the wheel or rely on external libraries.
      • It prevents code reuse by introducing incompatibilities if different developers roll their own implementations, or opt for different external libraries.
    • Rendering abstraction:
      • OpenGL entry points for geometry data types such as glVertex(p) or glMultMatrix(transform)
      • RAII support for OpenGL resources such as display lists, buffer or texture objects, or shaders
      • Multipipe-transparent OpenGL state management
  • Developers must be discouraged from "rolling their own" or reinventing the wheel by good abstractions with added value.
    • Ad-hoc implementations might be of lower performance, have subtle bugs, or cause compatibility problems.
• Vrui should support interoperability with other VR development toolkits, unless it limits functionality or efficiency.
  Lower-level abstractions must not be dependent on Vrui as a whole to support horizontally transferring units of functionality. For example, the GLMotif widget library must be fully usable by applications based on other VR toolkits (or even non-VR toolkits such as glut).
• Vrui's API should be as stable as possible, but new features, cleaner interfaces, or better approaches trump API stability.
  • APIs necessarily evolve as VR toolkit design is still research topic.
    Too strict adherence to API stability leads to stagnation and obsolescence.
  • API changes should be localized and incremental.
    • Applications not using a changed component API must recompile without code change, and should ideally not even require recompilation.
    • Adapting application code to changed APIs should be straightforward or even automatable.
  • Vrui must use version numbering scheme supporting maximum forward and backwards binary and source code compatibility.
  • Vrui must support multiple versions of itself installed and running at the same time.
    This enables fixing bugs in and running applications that have not (yet) been adapted to API changes.

Goals for Users

• It must be simple to install Vrui itself, and Vrui applications.
  • Dependencies on external libraries must be kept to minimum.
    • Only require libraries that are part of the operating system's standard installation.
    • Detect presence of optional libraries during toolkit installation, and disable use of missing libraries in the code.
    • Ideally, distribute toolkit as a package with automatic dependency resolution.
      • Use distributions' default package managers on Linux.
      • Unfortunately, this mechanism does not exist on Mac OS X.
  • Installation of Vrui or Vrui applications should be as simple as 1) unpack, 2) make, 3) make install; and make install should be optional for applications.
    • Vrui must be pre-configured for desktop environments, which are expected to be the most common environment type.
    • Vrui should contain template configurations for other common environment types (CAVEs, 3D TVs, ...).
    • Ideally, companies selling high-end virtual reality environments will install and configure Vrui as part of environment installation.
• It must be simple to run Vrui applications.
  • Vrui must not require users to start services, daemons, etc.
    Any required services or daemons must be administrator-level, and start automatically or run at all times.
    Exception: Some non-standard 3D input devices require, or benefit from, persistent drivers due to lengthy initialization or warm-up times, or stability problems if a device is started/stopped too often. In such cases it is tolerable to require a separate user-level driver.
  • Vrui must hide any user-visible differences between single-system and cluster environments.
    • Users only interact with the head node of a cluster, acting as the entire environment's "console."
    • Users must not be required to manually distribute executables, configurations, or data across the cluster.
    • Users must be able to start applications in cluster environments in the same way as in single-system environments.
      Replication, communication setup, and distribution of shared data must be handled by Vrui internally.
  • It should be possible to run a Vrui application by double-clicking on its executable in a file manager.
    This must work the same in any environment type, regardless whether single-system or cluster-based.
  • Vrui should support drag-and-drop of data files onto applications' executables in a file manager.
    Vrui applications must be able to parse command lines.
• It must be simple to use Vrui applications.
  • Vrui should provide a consistent "look&feel" between applications in the same environment.
    • Common 3D interaction metaphors for general tasks such as navigation
    • Common graphical user interface widget set
    • Common interaction principles between applications
      Similar features should work similarly. This is impossible to enforce via API; it requires agreed-upon and well-documented best practices.
  • Functionality and efficiency trump consistency between different environments.
    • Users don't expect identical interaction metaphors between vastly different environment types such as desktops and CAVEs.
    • Interaction metaphors should be optimized for each environment type.
    • "Principle of least surprise" still applies to all environments.
  • Users should be able to configure interaction metaphors to their liking.
    Vrui must support per-user configurations, and should support per-application or even per-use case configurations.

Architecture / Design Features

• Vrui is a layered toolkit with powerful lower-level abstractions and VR-specific functionality only in the topmost layer. Vrui's layers form a directed acyclic graph, i.e., there are no cyclic dependencies between layers. This fosters code reuse and toolkit interoperability since each library layer offers independent useful functionality, and any layer (except the Vrui layer itself) can be linked into applications based on other toolkits without having to import the entire Vrui toolkit as a monolithic unit (and without the concomitant main loop / window management / etc. conflicts).
  • Component library overview in order of dependency (although not every layer depends on every lower layer):
    • Misc - Basic abstractions to support OS independence or factor out commonly-used functionality
    • Threads - OS-independent threading abstraction with RAII
    • USB - RAII wrapper for low-level USB library
    • IO - Abstract high-performance file and directory access
    • Plugins - OS-independent management of dynamically loaded plug-ins
    • Realtime - OS-independent alarm timers for time-critical tasks
    • Comm - Abstract device port and network socket access based on the IO library
    • Cluster - Support for cluster distribution and intra-cluster communication, and cluster-transparent file and network access based on the IO library
    • Math - Templatized math functions and constants
    • Geometry - Templatized geometry library
    • MacOSX - RAII wrapper for Mac OSX core libraries
    • GLWrappers - Templatized wrappers for OpenGL API calls to support genericity in higher layers
    • GLSupport - RAII wrappers for OpenGL objects; multipipe management support; commonly-used functionality; OpenGL extension management
    • GLXSupport - Helper classes for OpenGL window system bindings
    • GLGeometry - OpenGL API bindings for the templatized geometry library
    • Images - Support classes to read and write image files in a variety of formats
    • GLMotif - 2.5D widget library for OpenGL "inspired by" the Motif toolkit
    • Sound - Support classes for low-level access to audio recording and playback devices
    • ALSupport - Templatized wrappers for OpenAL (spatial audio) API calls to support genericity in higher layers; RAII wrappers for OpenAL objects; multi-context management support; OpenAL API bindings for the templatized geometry library
    • Video - Support classes for low-level access to video recording and playback devices
    • SceneGraph - Simple scene graph library modeled after the VRML 97 specification
    • Vrui - VR application development toolkit
• The highest level of Vrui (the actual VR layer) features a "microkernel API" with a small core interface that delegates most work to independent managers to isolate applications from API changes in modules which they don't use.
• Vrui is threading-agnostic.
  • Vrui is centered around a sequential main loop handling every aspect of frame processing from input device updates and event processing to rendering.
  • Simple applications can use this main loop directly for all their processing to avoid the overhead and difficulty of inter-thread communication.
  • Complex applications can create any number of threads, at any time, and use the main loop for low-impact or inherently synchronous processing, or purely for synchronization.
  • Vrui itself uses a number of threads, invisible to applications, for purposes such as device communication or parallel rendering.
  • Application developers can use functionality of the Threads library to simplify parallel programming, or use internally-parallel classes or functions from other library layers (such as parallel kd-tree generation in the Geometry library or background read-ahead or gzip (de-)compression in the IO library).
• Vrui is cluster-transparent.
  • Vrui applications and the Vrui library itself are binary portable between single-system, shared-memory, and cluster environments. In other words, an application can be freely switched between any of the three modes without changing the source code, recompiling, or relinking.
  • When run in a cluster environment, a Vrui application will automatically distribute itself to all cluster nodes, without user or application developer interaction.
  • Vrui itself uses a hybrid split-first / split-middle approach.
    • At the coarsest level, Vrui uses replication and runs the mainloop code identically on all cluster nodes, synchronized by explicit broadcasting of input device and ancillary data.
    • On a finer level, Vrui uses split-middle approaches with explicit communication between cluster nodes in several components, sometimes provided by lower-level libraries. For example, Vrui relies on the split-middle distribution paradigm of the IO library for its own cluster-transparent file and network access.
  • Simple applications can ignore distribution and run entirely in split-first mode, as long as some simple rules are followed.
  • To better use cluster resources, and not be limited by the slowest node in a cluster, applications can follow a split-middle approach by either explicitly using cluster communication primitives offered by the Cluster library, or by using internally split-middle cluster-transparent classes and functions provided by the Cluster and higher libraries (such as GLMotif's split-middle file selection dialog).
• Vrui is multipipe-transparent.
  • Any node in a Vrui cluster (or the only node in a single-machine environment) can control any number of display windows, using any combination of mono or stereo rendering modes, controlled by any number of GPUs.
  • If a Vrui node has windows controlled by multiple GPUs, it can be set to render to these windows in parallel (Vrui can always render in parallel, but parallel rendering to a single GPU is usually significantly slower than sequential rendering due to OpenGL's context switching overhead).
  • Applications are completely isolated from the number of GPUs, OpenGL contexts, windows, eyes per window, and rendering modes on any cluster node.
    • Via the GLSupport library, Vrui provides automatic means to manage persistent per-context OpenGL state, such as shader objects, vertex buffers, texture objects, or display lists, so that an application always only sees one context and its associated state at a time, and never has to worry about when to initialize any of its persistent OpenGL state.
    • Vrui can automatically share OpenGL state data between contexts that are able to do so, without intervention by the application developer.
• Vrui's input architecture is based on a bipartite graph of input devices and so-called "tools." This is the most important feature enabling applications that are not merely "technically portable" between different environment classes, but are fully usable in each environment class, and often indistinguishable from applications natively written for a particular environment class. Particularly, Vrui does not have a "desktop simulator;" instead, Vrui applications, when run in a desktop environment, behave and are used exactly like native desktop applications.
  • Input devices are position tracking sources (2D, 3D, or 6D positions depending on input device type) and button or analog axis (valuator) event sources.
  • Tools are user interface components that either directly implement toolkit or application functionality (such as navigation, UI interaction, measurement, or selection/dragging), or transform their source input devices into derived input devices in some toolkit- or application-defined manner.
  • Tools offer independent bits of functionality that can be combined in inventive ways to create powerful interaction methods.
    For example, the toolkit provides a tool that measures its own tracked position, and an application could provide a tool that projects a source input device's position onto an application-defined surface (such as a topography model or polygon mesh). When combined, these two independent tools together create new functionality, namely measuring positions on the application-defined surface.
  • Application developers need only implement a small set of application-specific tool classes (such as surface projection), and enable users to use the large set of toolkit-provided tools in an application-specific fashion. This fosters a consistent "look&feel" because application developers do not have to implement their own versions of common tools such as navigation, measurement, or annotation.
  • The tool / input device graph can be manipulated by the user during runtime using a simple interface. While Vrui starts with a default input graph specific to each environment, users can completely change the way they interact with a Vrui applications from inside the application.
    For example, in environments with few available buttons, users can assign application functionality to buttons sequentially as needed, whereas in environments with many buttons, they can assign everything at once, and then use all functionality without having to switch contexts. Or, users can assign tools implementing different navigation metaphors according to personal preference, or as needed by particular tasks.