Documentation

The jReality scene graph

This document is designed to be used in conjunction with the jReality Javadoc pages , where the available Javadoc documentation for the classes mentioned here can (and should) be found.

Viewer interface

The jReality package provides viewers which can render 3D scene descriptions given in the form of scene graphs (directed acyclic graphs). The viewer capability is encapsulated in the Viewer interface. The most important methods here are

• setSceneRoot(), which sets the scene graph root to be rendered,
• setCameraPath() which describes the position in the scene graph, of the camera to use to render it, and
• render() which renders an image.

A class which implements the Viewer interface is generically called a backend. Depending on the rendering requirements, one can attach several different backends to the same scene graph to produce different sorts of images of the same scene. There are currently two interactive backends, one based on jogl and a software backend written completely in Java. Additionally, one can choose from variety of non-realtime backends for higher-resolution rendering.

Since working directly with backends can be time-consuming, jReality comes with a plugin system for assembling applications from reusable components. See this tutorial for an introduction to jReality plugins.

SceneGraphNode

The jReality scene graph is based on a relatively small number of classes, most of which inherit from the abstract superclass SceneGraphNode. The only subclass which can have children is SceneGraphComponent. The leaves of the scene graph include the following classes:

• Transformation: describes where
• Appearance: describes how
• Geometry: describes what (graphics)
• Audio source: describes what (sound)
• Light
• Camera

The first four classes are the most common; the final two are more specialized. An instance of SceneGraphComponent can have one each of the above nodes, in addition to a list of children (called proper children) which are also SceneGraphComponents. All these proper children plus any of the 5 fields present in a SceneGraphComponent are collectively known as the children of the component. To specify only the proper children, we say so!

In order to allow backends to optimize their performance by caching state, all the subclasses of SceneGraphNode broadcast change events to registered listeners.

A SceneGraphComponent also contains a possibly empty list of tools (implements the Tool interface). Tools are discussed in more detail below.

Transformation

The transformation is essentially given as a real 4×4 matrix. This matrix describes the projective transformation which converts from the local coordinate system into the parent coordinate system. This description includes euclidean as well as non-euclidean isometries, all of which are supported as equal citizens in the jReality package. (See below, “Metric neutrality”). Of course other sorts of matrices can also be applied, such as skews, scales, or perspectivities. Each backend maintains some sort of matrix stack where these transformations are appropriately concatenated during traversal. JReality includes a wide variety of classes to assist in the generation of 4×4 matrices. See the Matrix, MatrixBuilder, and P3 classes.

Appearance

The Appearance class itself is simple: essentially nothing more than a list of attributes, each consisting of a key-value pair, where the key is a String instance, and the value is some Object. But the rendering of appearances is the most complicated part of the jReality rendering process. It’s described in more detail below in the section on shading. Note: The class CommonAttributes contains definitions of the most commonly used appearance attribute keys, arranged according to their usage pattern.

Geometry

The abstract class Geometry includes two sorts of subclassess. First are surface based primitives such as Sphere and Cylinder, which have no points or lines, so only the polygon shader is invoked (see below). Secondly, are point-based primitives including PointSet, its subclass IndexedLineSet, and its subclass IndexedFaceSet. These point-based classes allow users to construct arbitrary simplicial 2-complexes.

Point, line, and face sets

These classes are based, like appearances, on attribute sets organized as a list of key/value pairs. Again the keys are strings, such as “coordinates”, “normals”, “colors”, “textureCoordinates”, etc. (See the Attribute class for all predefined attributes here). Conceptually, everything can be explained using the IndexedFaceSet class. It has three sets of attributes, one each for vertices, edges, and faces (don’t ask me how we decide when to use these terms instead of points, lines, and polygons!). If it has no face attributes, then it’s really an IndexedLineSet; if it additionally has no edge attributes, then it’s really a PointSet.

The values of the attributes are instances of class DataList. This is a very clever (but opaque) class which allows a variety of array formats to be used to represent the data. For example, if I have a list of 20 points, each of which is a 3-vector, I can use double [20][3] or double [60] to represent this data; the DataList handles the access, etc.

Factories

Partly because DataList’s are scary, jReality also provides a wide set of geometry factories which can be used without having to refer to the DataList class. These factories include PointSetFactory, IndexedLineSetFactory, IndexedFaceSetFactory, QuadMeshFactory, ParametricSurfaceFactory, BallAndStickFactory, and TubeFactory. All these factories function according the factory pattern: many set…() methods, a single update() method after a series of set calls, and finally a getGeometry() method to get the geometry instance which the factory produces. This geometry may subsequently be updated/changed using the same factory.

A final subclass of Geometry is ClippingPlane. A clipping plane does not cause any geometry to be rendered. It can be either global, or local to the scene graph where is occurs.

There are many static methods available in jReality for producing standard types of geometric primitives without having to explicitly use factories. See the classes Primitives, , PointSetUtliity, IndexedLineSetUtility, IndexedFaceSetUtility, QuadSetUtility.

Audio Sources

These paragraphs are merely an overview of the basic audio architecture. Please consult the Developer Tutorial for more detailed information. The user tutorial contains detailed information on how to run jReality applications with audio.

The core element of the audio architecture is a scene graph node called audio source that can be attached to a scene graph component just like a geometry or a light.

The responsibilities of an audio source are limited to maintaining a circular sample buffer, writing samples upon request, and handing out readers for its circular buffer to consumers of samples. A reader first fulfills sample requests with samples that are already in the circular buffer; when the reader reaches the current end of the buffer, it requests more samples from the audio source.

The audio source keeps track of time in terms of the number of samples requested so far. An audio source generates a mono signal at the sample rate of its choosing; sample rate conversion, spatialization, and multi-channel processing are the responsibility of the backend.

The deliberately simple nature of audio sources is intended to encourage developers to implement audio sources that draw their samples from a wide variety of inputs.

An audio backend traverses the scene graph, picking up a sample reader and transformation matrices for each occurrence of an audio source in the graph. For each occurrence of an audio source, it creates a {\em sound path} that is responsible for modeling the propagation of sound emitted by the source. It also keeps track of a microphone in the scene.

SceneGraphPath

Any instance of SceneGraphNode can appear at multiple places in the scene graph. In this way, geometry, appearances, and transformations can be repeatedly applied to different parts of the scene graph. Of course SceneGraphComponent instances cannot appear as descendents of themselves! But otherwise reuse is everywhere allowed.

To distinguish between different appearances of the same node in the scene graph, the class SceneGraphPath describes a path in the scene graph from the root to a final element. For the purposes of this discussion, all the elements of this list but the last must be SceneGraphComponents (such that each successive one in the child of the preceding one), while the final element must be a subclass of SceneGraphNode.

For example, the setCameraPath() method on Viewer takes such a SceneGraphPath as its argument, whose final element is a Camera instance; otherwise if it took a Camera instance alone it would not know which of possibly many occurrences of the Camera is intended. Similar considerations apply to many situations when dealing with the scene graph, for example, picking of geometric objects specifies its results using such paths.

Metric neutrality

The jReality scene graph is metric neutral. That is, it supports not only euclidean geometry but also hyperbolic and elliptic/spherical geometry, since all three of these geometries can be modeled on three dimensional projective geometry. The group of projective transformations is generated by the euclidean transformations (a subgroup) plus the perspective transformation, so modern rendering software and hardware is all based on the projective group, and the non-euclidean geometries can then be supported with little or no change in the basic code.

The choice of metric is sometimes called the signature, since it has to do with the signature of a certain quadratic form. The signature is propagated through the scene graph using appearance attributes (see section below on appearances). There are several jReality classes which are particularly useful for the care and feeding of the metric neutrality, for example, P2, P3, and Pn.

Homogeneous coordinates

All classes and methods in jReality are designed to handle coordinate data in either 3- or 4-dimensional form. It is generally well known that euclidean matrices can be written as 4×4 matrices which act on 3-dimensional points (x,y,z) by extending these to (x,y,z,1) and then performing a standard matrix-vector multiplication. These 4D coordinates are sometimes called homogeneous coordinates. For more on homogeneous coordinates, consult references on projective geometry. While not strictly necessary for euclidean setting, they are very useful (for example in rational b-splines); they are necessary for the support of non-euclidean geometry (especially elliptic geometry, which cannot be represented with a single non-homogeneous atlas).

Rendering process

When rendering the scene, a backend will first typically handle the camera which belongs to its camera path. Next, the scene graph is traversed and only the global lights are collected. jReality supports directional, point, and spot lights. A similar process occurs for global clipping planes. Then, the actual rendering traversal takes place. Leaf nodes are processed before the proper children are rendered. Among the leaf nodes, the geometry is rendered last, after the appearance and transformations. Here’s how the leaf nodes are rendered.

Shading

The Appearance class is used to specify all attributes for the shading process. Shading occurs when a geometry node is encountered in a scene graph component. In the simplest case, if there is only one appearance in the scene graph and it occurs in the same component where the geometry is, then the values of the shaders applied to this geometry, will be determined completely by the values of this appearance’s attributes, plus the default values specified by the shader interfaces themselves. This however is not typical; usually there will be more than one appearance which can influence the values taken by the shaders, due to the inheritance properties of the jReality scene graph.

Inheritance

Suppose the geometry is specified by a scene graph path, and suppose that on this path, there are several appearances. Then the values of the shaders is determined by the appearance attributes which are defined in the scene graph components appearing in this path. A shader is provided with a list of these appearances, sorted with the closest (along the path) appearances first. Such a list is represented by an instance of EffectiveAppearance.

In the simplest case, the shader queries this EffectiveAppearance for the value of a given key, for example, “diffuseColor”. The list of appearances is searched for the first occurrence of this key in the list of appearances. If a first occurrence is found, then the value of this occurrence is returned to the shader; otherwise a default value assigned to this key is returned.

Things are a little more complicated if the key contains the . character (period). This serves as a delimiter. Suppose the key is such a string: “polygonShader.diffuseColor”. First the list of appearances is searched for any occurrences of the complete string. If none is found, the next-to-last substring (defined by occurrences of .) is removed, along with one ., and the search if repeated for this shorter string, in this case, “diffuseColor”. The default value is returned only if all such searches find no match. This allows for a powerful (and sometimes confusing!) form of inheritance of attributes.

See also the class EffectiveAppearance, which encapsulates the inheritance mechanism described above.

Shaders

The shaders which appear in the jReality rendering process are defined as follows. First of all, all shaders are defined by interfaces. These interfaces are then used to automatically construct classes which fill their fields by querying the appearance lists as described above.

At the top level, every shading operation begins with an instance of GeometryShader, which in turns contains three sub-shaders: point, line, and polygon shaders, plus flags indicating whether points, lines, and polygons, respectively, are to be rendered. This allows for differentiated rendering of a single geometric primitive without having to repeat the geometry in order to render it as polygons, as lines, and as points. The jReality rendering process renders all three aspects of the same geometry, as desired.

Which shader interfaces provide the point, line, and polygon shaders can also be specified by appearance attributes. In general, for most generic rendering tasks, the default shaders are adequate. The attribute pair (“polygonShadername”, “default”) specifies this default polygon shader; other supported values include “constant”, “twoSide”, and “implode”.

Default shaders — notes

The default point shader allows the choice of rendering a 2D disk whose size is specified by the “pointShader.pointSize” attribute, or rendering spheres of a radius given by “pointShader.pointRadius” around each point. Typically the spheres look better, but the disks can be considerably smaller. In the JOGL backend, the 2D disks look like spheres (since they are represented by texture-mapped sprites) but they show artifacts when they overlap with real 3D geometry. If spheres are drawn, use the attribute prefix “pointShader.polygonShader.” to specify attributes for rendering them. Warning: since the attributes “pointSize” and “pointRadius” live in different coordinate systems (screen vs. object), switching between these two options can result in wide variation in the apparent size of the disks/spheres. If you think of a good solution, let us know. (This problem is limited to the JOGL backend.)

The default line shader, in a similar vein, allows the choice of rendering a flat bresenham representation of the line segment with a width specified by the “lineShader.lineWidth” attribute, or rendering tubes of a radius specified by the “lineShader.tubeRadius” attribute. If tubes are drawn, use the attribute prefix “lineShader.polygonShader.” to specify attributes for rendering them.

The default polygon shader implements a standard shader with ambient, diffuse, and specular contributions, plus an optional 2d texture (see Texture2D interface) which is active only if the geometry comes with texture coordinates. Play around with the ViewerApp navigator for the full set of shader attributes available. Additionally one can specify “smoothShading” on or off. The default is on. In this case, shading values at the vertices are smoothly interpolated across faces. If vertex normals are present they are used, as well as vertex colors. If smooth shading is off, then face normals and face colors are used, and the values are not interpolated across the face. For a nice effect for polyhedra (using JOGL backend), provide only face normals and face colors but turn on smooth shading. Then specular highlights look much closer to Phong shading than with traditional flat shading. Disclaimer: Your mileage with this trick may vary using other backends.

Warning: there may be difficulties with some backends when non-default (or default!) shaders are used, since there is no enforcement mechanism to make sure a given backend implements a given shader interface. If it doesn’t, it will typically use the default one instead.

Text shaders

jReality supports 3D text. Each of the standard point, line and polygon shaders contains a text shader. If the corresponding attribute named “labels” is present, then this shader is used to render 3D labels at the corresponding point, edge midpoint, or face center. See DefaultTextShader for possible parameters. The geometry factories can be used to generate automatic labels.

Picking

jReality provides picking infrastructure in the form of the PickSystem interface. There is an implementation called BruteForcePicking which does what its name implies. There’s also support for something called AABBPickTree’s. Brute force picking has recently been upgraded to support non-euclidean picking. Essentially picking is a projective task, but certain details about the ordering of the pick results along the pick line can be quite sensitive to non-euclidean isometries. As a result, the pick results which are returned from the computePick() method are sorted according to an affine coordinate along the pick line which runs from 0 to infinity through the positive values. This also avoids problems arising when geometries of different signature are present in the same scene, and the different distance functions are incompatible for sorting.

As with rendering, there is separate control over picking of points, lines, and polygons. This is done by setting the attributes “point.pickable”, “line.pickable”, and “polygon.pickable” in the appearance. These are inherited as described above with shader attributes. Or, to completely skip over a component, set the method (on SceneGraphComponent) setPickable(false). If points or lines are to be picked, then the sphere, resp. tube, representation is used to perform the picking (though the rendering may be done with the other representation — a not entirely correct result). Furthermore the picked point of the sphere or tube is returned rather that the point, resp. closest point on the edge. (We hope to provide methods to return the exact point on demand).

The PickResult class provides the information about the picked point. When the pick is on a face, barycentric coordinates of the pick point in a triangle are calculated and used to interpolate the attributes, such as object coordinates, color, and texture coordinates.

The jReality Tool system

Just as the Viewer interface provides an abstract description for the different backends, the Tool interface, and related classes, provide an abstract description of the different environments in which interaction can occur. This allows the same tool code to be used with a mouse device on a desk top or a wand device in a virtual reality, like the PORTAL installation of our group at TU-Berlin. There is a single configuration file describing the actual hardware devices in the environment; these devices are mapped on common virtual devices which then are the only objects needed to drive the actual tools. For example, the inputs of the mouse device and the wand device are both processed to create a virtual device called a “PointerTransformation” which is then accessed by a concrete jReality tool class.

A Tool is an object which is attached to the jreality scene graph in oder to manipulate it, the manipulation usually depends on user input, or time for animation tasks.

Devices

User input can come for instance from mouse, keyboard, joysticks, wand, space mouse, space navigator, speech recognition software, guesture recognition, tracking systems, ...

The jreality tool system abstracts all available input devices and maps them to a set of virtual devices. These virtual devices are usually available in every setup/environment, no matter if you run jreality on

• a desktop with mouse/keyboard
• a desktop with a joystick or a space navigator or a wii remote
• an immersive virtual environment (CAVE).

AxisStates and DoubleArrays

This set of virtual devices consists of AxisStates and DoubleArrays.

An AxisState represents a floating point value (ranging from -1 to 1), usually from a button or a joystick axis. Buttons are special AxisStates, which take only the values 0 and 1.

A DoubleArray always represents a 4x4 transformation matrix. 2- and 3-vectors are also represented as DoubleArrays, as the transformation's translation part.

InputSlots

Every virtual device is mapped to an InputSlot, which makes it availabe for the tools under a meaningful name.

PointerTransformation

The most important device to understand is the pointer transformation. Interaction with the 3D scene in jreality is usually pointer-based, that means the user can point to some geometry and then interact with it. For instance, if you want to drag a control point of a polygon, you point on that vertex (usually with the mouse pointer), then you press a button and drag the point. In a CAVE the user may have a wand which he uses to drag that vertex, or he might have a SensAble 6DOF input device.

To make this pointer interaction independent of the available devices, the tools will be driven by a virtual Pointer device, mapped to the input slot PointerTransformation. This transformation has the following meaning:

• translation part: the point in the 3D scene, where the pointer starts
• z-axis: the pointer direction in the 3D scene
• x- and y-axis: form together with the z-axis an ONB

From a mouse pointer, this transformation is obtained as follows:

• translation part: the mouse point on the near-clipping plane in scene coordinates
• z-axis: the direction in which the mouse pointer points
• x- and y-axis: computed in such a way that they are as close as possible to the screen coordinate system in scene coordinates

Tools

Conceptually, we destinguish two sorts of tools, that are:

• Pointer Tools: Tools that perform Pointer-based interaction (like dragging, rotating)
• Always-active Tools: Tools that perform interaction that is not associated to pointing at something (like navigation, animation)

Therefore, a tool defines a list of activation slots. This list may be empty, then we have an always-active tool.

Pointer tools are only activated when the user points to some geometry in the scene with the pointer device.

For the interaction with an active tool it defines also another list of slots, the current slots.

The list of activation slots of a tool is fixed, the list of current slots may be changed as desired. For instance, a tool may require the SystemTime slot for doing some animation, and when the animation is finished it removes this slot from its current slot list.

Activation slots always need to correspond to an AxisState. The tool becomes active as soon as one of its activation slots changes its state to AxisState.PRESSED. It gets deactivated again as soon as none of its activation slots corresponds to an axis state with value AxisState.PRESSED.

Current slots can be both AxisStates and DoubleArrays. Whenever the value corresponding to a current slot changes, the tool will be notified (but only if the tool is active).

The basic API of a tool are the three methods

• activate(ToolContext context): called as soon as the tool gets activated
• perform(ToolContext context): called when a tool is active and the value of one of the current slots has changed
• deactivate(ToolContext context): called as soon as the tool gets deactivated

Tool Context

The ToolContext gives access to all information a tool may require. It allows querying the states of all InputSlots, it provides information about the current pick result and gives the SceneGraphPaths a tool may require.

The ToolContext also gives access to the location of the tool inside the scene graph. Since the same tool may occur in different places of the scene, it is necessary to obtain the information from the ToolContext. The corresponding information is always a SceneGraphPath, starting at the scene root. There are always two paths, one is to the component where the tool is located, the other one to the component which was picked for activation (for always active tools both paths are the same). The code is

// path to the picked component at activation:
SceneGraphPath pathToActivationComponent = context.getRootToLocal();
// path to the tool's component
SceneGraphPath pathToTool = context.getRootToToolComponent();

Since the root to local path corresponds to the activation path, it will not change during successive calls of the perform method.

Audio

We present here an overview of the audio architecture of jReality, based in part on Spatialisation - Stereo and Ambisonic by Richard W.E. Furse. Details can be found in other tutorials.

The jReality audio rendering pipeline

The audio rendering pipeline starts with a subclass of AudioSource, which only needs to write mono samples to a circular buffer upon request, at a sample rate of its choosing. The audio backend collects all audio sources in the scene graph and requests samples from them when necessary. Audio sources are designed in such a way that a single audio source can appear in several places in the scene graph, and multiple backends can operate on the scene graph at the same time.

The audio backend creates one rendering pipeline for each occurrence of each audio source. In particular, all the optional processing described below (preprocessing, gain control, distance cues, etc.) can be chosen for each occurrence individually. They are defined through appearances, so that they can be changed on the fly. The class AudioAttributes lists the tags under which those options are listed in appearances. For an example of how to use them, see AudioOptions.

The audio backend will implicitly convert any input samples to the sample rate of the output device. After the sample rate conversion, the user may insert optional effects, such as early reflections or pitch shifting. Such effects are implemented as subclasses of AudioProcessor. Multiple effects can be plugged in a chain of processors, implemented in the convenience class AudioProcessorChain.

After the initial processing, the audio signal is multiplied by a gain factor and the first (optional) distance cues are applied. A typical application of the first round of distance cues is distance-dependent low-pass filtering, making distant sound sources sound duller than nearby ones.

Now the signal is fed through a distance-dependent delay line, giving rise to physically accurate and subjectively convincing Doppler effects. The speed of sound defaults to 340m/s, but it can be set to arbitrary values through appearances. A speed of sound of zero or less is interpreted as infinite speed of sound, i.e., sound propagation is instantaneous in this case and Doppler effects do not occur.

Now the audio signal is split into two streams. One stream is sent through another optional distance cue (a typical example of a second distance cue is distance-dependent attenuation) and then sent on to an encoder that handles spatialization. In the simplest case, this encoder will compute a stereo signal, based on the relative position of source and microphone. In a 3D audiovisual environment, the encoder will compute an Ambisonics B-signal.

The second stream is intended for directionless processing. It is multiplied by a second gain factor and then fed through an optional effect (e.g., reverberation) that is then rendered without any spatial encoding. For an explanation of this design (directed and directionless stream, two sets of distance cues applied at different times), see Furse's paper cited above.

Finally, the directed and the directionless signal are added up and sent to the output device. In the case of stereo encoding, the output device may already be the physical output of the soundcard. In the case of Ambisonics encoding, the output device will be an Ambisonics decoder configured for the local speaker rig.

Audio Sources

All audio sources in jReality are subclasses of AudioSource. In practice, virtually all audio sources will be subclasses of RingBufferSource, a subclass of AudioSource that maintains a circular sample buffer and takes care of everything except the actual writing of samples to the sample buffer.

The responsibilities of an audio source are limited to maintaining a buffer for mono samples, writing samples upon request, and handing out readers for its sample buffer to consumers of samples. A reader is an instance of SampleReader. A reader first fulfills sample requests with samples that are already in the buffer; when the reader reaches the current end of the buffer, it requests more samples from the audio source. Several readers must be able to work concurrently in a thread-safe manner. If you derive your audio source from RingBufferSource, then all this is already taken care of and you can focus on the main job of an audio source, i.e., computing audio samples.

When asked to render n samples, an audio source may choose to render more than n samples. For instance, if an audio source wraps a software synthesizer, it may be convenient to render a multiple of the buffer size of the synthesizer. Our approach implicitly reconciles buffer sizes across audio sources and output devices.

The audio source keeps track of time in terms of the number of samples requested so far. An audio source generates a mono signal at the sample rate of its choosing; sample rate conversion, spatialization, and multi-channel processing are the responsibility of the backend (see The jReality audio rendering pipeline).

The deliberately simple nature of audio sources is intended to encourage developers to implement audio sources that draw their samples from a wide variety of inputs.

In order to get started, you may want to look at the SampleBufferAudioSource or SynthSource and see how they work. Both are quite short, just a few dozen lines, with all the boilerplate taken care of by RingBufferSource.

Odds and ends

The jReality audio rendering pipeline lets you patch in two types of objects, processors and distance cues. Processors are intended to handle an entire buffer's worth of samples at once, and so they are used for the initial preprocessing as well as the final directionless processing of samples. Distance cues are intended for distance-dependent processing and handle one sample (along with individual location information for this sample) at a time.

Audio processors

An audio processor is initialized with an instance of SampleReader from which it reads its input. The AudioProcessor extends the SampleReader interface, so that audio processors can be chained, and the convenience class AudioProcessorChain simplifies this.

In order to get an idea how to write an audio processor, take a look at LowPassProcessor.

Distance cues

Inplementations of the DistanceCue interface are intended to handle individual samples one at a time, along with location information. Like audio processors, distance cues are intended to be chained, and the convenience class DistanceCueChain facilitates this.

The parameters of the nextValue method seem redundant (they include a distance r between microphone and audio source, as well as x, y, and z coordinates that indicate the position of the microphone relative to the source), but they are not because r is given in microphone coordinates, while x, y, and z are given in source coordinates. Depending on the transformation matrices between source and microphone, r may not be given by a simple function of x, y, and z.

The DistanceCue interface itself includes a number of sample implementations. The LowPassFilter class is a slightly more complex example.