The tinySG Streaming Server

Friends of tinySG,

tinyScenegraph comes with a streaming server implementation that allows to take tinySG basically everywhere, easily: Onto a Tablet or smart phone, the web, into a browser or just another remote machine, without the need of a remote desktop solution like vnc or even a real virtual machine, like remoteFX, parallels or vmware ESX.
tinySG scenes may be managed on a server by the tsgAsset Manager , searching for a scene or browsing available datasets can be done with a regular web browser. A simple link to a java applet or a helper app can take the user to a real-time and interactive 3D renderer for a selected scene.
In contrast to a WebGL implementation, a streaming solution avoids transferring huge scene data to the client on every viewer session. It neither requires a WebGL implementation, nor much processing power on the client, because all rendering is done on the server.

The Streaming Server

Overview

tsgStreamer is based on Qt network and communicates with it's clients via TCP socket connections. It listens on a well-known port and spawns threads for each client connecting. For stability reasons, this design is not suitable for high availability and would need to change to use different processes per client so a crash would not affect other clients.
The server loads all plugins and scenes on startup, as configured per .ini file. Although this requires quite a lot memory, it avoids keeping clients waiting while loading large scenes.
The tsgServer collaborates with different renderers via a plugin model.

The Streaming Server	The tinySG Software Stack

tsgStreamer plugins

File loaders and scene renderers are implemented in shared libraries and loaded as plugins when tsgStreamer starts. This way, the server is not limited to streaming tinySG scenes, but could also use other file formats and render back-ends. In fact, a proof of concept has been implemented for the RTT renderer and tinyRT in the past.
Today, there are two plugins:

The tinySG plugin supports the .csg, .iv, .vrml and .collada file formats and uses tinySG as it's renderer.
The Remote Desktop plugin captures the entire desktop and sends

is just one plugin using tinySG as it's renderer and associated loaders for . However, a prototype of the RTT renderer and csb-loader has been demonstrated in the past as well.

Network Protocol

The communication between clients and server is based on the client sending commands and the server sending responses. Basic commands are build into the server, but client implementations may define protocol extensions that are handled by specific plugins.

The protocol is still experimental and subject to changes all the time. For now, few commands are build in to select/change a scene for rendering, update the 3D camera position and orientation, and retrieve a frame based on the current camera settings. The server basically acts as a state machine, with state being updated by the command strings.
For example, the LIST command requests a list or available scene IDs, the SCENE,<ID> command directs the server to apply render and navigation requests to the scene with the given ID from now on.
The client can choose between different formats for encoding images sent by the server. This allows to balance the trade off between image quality, bandwidth requirements and processing power available on the client.
The current implementation supports just raw rgb data and png or jpeg images, but it would also be possible to use more efficient codecs, like mpeg or H.264.
To allow both client and server to work asynchronously to network transfers, the protocol tags each command with an ID. This way, a reply can refer to the associated request, allowing for a handler-implementation that reacts on incoming packets, rather than handshaking each command.

Configuration File, On-demand Streaming

The configuration file contains of three sections

Global settings: Network parameters, default settings, number of plugins and scenes.
Plugins/Modules: Each module is implemented as a shared library and may provide several loaders for different file formats and renderers.
Scenes: Scene data is composed of the main scene file name and hints what loaders and renderers to use.

This is what a simple configuration file may look like:

	[ Network ]
	tcpPort = 2970

	[ Global ]
	enableAccessLog = 1
	imgMode = raw
	numModules = 1
	numScenes = 1

	[ Module_1 ]
	loaders = "csg,iv"
	module = tsgStreamerModule.dll
	renderers = csg

	[ Scene_1 ]
	ID = 1
	description = "cubes.html"
	file = cubes.csg
	loader = csg
	meta = "tsg"
	preferedHeight = 240
	preferedStreamer = raw
	preferedWidth = 320
	renderer = csg

Streaming Clients

Since the communication protocol relies on just sending ASCII commands, clients can be implemented easily on any device that supports TCP/IP networking. The oldest client implementation is a prototype Java Applet that embeds an interactive 3D viewer in any web page. The plugin receives all connectivity data as it's parameters:

  ...
  <applet code=csgApplet.class width="644" height="580">
	<PARAM NAME="port" VALUE="2970">
	<PARAM NAME="scene" VALUE="0">
	<PARAM NAME="prefStreamer" VALUE="png">
	<PARAM NAME="prefWidth" VALUE="640">
	<PARAM NAME="prefHeight" VALUE="480">
  </applet>
  ...

The Android client is an app that either runs on it's own or plugs into a browser as a helper application. The latter will receive the address and port of the server and connect to a given scene to start streaming.
To aid easy development and debugging, there is also a native C++ client that implements a remote viewer as a MDI desktop application, running on Windows and Linux.


The Streamer Applet, run inside FireFox	The C++ MDI Client, running on Windows 7	The Android Client App

Notes on Performance

The user experience of a remote rendering solution on the client depends on just few basic parameters:

Usability: Ideally, the rendered scene is just one click away. Thus, the streamer collaborates with the tsgAsset Manager:
Latency/interactivity:
Framerate: Obviously, the more frames are presented per second the

[insert a row with charts: x=resolution, y=network (gigE, 100MBit, Wireless, UMTS) framerate/resolution, framerate/network speed]

scene	Resolution	Network	Performance (fps)

	320x240
	640x480
	1280x960
	1920x1200

Keep rendering,
Christian

Acknowledgments:

Java.
Android
Datasets in screenshots