Simulation Engines TDA571|DIT030 Multimedia and scenegraphs - PowerPoint PPT Presentation

Simulation Engines TDA571|DIT030 Multimedia and scenegraphs Tommaso Piazza 1

Administrative stuff  Student Representatives  People with orange or red extension IDC | Interaction Design Collegium 2

Introduction  Multimedia is everything about the external appearance of the game that is not directly related to the real-time rendering  More specifically  Music  MP3 and Ogg audio formats  Contextual event-driven music  Sound effects  File formats  3D sound  Environmental audio  Video  Cutscenes (using video or in-game rendering and scripting)  Graphical User Interface (GUI) (Covered in another lecture) IDC | Interaction Design Collegium 3

Example: Data screens in Doom3  Doom3 uses a lot of interesting multimedia  In-game computer screens are interactive and show animations that help progress the game  Cutscenes are done in-game with the engine IDC | Interaction Design Collegium 4

Fundamentals of digital audio  Audio is a pressure wave in the air  Digitally represented by discrete samples  Each sample represents the amplitude of the wave at a given time  The frequency of samples  Sample rate  The sample size  8 or 16 bit mostly  Number of channels  1 for mono, 2 for stereo IDC | Interaction Design Collegium 5

Sample rates  Telephone quality  11kHz, 1 channel  Radio quality  22kHz, 1 or 2 channels  CD quality  44kHz, 2 channels  Studio quality  > 48kHz, 24 bits, up to 24 tracks  Human ear is sensitive up to 20kHz, human voice has a spectrum of ~4kHz  Nyquist-Shannon theorem IDC | Interaction Design Collegium 6

Windows WaveOut Standard component on all flavors of Windows   Provides basic functionality for playing raw sound data  Easy to use and works on much hardware  Corresponding WaveIn also part of the WaveForm API How to use  1. Open the desired sound device with waveOutOpen() 2. Allocate a memory buffer and a WAVEHDR structure 3. Call waveOutPrepareHeader() to prepare for playback 4. Fill in the buffer with sound data 5. Call waveOutWrite() to play it 6. Call waveOutReset() to force the device to relase all buffers 7. Call waveOutUnprepareHeader to release resources 8. Deallocate the buffer memory 9. Calls waveOutClose to release the sound device Typically not used in games except for basic fallback   Slow, not optimized for games and restricted to the Windows platform IDC | Interaction Design Collegium 7

3D and environmental sound  Throughout the years sound playback has been improved with support for 3D sound (positioning)  Environmental sound (effects that emulate the size, shape and  properties of the surrounding area)  Sources Physical sound sources. Includes information about 3D position,  velocity and direction  Listeners Global listeners instance. Represents the 3D position and velocity of  the player receiving the sound  Environment Information about the properties of the environment , including its size,  shape and other physical attributes. Often tied to the listener IDC | Interaction Design Collegium 8

3D sound – Speaker configuration  3D sound must take in considerations the hardware configuration of the listener  There are many standards setups and formats  Headphones  Stereo systems  Dolby Digital 5.1  DTS  ...  Usually this is abstracted away by the sound API IDC | Interaction Design Collegium 9

Creative EAX Creative's API for environmental audio   One of the first with commercial support Handles environmental effects   3D is handed off to DirectSound, OpenAL, etc 3.0 was called EAX Advanced HD, latest version on the X-fi is 5.0 with 128  voices and up to 4 effects on each voice Multiple environments   Instead of a single set of environment properties, tied to the listener, the later EAXs support concurrent environments that affect sound differently Morphing   Morphing of environmental properties as the listener moves Panning   Advanced control of the movement of sound in a 3D acoustic space Reflection   Functionality for simulating reflections and echoes of sound against surfaces in 3D IDC | Interaction Design Collegium 10

DirectSound  The sound component of the DirectX SDK  Optimized for real-time applications  Better than WaveOut/In but less hardware support  DirectSound3D gives support for 3D positioning and environmental effects  Eats more CPU but can be hardware accelerated  Slowly being replaced by Xaudio and XACT  XP, Vista and the XBOX 360 (part of XNA)  DirectSound on Vista lacks support for hardware acceleration IDC | Interaction Design Collegium 11

OpenAL  Designed to be the “OpenGL of audio”  Provides a standardized API for accessing 3D audio  Uses extensions for adding future functionality (same as OpenGL)  No standardized support for environmental effects, but available with extensions IDC | Interaction Design Collegium 12

OpenAL: Fundamentals  OpenAL uses a simple model of 3D acoustics  Three kinds of entities  Sources  A source of sound in the world. Linked to one or more buffers containing sound data . Also contains information about position and velocity.  Listener  An “ear” in the world representing the player, including position and velocity.  Buffers  Memory buffers for the storage of sound date for playback . Controlled by sources and are never manipulated directly. In many cases they are transferred to the sound card prior to playback and are thus immutable. IDC | Interaction Design Collegium 13

OpenAL: Buffer creating & loading  After initializing OpenAL we need to create our buffers ALuint buffer; alGenBuffers(1, &buffer);  Each buffer is denoted by a handle (integer)  Fill buffers with data with alBufferData(buffer, format, data, size, freq);  OpenAL does not in itself contain loaders for common file formats IDC | Interaction Design Collegium 14

ALUT  Luckily, OpenAL, like OpenGL, contains a simple portable API called ALUT with support for a lot of common tasks ALenum format; ALsizei size, freq; ALboolean loop; Alvoid *data; alutLoadWAVFile(“sound.wav”, &format, &data, &size, &freq, &loop); IDC | Interaction Design Collegium 15

OpenAL: Source creation  To create a source ALuint source; alGenSources(1, &source);  Bind the buffer to a source alSourcei(source, AL_BUFFER, buffers);  Set 3D properties alSourcefv(source, AL_POSITION, sourcePos); alSourcefv(source, AL_VELOCITY, sourceVel); alSourcefv(source, AL_DIRECTION, sourceDir); IDC | Interaction Design Collegium 16

OpenAL: Listeners  Since there can only be one listener, it does not need to be created.  Configure its 3D position alListenerfv(AL_POSITION, listenerPos); alListenerfv(AL_VELOCITY, listenerVel); alListenerfv(AL_DIRECTION, listenerDir);  listenerDir is an array of 6 floats defining both the “forward” and “up” vectors IDC | Interaction Design Collegium 17

OpenAL: Playing sounds  Playing operations alSourcePlay(source); alSourceStop(source); alSourceRewind(source); alSourcePause(source);  To cleanup alDeleteSources(NUM_SOURCES, source); alDeleteBuffers(NUM_BUFFERS, buffer); IDC | Interaction Design Collegium 18

FMOD  FMOD is a cross-platform audio library supporting the same functionality on a wide range of hardware and platforms  Includes support for software sound mixing, 3D audio, music playback, Dolby support, etc  Free for non-commercial use (no sources)  $1000-4000 per platform for commercial use  $100 “shareware/hobbyist” license exists as well IDC | Interaction Design Collegium 19

Music  Sound effects suspend disbelief  Music helps with immersion  Extremely important  Older games used CD audio  Sometimes even for voiceovers (Loom)  Gives little opportunity for processing IDC | Interaction Design Collegium 20

MP3  Developed by the Fraunhofer Institute in 1997  ISO-MPEG Audio Layer 3, MP3  Non-audible frequency principle  Sound sources next to other sound sources with higher intensity are masked. Example: A bird singing next to a running car is hard to hear and can be eliminated  Principle of less important signals  The parts of the sound spectrum less important to fidelity can be eliminated  Commonly used in games, but the format is proprietary and jealously guarded by Fraunhofer, so using it requires a substantial licensing fee IDC | Interaction Design Collegium 21

Ogg Vorbis  Open source equivalent to MP3  Freely available  Arguably better quality sound than MP3 on equivalent bitrates  There are free libraries available for using with Ogg Vorbis IDC | Interaction Design Collegium 22

MOD music  Short for “module”  Devised for the Amiga computers  Capable of playing four simultaneous sounds  Stores both samples and the actual melodies  Generally smaller than MP3/OGG  Relatively easy to manipulate  Change pitch  Change tempo  Inject tunes IDC | Interaction Design Collegium 23