Documentation/Maemo 5 Developer Guide/Using Multimedia Components/Introduction
The following code examples are used in this chapter:
Maemo offers lots of possibilities for multimedia applications. Maemo was built with the Internet Tablet devices in mind - hardware in this form factor provides a big high resolution touch screen, audio input and output, video and image capturing through the camera, fast network connections for streaming and a digital signal processor for efficient audio and video manipulation.
For developers, there are open programming interfaces to make use of these features apart from directly programming the DSP. The preferred way of doing video and audio programming for Maemo is using the GStreamer framework. There are several libraries available for manipulating images. Interactive multimedia for applications like games can be done using the SDL framework.
Contents |
GStreamer - Multimedia Framework
The main multimedia framework in Maemo is GStreamer. It is based on the concept of pipelines, which are made of multiple elements. Elements themselves can be practically anything that do something with a data stream.
Maemo includes a variety of these elements to support audio and video effects, encoding and decoding and interfacing with the device's hardware. GStreamer also provides high level elements for specific use cases, like playbin2 for playback and camerabin for image and video capture. Using this approach, a developer does not have to bother with details like low-level hardware management, audio and video compression schemes, etc. If the elements included with the OS are not enough, more can be either compiled for Maemo, or new ones can be developed.
Developer documentation for GStreamer can be found at the project's website.
MAFW
Media Application Framework (MAFW) is a middleware layer for all mediaplayer applications. It provides generic APIs for browsing and searching multimedia content (music, video, images) from various services (e.g. local filesystem, UPnP, etc.) and an extensible plugin architecture which allows providing new content sources via new service plugins. In addition to content metadata access, MAFW provides also generic APIs for playback of media (music, video, images) via various rendering services (e.g. gstreamer, UPnP, etc.). Also new rendering services can be added via a plugin mechanism. MAFW allows 3rd party developers flexible ways to implement new content services support for the Maemo platform.
Stream Encoding and Decoding
Maemo devices include a wide arsenal of video and audio codecs, which enable the Maemo applications to read and write nearly all commonly used video and audio formats. These are supported also by the GStreamer framework. For many purposes the developer does not even have to specify the used codecs implicitly, since GStreamer automatically detects the format and the codec to use. The playbin2 GStreamer base plug-in provides a convenient abstraction layer for all audio and video content.
Due to non-technical reasons, most of the codecs are not distributed with the SDK. This is good to keep in mind when developing applications relying on such features.
Digital Signal Processor
Inside the Maemo device, there is a dedicated digital signal processor (DSP). Its design is optimized for tasks like stream encoding/decoding and audio effects. Maemo has high-level programming support for the DSP in the form of GStreamer elements, which can decode most of the supported file formats. By using the DSP, the computing load is also taken off from the main processor, greatly enhancing system performance and responsiveness.
Audio
For audio programming, Maemo has several main APIs: GStreamer and PulseAudio. Usually system sounds, such as sounds for notifying user of an event, e.g. battery low, are played through PulseAudio. One can use libcanbera if a simple API is needed. More sophisticated operations, e.g. playing music files or recording audio, should be generally performed using GStreamer, which a much more flexible API. Most of Maemo's computationally intensive GStreamer elements are implemented using the device's DSP which greatly enhances their performance.
The Linux kernel has also two lower-level audio interfaces: ALSA and OSS. Of these, ALSA is supported. The legacy API OSS is not supported by the kernel, but ALSA has an OSS emulation system that works for most purposes.
For the audio APIs' documentation, see the GStreamer web site, PulseAudio web site and ALSA project's web site.
Video
Although the framework hides much of the implementation and hardware details, it is good for a developer to know what happens beneath the interfaces. Video capturing is performed via the Linux kernel's Video4Linux2 API, and graphics are displayed using the X Window System, namely the XVideo extension. Practically all GNU/Linux applications rely on these components for video tasks, so porting existing applications should be quite effortless, and support easy to find.
Hands-on instructions for using capture and output features are given in the article on camera API usage.
Graphics and Images
Maemo includes several libraries for working with images and graphics:
- Cairo is a library for making 2D vector graphics. It features, for example, high quality vector-based graphics, support for importing and exporting a variety of formats, such as SVG, PNG, PDF and Postscript, and bindings for many programming languages
- GDK and GdkPixbuf are the bitmap graphics libraries that GTK+ is built on. Together, they provide opening and saving for bitmap images in JPEG, PNG, TIFF, ICO and BMP formats, tight integration into GTK+ and tools for drawing with graphic primitives. Also loading of SVG images is supported
- For more detailed control over the different image formats, there are many more specialized libraries, e.g. libpng, libjpeg and libwmf.
Visit Cairo website for guides, tutorials and references. GDK's and GdkPixbuf's documentation can be found at GTK+ project's website.
Games
For game developers, maemo offers a common start-up and settings screen, and a framework called osso-games-startup. Using this library, game developers and porters can reduce the common start-up screen coding, and concentrate on the real game programming. Osso-games-startup's operating system communication features also offer an interface to make games behave correctly. It eases, for example, the management of full screen mode and exceptional situations, such as battery low notification, in a unified and user friendly manner. See the article on games start-up screen usage for more in-depth description.