Nicholas Gaffney and Maja Kuzmanovic Paper for the International Conference on Media Futures, May 2001

ABSTRACT

This paper discusses the need for a shift of focus from multimedia to 'intermedia' and has two primary threads:

• In Multiplex Translations we suggest that current computer based systems which separate different media, and reinforce the traditional approach of separating authoring from representation do not meet the needs of developers. We look at the problems and the opportunities of a 'media synesthesia' and discuss properties of an intermedia environment
• Entangled Aphasia analyses the current problem of tools and environments that do not degrade gracefully, aphasic systems unable to remember their past or current conditions. Systems not even able to make basic associations or interconnections between the myriad of content which flows through them. We examine the possibility of viewing the computer as an interwoven mnemonic space, where the techniques of remembering and forgetting information may lead to [re|de]construction of media rich environments, unique to the computer systems and responsive to the needs of the people that use them.

In conclusion, we will propose the necessity of looking at computer supported collaborative tools, rather than focusing on tools for single authors and disassociated media.

Keywords: context, translation, synesthesia, mnemonics, construction, collaboration

In the past few years, the development of digital experiences increasingly stumbles across rigid paradigms used in industry and academia to describe the processes and products of digital art and design. The term 'content' has become the signature of artists, designers and producers working in the digital realm and this term is often applied with various meanings (for example information, media, artworks, meta-language and even advertising fodder). However, the clearly understandable lexical meaning of this word is 'something contained'. If we focus on the widespread push for 'content production for multimedia' we will possibly end up with a collection of linked containers, with their contents never truly mixing. The distinction between the 'container' and the 'contained' might in some cases be an obstacle for true interaction of different media.

Instead of 'content', we propose that 'context' should be the driving force behind the future media productions. We understand 'context' to be the interrelated conditions in which an event occurs, a framework or a setting that actively transforms the objects and actions that it encompasses. Context is not a passive or neutral repository of related content parts, but a generative potential that can bring an 'intermedia' environment into existence. Following this line of thought, the term 'author', often also labeled as 'content provider', should be replaced by the term ‘context provider’; an entity (or group of entities) facilitating the generation of coherent and responsive environments, in which media are not containers, but actuators of interconnected events. In this case, the terms 'origin' and 'creation' (usually attributed to the notion of authorship) are distributed between the facilitators, the entities experiencing the environment as well as a range of computational subsystems, protocols and devices responsible for bringing the environment into being.

If authorship is dispersed, the 'authoring tools' include not only “programs to create content”, but programs that generate, communicate, translate, evolve, build and break media environments based on their recall of linked information. In this model, the relationships between the components of the environment are extremely important, and should be implicitly made by the system, as well as explicitly in the process of creating a ‘context’. The future of media tools does not lie in developing more specialised tools for representation, but in supporting the emergence of tools that enable deep interconnnection, that enhance complex relationships of multiple components or subsystems; and that allow a more generative or evolving media communication, transformation and translation.

“Error during translation: (-3030) Translation path does not exist.”

Why does the translation path not exist? The systems presumed to assist humans in their global and local communications often can't even communicate effectively among each other. They are built upon protocols that tend to block information flow when a particular data format or protocol is not understood. These weaknesses could be an artifact of deeper problems that arise from hierarchical file systems, systems riddled with monolithic applications, or systems which address data by location or name, rather than by content. These systems are based on languages that can contain media, but can not communicate (with) media. They are built upon interfaces that treat the media as defined and static chunks, unable to transform their bits. We quote a message sent from FarmersManual, a globally dispersed collective, exploring the interface between live digital audio and video synthesis: “The interesting thing for me is that changing something in one of the domains (audio, video, graphics) automatically affects the output of the other domain. I hope the outcome of such an experiment would be a very “elastic” system that can be pushed into various extreme/frantic states that could change according to real-world sensor data or internet-sucked analytical / web-user-based data and than re-develops into its sleeping state after stopping firing data at it.”

One of the major problems with the current authoring and representation tools is their inability to translate between data-formats, protocols, interfaces and devices. The languages, systems and logic they are built upon do not allow for graceful degradation of their functionality. If a particular data-format is not understood for example, there seem to be 2 categories of response: output gibberish or crash, both equally useless.

A future media environment should be able to handle this problem in a more elegant manner. It should be prepared for perpetual (erratic) technology and media upgrades, while retaining a stable framework adapted to the needs of their users. This is where an intermedia tool could come to the fore. An intermedia tool focuses on the processes that collate, relate and analise media, enabling connections and providing constant operations on a range of media elements. Such tools or environments are context sensitive and would be built upon extendible and flexible frameworks. They should be data driven, and handle the previously mentioned problem of unknown data-formats, devices or protocols through a negotiation/translation between the entity demanding information and a dynamic network of computational entities providing information, or providing possible solutions for handling this particular type of information. The system should begin to learn and remember, disentangling itself from the current mess.

Ideally the intermedia tools would encompass multiple platforms, and be scalable from a text only interface to an immersive Virtual Environment. The media should be scaled or transformed by the environment in which they are experienced. The final media output should be the result of negotiations between internal rules associated with each element (or group of elements) and the external conditions/obstacles created by the environment.

An extreme example could be a meditative augmented reality installation in an office building, whose media should be displayed as ambient sound and light, while its transformation parameters are accumulated from an automatic text generator (logfiles), a film archive and a skin conductivity sensor. A more practical example could be a training program for facility monitoring which incorporates real time data input (such as surveillance cameras, electric fences and temperature metres), together with a context sensitive library system. This program should scale from a SMS message for transmission on a mobile phone, to a video presentation and web pages.

Another aspect of scalability required is the customisation of the UI to the particular needs/skills of the user. At the present moment, developers of media worlds (such as games) are confronted with the challenge of learning (or creating) several interface paradigms for authoring different aspects of this creation. Another approach to this process is that a range of different media could be transformed through the same interface. For example, a developer comfortable with video editing systems, could edit sound, graphics, text and scripts through a video-editing interface; or a sculptor could have a tool that allows him/her to model digital media as if it were a physical matter. This area is beyond the scope of this paper, but should be mentioned as another possible tangent of future media research.

Discussing integration and interaction between media elements does not solve the problem of [de|re]constructing all relevant information into a coherent media environment. In order to build such an environment, tools and processes of dynamic composition are required. Instead of working with static representation systems, we might need to look at systems that can create associations, observe patterns or represent information dynamically in a consistent manner. Arranging information spatially has a long tradition of being particularly useful as an aid to memory. People are often able to fill their mind’s eye with images to enhance recall, or their mind’s ear with a particular rhythm or rhyme for reinforcement. These techniques allow us to do more than just remembering particular concepts, but also to relate, or associate these concepts by using similar methods. We have seen that even a small amount of spatial structure can enable significantly larger increases in the knowledge about the structure of interrelated contents. High dimensional geometries can be quite easily represented in a computer, and if carefully structured, could contain a large amount of implicit information about the contents of the space. Manipulations and traversals of this geometry would be the computers equivalent of ‘My Very Elderly Mother Just Served Us Pistachio Nuts’.

(sutChwon = SUbject To CHange WithOut Notice)

Since a complete operable intermedia system (and all its required subsystems) is a daunting task, we have decided to focus on the implementation of a small section of the system described this paper.

In the increasing number of international collaborative projects, most of the design, development and distribution tasks are communicated through local and global networks. These networks consist of a variety of systems or entities, and of interfaces that allow these systems to communicate to the human-readable reality. A range of possibly incompatible protocols, data formats and platforms can be expected. The networks can fluctuate, connections can be lost and new entities can enter the network. As this is becoming the standard condition of current practice with telecommunications technology, the development of techniques to deal gracefully with these problems is necessary.

With SutChwon, we aim to develop a framework for the construction of a flexible system for remote collaboration (CSCW/CSCD). This system should act as an interconnecting layer between several autonomous computational entities, and should allow effortless communication between platforms, protocols, data-formats and interfaces. Automated content and protocol negotiation are a central aspect of this, as is service discovery or learning. With the further integration of machine learning techniques, the system should be able to adapt to new and unknown computational environments it needs to operate within. Our goal is to enable open and adaptive models of communication and interaction within complex and multi-layered networked environments.

In this paper we have attempted to describe some of the weaknesses of current multimedia authoring and representation tools and propose a possible trajectory towards ‘intermedia’ environments. These environments have the capability to adapt to the needs of their users. The methods we propose for construction of these environments involve the use of mnemonic techniques that can operate on two levels: media transformation and composition. In practice this means that production and representation of media would converge resulting in tools that evolve through time and usage. As an example subsystem, we described sutChwon, a work in progress, dealing with complex media transformation and translation, graceful system degradation and computer supported collaborative environment.