Figure 1: The score displayed by the plug-in
Anton Eliëns,
Martijn van Welie,
Jacco van Ossenbruggen and
Bastiaan Schönhage
Vrije Universiteit, Fac. of Mathematics and Computer Sciences
De Boelelaan 1081a, 1081 HV Amsterdam, The Netherlands
email: {eliens,martijn,jrvosse,bastiaan}@cs.vu.nl
Our framework consists of a browser plug-in that supports the display and editing of scores as well as playing scores by connecting to a shared MIDI server. To participate in a jam session, clients of the MIDI server can also send data in real time, for instance by using a keyboard embedded in a Web page. We discuss the issues involved in displaying musical material on the Web and we sketch the technical architecture of our framework.
Compared to textual and graphical material, the capabilities of the Web for musical information are rather poor. The embedding of music, or sound in general, rarely goes beyond links to raw audio and MIDI files or to streamed audio connections. To display a musical work, HTML authors have to use images containing the score. All of these solutions are very low level as they basically regard music as being just sound (or a picture in the case of a score).
True score files are usually a few orders of magnitude smaller, and the audio signal can be synthesized at the client side at any appropriate sample rate. Additionally, a high-level description of music provides the browser with far more information when compared to the raw samples. In previous work we proposed to transmit musical scores (instead of the raw samples) across the Internet and to add sound synthesis functionality to Web browsers [10], and the use of generic SGML to encode structured documents [2].
In this paper, we describe an experimental framework that offers many of the ingredients for true networked music support including facilities for editing, displaying and playing musical scores as well as facilities for high level exchange of musical material and real-time collaborative work involving music and sound. Our approach is based on traditional music notation and on MIDI for playing facilities. The framework builds upon the work done in the DejaVu project at the Software Engineering section of the Vrije Universiteit (Amsterdam), which resulted in a suite of components for developing distributed Web-aware hypermedia applications [1,2,7,8,9,10,12].
The structure of this paper is as follows. We first describe our solution to exchanging musical information. In particular, we describe our score format and the associated score editor, and how musicians may connect to a shared MIDI server to join a real-time collective improvisation. Next, we outline the architecture underlying our approach to networking music and the web components used for its realization. After sketching some applications of networked music we discuss the merits and shortcomings of our solution and indicate some directions for future research.
The most ambitious markup language for the dissemination of music on the Web is probably the Standard Music Description Language [4]. SMDL expresses a musical work in terms of four basic domains. The logical domain --- the primary focus of SMDL --- is, according to the standard, describable as ``the composer's intentions with respect to pitches, rhythms, harmonies, dynamics, tempi, articulations, accents, etc.''. The central element of the logical domain, the cantus element is an abstract, one-dimensional finite coordinate space onto which musical and non-musical events can be scheduled. This allows for the inclusion of any dependent time sequences (such as automated lighting information) in a musical work. The standard uses HyTime [3] hyperlinking to specify the relations with information from the other three domains: the gestural domain --- describing any number of particular performances (e.g. MIDI files or digital audio) of the work, the visual domain --- describing any number of scores (a printable/displayable version) of the work, and the analytical domain --- comprised of any number of theoretical analyses or commentaries about the information in the three other domains. The addressing power of HyTime makes it possible to link directly into information expressed in other formats, including MIDI files, digital audio recordings or specific score notations, without modification.
Figure 1: The score displayed by the plug-in
Our approach is more modest and we deploy a much simpler SGML representation, primary geared to encode printable/displayable versions of the score (i.e. SMDL's visual domain). However, the format used is sufficiently rich to be able to generate a playable MIDI representation as well. Information which is usually added by performers (in SMDL this is represented in the gestural domain) such as explicit interpretations of tempi, articulations and accents are not supported in the current version.
<SCORE>
<TITLE>Corrente</TITLE>
<COMPOSER>Antonio Vivaldi</COMPOSER>
<STAFF>
<MEASURE Sig="3,4" Key=F Clef=Gclef>
<NOTE Pos="1,3" Stem=down>d6 4 0
<REST Pos="3,6">C6 8 0
<NOTE Pos="4,6" Stem=up>a5 8 0
<NOTETUPLE Stem=down>
<NOTE Pos="5,6">f5 8 0</NOTE>
<NOTE Pos="6,6">a5 8 0</NOTE>
</NOTETUPLE>
</MEASURE>
...
</STAFF>
</SCORE>
To support display and editing of SGML scores on the Web, we developed the Amuse score editor as a plug-in for our Web browser (see figure 1). The editor has a graphical user interface and does not require any SGML knowledge from the user. Figures 2 and 3 show, respectively, a fragment of an example score file, and the associated style sheet with a CSS1-like syntax [5]. Both documents can be edited by the graphical score editor plug-in. Changes in the style sheet are dynamically reflected in the display of the score. A significant enlargement of the page-width parameter, for example, will allow for more measures on a single staff, and will result in a redraw of the complete score.
SCORE {
margin-left : 30;
margin-right : 30;
margin-top : 80;
margin-bottom : 20;
page-height : 1000;
page-width : 920;
}
TITLE {
title-align : Center;
title-font : -*-Times-Bold-R-Normal--*-240-*;
}
COMPOSER {
composer-align : Center;
composer-font : -*-Times-*-R-Normal--*-180-*;
}
We developed the keyboard applet as an alternative input device to be able to send ``live'' MIDI data to our server. Since multiple applications can have access to the MIDI-server, a user can have a score edit session running, and simultaneously be playing a keyboard applet.
To engage in a jam session, the keyboard applet connects to the JamServer instead of the MIDI server. The JamServer acts as the central point of a jam session, keeping track of all clients engaged in the session.
To start a jam session, all jam clients connect to a single JamServer and send it their MIDI data. The JamServer is connected to one or more MIDI servers, as depicted in figure 4. By having the JamServer separate from the MIDI server itself, the latter is relieved from the burden of jam session management. Every connected MIDI device will receive all the MIDI data submitted by the jam clients. This data is relayed to these devices by the MIDI server(s), through the virtual MIDI data stream that is created when registering as a jam client.
In figure 4 we see three jam clients connected to a single JamServer (on machine B). The MIDI server is running on the same machine as the JamServer. Both the clients on machine A and C have registered a MIDI-out device (a software sound synthesis MIDI program developed for Solaris [6]) with the MIDI server on B. The user on A has additionally registered a MIDI-in device (the keyboard). Using the keyboard, the user on A can contribute to the jamming. The score editor on C is directly connected to the MIDI server and is not engaged in the jam session. The MIDI server will redirect MIDI request from the score editor only to the MIDI device on C.
Figure 5: Overview of the Web components
Figure 5 shows an overview of the basic Web-related components of the hush library. The browser provides the top level user interface for all Web components, including a viewer, a scrollbar, navigation buttons (back, forward, home, reload) and an entry box to enter URLs. The netclient, web and MIMEviewer components form the conceptual base of our approach of connecting to the Web:
The MIMEviewer component provides an abstract interface to viewers for several MIME types. The web widget only knows about the (abstract) MIMEviewer class while the actual functionality is implemented in several concrete viewer classes, one per MIME type. Specific viewers for new MIME types can be plugged dynamically into the MIMEviewer object.
When the MIMEviewer gets the instruction to display a document of a certain MIME type, it changes its role and becomes a viewer for that particular MIME type. This dynamic role-switching idiom is discussed in more detail in [2]. As a result, the addition of new viewers can be done without changing the web widget.
The netclient component builds the bridge between the local web widget and the World Wide Web by providing an abstract and uniform interface to network (file) access and transport protocols. In the realization of the netclient components we have employed the dynamic role-switching idiom in the same way as in the implementation of the MIMEviewer components.
The web object creates a MIMEviewer object and tells which role it should play (e.g. SGML, Amuse or VRMLviewer). This role can be changed during the lifetime of a single MIMEviewer object by calling a method to change its role. A browser typically uses only one single MIMEviewer object that changes its role according to the type of data that should be displayed. The SGMLviewer is the default viewer, it displays generic SGML documents by using style sheets for each document type. By default, a style sheet for HTML is used [11]. Since our generic SGMLviewer is more suited for textual documents and does not offer editing support, we developed a separate viewer/editor to process our Amuse/SGML score files.
Since the MIMEviewer provides no network functionality at all, it generates events whenever it needs to retrieve data pointed to by a URL. Such events are generated as a response to user interaction (e.g. clicking an anchor) or to fetch inline data during the parsing process. These events are typically handled by the web component which plays a central role in our approach because it combines the functionality of the MIMEviewer and the netclient components. Additionally, the web component adds a history and caching mechanism to the MIMEviewer. The web component's behavior is similar to the standard widgets of the hush framework, and can be conveniently used as a part of an application's GUI. Because the web widget has both a C++ class interface and a script interface, it is easy to create, or extend, applications with Web functionality.
Music can significantly enhance the perception of HTML pages, especially in a commercial or educational environment. Possible application areas which might benefit from the use of high-level music encodings and networked MIDI include:
The current version of the score editor and keyboard applet only works on our hush Web browser. We are currently developing a Netscape plug-in version of the score editor and experimenting with a Java version of the keyboard applet.
Because of the textual format of music description languages, it will be possible to employ anchoring and link facilities within musical documents as well. We plan to support both HTML as well as HyTime hyperlinks in future versions of our score editor.
We have described a framework offering a high-level description for the exchange of musical scores which can also be used for the generation of MIDI data. Furthermore, with our MIDI server architecture we can connect several musicians to share their music. Although both aspects of our system are in their beginning stages and need to be elaborated further, they indicate the new possibilities of music on the Web. In other words, it is time for the Web to become aware of music!
First of all we like to thank S.A. Megens (SAM) who developed the MIDI Jam server as part of his MSc. thesis project. Further we like to thank all those students who participated in one of the jamming sessions, devoting time to our project while they were supposed to do something useful.
