Content-based image retrieval often yields surprising results in the first series of
"similar" images. If we put an image of the Eiffel Tower in Paris into the machine to
be matched by similar images, we might get an image of "Big Ben" in London which
obviously looks different.3 But instead of aiming at eliminating such mismatches,
f. e. by making the program "learn" the user preferences, as being attempted by most
user-orientated commercial image sorting software, we might aesthetically learn
from the logic of the computer, looking at an image from a different perspective,
asking: what is it that makes an apparent different image "similar" to the reference
images from the point of view of info-aesthetics.
An equivalent for such pictorial matching by similarity in the dynamic field is the
music finder mufin which chooses a cluster of songs in the databank according to
the requested moods (respecting tempo, style, instrumentation and so forth).
Resulting in findings which have never been searched for, this is genuine "info-
aesthetics", according to which the degree of surprise corresponds with the measure
of informative quality.
In the Eifel Tower Big Ben case, there is a similarity in the grid structure of the
construction, which becomes apparent once we look structurally, that is: media-
archaeologically at the object; 'structure as an analytic tool' and 'structure as the
image subject' here coincide).
Apparent (mis)application in automated sound matching as well may open our ears
for a different notion of what music is, when the described program "mis"identifies
a song with predominant drums and singing as "speech", leading to reconsidering
the sonic aspect of (phonetic) language - rather than being a misinterpretation, this
computer-based classification reveals the truly mediatic essence of speech (in Indo-
European languages at least).
What is informatically at work here is the classification of (micro-)temporal objects
(f. e. in spectral analysis), of smallest intervals (by time-discrete sampling), of delta-
ts. The most relevant spectral components of speech and singing ranges from around
100 to 2000 Hz - which is a temporal, periodic measure.
Fuzzy search
Exploring and developing new options in navigating trans-alphabetical archives
depends on two conditions: the technomathematical and an epistemological
opening of a different, almost thermodynamical search aesthetics. In his lecture
called "The Storm-Cloud of the Nineteenth Century", in 1884 Ruskin answered to
the challenge that at his time the library concept of classification by key terms
increasingly became substituted by a theory of balance in motion (R0ssaak, 2010),
oriented rather at weather phenomena which bring forces into play that radically
alter the traditional order of knowledge: "order by fluctuation, a form of order
understood as process rather than state", where entropy is not the negation of order
but rather its epistemological alternative, "an organizing principle of disorder that
only made sense when observed from on high" (Richards, 1993, p. 86f) - just like
so-called distant reading of big data in Digital Humanities.
wolfgang ernst order by fluctuation? classical archives and their
audiovisual counterparts
When a search engine such as Google is able to predict an influenza, it is because of
calculating so-called "swarm intelligence": a growing number of research entries on
medicamentation against cold indicates regional agglomeration of disease. Instead
of hierarchical classification based on a thesaurus of fixes terms, knowledge is based
on statistical probabilities. Data are not being distributed into fixed, unchangeable
addresses anymore, but form a cloud.
Classification by correlation: Pockets full of Memories
The media artist George Legrady explores new forms of cultural narratives. In his
media-technological installation Pockets Full of Memories4, Legrady - in the best
tradition of Bertolt Brecht's "radio theory" from around 1930 - focuses on the
potential changing role of the archival reader from receivers to producers, in fact the
shift from passive reading to the active archive. In Legrady's installation at the Paris
Centre Pompidou, 2001, the audience created an archive by contributing a digitised
image of an object in their possession at the time of the exhibition visit. The sum of
the archive of objects, organised through a self-organising map algorithm, has been
projected on a large gallery wall and the audience will be able to interact, regroup,
and reformulate relationships through digital devices, according to both intuitive
and classificatory parameters. This true media archive aimed at exploring digital
data structures as a site of literally "collective" memory (thus closer to the museum
and the library than to the archive in its strict sense).
The archive of objects, once having been converted from analogue (physics) to
digital (information), has been stored in a continuously growing database sorted
through a complex algorithm and was then projected at a large scale on the walls of
the gallery space. The key component, that is: the generative archive of this
mechanism is the implementation of the Kohonen Self-Organizing Map (SOM)
algorithm that continuously organises the data within a two-dimensional map,
positioning objects of similar values near each other to arrive at an overall "ordered"
state. This arrival, of course, is permanently being deferred by additional objects:
order in fluctuation indeed. Developed by Teuvo Kohonen, a SOM is an algorithm
used for representing large high-dimensional data sets. It is an artificial neuronal net
capable of adapting to inputs in a self-learning way. The topological model is based
on the binary neuronal function which consists of inhibition (hindrance) and
coupling. Variations lead to temporary, generative and fuzzy SOMs. So, let us not
forget: what looks like iconographically "similar", is in fact a function of
mathematical values; similarity is measured by so-called "distance", a numerical
parameter.5
archives in liquid times
170
3 See for example the software tool MoViMos (Modulare Visuelle Mobile Suche), developed by the German
Research Center for Artificial Intelligence, Kaiserlautern (www.iupr.org)
4 Produced in collaboration with Timo Honkela, medialab, University of Helsinki, applying the Kohonen
self-organizing map algorithm (SOM)
5 See for example the tool MoVIMoS for content-based image retrieval, developed by the Forschungsbereich
Bildverstehen und Mustererkennung at the Deutsches Forschungszentrum für Künstliche Intelligenz,
Saarbrücken (www.dfki.de)
171
