Blog posts tagged as 'prototyping'

At the beginning of 2011 we started a wide-ranging conversation with Google Creative Lab, discussing near-future experiences of Google and its products.

We’ve already discussed our collaboration on “Lamps”, the conceptual R&D around computer vision in a separate post.

They had already in mind another brief before approaching us, to create a physical product encapsulating Google voice/video chat services.

This brief became known as ‘Connection Box’ or ‘Connbox’ for short…

For six months through the spring and summer of 2011, a multidisciplinary team at BERG developed the brief based on research, strategic thinking, hardware and software prototyping into believable technical and experiential proof of a product that could be taken to market.

It’s a very different set of outcomes from Lamps, and a different approach – although still rooted in material exploration, it’s much more centred around rapid product prototyping to really understand what the experience of physical device, service and interface could be.

As with our Lamps post, I’ve broken up this long report of what was a very involving project for the entire studio.

• The Connbox Backstory
• Unpacking the brief
• Material Exploration
• Prototyping: Technical and Experiential
• Proto #1: Polar Bear
• Proto #2: Domino
• Product/industrial design development
• Proto #3: Testing the experience and the UI
• Connbox: Film
• The importance of legible products
• Conclusion: What might have been next?
• Then and now – how technology has moved on, and where we’d start now
• Notable Precedents

The videophone has an unusually long cultural legacy.

It has been a very common feature of science fiction all the way back to the 1920s. As part of our ‘warm-up’ for the project, Joe put together a super-cut of all of the instances he could recollect from film and tv…

Videophones in film from BERG on Vimeo.

The video call is still often talked about as the next big thing in mobile phones (Apple used FaceTime as a central part of their iphone marketing, while Microsoft bought Skype to bolster their tablet and phone strategy). But somehow video calling has been stuck in the ‘trough of disillusionment’ for decades. Furthermore, the videophone as a standalone product that we might buy in a shop has never become a commercial reality.

On the other hand, we can say that video calls have recently become common, but in a very specific context. That is, people talking to laptops – constrained by the world as seen from webcam and a laptop screen.

This kind of video calling has become synonymous with pre-arranged meetings, or pre-arranged high-bandwidth calls. It is very rarely about a quick question or hello, or a spontaneous connection, or an always-on presence between two spaces.

Unpacking the brief

The team at Google Creative Lab framed a high-level prototyping brief for us.

The company has a deep-seated interest in video-based communication, and of course, during the project both Google Hangouts and Google Plus were launched.

The brief placed a strong emphasis on working prototypes and live end-to-end demos. They wanted to, in the parlance of Google, “dogfood” the devices, to see how they felt in everyday use themselves.

I asked Jack to recall his reaction to the brief:

The domain of video conferencing products is staid and unfashionable.

Although video phones have lived large in the public imagination, no company has made a hardware product stick in the way that audio devices have. There’s something weirdly broken about taking behaviours associated with a phone: synchronous talking, ringing or alerts when one person wants another’s attention, hanging up and picking up etc.

Given the glamour and appetite for the idea, I felt that somewhere between presence and video a device type could emerge which supported a more successful and appealing set of behaviours appropriate to the form.

The real value in the work was likely to emerge in what vehicle designers call the ‘third read’. The idea of product having a ‘first, second and third read’ comes up a lot in the studio. We’ve inherited it by osmosis from product designer friends, but an excerpt from the best summation of it we can find on the web follows:

The concept of First, Second, Third Read which comes from the BMW Group automotive heritage in terms of understanding Proportion, Surface, and Detail.

The First Read is about the gesture and character of the product. It is the first impression.

Looking closer, there is the Second Read in which surface detail and specific touchpoints of interaction with the product confirm impressions and set up expectations.

The Third Read is about living with the product over time—using it and having it meet expectations…

So we’re not beginning with how the product looks or where it fits in a retail landscape, but designing from the inside out.

We start by understanding presence through devices and what video can offer, build out the behaviours, and then identify forms and hardware which support that.

To test and iterate this detail we needed to make everything, so that we can live with and see the behaviours happen in the world.

We use the term ‘material exploration’ to describe our early experimental work. This is an in-depth exploration of the subject by exploring the properties, both inate and emergent of the materials at hand. We’ve talked about it previously here and here.

What are the materials that make up video? They are more traditional components and aspects of film such as lenses, screens, projectors, field-of-view as well as newer opportunities in the domains of facial recognition and computer vision.

Some of our early experiments looked at field-of-view – how could we start to understand where an always-on camera could see into our personal environment?

We also challenged the prevalent forms of video communication – which generally are optimised for tight shots of people’s faces. What if we used panoramic lenses and projection to represent places and spaces instead?

In the course of these experiments we used a piece of OpenFrameworks code developed by Golan Levin. Thanks Golan!

We also experimented with the visual, graphic representation of yourself and other people, we are used to the ‘picture in picture’ mode of video conferencing, where we see the other party, but have an image of ourselves superimposed in a small window.

We experimented with breaking out the representation of yourself into a separate screen, so you could play with your own image, and position the camera for optimal or alternative viewpoints, or to actually look ‘through’ the camera to maintain eye contact, while still being able to look at the other person.

One of the main advantages of this – aside from obviously being able to direct a camera at things of interest to the other party – was to remove the awkwardness of the picture-in-picture approach to showing yourself superimposed on the stream of the person you are communicating with…

There were interaction & product design challenges in making a simpler, self-contained video chat appliance, amplified by the problem of taking the things we take for granted on the desktop or touchscreen: things like the standard UI, windowing, inputs and outputs, that all had to be re-imagined as physical controls.

This is not a simple translation between a software and hardware behaviour, it’s more than just turning software controls into physical switches or levers.

It involves choosing what to discard, what to keep and what to emphasise.

Should the product allow ‘ringing’ or ‘knocking’ to kickstart a conversation, or should it rely on other audio or visual cues? How do we encourage always-on, ambient, background presence with the possibility of spontaneous conversations and ad-hoc, playful exchanges? Existing ‘video calling’ UI is not set up to encourage this, so what is the new model of the interaction?

To do this we explored in abstract some of the product behaviours around communicating through video and audio.

We began working with Durrell Bishop from LuckyBite at this stage, and he developed scenarios drawn as simple cartoons which became very influential starting points for the prototyping projects.

The cartoons feature two prospective users of an always-on video communication product – Bill and Ann…

This single panel from a larger scenario shows the moment Bill opens up a connection (effectively ‘going online’) and Ann sees this change reflected as a blind going up on Bill’s side of her Connbox.

Our early sketches on both whiteboards and in these explorations then informed our prototyping efforts – firstly around the technical challenges of making a standalone product around google voice/video, and the second more focussed on the experiential challenges of making a simple, pleasurable domestic video chat device.

For reasons that might become obvious, the technical exploration became nicknamed “Polar Bear” and the experimental prototype “Domino”.

Prototype 1: A proof of technology called ‘Polar Bear’

In parallel with the work to understand behaviours we also began exploring end-to-end technical proofs.

We needed to see if it was possible to make a technically feasible video-chat product with components that could be believable for mass-production, and also used open-standard software.

Aside from this, it provided us with something to ‘live with’, to understand the experience of having an always-on video chat appliance in a shared social space (our studio)

Andy and Nick worked closely with Tom and Durrell from Luckybite on housing the end-to-end proof in a robust accessible case.

It looked like a polar bear to us, and the name stuck…

The software stack was designed to create something that worked as an appliance once paired with another, that would fire up a video connection with its counterpart device over wireless internet from being switched on, with no need for any other interface than switching it on at the plug.

We worked with Collabora to implement the stack on Pandaboards: small form-factor development boards.

Living with Polar Bear was intriguing – sound became less important than visual cues.

It reminded us all of Matt Webb’s “Glancing” project back in 2003:

Every so often, you look up and look around you, sometimes to rest your eyes, and other times to check people are still there. Sometimes you catch an eye, sometimes not. Sometimes it triggers a conversation. But it bonds you into a group experience, without speaking.

We needed to come up with new kinds of behaviours for an always on, domestic device.

This was the biggest challenge by far, inventing ways in which people might be comfortable opening up their spaces to each other, and on top of that, to create a space in which meaningful interaction or conversation might occur.

To create that comfort we wanted to make the state of the connection as evident as possible, and the controls over how you appear to others simple and direct.

The studio’s preoccupations with making “beautiful seams” suffused this stage of the work – our quest to create playful, direct and legible interfaces to technology, rather than ‘seamless’ systems that cannot be read or mastered.

In workshops with Luckybite, the team sketched out an approach where the state of the system corresponds directly to the physicality of the device.

The remote space that you are connecting with is represented on one screen housed in a block, and the screen that shows your space is represented on another. To connect the spaces, the blocks are pushed together, and pulled-apart to disconnect.

Durrell outlined a promising approach to the behaviour of the product in a number of very quick sketches during one of our workshops:

Denise further developed the interaction design principles in a detailed “rulespace” document, which we used to develop video prototypes of the various experiences. This strand of the project acquired the nickname ‘Domino’ – these early representations of two screens stacked vertically resembling the game’s pieces.

As the team started to design at a greater level of detail, they started to see the issues involved in this single interaction: Should this action interrupt Ann in her everyday routine? Should there be a sound? Is a visual change enough to attract Ann’s attention?

The work started to reveal more playful uses of the video connection, particularly being able to use ‘stills’ to communicate about status. The UI also imagines use of video filters to change the way that you are represented, going all the way towards abstracting the video image altogether, becoming visualisations of audio or movement, or just pixellated blobs of colour. Other key features such as a ‘do not disturb blind’ that could be pulled down onscreen through a physical gesture emerged, and the ability to ‘peek’ through it to let the other side know about our intention to communicate.

With Luckybite, we started working on turning it into something that would bridge the gap between experience prototype and product.

The product design seeks to make all of the interactions evident with minimum styling – but with flashes of Google’s signature colour-scheme.

The detachable camera, with a microphone that can be muted with a sliding switch, can be connected to a separate stand.

This allows it to be re-positioned and pointed at other views or objects.

This is a link back to our early ‘material explorations’ that showed it was valuable to be able to play with the camera direction and position.

Final technical prototypes in this phase make a bridge between the product design and experience thinking and the technical explorations.

This manifested in early prototypes using Android handsets connected to servers.

Durrell Bishop narrates some of the prototype designs that he and the team worked through in the Connbox project.

The Connbox design project had a strong thread running though it of making interfaces as evident and simple as possible, even when trying to convey abstract notions of service and network connectivity.

I asked Jack to comment on the importance of ‘legibility’ in products:

Connbox exists in a modern tradition of legible products, which sees the influence of Durrell Bishop. The best example I’ve come across that speaks to this thinking is Durrell’s answering machine he designed.

When messages are left on the answering machine they’re represented as marbles which gather in a tray. People play the messages by placing them in a small dip and when they’ve finished they replace them in the machine.

If messages are for someone else in the household they’re left in that persons bowl for later. When you look at the machine the system is clear and presented through it’s physical form. The whole state of the system is evident on the surface, as the form of the product.

Making technology seamless and invisible hides the control and state of the system – this path of thinking and design tries to place as much control as possible in the hands of the end-user by making interfaces evident.

In the prototype UI design, Joe created some lovely details of interaction fusing Denise’s service design sketches and the physical product design.

For instance, I love this detail where using the physical ‘still’ button, causes a digital UI element to ‘roll’ out from the finger-press…

A very satisfying dial for selecting video effects/filters…

And here, where a physical sliding tab on top of the device creates the connection between two spaces

This feels like a rich direction to explore in future projects, of a kind of ‘reverse-skeuomorphism‘ where digital and physical affordances work together to do what each does best rather than just one imitating the other.

At the end of this prototyping phase, the project was put on hiatus, but a number of directions seemed promising to us and Google Creative Lab.

Broadly speaking, the work was pointing towards new kinds of devices, not designed for our pockets but for our homes. Further explorations would have to be around the rituals and experience of use in a domestic setting.

Special attention would have to be given to the experience of set-up, particularly pairing or connecting the devices. Would this be done as a gift, easily configured and left perhaps for a relative who didn’t have a smartphone or computer? How could that be done in an intuitive manner that emphasised the gift, but left the receiver confident that they could not break the connection or the product? Could it work with a cellular radio connection, in places where there no wireless broadband is found?

What cues could the physical product design give to both functionality and context? What might the correct ‘product language’ be for such a device, or family of devices for them to be accepted into the home and not seen as intrusive technology.

G+ and Hangouts launched toward the end of the project, so unfortunately there wasn’t time in the project to accommodate these interesting new products.

However we did start to talk about ways to physicalize G+’s “Circles” feature, which emphasises small groups and presence – it seemed like a great fit with what we had already looked at. How might we create a product that connects you to an ‘inner circle’ of contacts and the spaces they were in?

Since we started the Connbox project in the Spring of 2011, one could argue that we’ve seen a full cycle of Moore’s law improve the capabilities of available hardware, and certainly both industry and open-source efforts in the domain of video codecs and software have advanced significantly.

Making Connbox now would be a very different endeavour.

Here Nick comments on the current state-of-the-art and what would be our starting points were we (or someone else) to re-start the project today…

Since we wrapped up this project in 2011, there’s been one very conspicuous development in the arena of video chat, and that is the rise of WebRTC. WebRTC is a draft web standard from W3C to enable browser to browser video chat without needing plugins.

As of early 2013, Google and Mozilla have demonstrated this system working in their nightly desktop browser builds, and recorded the first cross-browser video call. Ericsson are one of the first groups to have a mobile implementation available for Android and iOS in the form of their “Bowser” browser application.

WebRTC itself is very much an evolution of earlier work. The brainchild of Google Hangout engineers, this single standard is implemented using a number of separate components. The video and audio technology comes from Google in the form of the VP8 and iLBC codecs. The transport layer has incorporated libjingle which we also relied upon for our Polar Bear prototype, as part of the Farsight 2 stack.

Google is currently working on enabling WebRTC functionality in Chrome for Android, and once this is complete, it will provide the ideal software platform to explore and prototype Connbox ideas. What’s more, it actually provides a system which would be the basis of taking a successful prototype into full production.

While not exhaustive, here are some projects, products, research and thinking we referenced during the work…

• RCA Helen Hamlyn Centre Video Chat Work
  http://www.hhc.rca.ac.uk/3994-4006/all/1/Window-on-the-World.aspx
• Push-to-talk and the intimacy it creates
  http://www.purselipsquarejaw.org/2003_11_01_blogger_archives.php#106787988957888590
• Clearboard
  http://web.media.mit.edu/~ishii/CB.html
• Ambient Awareness
  http://en.wikipedia.org/wiki/Ambient_awareness
• Leisa Reichelt on Ambient intimacy
  http://www.disambiguity.com/ambient-intimacy/
• Project Stargate (always-on Skype)
  http://danielodio.com/project-stargate-always-on-skype-video-connection-for-remote-offices
• Matt Webb’s Glancing project
  http://interconnected.org/notes/2003/09/glancing/

Thanks

Massive thanks to Tom Uglow, Sara Rowghani, Chris Lauritzen, Ben Malbon, Chris Wiggins, Robert Wong, Andy Berndt and all those we worked with at Google Creative Lab for their collaboration and support throughout the project.

Thanks to all we worked with at Collabora and Future Platforms on prototyping the technology.

Big thanks to Oran O’Reilly who worked on the films with Timo and Jack.

Preface

This is a blog post about a large design research project we completed in 2011 in close partnership with Google Creative Lab.

There wasn’t an opportunity for publication at the time, but it represented a large proportion of the studio’s efforts for that period – nearly everyone in the studio was involved at some point – so we’ve decided to document the work and its context here a year on.

I’m still really proud of it, and some of the end results the team produced are both thought-provoking and gorgeous.

We’ve been wanting to share it for a while.

It’s a long post covering a lot of different ideas, influences, side-projects and outputs, so I’ve broken it up into chapters… but I recommend you begin at the beginning…

• Introduction
• Material Exploration
• Influences and Starting Points
• The $700 Lightbulb
• “Lamps that see”
• Sketching in Video, Code and Hardware
• Little Brain, Big Brain
• Rules for Smart Light
• Smart Light, Dumb Products
• What happened next: Universe-B
• Postscript: the scenius of Lamps

Introduction

At the beginning of 2011 we started a wide-ranging conversation with Google Creative Lab, around near-future experiences of Google and its products.

Tom Uglow, Ben Malbon of Google Creative Lab with Matt Jones of BERG

During our discussions with them, a strong theme emerged. We were both curious about how it would feel to have Google in the world with us, rather than on a screen.

If Google wasn’t trapped behind glass, what would it do?

What would it behave like?

How would we react to it?

This traces back to our studio’s long preoccupation with embodied interaction. Also, our explorations of the technologies of computer vision and projection that we’ve talked about previously under the banner of the “Robot-Readable World”.

Our project through the spring and summer of 2011 concentrated on making evidence around this – investigating computer vision and projection as ‘material’ for designing with, in partnership with Google Creative Lab.

Material Exploration

We find that treating ‘immaterial’ new technologies as if they were physical materials is useful in finding rules-of-thumb and exploring opportunities in their “grain”. We try as a studio to pursue this approach as much as someone trying to craft something from wood, stone, or metal.

Jack Schulze of BERG and Chris Lauritzen, then of Google Creative Lab

We looked at computer-vision and projection in a close relationship – almost as one ‘material’.

That material being a bound-together expression of the computer’s understanding of the world around it and its agency or influence in that environment.



Influences and starting points

One of the very early departure points for our thinking was a quote by (then-)Google’s Marissa Meyer at the Le Web conference in late 2010: “We’re trying to build a virtual mirror of the world at all times”

This quote struck a particular chord for me, reminding me greatly of the central premise of David Gelernter‘s 1991 book “Mirror Worlds“.

I read “Mirror Worlds” while I was in architecture school in the 90s. Gelernter’s vision of shared social simulations based on real-world sensors, information feeds and software bots still seems incredibly prescient 20 years later.

Gelernter saw the power to simply build sophisticated, shared models of reality that all could see, use and improve as a potentially revolutionary technology.

What if Google’s mirror world were something out in the real world with us, that we could see, touch and play with together?

Seymour Papert – another incredibly influential computer science and education academic – also came to our minds. Not only did he maintain similar views about the importance of sharing and constructing our own models of reality, but was also a pioneer of computer vision. in 1966 he sent the ‘Summer Vision Memo‘ “Spend the summer linking a camera to a computer, and getting the computer to describe what it saw…”


Nearly fifty years on, we have Kinects in our houses, internet-connected face-tracking cameras in our pockets and ‘getting the computer to describe (or at least react to) what it saw seems to be one of the most successful component tracks of the long quest for ‘artificial intelligence’.

Our thinking and discussion continued this line toward the cheapness and ubiquity of computer vision.

The $700 Lightbulb

Early on, Jack invoked the metaphor of a “$700 lightbulb”:

Lightbulbs and electric light went from a scientific curiosity to a cheap, accessible ubiquity in the late 19th and early 20th century.

What if lightbulbs were still $700?

We’d carry one around carefully in a case and screw it in when/where we needed light. They are not, so we leave them screwed in wherever we want, and just flip the switch when we need light. Connected computers with eyes cost $500, and so we carry them around in our pockets.

But – what if we had lots of cheap computer vision, processing, connectivity and display all around our environments – like light bulbs?

Ubiquitous Computing has of course been a long held vision in academia, which in some ways has been derailed by the popularity of the smartphone

But smartphones are getting cheaper, Android is embedding itself in new contexts, with other I/Os than a touchscreen, and increasingly we keep our data in the cloud rather than in dedicated devices at the edge.

Ubiquitous computing has been seen by many as in the past as a future of cheap, plentiful ‘throw-away’ I/O clients to the cloud.

It seems like we’re nearly there.

In 2003, I remember being captivated by Neil Gershenfeld’s vision of computing that you could ‘paint’ onto any surface:

“a paintable computer, a viscous medium with tiny silicon fragments that makes a pour-out computer, and if it’s not fast enough or doesn’t store enough, you put another few pounds or paint out another few square inches of computing.”

Professor Neil Gershenfeld of MIT

Updating this to the present-day, post web2.0 world – where if ‘it’s not fast enough or doesn’t store enough’ we request more resources from centralised, elastic compute-clouds.

“Clouds” that can see our context, our environment through sensors and computer vision, and have a picture of us built up through our continued interactions with it to deliver appropriate information on-demand.

To this we added speculation that not only computer-vision would be cheap and ubiquitous, but excellent quality projection would become as cheap and widespread as touch screens in the near-future.

This would mean that the cloud could act in the world with us, come out from behind the glass and relate what it sees to what we see.

In summary: computer vision, depth-sensing and projection can be combined as materials – so how can we use them to make Google services bubble through from the Mirror World into your lap?

How would that feel? How should it feel?

This is the question we took as our platform for design exploration.

“Lamps that see”

One of our first departure points was to fuse computer-vision and projection into one device – a lamp that saw.

Here’s a really early sketch of mine where we see a number of domestic lamps, that saw and understood their context, projecting and illuminating the surfaces around them with information and media in response.

We imagined that the type of lamp would inform the lamp’s behaviour – more static table lamps might be less curious or more introverted than a angle-poise for instance.

Jack took the idea of the angle-poise lamp on, thinking about how servo-motors might allow the lamp to move around within the degrees of freedom its arm gives it on a desk to inquire about its context with computer vision, track objects and people, and surfaces that it can ‘speak’ onto with projected light.



Early sketches of “A lamp that sees” by Jack Schulze



Early sketches of “A lamp that sees” by Timo Arnall

Of course, in the back of our minds was the awesome potential for injecting character and playfulness into the angle-poise as an object – familiar to all from the iconic Pixar animation Luxo Jr.


And very recently, students from the University of Wellington in New Zealand created something very similar at first glance, although the projection aspect is missing here.

Alongside these sketching activities around proposed form and behaviour we started to pursue material exploration.

Sketching in Video, Code & Hardware

We’d been keenly following work by friends such as James George and Greg Borenstein in the space of projection and computer vision, and a number of projects in the domain emerged during the course of the project, but we wanted to understand it as ‘a material to design with’ from first principles.

Timo, Jack, Joe and Nick – with Chris Lauritzen (then of Google Creative Lab), and Elliot Woods of Kimchi & Chips, started a series of tests to investigate both the interactive and aesthetic qualities of the combination of projection and computervision – which we started to call “Smart Light” internally.

First of all, the team looked at the different qualities of projected light on materials, and in the world.

This took the form or a series of very quick experiments, looking for different ways in which light could act in inhabited spaces, on surfaces, interact with people and things.

In a lot of these ‘video sketches’ there was little technology beyond the projector and photoshop being used – but it enabled us to imagine what a computer-vision directed ‘smart light’ might behave like, look like and feel like at human scale very quickly.

Here are a few example video sketches from that phase of the work:

Sketch 04 Sticky search from BERG on Vimeo.



Sketch 06: Interfaces on things from BERG on Vimeo.

One particularly compelling video sketch projected an image of a piece of media (in this case a copy of Wired magazine) back onto the media – the interaction and interference of one with the other is spellbinding at close-quarters, and we thought it could be used to great effect to direct the eye as part of an interaction.



Sketch 09: Media on media from BERG on Vimeo.

Alongside these aesthetic investigations, there were technical explorations for instance, into using “structured light” techniques with a projector to establish a depth map of a scene…



Sketch 13: Structured light from BERG on Vimeo.

Quickly, the team reached a point where more technical exploration was necessary and built a test-rig that could be used to prototype a “Smart Light Lamp” comprising a projector, a HD webcam, a PrimeSense / Asus depth camera and bespoke software.

Elliot Woods working on early software for Lamps

At the time of the project the Kinect SDK now ubiquitous in computer vision projects was not officially available. The team plumped for the component approach over the integration of the Kinect for a number of reasons, including wanting the possibility of using HD video in capture and projection.

Testing the Lamps rig from BERG on Vimeo.

Nick recalls:

Actually by that stage the OpenNI libraries were out (http://openni.org/), but the “official” Microsoft SDK wasn’t out (http://www.microsoft.com/en-us/kinectforwindows/develop/developer-downloads.aspx). The OpenNI libraries were more focussed on skeletal tracking, and were difficult to get up and running.

Since we didn’t have much need for skeletal tracking in this project, we used very low-level access to the IR camera and depth sensor facilitated by various openFrameworks plugins. This approach gave us the correct correlation of 3D position, high definition colour image, and light projection to allow us to experiment with end-user applications in a unified, calibrated 3D space.

The proto rig became a great test bed for us to start to explore high-level behaviours of Smart Light – rules for interaction, animation and – for want of a better term – ‘personality’.

Little Brain, Big Brain

One of our favourite things of the last few years is Sticky Light.


It’s a great illustration of how little a system needs to do, for us to ascribe personality to its behaviour.

We imagined that the Smart Light Lamp might manifest itself as a companion species in the physical world, a creature that could act as a go-between for you and the mirror-worlds of the digital.

We’ve written about digital companion species before: when our digital tools become more than just tools – acquiring their own behaviour, personality and agency.


Bit, Flynn’s digital companion from the original Tron

You might recall Bit from the original Tron movie, or the Daemons from the Philip Pullman “His Dark Materials” trilogy. Companions that are “on your side” but have abilities and senses that extend you.

We wanted the Lamp to act as companion species for the mirror-worlds of data that we all live within, and Google has set out to organise.

We wanted the lamp to act as a companion species that illustrated – through its behaviour – the powers of perception that Google has through computer vision, context-sensing and machine-learning.

Having a companion species that is a native of the cloud, but on your side, could make evident the vast power of such technologies in an intuitive and understandable way.

Long-running themes of the studio’s work are at play here – beautiful seams, shelf-evidence, digital companion species and BASAAP – which we tried to sum up in our Gardens and Zoos talk/blog post , which in turn was informed by the work we’d done in the studios on Lamps.

One phrase that came up again and again around this areas of the lamps behaviour was “Big Brain, Little Brain” i.e. the Smart Light companion would be the Little Brain, on your side, that understood you and the world immediately around you, and talked on your behalf to the Big Brain in ‘the cloud’.

This intentional division, this hopefully ‘beautiful seam’ would serve to emphasise your control over what you let the the Big Brain know in return for its knowledge and perspective, and also make evident the sense (or nonsense) that the Little Brain makes of your world before it communicates that to anyone else.

One illustration we made of this is the following sketch of a ‘Text Camera’

Text Camera from BERG on Vimeo.

Text Camera is about making the inputs and inferences the phone sees around it to ask a series of friendly questions that help to make clearer what it can sense and interpret.

It reports back on what it sees in text, rather than through a video. Your smartphone camera has a bunch of software to interpret the light it’s seeing around you – in order to adjust the exposure automatically. So, we look to that and see if it’s reporting ‘tungsten light’ for instance, and can infer from that whether to ask the question “Am I indoors?”.

Through the dialog we feel the seams – the capabilities and affordances of the smartphone, and start to make a model of what it can do.

The Smart Light Companion in the Lamp could similarly create a dialog with its ‘owner’, so that the owner could start to build up a model of what its Little Brain could do, and where it had to refer to the Big Brain in the cloud to get the answers.

All of this serving to playfully, humanely build a literacy in how computer vision, context-sensing and machine learning interpret the world.

Rules for Smart Light

The team distilled all of the sketches, code experiments, workshop conversations and model-making into a few rules of thumb for designing with this new material – a platform for further experiments and invention we could use as we tried to imagine products and services that used Smart Light.

Reflecting our explorations, some of the rules-of-thumb are aesthetic, some are about context and behaviour, and some are about the detail of interaction.

24 Rules for smart light from BERG on Vimeo.

We wrote the ‘rules’ initially as a list of patterns that we saw as fruitful in the experiments. Our ambition was to evolve this in the form of a speculative “HIG” or Human Interface Guideline – for an imagined near-future where Smart Light is as ubiquitous as the touchscreen is now…

Smart Light HIG

1. Projection is three-dimensional. We are used to projection turning a flat ‘screen’ into an image, but there is really a cone of light that intersects with the physical world all the way back to the projector lens. Projection is not the flatland display surfaces that we have become used to through cinema, tv and computers.
2. Projection is additive. Using a projector we can’t help but add light to the world. Projecting black means that a physical surface is unaffected, projecting white means that an object is fully illuminated up to the brightness of the projector.
3. Enchanted objects. Unless an object has been designed with blank spaces for projection, it should not have information projected onto it. Because augmenting objects with information is so problematic (clutter, space, text on text) objects can only be ‘spotlighted’, ‘highlighted’ or have their own image re-projected onto themselves.
4. Light feels touchable (but it’s not). Through phones and pads we are conditioned into interacting with bright surfaces. It feels intuitive to want to touch, grab, slide and scroll projected things around. However, it is difficult to make it interactive.
5. The new rules of depth. A lamp sees the world as a stream of images, but also as a three-dimensional space. There is no consistent interaction surface to interact with like in mouse or touch-based systems, light hits any and all surfaces and making them respond to ‘touch’ is difficult. This is due to finger-based interaction being very difficult to achieve with projection and computer vision. Tracking fingers is technically difficult, fingers are small, there is limited/no existing skeletal recognition software for detecting hands.
6. Smart light should be respectful. Projected light inhabits the world alongside us, it augments and affects the things we use every day. Unlike interfaces that are contained in screens, the boundaries of the lamps vision and projection are much more obscure. Lamps ‘look’ at the world through cameras, which mean that they should be trustworthy companions.

Next, we started to create some speculative products using these rules, particularly focussed around the idea of “Enchanted Objects”…

Smart Light, Dumb Products

These are a set of physical products based on digital media and services such as YouTube watching, Google calendar, music streaming that have no computation or electronics in them at all.

All of the interaction and media is served from a Smart Light Lamp that acts on the product surface to turn it from a block into an ‘enchanted object’.

Joe started with a further investigation of the aesthetic qualities of light on materials.



Projection materials from BERG on Vimeo.

This lead to sketches exploring techniques of projection mapping on desktop scales. It’s something often seen at large scales, manipulating our perceptions of architectural facades with animated projected light, but we wanted to understand how it felt at more intimate human scale of projecting onto everyday objects.

In the final film you might notice some of the lovely effects this can create to attract attention to the surface of the object – guiding perhaps to notifications from a service in the cloud, or alerts in a UI.

Then some sketching in code – using computer vision to create optical switches – that make or break a recognizable optical marker depending on movement. In a final product these markers could be invisible to the human eye but observable by computer vision. Similarly – tracking markers to provide controls for video navigation, calendar alerts etc.



Fiducial switch from BERG on Vimeo.

Joe worked with physical prototypes – first simple nets in card and then more finished models to uncover some of the challenges of form in relation to angles of projection and computer vision.



For instance in the Video object, a pulley system has to connect the dial the user operates to the marker that the Lamp sees, so that it’s not obscured from the computer vision software.

Here’s the final output from these experiments:

Dumb things, smart light from BERG on Vimeo.

This sub-project was a fascinating test of our Smart Light HIG – which lead to more questions and opportunities.

For instance, one might imagine that the physical product – as well as housing dedicated and useful controls for the service it is matched to – could act as a ‘key’ to be recognised by computer vision to allow access to the service.

What if subscriptions to digital services were sold as beautiful robot-readable objects, each carved at point-of-purchase with a wonderful individually-generated pattern to unlock access?

What happened next: Universe-B

From the distance of a year since we finished this work, it’s interesting to compare its outlook to that of the much-more ambitious and fully-realised Google Glass project that was unveiled this summer.

Google Glass inherits a vision of ubiquitous computing that has been strived after for decades.

As a technical challenge it’s been one that academics and engineers in industry have failed to make compelling to the general populace. The Google team’s achievement in realising this vision is undoubtedly impressive. I can’t wait to try them! (hint, hint!)

It’s also a vision that is personal and, one might argue, introverted – where the Big Brain is looking at the same things as you and trying to understand them, but the results are personal, never shared with the people you are with. The result could be an incredibly powerful, but subjective overlay on the world.

In other words, the mirrorworld has a population of 1. You.

Lamps uses similar techniques of computer vision, context-sensing and machine learning but its display is in the world, the cloud is painted on the world. In the words of William Gibson, the mirrorworld is becoming part of our world – everted into the spaces we live in.

The mirrorworld is shared with you, and those you are with.

This brings with it advantages (collaboration, evidence) and disadvantages (privacy, physical constraints) – but perhaps consider it as a complementary alternative future… A Universe-B where Google broke out of the glass.

Postscript: the scenius of Lamps

No design happens in a vacuum, and culture has a way of bubbling up a lot of similar things all at the same time. While not an exhaustive list, we want to acknowledge that! Some of these projects are precedent to our work, and some emerged in the nine months of the project or since.

• Steve Mann’s Telepointer project (1998) http://en.wikipedia.org/wiki/Telepointer
• Moveable Projected Displays using Projector Based Tracking (2005-2007) by Jonny Chung Lee, Scott E. Hudson, Jay W. Summet and Paul H. Dietz
• Microsoft research: 6th sense (2009)
• Jamie Tunnard’s anglepoise projector (2010) http://www.hhc.rca.ac.uk/3147/all/1/Desklampprojector.aspx
• Omnitouch by Chris Harrison, Hrvoje Benko and Andrew D. Wilson (2011)
• Camara-Lucida by Christian Parsons and Mariano Tepper (2011)
• University of Tokyo: “Invoked Computing” (2011)
• Pico projection of interfaces onto surfaces from Darmstadt University (2012)
• SketchSynth by Billy Keyes of CMU (2012)
  

Here are a selection of less-academic projects using projection and computer-vision that Joe picked out from the last year or so:

• Display Cabinet by Ben Bashford : http://journal.benbashford.com/post/4530562245
  
• TSUMIKI by PPP : http://vimeo.com/31836098
• Evolution of Mobile by VodaphoneUK : http://www.youtube.com/watch?v=883UlMNOuX4
• Receipt racer by undef and Joshua Noble : http://vimeo.com/24987120
• Real-life Minecraft Block by Ben Purdy : http://vimeo.com/28124800
• SHOWROOM RICHTI-AREAL by PROJEKTIL : http://vimeo.com/22946428
• Playstation 3 Promos : http://www.youtube.com/watch?v=VrgWH1KUDt4, http://www.youtube.com/watch?v=N_cKsOe7hLI

Huge thanks to Tom Uglow, Sara Rowghani, Chris Lauritzen, Ben Malbon, Chris Wiggins, Robert Wong, Andy Berndt and all those we worked with at Google Creative Lab for their collaboration and support throughout the project.

Headliner is an experiment in online reading that BERG conducted in a short project with The Guardian. It uses face detection and term extraction to create “a newspaper that looks back at you”

It’s part of a series of experiments and prototypes that they are initiating with internal and external teams. You can try it for yourself here: http://headliner.guardian.co.uk

Jack led the project and we got a dream-team of past collaborators to work with on it: Phil Gyford, who had already done loads of thoughtful designing of new reading experiences for the Guardian with his ‘Today’s Guardian‘ project, and brilliant designer James King who we had worked with previously on the Here & There Maps.

I asked Jack, Phil and James to share their thoughts about the process and the prototype:

Jack Schulze:

Faces come up in News articles a lot, editors exercise artistry in picking photos of politicians or public figures at their most desperate. Subjects caught glancing in the wrong direction or grimacing are used to great effect and drama alongside headlines.

Headliner makes use of face detection to highlight the eyes in news photographs. It adds a second lens to the existing photo, dramatising and exaggerating the subject. It allows audiences to read more meaning into the headline and context.

Graphically Headliner departs from the graphic rules and constraints news has inherited from print. It references the domain’s aesthetic through typography but adopts a set of behaviours and structures only available in browsers and on the web.

Phil Gyford:

We wanted to retain much of what makes Today’s Guardian a good reading experience but find more in the text and images that we could use to make it less dry. We decided to rely solely on the material we can get from the Guardian’s API, alongside other free services and software.

We looked at various ways of extracting useful data from the text of articles. It had been some years since I’d last dabbled with term extraction and I was surprised that it didn’t seem wildly better than I remembered. We settled on using the free Calais API to pull useful terms out of articles, but it’s quite hit and miss — some places and peoples’ names that seem obvious to us are missed, and other words are erroneously identified as significant. But it gave us a little something extra which we could use to treat text, and also to guess at what an article was about: we could guess whether an article was focused on a person or a place, for example.

We wanted to do more with the articles’ images and focusing on faces seemed most interesting. We initially used the Face.com API to identify faces in the images and, in particular, the eyes. This worked really well, and with a bit of rotating and cropping using PIL we could easily make inevitably small pictures of eyes. (All the article text and images are pre-processed periodically on the back-end using Python, to create a largely static and fast front-end that just uses HTML, CSS and JavaScript.)

Unfortunately for us Face.com were bought by Facebook and promptly announced the imminent closure of their API. We replaced it with OpenCV using Python, which is trickier, and we don’t yet have it working quite as well as Face.com’s detection did, but it’s a good, free, alternative under our control.

Enlarging the cropped eye images looked great: eyes that seemed guilty or angry or surprised, emphasising the choices of picture editors, stared out at you below a headline. We tried giving these images a halftone effect, to emphasise the newspaper printing context of the stories, but unfortunately it didn’t work well with such tiny source images. (Here’s the code for the half toning effect though.)

Browsers treated the drastically zoomed images differently. Chrome and Safari tried to smooth the grossly enlarged images out, which sometimes worked well, but we liked the effect in Firefox, which we could force to show the now-huge pixels using `image-rendering: -moz-crisp-edges;`. The chunky pixels made a feature of the cropped portions of images being so small, and we wanted to show this very raw material on all browsers. This was easily done on the front-end using the excellent Close Pixelate JavaScript.

If we didn’t have any detected eyes to use, we didn’t only want to enlarge the supplied photo — we wanted some variety and to use more of the data we’d extracted from the text. So, if we’d determined that the article was probably focused on a place, we used Google’s Static Maps API to display a satellite image centred on the location Calais had identified.

We put all that together with a front-end based, for speed, on the original Today’s Guardian code, but heavily tweaked. We make images as big as we possibly can — take advantage of that huge monitor! — and enlarge the headlines (with the help of FitText) to make the whole thing more colourful and lively, and an interesting browsing experience.

James King:

To start with, we were most interested in how we might integrate advertisements more closely into the fabric of the news itself. Directing the readers attention towards advertising is a tricky problem to deal with.

One of the more fanciful ideas we came up with was to integrate eye-tracking into the newspaper (with support for webcams) so that it would respond to your gaze and serve up contextually relevant ads based on what you were reading at any particular moment.

This idea didn’t get much further than a brief feasibility discussion with Phil who determined that, given the tight deadline, building this would be unlikely! What did survive however, was the idea that the newspaper looks back at you.

Eyes are always interesting. Early on, we experimented with cropping a news photo closely around the eyes and presenting it alongside a headline. This had quite a dramatic effect.

In the same way that a news headline can often grab the attention but remain ambiguous, these “eye crops” of news photos could convey emotion but not the whole story. Who the eyes belong to, where the photo is taken and other details remain hidden.

In the same way that we were summarising the image, we thought about summarising the story, to see if we could boil a long story down to a digestible 500 words. So we investigated some auto-summarising tools only to find that they didn’t do such a good job of selecting the essence of a story.

Perhaps they take a lot of customisation, or need to be trained with the the right vocabulary, but often the output would be comical or nonsensical. We did discover that Open Calais did a reasonably reliable job of selecting phrases within text and guessing whether it referred to a person, a place, an organisation etc. While we felt that Open Calais wasn’t good enough to draw inferences from the article, we felt we could use it to emphasising important phrases in the headlines and standfirsts.

Typographically, it made sense to use Guardian Egyptian for the headlines, although we did explore some other alternatives such as Carter One – a lovely script face available as a free Google font.

Headliner was a two-week experiment to explore the graphic possibilities of machine-learning and computer vision applied to content.

Not everything works all the time, it’s a prototype after all – but it hints at some interesting directions for new types of visual presentation where designers, photo editors and writers work with algorithms as part of their creative toolbox.

Back in 2009, we started the work that would become http://howbigreally.com and http://howmanyreally.com with the BBC, releasing those two prototypes over the last two years under the banner of “BBC Dimensions“.

Our intention from the beginning was to design the service as a module that could be integrated into bbc.co.uk at a later date if they proved successful with audiences.

Earlier this year, Alice worked with the engineers at BBC News to do just that, and now the first BBC News stories featuring the “How Big Really” module are starting to appear.

Here are a couple of examples – a story on the vast amount of space given over to car parking in the world, illustrated with the module juxtaposing the total space used by parked cars over the island of Jamaica!

…and a more recent example showing the size of a vast iceberg that has just broken free of a glacier on Greenland.

Of course, as with the original http://howbigreally.com prototype, you can customise the juxtaposition with the post-code of somewhere you’re familiar with – like your home, school or office.

The team worked hard to integrate the prototype’s technology with BBC News’s mapping systems and the the look/feel of the site overall.

Here’s Alice on some of the challenges:

We worked with the BBC Maps team to create a tool that could be used by editors, journalists and developers to create How Big Really style maps. Chris Henden and Takako Tucker from the team supplied me with the BBC Maps Toolkit and did a great job of explaining some of its more nuanced points, particularly when I got into trouble around Mapping Projections.

The tool takes an SVG representation of an area, including a scale element, converts it to a JSON object that is then rendered onto a map using the BBC Maps Toolkit. Immediate feedback allows the map creator to check their SVG is correct, and the JSON representation of the shape can then be used to build the map in future.

It’s really satisfying for us to see something that started as a conceptual prototype back in 2009 find it’s way into a daily media production system of the scale and reach of BBC News.

Thanks to all the team there – also Chris Sizemore, Lisa Sargood and Max Gadney for shepherding the project from whiteboard sketches to become part of the BBC journalist’s digital toolkit.

This week saw us filming the prototypes for one of our clients, Chaco, that meant two days with a studio full of people, cameras, lights, product models and as it turns out, a huge amount of extra radio waves.

(click for larger image)

This is a visualisation by Phil Wright who is working with us. It shows the usual BERG wifi network versus the monstrous chunk of the spectrum taken up by the ‘CHACONET’. That’s what happens when you have experience prototypes that use four wifi phones, two wireless baby monitors and eight bluetooth connections.

Suwappu – the augmented-reality toy we invented with Dentsu London is a nominee this year in the Digital category of the Designs Of The Year show at London’s Design Museum.

It’s in great company – with other nominees in the category such as the Kinect, the Guardian’s iPad app (which we also consulted on, with Mark Porter and the brilliant internal team at the paper), High Arctic by UVA and others.

The Suwappu certainly get around a bit – here they are last year where they went to Pop!Tech with me to speak about toys, play and learning in a Robot-Readable World.

And last year they also lived for a while at MoMA, at the Talk To Me exhibit…

We worked with Dentsu London from their original idea to bring them to life through model-making and animation, and then build working prototype software on the cutting-edge of what’s possible in computer-vision on smartphones.

It’s great to have partnerships like this that can rapidly get all the way from a strategic idea ‘What if toys were a media channel’ through to working, real things that can be taken to market.

That’s our favourite thing!

Of course – it’s a lovely bonus when they get recognised in a wider cultural context such as MoMA or the Design Museum.

As well as making our own products, we spend most of our time in the studio working closely in partnership with clients to create new things for them – making strategy real through research, design, making and communication.

Do get in touch if you and your company would like to work with us this way.

This is a blog post about a type of work we find successful – namely, video prototyping – and why we think it’s valuable.

We’ve made quite a few films in the last couple of years, that have had some success – in how they describe products, technologies and contexts of their use in public.

We’re lucky enough to work with Timo Arnall, as creative director, who guides all of our film output and is central to the way that we’ve been able to use the moving image as part of our design process – more of which later.

Film is a great way to show things that have behaviour in them – and the software, services and systems that literally animate them.

A skilled film-maker can get across the nature of that behaviour in a split-second with film – which would take thousands of words or ultra-clear infographics.

They can do this along with the bonuses of embedding humour, emotional-resonance, context and a hundred other tacit things about the product.

Film is also an easy way to show things that don’t exist yet, or can’t exist yet – and make claims about them.

We’ve all seen videos by corporations and large design companies that are glossy and exciting illustrations of the new future products they’ll almost certainly never make.

Some are dire, some are intriguing-but-flawed, some are awesome-but-unbelievable.

This is fine!

More than fine!

Brilliant!

Ultimately they are communications – of brand and ambition – rather than legal promises.

Some of these communications though – have enormous purchase on our dreams and ambitions for years afterwards – for better, or for worse.

I’m thinking particularly of the Apple ‘Knowledge Navigator’ film of 1987, important in some of the invention it foreshadowed, even while some of the notions in it are now a little laughable.It was John Sculley‘s vision – not Jobs – and was quite controversial at the time.

Nevertheless, designers, technologists and businesses have pursued those ideas with greater and lesser success due to the hold that film had over the collective psyche of the technology industry for, say, 20 years.

Hugh Dubberly was working at Apple at the time points out some of the influences the film in a piece on his studio’s website:

“We began with as much research as we could do in a few days. We talked with Aaron Marcus and Paul Saffo. Stewart Brand’s book on the “Media Lab” was also a source—as well as earlier visits to the Architecture Machine Group. We also read William Gibson’s “Neuromancer” and Verber Vinge’s “True Names”.

Of course the company that authored it, Apple, I’d argue built it eventually to some extent with the iPhone.

The gravity well of the knowledge navigator was enormous, and fittingly, Apple punched out of it first with real product.

As Andy Baio and Jason Kottke has pointed out – their predicted time horizon for some of the concepts realised in the iPhone 4S and particularly Siri was uncannily accurate.

This ‘communications gravity’ – the sheer weight of the ‘microfuture’ portrayed shifts the discussion, shifts culture and it’s invention just a little bit toward it.

They are what Webb calls (after Victor Papanek, I believe) ‘normative’ – they illustrate something we want to build toward.

They are also commercial acts – perhaps with altruistic or collegiate motives woven in – but commercial all the same.

They illustrate a desirable microfuture wherein Brand-X’s product or services are central.

Dubberly, in his piece about Knowledge Navigator points out the importance of this – the influence the film had on the corporate imagination of the company, and of competitors:

“What is surprising is that the piece took on a life of its own. It spawned half a dozen or more sequels within Apple, and several other companies made similar pieces. These pieces were marketing materials. They supported the sale of computers by suggesting that a company making them has a plan for the future.

…

One effect of the video was engendering a discussion (both inside Apple and outside) about what computers should be like. On another level, the videos became a sort of management tool.

They suggested that Apple had a vision of the future, and they prompted a popular internal myth that the company was “inventing the future.”

Very recently, we’ve seen the rise of two other sub-genres of concept video.

It’s very early days for both, but both are remarkable for the ‘communications gravity’ they generate for very different commercial endeavours.

First of all – what Bruce Sterling has called the ‘vernacular video’ – often of products in use – created for startups and small companies.

Adam Lisagor has been hailed as the leader in this genre by Fast Company – and his short films for the like of Flipboard, Square and Jawbone have in many ways been defining of the vernacular in that space. They are short, and understated – and very clear about the central benefit of the product or service. Perfect for the sharing and re-sharing. Timo’s written about Adam’s work previously on his personal blog, and I’d agree with him when he says “He’s good at surfacing the joy and pleasure in some of the smallest interactions”. They serve as extraordinarily elegant pitches for products and services that are ‘real’ i.e. has usually already been made.

Secondly – the short videos that illustrate the product intentions of people on Kickstarter, often called ‘campaign videos’ – outlining a prototype or a feasibility investigation into making a product at small scale.

They are often very personal and emotive, but mix in somewhat of a documentary approach to making and construction around prototypes. They serve as invitations to support a journey.

So far, so what?

Video is a well-known way of communicating new or future products & services that reaches the mainstream – and we are seeing a boom in the amount of great short communication about design, invention and making with ever-higher production value as the tools of creation fall in cost, and the techniques of using them become available to small, nimble groups of creators.

Well, we think that’s just half of the potential of using video.

There is a great deal of potential in using video as a medium for design itself – not just communicating what’s been designed, or imagined.

Jack and Timo drew this for me a couple of months ago when we were discussing an upcoming project.

We were talking about the overlap between invention and storytelling that occurs when we make films, and how and why that seems to happen.

On the right is the ‘communications gravity’ that I’ve already talked about above – but the left-hand circle of the Venn is ‘product invention’.

During a project like Mag+ we used video prototyping throughout – in order to find what was believable, what seemed valuable, and how it might normalise into a mainstream product of worth.

In the initial workshopping stages we made very quick sketches with cut-up magazines, pasted together and filmed with an iPhone – but then played back on an iPhone to understand the quality of the layout and interaction on a small screen.

From these early animatics to discuss with our client at Bonnier R&D, we moved to the video prototype of the chosen route.

There were many iterations of the ‘material quality’ of the interface – we call it the ‘rulespace’ – the physics of the interactions, the responsiveness of the media – tuned in the animation and video until we had something that felt right – and that could communicate it’s ‘rightness’ in film.

You find what is literally self-evident.

You are faking everything except this ‘rulespace’ – it’s a block of wood, with green paper on it. But as we’ve written before, that gets you to intuitions about use and gesture – what will make you tired, what will feel awkward in public places, how it sits on the breakfast table.

Finding the rulespace is the thing that is the real work – and that is product invention through making a simulation.

We are making a model of how a product is, to the degree that we can in video. We subject it to as much rigour as we can in terms of the material and technological capabilities we think can be built.

It must not be magic, or else it won’t feel real.

I guess I’m saying sufficiently-advanced technology should be distinguishable from magic.

Some of that is about context – we try and illustrate a “universe-next-door” where the new product is the only novelty. Where there is still tea, and the traffic is still miserable.

This increases belief in our possible microfuture to be sure – but it also serves a purpose in our process of design and invention.

The context itself is a rulespace – that the surface and behaviour of the product must believably fit into for it to be successful. It becomes part of the material you explore. There are phenomena you discover that present obstacles and opportunities.

That leads me to the final, overlapping area of the Venn diagram above – “New Grammar”

This summer I read “The Nature Of Technology: What it is and how it evolves” by W. Brian Arthur. I picked it up after reading Dan Saffer’s review of it, so many thanks to him for turning me on to it.

In it, Arthur frames the realtionship between ‘natural phenomena’ as discovered and understood by science, and how technology is that which ‘programs phenomena to our use’.

“That a technology relies on some effect is general. A technology is always based on some phenomenon or truism of nature that can be exploited and used to a purpose. I say “always” for the simple reason that a technology that exploited nothing could achieve nothing.”

“Phenomena are the indispensable source from which all technologies arise. All technologies, no matter how simple or sophisticated, are dressed-up versions of the use of some effect—or more usually, of several effects.”

“Phenomena rarely can be used in raw form. They may have to be coaxed and tuned to operate satisfactorily, and they may work only in a narrow range of conditions. So the right combination of supporting means to set them up for the purpose intended must be found.”

“A technology is a phenomenon captured and put to use. Or more accurately I should say it is a collection of phenomena captured and put to use. I use the word “captured” here, but many other words would do as well. I could say the phenomenon is harnessed, seized, secured, used, employed, taken advantage of, or exploited for some purpose. To my mind though, “captured and put to use” states what I mean the best.”

“…technology is more than a mere means. It is a programming of phenomena for a purpose. A technology is an orchestration of phenomena to our use.”

This leads me to another use of film we find valuable – as documentary evidence and experimental probe. What Schulze calls ‘science on science’.

The work that he and Timo did on RFID exploring it’s ‘material’ qualities through film is a good example of this I think.

It’s almost a nature documentary in a way, pointing and poking at a phenomena in order to capture new (often visual) language to understand it.

Back to W.Brian Arthur:

“…phenomena used in technology now work at a scale and a range that casual observation and common sense have no access to.”

I think this is what Jack and Timo are trying to address with work such as ‘Immaterials’, and reffering to in the centre of their Venn – creating new grammar is an important part of both design investigation, and communication. It is an act of synthesis that can happen within and be expressed through the film-making process.

Arthur’s book goes on to underline the importance of such activities in invention:

“A new device or method is put together from the available components—the available vocabulary—of a domain. In this sense a domain forms a language; and a new technological artifact constructed from components of the domain is an utterance in the domain’s language. This makes technology as a whole a collection of several languages, because each new artifact may draw from several domains. And it means that the key activity in technology—engineering design—is a form of composition. It is expression within a language (or several).”

He goes on to quote Paul Klee on the the importance of increasing the grammar we have access to:

“…even adepts can never fully keep up with all the principles of combination in their domain. One result of this heavy investment in a domain is that a designer rarely puts a technology together from considerations of all domains available. The artist adapts himself, Paul Klee said, to the contents of his paintbox. “The painter… does not fit the paints to the world. He fits himself to the paint.” As in art, so in technology. Designers construct from the domains they know.”

I think one of the biggest rewards of this sort of work is finding new grammar from other domains. Or what Arthur calls the importance of ‘redomaining’ in invention.

“The reason… redomainings are powerful is not just that they provide a wholly new and more efficient way to carry out a purpose. They allow new possibilities.”

“A change in domain is the main way in which technology progresses.”

“…a single practitioner’s new projects typically contain little that is novel. But many different designers acting in parallel produce novel solutions: in the concepts used to achieve particular purposes; in the choice of domains; in component combinations; in materials, architectures, and manufacturing techniques. All these cumulate to push an existing technology and its domain forward.”

“At the creative heart of invention lies appropriation, some sort of mental borrowing that comes in the form of a half-conscious suggestion.”

“…associates a problem with a solution by reaching into his store of functionalities and imagining what will happen when certain ones are combined.”

“Invention at its core is mental association.”

It’s not necessarily an end product we are after – that comes through more thinking through making. And it also comes from a collegiate conversation using new grammars that work unearths.

But to get a new language, a map, even if it’s just a pirate map, just a confident sketch in an emerging territory – is invaluable in order to provoke the mental association Arthur refers to.

We’re going to continue to experiment with video as a medium for research, design and communication.

Recent efforts like ‘Clocks for Robots‘ are us trying to find something like a sketch, where we start a conversation about new grammar through video…

About a decade ago – I saw Oliver Sacks speak at the Rockerfeller Institute in NYC, talk about his work.

A phrase from his address has always stuck with me since. He said of what he did – his studies and then the writing of books aimed at popular understanding of his studies that ‘…sometimes the stories are the science’.

Sometimes our film work is the design work.

Again this is a commercial act, and we are a commercial design studio.

But it’s also something that we hope unpacks the near-future – or at least the near-microfutures – into a public where we can all talk about them.

Congratulations to all the team at The Guardian for launching their iPad app this week.

BERG played a small role at the very beginning of the process with initial product workshops, Nick contributing his experience on iOS prototyping and Jack consulting on the interaction design with Mark Porter and the team.

Andy Brockie who led the internal design team there has put together a great ‘behind-the-scenes’ gallery of the process, and newspaper design guru Mark Porter has an in-depth blog post about his involvement here.

From that post, a snippet about some of the ‘algorithmic-art-director‘ workflow the team invented:

Unlike the iPhone and Android apps, which are built on feeds from the website, this one actually recycles the already-formatted newspaper pages. A script analyses the InDesign files from the printed paper and uses various parameters (page number, physical area and position that a story occupies, headline size, image size etc) to assign a value to the story. The content is then automatically rebuilt according to those values in a new InDesign template for the app.

It’s not quite the “Robot Mark Porter” that Schulze and Jones imagined in the workshops, but it’s as close as we’re likely to see in my lifetime. Of course robots do not make good subs or designers, so at this stage some humans intervene to refine, improve and add character, particularly to the article pages. Then the InDesign data goes into a digital sausage machine to emerge at the other end as HTML.

Fascinating stuff, and perhaps a hint of the near-future of graphic design…

It was a pleasure working with the team there, and Mark especially. The final result looks fantastic, and more importantly perhaps reads beautifully and downloads extremely quickly. Well done to all involved!

And now, we can now finally exclusively reveal our prototype sketch for Robot Mark Porter…

As a studio we have recently been quite pre-occupied with two themes. One is new systems of time and place in interactive experiences. The second is with the emerging ecology of new artificial eyes – “The Robot Readable World”. We’re interested in the markings and shapes that attract the attention of computer vision, connected eyes that see differently to us.

We recently met an idea which seems to combine both, and thought we’d talk about it today – as a ‘product sketch’ in video to start a conversation hopefully.

Our “Clock for Robots” is something from this coming robot-readable world. It acts as dynamic signage for computers. It is an object that signal both time and place to artificial eyes.

It is a sign in a public space displaying dynamic code that is both here and now. Connected devices in this space are looking for this code, so the space can broker authentication and communication more efficiently.

The difference between fixed signage and changing LED displays is well understood for humans, but hasn’t yet been expressed for computers as far as we know. You might think about those coded digital keyfobs that come with bank accounts, except this is for places, things and smartphones.

Timo says about this:

One of the things I find most interesting about this is how turning a static marking like a QR code into a dynamic piece of information somehow makes it seem more relevant. Less of a visual imposition on the environment and more part of a system. Better embedded in time and space.

In a way, our clock in the cafe is kind of like holding up today’s newspaper in pictures to prove it’s live. It is a very narrow, useful piece of data, which is relevant only because of context.

If you think about RFID technology, proximity is security, and touch is interaction. With our clocks, the line-of-sight is security and ‘seeing’ is the interaction.

Our mobiles have changed our relationship to time and place. They have radio/GPS/wifi so we always know the time and we are never lost, but it is at wobbly, bubbly, and doesn’t have the same obvious edges we associate with places… it doesn’t happen at human scale.

^ “The bubbles of radio” by Ingeborg Marie Dehs Thomas

Line of sight to our clock now gives us a ‘trusted’ or ‘authenticated’ place. A human-legible sense of place is matched to what the phone ‘sees’. What if digital authentication/trust was achieved through more human scale systems?

Timo again:

In the film there is an app that looks at the world but doesn’t represent itself as a camera (very different from most barcode readers for instance, that are always about looking through the device’s camera). I’d like to see more exploration of computer vision that wasn’t about looking through a camera, but about our devices interpreting the world and relaying that back to us in simple ways.

We’re interested in this for a few different reasons.

Most obviously perhaps because of what it might open up for quick authentication for local services. Anything that might be helped by my phone declaring ‘I am definitely here and now’ e.g., as we’ve said – wifi access in a busy coffee shop, or authentication of coupons or special offers, or foursquare event check-ins.

What if there were tagging bots searching photos for our clocks…

…a bit like the astrometry bot looking for constellations on Flickr?

But, there are lots directions this thinking could be taken in. We’re thinking about it being something of a building block for something bigger.

Spimes are an idea conceived by Bruce Sterling in his book “Shaping Things” where physical things are directly connected to metadata about their use and construction.

We’re curious as to what might happen if you start to use these dynamic signs for computer vision in connection with those ideas. For instance, what if you could make a tiny clock as a cheap solar powered e-ink sticker that you could buy in packs of ten, each with it’s own unique identity, that ticks away constantly. That’s all it does.

This could help make anything a bit more spime-y – a tiny bookmark of where your phone saw this thing in space and time.

Maybe even just out of the corner of it’s eye…

As I said – this is a product sketch – very much a speculation that asks questions rather than a finished, finalised thing.

We wanted to see whether we could make more of a sketch-like model, film it and publish it in a week – and put it on the blog as a stimulus to ourselves and hopefully others.

We’d love to know what thoughts it might spark – please do let us know.

Clocks for Robots has a lot of influences behind it – including but not limited to:

Josh DiMauro’s Paperbits
e.g. http://www.flickr.com/photos/jazzmasterson/3227130466/in/set-72157612986908546
http://metacarpal.net/blog/archives/2006/09/06/data-shadows-phones-labels-thinglinks-cameras-and-stuff/

Mike Kuniavsky:

• http://www.orangecone.com/archives/2010/06/smart_things_ch_9.html
• http://www.ugotrade.com/2009/03/18/dematerializing-the-world-shadows-subscriptions-and-things-as-services-talking-with-mike-kuniavsky-at-etech-2009/

Warren Ellis on datashadows

Bruce Sterling: Shaping Things

Tom Insam‘s herejustnow.com prototype and Aaron Straup Cope’s http://spacetimeid.appspot.com/, http://www.aaronland.info/weblog/2010/02/04/cheap/#spacetime

We made a quick-and-dirty mockup with a kindle and http://qrtime.com

I first came across Alvin Lucier’s “I am sitting in a room” through the Strictly Kev/Paul Morley masterpiece mix “Raiding The 20th Century”.

It’s an incredibly simple but powerful piece, that becomes hypnotic and immersing as his speech devolves into a drone through the feedback loop he sets up in the performance.

The space that he performs in becomes the instrument – the resonant frequencies of the room feeding back into the loop.

From wikipedia:

I am sitting in a room (1969) is one of composer Alvin Lucier’s best known works, featuring Lucier recording himself narrating a text, and then playing the recording back into the room, re-recording it. The new recording is then played back and re-recorded, and this process is repeated. Since all rooms have characteristic resonance or formant frequencies (e.g. different between a large hall and a small room), the effect is that certain frequencies are emphasized as they resonate in the room, until eventually the words become unintelligible, replaced by the pure resonant harmonies and tones of the room itself. The recited text describes this process in action—it begins “I am sitting in a room, different from the one you are in now. I am recording the sound of my speaking voice,” and the rationale, concluding, “I regard this activity not so much as a demonstration of a physical fact, but more as a way to smooth out any irregularities my speech might have,” referring to his own stuttering.

Playing around with the kinect/makerbot set-up at Foo set me thinking of Lucier’s piece, and how the sensor-vernacular interpretation could play out as a playful installation…

First, we need a Kinect chandelier.

Then, we scan the original ‘room’ with it.

Next, we print a new space using a concrete printer.

Which we then scan with the Kinect chandelier…

And so on…

One could imagine the degradation of the structure over the generations of scanning and printing might become quite beautiful or grotesque – a kind of feedback-baroque. And, as we iterate, printing spaces one after the other – generate a sensor-vernacular Park Güell…

If anyone wants to give us an airship hanger and a massive concrete printer this summer, please let us know!