The World Is Your Interface

Is our planet ready to be augmented with a digital operating system?

Written by Gregory Cornelius (2017).

First delivered at the CITEC-I2R Mixed Reality for Human Enhancement (MRHE) workshop held at A*STAR Institute for Infocomm Research, Singapore on August 28, 2017

I was a teenager in the 1980s when operating electronic appliances like video recorders meant programming them manually with a clumsy hand-held infrared remote control. It wasn’t easy. 
Decades later computer operation remains a mystery to significant segments of the population. I’m here to reassure you: this is a temporary situation. We live in a period of active change in Human Computer Interface design, from manual operation to automatic operation. Soon, computer interfaces will be integrated seamlessly with our lives.

My name is Gregory Cornelius and I am the founder of a design science community here in Singapore called ‘Protiotype’. Protiotype is a community of enthusiastic individuals, companies and institutions. We hold events - workshops, meet-ups and exhibitions - we speak at conferences and publish our thoughts in the media but we do more than talk. We explore our ideas by building functional prototypes. We stitch together emergent technologies with in-market products and services to create something new. Our main project is called Intelligent Interiors. An Intelligent Interior is where IoT sensor deployments meet computer science and Augmented Reality (AR) to improve human centered experiences in a smart space to create the workplace of the future. However, today I’m going to introduce one of the conceptual by-products that has arisen from our project.

My subject today is the Augmented Reality Operating System: AROS. Today, I will make a distinction between holographic foreign objects and AR; I’ll explain why the likely transition from 2D to 3D computing interface is transformational for human culture. Importantly, by 3D I do not mean the illusion of 3D on a flat screen. I mean digital assets extended in 3 dimensions over top of the physical world. Finally, I’ll introduce Protiotype’s concept of Solid State City - an algorithmic Architecture. Solid State City is our concept of Intelligent Interiors at scale. It’ll be a very light touch on these concepts but, my hope is to spark some interest in the researchers gathered here to help us unpack these ideas. So let’s begin.

A hidden environment of services

The twentieth century transformational media theorist Marshall McLuhan gave us the famous phrase ‘the medium is the message’. We call McLuhan a transformational media theorist because he theorised how technology transforms culture. By the way, when we use the word ‘technology’ in language we just mean our “tools”. The two words are interchangeable. Anyway, what McLuhan meant by ‘the medium is the message’ is the impact of a medium on society is greater than any single instance of the medium. McLuhan famously argued the societal impact of TV is greater than any specific TV show. If he were alive today, he would say the societal impact of email is greater than any email ever sent; the impact of the web is greater than any website; the impact of social media on society is greater than any single tweet. This rings true, right? For McLuhan the medium is the message.

From McLuhan’s perspective a medium is a ‘hidden environment of services’. It’s the field or ground or network upon which radio, TV, the internet — electronic media — exist. For McLuhan, even technologies (tools) like money clothing or motor cars can be considered mediums.

In the case of the motor car the medium is not the motor car per se. The medium is the roads, highways, street lights, factories, retail sales and service. Further, the medium is a hidden environment of services that determines which new technological actors can be born through ‘innovation’. Innovations do not spring from fresh air.

McLuhan tells us that bicycles, in being made from tyres chains and ball-bearings, presented all the necessary raw materials for the invention of the motor car. The motor car piggy-backed on the bicycles’ hidden environment of services: the roads, factories, retail sales and service. 

Further, McLuhan said the bicycle also precipitated the invention of the aeroplane. “The Wright Brothers were bicycle men” he said.

The bicycles’ environment of services paved the way for the automotive industry, and the aviation industry, and it’s a really good analogy for the potential existing today to build new tools on top of the environment of services that support electronic communications.

In my opinion, the necessary pre-conditions - a hidden environment of services - already exist for the development of an important new tool: a digital Augmented Reality Operating System for the physical world, which Protiotype calls “AROS”. The internet and its supporting electronic components -- telecommunication satellites, data centres, GPS, software applications and APIs -- are an equivalent hidden environment of services and pre-conditions that, in my opinion, when coupled with the Internet of Things sensor deployments, Building Information Models (BIM) and Augmented Reality visualisations, could lead to a new and transformational medium for our time. This new actor would be an ‘asset element class’ or ‘parametric object’ leveraging the hidden environment of services to empower a digital operating system to control the assets of the physical world.

A distinction between Augmented Reality and Holographic Foreign Objects

Today I define the concept of AROS, but before I do that, I want to make a distinction between holographic foreign objects and Augmented Reality. Why? Because there are at least two possible paths forward from where we are today. In my opinion one is more useful than the other.

Currently, the majority of virtual and mixed reality work produced involves the creation of what we might call ‘possible worlds’. The positing of fanciful holographic foreign objects in the physical world. This trajectory is of less interest to me. Yes, there are valid use cases for visualising holographic foreign objects in environments - mostly in gaming and entertainment, as well as education training and sales - where VR headsets offer a cheaper entry point compared to AR smart glasses - but for me, that’s a less useful direction to pursue than an augmentation of the physical world with practical utilitarian information.

As a designer I want to make the world a better place. And I think we can improve the quality of life by layering useful digital information over top of the physical world. In fact, we augment the world already with symbolic systems including typography on street signs: way finding directions and safety information for example. Maps are another form of augmentation, as are audible alarms, and thermal cameras which visualise surface temperatures for us. All these tools augment our world with useful information. Augmenting the augmentation that already exists is a very good place to begin the journey towards AROS. We could add digital data to physical sign boards, say. When I’m presented with information about the place I am in and the artefacts that I encounter then we are heading in a useful direction. If I’m presented a set of digital tools to control those things - even better.

With controls in mind I want to highlight what I call the ‘Augmented Reality Operation System’ or AROS. It’s a useful augmentation that’s laid out in three-dimensions across the physical world. Perhaps I should say four dimensions because the augmentation will change over time. If I’m in a restaurant I’ll see the breakfast menu in the morning, lunch menu at lunch time, dinner menu in the evening, and so on. Then again the augmentation itself will adjust relative to my own data profile: so let’s call it ‘multi-dimensional data’. In Protiotype’s vision of Intelligent Interiors AROS is a real-time layer of sensor data and interactive control. AROS is read-write send and receive. In theory you could reveal that digital data layer with any device: phone, glasses, contact lens, interactive projectors, microphones, speakers. Eventually we shouldn’t need a device at all. It’ll be possible to add a layer of control logic to physical surfaces, to geo-coordinates, to create fluid ambient spatial computing environments.It’s important that AROS includes an actuating function so we can change the state of the physical world as we encounter it. This is the functionality I’m proposing.

Let’s look at an example: commercial buildings. Commercial buildings have technological components including Mechanical Electrical Plumbing (MEP), Air Con (HVAC), Audio Visual (AV) and Information Technology (IT). These components can be monitored and measured and connected to web services like ACA Engine and interactive controls output to a browser on flat screen devices. This system could become even smarter when connected to tools like Building Information Modelling (BIM). BIM is one likely foundation upon which we could add this new layer of digital control. A Facilities Manager or security guard could actuate the physical state of a building by interacting with this digital layer of tools made visible to them. We can already visualise BIM in smart glasses which gives an advantage over flat screens for operators in-the-field, by overcoming the intrusion of dust, glare, gloves, and allowing hands-free operation. I believe the necessary technology -- the hidden environment of services -- already exists today to build AROS and digitally control our built environment.

“We look at the present through a rear-view mirror. We march backwards into the future.” McLuhan

McLuhan tells us that all new mediums are initially populated with existing content from old mediums. The first radio shows were a man reading a printed newspaper aloud. He read the news. The first TV shows pointed a camera at a radio announcer reading the news. The first websites had web “pages” as if the internet was a kind of book. McLuhan said ‘when a new technological medium (a new tool) arrives, we don’t know what to do with it’. We understandably fill the new medium with content from the old medium until we work out how to use it better. Compelling content develops over a period of time. This will be the case with AR and therefore the same with AROS. It’s likely that the first examples of the AROS interface will be inherited from existing software conventions - the things we know.

The AR start-up Meta populates their device with office software applications lifted directly from flat screen PCs. So wrong. However, there is some experimentation in more promising areas. The designers at Argo Design have given us some ground breaking projection interfaces. The Ishikawa Watanabe Lab at University of Tokyo has presented the ‘Anywhere Surface Touch’ utilizing any surface as an input area with a wearable device. Our own Protiotype Intelligent Interiors at Singapore Polytechnic, where we are visualising data with Maxime Depondt’s AR task management application for Facilities Management. And we are building upon the CORE research of Atlas Lim, who described an alternate models of human computer interaction. CORE has become AROS. As time goes by, we’ll iterate again, and add a layer of digital controls to AR as we prototype AROS. AROS will be more than a digital layer of information. It’s a digital layer of meters menus and buttons that control the physical world through web sockets and APIs. It’s a set of controls. AROS could turn switches and operate appliances. AROS is a parametric object. It’s a cyber-physical system.

AROS epitomises a transformation from 2D to 3D computer interface. I don’t mean the illusion of 3 dimensions on a flat screen, I mean 3D multi-dimensional interfaces, on top of the physical world. This change to spatial computing could be equivalent to Euclidean geometry’s impact on the cultural Renaissance of the 14th to 17th centuries by the introduction of perspective. This led to spatial representational painting in the arts and I don’t think this is well understood.

Think of hieroglyphics. Before the Renaissance all drawings were two dimensional representations and this reflected our world view. We lived on a flat earth, figuratively!

The vanishing point changed our perception of the world. We became detached observers outside the picture frame. Our conception of space changed. Our perception of our place in the world changed.

This change to spatial representation in the arts wasn’t the cause but it was symptomatic of a change in our thinking, and precipitated the Enlightenment of the 17th century, the dawn of the scientific revolution, and an extended period of dominance for Western civilisation. I suggest this kind of transformation can happen again.

Our current computing environment is dominated by two-dimensional flat screens. As we liberate human computer interface from the flat-screen, and map the world in 3D with a set of controls, the change could have a similarly profound impact on the way we think about the world - the way we perceive space and time. If this transformation does happen (and there is no guarantee that it will) it would change our perspective, and prepare the field for new technological actors, whatever they might be, just like Marshal McLuhan would expect.

I’m saying: just like the bicycle precipitated the automotive industry and aviation industries, the necessary components - a hidden environment of services - are present now to develop a new medium determining the shape of the new industries to follow.

Protiotype is contributing to a smart sustainable building showcase at Singapore Polytechnic School of Architecture. It’s there we will attempt to build a prototype of AROS. And that’s really all I should safely say today. I should end here with an introduction to AROS because it doesn’t seem too far fetched right? But as I have a few minutes remaining let’s speculate wildly...

Solid State City

What if we did change our computing interface from 2D to 3D? What if we did have an AROS that extended the idea of a digital twin, that aggregated and negotiated data assets, that extended BIM with a set of digital controls and could change the state of assets in the physical world around us like a digital nervous system? At Protiotype we think about this future world. It’s going to need some serious computing power. There is so much focus on mobility yet we spend 90% of our lives indoors. And the majority of that time is spent in just two places: work and home. Instead of making computers smaller all the time, so we can carry our computers around, why don’t we embed computing power inside the buildings around us instead?

Today’s buildings are equivalent to the vacuum tube. And just like the vacuum tube became silicone - and became solid state - what if buildings were constructed from computing components? Computational buildings can become “solid state”. This is another possible world. Computer hardware built out at super-scale. An algorithmic architecture that scales Intelligent Interiors to intelligent buildings precincts and cities. That’s what we call the ‘Solid State City’ - a term coined by Protiotype’s David Ivory in his 2016 essay of the same name. AROS is Solid State City’s unifying layer of data, logic, and control and it leads to generative design and robotic construction. This means, data aggregated through AROS, through digital twins, BIM, is fed to algorithms that consider the terrain and design the optimal architectural solution. In the future AI cities will autonomously and independently design and construct themselves from modular and additive constructed computer components.

Over time the built environment will transform to amplify AROS, to become the Solid State City, a new environment of services, a new electronic medium, that ultimately re-articulates the built surface of the planet in its entirety as computer architecture, to remove all the friction from Human Computer Interaction, and transform human culture along with it, to achieve our collective goals of environmental and economic sustainability, or whatever else we choose as our collective priorities. And that’s our Protiotype journey. To debate these ideas, and some others, and then build functioning research laboratories to explore them. Our next step is to try and prototype AROS at Singapore Polytechnic, or somewhere else, and to look for opportunities to build our scenarios and test our hypothesis. We are open to working with anybody who is interested to work with us. We are a bridge between research and commercialisation. And hopefully today I have sparked an interest in researchers here today to participate in our project? Thank you.

Gregory Cornelius (2017)


Marshall McLuhan, Understanding Media: The Extensions of Man. (McGraw-Hill, 1964)

David Ivory, Solid State City. (Protiotype, 2016)

Protiotype's Related content:

David Ivory, 2016. Solid State City.

Rob Peart, 2016 Developing urban Smarts .