3D NUI (Natural User Interface)

September 23, 2009

I came across an interesting blog article while doing research into multi-touch interfaces.  Joshua Blake, a .NET, SharePoint, and Microsoft Surface Consultant for InfoStrat is petitioning the World Wide Web to participate in what he's calling The MOADtouch Project.  If you're one of the many folks (like me) who are scratching your head as to how multi-touch computers can apply to your everyday experience, Joshua feels your pain. Why smudge the screen when you can quickly point and click?  Today's computers were designed to be interfaced with using a mouse, a keyboard or a digitizer (tablet PC).  

Essentially, Joshua is wanting to help reinvent the way humans interact with a computer and he's asking all to participate. "Pretend touch was the only acceptable input device to the mass market. Forget the Windows, Icons, Menus, Pointers. Forget the rectangles. Start with your fingertips and design out from there," Joshua says.  OK Joshua, you're on…


The 3D NUI 

To date, input devices such as mice, keyboards and digitizers offer limited interaction with the computer, right?.  The monitor literally acts as a barrier to entry disallowing a user the ability to reach inside and interact with an application beyond a one-dimensional plane.  But what if the dimensions of the computer extended further than the screen?  Imagine a three-dimensional NUI sans the stupid red and blue paper glasses.  It's like an advanced Wii experience, except not only can you use applicable hardware (see Fig 3), but it's better because your body can essentially step inside the application.  No, I don't mean like a character you create on the Wii.  Your hand or whatever body part actually interacts with an application that is presented in 3D.  Thus, your hand remains your hand, not a character-generated interpretation. The application itself is three-dimensional. For example, if you're inside CAD building the next space shuttle, you could pick up the parts you're working with and place them accordingly. 

FIG 1  (3D NUI)

Now take the above picture one step further and imagine a large 60-inch monitor offering a grander scale and the ability to interact using the entire human body. Need to prep for an operation you've never performed Doc?  Open up the cadaver application and practice, using an electronic scalpel as an input device. 




Fig 2 (3D NUI – Massive Scale)

Fig 3  (3D NUI Input Devices etc. etc.)

Forgive my entrepreneurial zeal, but my brain immediately started mind-mapping the endless horizontal and vertical markets a 3D NUI could apply:

  • CAD
  • Interior Design
  • Cartography
  • Digital Sculpting and Painting???  (Takes art degrees over the internet to a whole new level)
  • Auto Design
  • NASA
  • Facial Recognition (Social networking never looked so good)
  • Surgical Practice (Use scalpels, scissors, tweezers, etc as input devices)
  • Education (Imagine the possibilities: applied physics, trigonometry, geometry, etc.)

Okay Joshua, that's my two pennies.  It's late… All these possibilities have kept me at the office later than I planned on staying this evening.

I enjoyed participating in The MOADtouch Project and encourage the LitCentral Founders' Blog readers to give it a go.  BTW…Have I mentioned before on this blog how much I love technology? We live in such an exciting era.  The possibilities are endless!

Be Sociable, Share!

{ 4 comments… read them below or add one }

Frank K. September 29, 2009 at 1:47 pm

Robin this is a very interesting concept. It allows for not only total body gesturing, but unlimited input devices as you suggested. The larger the surface area, the more possibilities. Very interesting.


Doc_Brad September 29, 2009 at 4:22 pm


How would you apply this NUI concept to writing?


Robin September 29, 2009 at 5:39 pm

Doc_Brad – I don’t, actually. I still think the best input device for novel writers is the keyboard. However… Screenwriting is a different medium, because the only part of the script that reaches the audience’s ear is dialogue. Everything else is interpreted or improvised. I think there is a vertical market for speech recognition amongst current screenwriting applications.

Yes, I know the technology is already there in Vista with applications such as Dragon. That’s not what I’m talking about. That’s a broad application and hardware combo meant to capture as many users as possibles using common features. I envision something more customized for screenwriting.

Imagine if the input device (keyboard) had separate hot keys for character, action, dialogue, and transitions. When the cursor is blinking in a dialogue area you can input with speech. All the rest (i.e. action, character, transition) is keyed in.

You can almost do this with Dragon and MS Word but the repetitive grammar required to format correctly gets in the way of productivity and creativity – two big no-nos. This is where Final Draft, Mariner and Movie Magic should be five years from now. Tailor these screenwriting programs specifically for speech recognition and you just might improve dialogue in some scripts because the writer would essentially PERFORM the dialogue out loud as it is being written.

Now push this app to a mobile device and watch productivity go through the roof! How? Imagine an iPhone app that helps you create dialogue for scenes. Say John and Jennifer are going back and forth in a scene. Click the appropriate character and speak their dialogue. He said, then she said… The app will transcribe everything into text and once you’re back inside a Wi-Fi it syncs up to your screenwriting app. You could essentially create great dialogue day and night AND I BET the result is better – with tighter exchanges. A writer becomes unfettered, moving away from the computer. He can pace, move to a different location, gesticulate and really get into character.

A NUI should improve speech recognition because it could utilize video as well as audio to recognize what is being spoken. Can a computer read lips? I think anything is possible if, like Joshua suggests, we re-imagine how we interact with a computer.

[This probably should have been a followup article, but oh well.]


Doc_Brad October 1, 2009 at 10:27 am

Wow thanks for the response.


Leave a Comment

Previous post:

Next post: