Binary Body Double : Microsoft exposes the Science Behind Project Natal for Xbox 360 The software company studied the body to coach its latest motion-tracking gaming technology to work without the necessity for controllers When Nintendo's Wii game console debuted in Nov 2006, its motion-sensing hand-held "Wiimotes" got players off the sofa and onto their feet.

Now Microsoft is attempting to outdo its rival by getting rid of the controller altogether : It has exposed details of how it developed Project Natal, which gives Xbox 360 players the facility to manipulate on-screen characters via natural body movements. The machine-learning technology is going to enable players to do things like kick a digital football ball or swat a handball in their living rooms by mimicking the motion. "Instead of a controller, your body becomes the game input," claims Alex Kipman, Microsoft's director of incubation for Xbox 360. Microsoft introduced its formidable Xbox upgrade in June 2009 and is expecting to ship the technology in time for the year-end 2010 vacation season. Natal will be made of a depth sensor that uses infrared signals to make a digital 3-D model of a player's body as it moves, a camcorder that may pick up small details like facial expressions, and a microphone that will identify and find individual voices.

Programming a game system to discern the human body's just about limitless combos of joint positions is a fearful computational problem. "Every single motion of the body is an input, so you'd need to program near infinite reactions to actions," Kipman says.

Rather than making an attempt to preprogram actions, Microsoft made a decision to teach its gaming technology to recognize gestures in realtime just like a human does : by extrapolating from experience. Jamie Shotton, an analyst at Microsoft Research Cambridge in Britain , invented a machine learning algorithm for that purpose. It also recognizes poses and renders them in the game space on-screen at thirty frames per second, a rate that conveys smooth movement. Fundamentally , Natal-enhanced Xboxes will do motion capture on an ad-hoc basis, without the requirement for the mirror-studded spandex suit of traditional motion-capture approaches. Training Natal for this job needed Microsoft to assemble a huge quantity of biometric info.

The firm sent observers to houses around the world, where they videotaped basic motions like turning a wheel or catching a ball, Kipman says. Microsoft analysts later laboriously selected key frames inside this footage and marked each joint on each person's body. Kipman and his team also went into a Hollywood motion-capture studio to assemble information on more acrobatic movements. "During coaching, we want to give the algorithm with 2 things : realistic-looking photographs that are synthesized and, for each pixel, the matching part of the body," Shotton says.

The algorithm processes the information and changes the values of different elements to attain the best performance. To keep the quantity of information controllable, the team required to work out which elements were most important for coaching. For instance, the system doesn't need to recognize the whole body mass, but only the spacing of skeletal joints. After cutting down the information to the necessary motions, the analysts mapped each unique pose to twelve models representing different ages, genders and body types.

The A. I behind Microsoft's Xbox 360 motion-sensing game controller.Deep in Microsoft's lairs, the Xbox 360 team is working on more than a new video-game system. They are essentially making an attempt to resolve an incredibly tough problem in AI. Their prototype Project Natal allows you to control a game just with your body movementsno buttons or Wii-like wandsby watching you with a 3-D camera.

Project Natal programmed

Sounds easy enough, but most cameras just snap photographs without having any idea what they are having a look at. To make Natal work, Microsoft has to coach its camera to realise what it sees. Here at CES, Microsoft narrated yesterday evening that Natal will go up for sale "by the holidays." Before the show, we got an exclusive glance at the smarts which make Natal tick. The Brain The part of Natal that players see is like a webcam.

( Microsoft's not revealing details about this hardware yet, possibly as the release is many months in the future, but we do know that it measures relative distances employing a black-and-white camera sensor and a near-infrared beam. ) but it is the software within, which Microsoft casually pertains to as the brain, that seems sensible of the pictures caught by the camera.

It has been programmed to investigate photographs, look for a basic human form, and identify about thirty necessary parts ,eg your head, torso, hips, knees, elbows, and thighs. In programming this brain--a process that is still going onMicrosoft relies on an advancing field of AI called machine learning. The grounds is this : Feed the PC enough datain this case, millions of pictures of peopleand it can learn for itself the easiest way to understand it. That saves programmers the near-impossible task of coding rules that describe all of the millions of possible

Natal Learning Brain

movements a body can make. The method is a lot like a parent pointing to several different folks's hands and announcing "hand," until a baby gradually works out what hands looks like, how they can move, and that, for example, they do not disappear into thin air when they are briefly out of view. The simple way to Teach A Machine To See Microsoft is now training and making improvements to the version of the brain which will at last go into the final product. How? By painstakingly gathering photos of folks in several different poses, and then running all this information through big clusters of PCs ( as shown in the studio ) where the learning brain lives. The method of gathering the information basically needs plenty of manual work. First, reps went into homes around the planet and recorded folks moving in front of a specifically built rig. The photographs caught are real folk moving the way any standard person would. But those recordings can't tell the PC anything helpful about joints and limbs all alone, so programmers dive into the raw information and hand-code it to pigeonhole each body part ( at every frame ). Microsoft also uses professionally staged motion-capture scenes, which gives similar information but without all of the manual work of coding by hand ( since the systems use sensors that mark individual body parts ). And Microsoft has a mini mo-cap studio of its own, where staff can make a fast recording when a new chunk of information is required. All these marked-up pictures comprise tens of terabytes of info. Microsoft's PC farms seive thru this gigantic information set, letting the brain come up with chances and statistical data about the human form. Once the brain is done learning, it and its figures get packed into the Natal system.

An early version is now making the rounds of trade shows, and later, more-accurate versions will ultimately show up in your living room.

Next, read about how it applies its precious data to decode your game-playing moves. Within Natal's Thought Process.

What is the Brain Thinking? : what is the brain thinking as it watches you jump around, swinging hypothetical bats or head-butting hypothetical football balls? As you stand in front of the camera, it judges the distance to different points on your body. Then the brain guesstimates which parts of your body are which. What is the brain thinking as it watches you jump around, swinging hypothetical bats or head-butting hypothetical football balls? The above screen-capture shows what's occurring in it's headthe different photographs represent different stages of Natal's computational process. Here's the step by step : Step one : As you stand in front of the camera, it judges the distance to different points on your body. In the image on the far left, the dots show what it sees, a supposed "point cloud" representing a 3-D surface ; a skeleton drawn there is just a basic guess. ( The image on the top shows the image understood by the color camera, which may be employed like a webcam. ) Step two : Then the brain guesstimates which parts of your body are which. It does this primarily based on all of its experience with body posesthe experience explained above.

Dependent on how similar your pose is to things it's seen before, Natal can be nearly assured of its guesstimates.

Natal Maps Body Positions

In the color-coded person above [bottom center], the darkness, lightness, and size of different squares represent how certain Natal is that it knows what body-part that area belongs to. ( for instance, the 3 enormous red squares indicate that it's highly likely that those parts are left shoulder, left elbow and left knee" ; as the pixels become smaller and muddier in color , for example the grayish pixels round the hands, that is a hint that Natal is hedging its gambles and isn't extraordinarily sure of its identity. ) Step three : Then, based totally on the chances assigned to different areas, Natal comes up with all possible skeletons that might fit with those body parts. ( This step isn't shown in the image above, but it's like the stick-figure drawn on the left, except there are many possible skeletons overlaid on one another. ) It eventually settles on the likeliest one. Its reasoning here is partially based totally on its experience, and partially on more formal kinematics models that programmers added in. Step four : Once Natal has determined it has enough certainty about enough body parts to choose the most likely skeletal structure, it outputs that shape to a simplified 3D avatar [image at right].

That is the last skeleton that'll be skinned with garments, hair, and other features and shown in the game. Step five : Then it does this all over again30 times a second! As you move, the brain generates all possible skeletal structures at each frame, at last deciding on, and outputting, the one that's most likely. This concept process takes just one or two milliseconds, so there's masses of time for the Xbox to take the data and use it to manage the game.

With the announcement that Project Natal would dump the image processing to the Xbox console vs handling those requirements in hardware on the device itself and consume as much as ten to fifteen % of the console's overall "computing resources," we became troubled that the capability for "supplemental" input in core games would be blunted. When asked what % of the Project Natal enabled games under development would be Natal exclusive versus Natal boosted titles, Microsoft's Aaron Greenberg said,

"Our focus is on most or all them will fall into that class of fully unique, new experiences for Natal." "We're going to manage the portfolio extraordinarily rigorously and our focus is going to be on bringing fully brand spanking new original experiences to the market when we launch Natal,"

Greenberg claimed. "We're not taking a look at just adding tiny Natal elements to games, we are taking a look at how can we basically bring a wholly new class of controller-free games and entertainment to the market so I believe that is where we intend to continue to focus."

But that doesn't suggest that other developers are not able to supplement their game's input with Natal controls. After the release this vacation, "developers will have that out there at their fingertips if they need to enable those features," Greenberg told us. "But I cannot talk to what that implies from a game development perspective. As far as I am aware, there is not any real sacrifice." But Greenberg was not just chatting up "new games" and "new methods to navigate" the console, but Microsoft is mysteriously promising "new entertainment-like experiences that'll be enabled" thanks to the new interface. With the "focus" on "controller free games" and not just adding "little Natal elements to games" we are left thinking how Peter Molyneux's Myth three which he confirmed will have Natal support will implement that support.

With a 2010 launch for both Story three and Project Natal, we are expecting it's going to be one of the device's bellwether games.

Develop Magazine has obtained what it considers to be confirmed information that Microsoft will show much more about their Natal platform at their February Gamesfest conference. This comes after a 2009 hiatus for the series of events for the development community held in Seattle, London, and Tokyo.

Via Develop:

The description for the design track reads: "Project Natal not only revolutionizes the way people play games, but also changes the way games are designed and created. The Project Natal Design track will present innovative thinking and ideas to help you take your game from office to living room—creating new ways to work, building showcase experiences, divining user intent, and designing gestures for UI versus game interactions. Discover best practices and what makes the “magic” in a Project Natal game."

Meanwhile the tech track's blurb reads: "Project Natal provides a groundbreaking new way for games to use natural user motion to interact with the Xbox 360. Experience the future now, with this cutting-edge technology! Join us to learn how to develop world-class titles using Project Natal, which provides many exciting new features that can be challenging to programmers. In the Project Natal Technical track, we will walk you through how to overcome these challenges with a combination of classic techniques and new thinking. We will explore the depths of this exciting technology and dive deep into gesture recognition, avatar retargeting, speech recognition, advanced raw stream processing, handling different player environments, and many other topics. No controller required!"

This follows earlier reports this month that Microsoft has secretly been visiting major publishers to sell them on developing for Natal in time for the system's predicted fiscal Q3 launch next year, and their recent postings for development jobs for Natal and their Halo-focused 343 studios. Ubisoft also specified that they'll dedicate significant support to the Natal platform in their earnings call this week, with plans to ship up to 10 titles within 6 months of its launch.

Every year Times Magazine picks the top 50 inventions of the year.
not surprisingly, the natal made the list for 2009 of course the $10 million light bulb won over Natal
ranking in at third.

Although the xbox natal 360, did get a higher ranking then the AIDS vaccine!!!, and the electric Eye?

Well I guess the vaccine is only 31% successful, I am certain, Microsoft will have a higher success rate. LOL

Don't forget to leave us your thoughts on this at the bottom of the page.

you can check it out to Time magazine Project natal comes in the fifth