October 28, 2020

Watch Kinect Perform with Human Musicians from Seattle Symphony




I have posted a while ago that Seattle Symphony to Perform with Kinect Instruments. Now video features this performance and some stories is posted. Check here for more details, and enjoy the video.




Seattle Symphony to Perform with Kinect Instruments




I am not joking. It’s true, Kinect controlled instruments are to perform with human musicians in an orchestra.

From GeekWire,

On Friday, May 1, the Seattle Symphony will present the world premiere of a new work. It will be conducted by Music Director Ludovic Morlot and feature ten human musicians and three “kinetic instruments.” All will be directed, as is tradition, by gesture.

It will be in Benaroya Hall, downtown Seattle, which is Seattle Symphony’s home theatre. The report also said, after performance, the Kinect instruments will be left there on display for exhibition.

The concert will be held in the grant lobby, not the concert hall. There will be 11 human musicians: 9 instrumental musicians, 1 soprano, the conductor (Morlot), and three Kinect instruments: a 24-reedhorn sculpture, a set of concert chimes, and a customized grand piano. So, it is a small orchestra, more of chamber setup. The piece is to performed lasts 22 minutes.

Those three robotic instruments are controlled by Kinect, the one comes with Xbox One. It doesn’t say whether there are 3 or 1 Kinect. The challenge here is how the robots understand the conductor’s gestures. Think about this for a second, those are not predefined, limited set of gestures like that are used on Xbox One. Those are the gestures both human and Kinect understand in real time, for 22 minutes. I guess the conductor has to limit type of gestures but still, it is not an easy thing.

I am not sure if the conductor is using a baton, or just hand. Since if it is a baton, it would be more important, because that would mean object recognition other than hand recognition. I will write about object recognition(it is an excite topic) later.

Here is the big deal: how to keep the human musicians and robotic musicians in sync? If there is any small latency caused by Kinect, the performance would be out of sync. It won’t work. I think there must be a very small latency in Kinect. If there wasn’t a conductor, musicians can usually adjust speed to catch each other up, with the conductor, the robotic instruments have to respond in real time.

This gives me more confidence to HoloLens, and that’s the whole point I am writing this.


Some Business Ideas with HoloLens – (20) Holo-Golf Swing

Is HoloLens Going to Be a Commodity?

Review of Surface 3 Reviews, and My Thoughts

Jerry Nixon: “Why you must build a HoloLens app”




Understand Depth Sensor, Gesture Recognition – Some Interesting Videos





Found some interesting videos about Kinect, but the technology is directly related to HoloLens. Hope you enjoy.


The first video is How the “Kinect Depth Sensor Works in 2 Minutes”. The video illustrates how depth sensor works.

Two cameras from two different view points are required, plus the object to form a triangle, so you can calculate the depth of the object. The two camera approach has some limitations. Kinect uses infrared light to improve that by project a speckle pattern to code each spot, so that the RGB camera can identify an object even they are the same color. I am not sure if the sun light limitation applies to the IR sensor in HoloLens though.


The second video here is the work done by Microsoft MVP Josh Blake of InfoStrat. He set up a test on Kinect v1 and Kinect v2, found drastic improvement in response time and details. As you can see from the video, Kinect v2 detects much more skeletal details of his body (ignore the legs), especially his hands: not only hand-close, hand-open, but finger pushes, two fingers up or one finger up. I am assuming HoloLens is in the v3 class.


The third one demonstrates the use of Kinect in retail clothing scenario: try on clothes without go to the fitting room. The concept is old, but it’s value is there. I hope HoloLens can popularize this idea.


HoloLens, Recommended Readings – Business Ideas Collection


Why HoloLens Is Not Kinect

Since HoloLens was accounced back in Jan 21st, many articles on the Internet have referred Kinect while reporting on HoloLens in various ways. Most of them tend to put a shade of skepticism over HoloLens from the comparison. In this post, I will try to give a little inside look at why this comparison comes so often and why it’s still invalid.

Sure, both HoloLens and Kinect use hand gesture for interaction, both use voice command for action control. Both are developed by the same group of people in Microsoft, and both happen to be amazing technology with no peer in the world. Indeed, HoloLens has a couple of Kinect sensors inside it, miniaturized of course. However, amazing technology doesn’t always lead to great consumer adoption. Despite great initial sales, Kinect’s momentum hasn’t been able sustained, which leads some people in the media draw comparison with HoloLens in the market department. I think people who draw comparison between the two are missing the point.

Kinect is a sensor, it does a lot of things, like body movement recognition, field depth mapping, etc., but it is still a sensor. HoloLens is a full featured computer, Kinect is just one of the its many sensors, but that’s not my point. HoloLens is a display device. As I have pointed out in another post, what distinguishes Hololens from anything else is its display technology. It breaks the traditional screen barrier: Contents are no longer contained in a fix-sized screen, not to mention its 3D holographic capability. But this is still not quite my point.

In essence, Kinect is an input device. HoloLens is a output device first, input device second. And that is my point. So, what does it mean? What difference does it make? Lets take a closer look (I know I have used this sentence too much, but I am not tired yet).

HoloLens Technology – 3D World Mapping

Kinect is not a great input device, neither is HoloLens. To understand this, please look at another post of mine: Natural User Interface: Is Natural Always Better? Sure, Natural User Interface (NUI) is great concept, however, no matter how we label it, in the presence of mouse, keyboard and game controller, every other ways of interactions look inferior, that includes hand gesture, voice, gaze and touch. Those NUIs are only valuable when you don’t have access to a mouse, keyboard or game controller, like the way you use a phone, a tablet on the go. Whenever you’re siting at a desk, a mouse is always the best way of human-computer interaction. I think every gamers or game developers agree with me that use gesture to play a hard core game is close to impossible. So, basically, Kinect is trying to use an inferior input method to replace great input methods. And we know the results.

HoloLens is an input device as is Kinect, but its main function is an output device: a display. It is its own display, it can also be a display for a PC, a display for Xbox. The key point here, it doesn’t exclude other input methods: mice, keyboards and game controllers. You can still use a PC the way you use it before, HoloLens just adds another great display. You can play games on Xbox the way you always do, with a game control in hand, the game can be streamed to a giant display on Hololens. Even you use HoloLens as a standalone machine, you can still use a physical keyboard, a mouse, or a game controller. What does it mean, really?

HoloLens Technology – Eye Tracking

It means every application that works on a PC should work with HoloLens, because you consider HoloLens as a mere display. Every Xbox games will be instantly available to HoloLens, I said every.

Yes, I heard your argument: Why do I use HoloLens only as a display? I buy HoloLens for 3D holograms. I believe there will be 3D holographic games, and 3D holographic applications. DirectX has been used for game development, I believe having those games appear as 3D holograms wouldn’t be a major problem, but my point is even without those 3D stuff, HoloLens gives you a giant display which is can be a huge boost to your PC or gaming experience.

In summary, Kinect is a input device, which is its weak point, in my opinion. HoloLens still has the same input device function, but it doesn’t exclude other input methods. HoloLens’ display is what makes it great: 3D or 2D.


PC + HoloLens: Would It Work?

Why HoloLens’ Most Remarkable Achievement Is Its Display

Xbox + HoloLens: How Can They Work Together?