March 18, 2019

Why Is There No HoloLens SDK?

Gallo-blog-1-v2

 

This post is long over due. I have had it in the backlog for a while, now I can’t wait anymore.

 

A Bit of History…

Since the day HoloLens went public, which was January 21, 2015, the developer community have been discussing the possibility of HoloLens SDK. People posted on  Channel 9 of the up coming Windows Holographic APIs from Windows 10 build 9926. They have discovered that there are many holographic related namespaces, even holographic controls (in C++):

Windows::Mirage::HologramFramework::Controls::Button
Windows::Mirage::HologramFramework::Controls::ListView
Windows::Mirage::HologramFramework::Controls::TextBlock
Windows::Mirage::HologramFramework::Controls::TextBox

The good people from NUI World had discovered namespaces like the following (in C#):

Windows.Human
Windows.Human.Gestures
Windows.Human.Hands
Windows.World
Windows.World.Lighting.Prototype
Windows.World.Sensors
Windows.World.Surfaces
Windows.World.Surfaces.Prototype

I had explored Windows 10 debug symbols myself. All those made us to believe that Windows Holographic Platform is well defined from the beginning. There were three distinct namespaces: Human, World, and Mirage, which represent the whole picture of the mixed reality. Please check my early post here.

 

Then It Happened…

We understand at the beginning that HoloLens is one of many Windows 10 devices. It is said from January 21 that all Windows 10 universal apps will run on HoloLens. We also understand HoloLens is a special Windows 10 device, maybe more special than a phone or a tablet. We understand again, for special device families, there is something called Extension SDK to deal with the specialties of certain features that are not implemented on all devices. Naturally, we have guessed this must be the case for HoloLens.

General consensus is there should a set of special APIs for Hololens, or more broadly for Windows Holographic Platform. Even today, you go to HoloLens official website, click Developers on top, then scroll to the bottom to Q&A section, expend “When will the SDK available?”, the answer shows,

We look forward to sharing information on the SDK for Microsoft HoloLens soon. The best way to ensure you receive developer updates and holographic APIs when available is to join the Windows Insider Program.

As it turns out this is false.

Yesterday, Alex Kipman (you don’t know who he is ?!) tweeted in response to #HoloLensDevs,


He actually said the same thing long before yesterday. He had made the clear at Build 2015, there is no special APIs that is specific for HoloLens, therefore SDK won’t be needed. Please check the embedded video at end of the post (@2:20).

Interesting thing is: Out of curiosity, I checked the latest Windows 10 debug symbol packages, well, the latest is still very old, it corresponds to Build 10074, the pdbs that contain afore-mentioned namespaces no longer exist.  For example there use to be,

windows.mirage.hologramframework.pdb

and many other hologram related pdbs, now only a few suspected ones are left, like,

windows.mirage.holgramframwork.query.pdb
holoshellruntime.pdb

In any case, this is the very much a confirmation, that HoloLens APIs were planned, and then removed.

 

To Understand the Fact…

Knowing the fact there will be no HoloLens API, or HoloLens SDK, or Windows Holographic SDK in general is one thing, understanding it is another.

It’s kind of strange why there is no need for HoloLens SDK at all? The way I understand this is: HoloLens is no different than any other Windows 10 devices. It is a PC indeed, by any definition. The only difference, apart from worn on head and look nothing like a computer, is: it is a computer with a monitor, a keyboard, a mouse, touch interface, voice interface all built-in. As it happened, that’s a special monitor, it shows 2D objects and 3D objects, and they are called holograms. That’s all, why do we need any specially SDK at all?

One possible implication is developers will not have access to the spatial mapping data. This can be either a good thing or bad thing. Good thing because the OS hide the complexity, so developers wouldn’t have to worry about the contours of the environment. It’s real hard to write good games or apps if you consider that. Bad thing because sometimes, the environment matters, if you develop a virtual pet, you don’t want it to walk in the air right? This is unknown to me.

When you think of OEMs, who are developing VR devices or AR devices other than HoloLens. They need to write drivers in Windows 10 for their devices, there might be special DDKs for them, which is generally not what we are worrying about.

But wait, how about mixed reality video capturing. That was the thing appeared in the old debug symbols, but disappeared now. Alex had said the function is there (@ the end of the video). Will it be exposed to developer as an API, or to the end user through the OS, or both? We don’t have answer yet.

 

 

Microsoft Is Working to Implement Holo-Video Recording For HoloLens

Tech In Short: Surface 3

 

 

 

Microsoft Will Update Windows 10 SDK before End of June

w10

 

According to Mary Jo Foley, Microsoft will update Windows 10 SDK Insider Preview by end of this month. Last update was before Build in April.

Starting from there, Microsoft will try to pair each Windows 10 build with matching SDK and mobile emulators. Not stated clearly, but looks like SDK and mobile emulators will come with Windows 10 build automatically, so you don’t have to install it separately. So that when restart Visual Studio, new SDK and emulators are already there. Also, the Microsoft blog says it goes with fast ring updates.

I think most important thing would be adding new Visual Studio templates for UWP apps. Not sure if that’s part of it.

 

 

Digest the Windows 10 Technical Preview SDK (Updated)

 

Fig. 1 Convergence

Fig. 1 Convergence

 

Windows 10 Technical Preview SDK was released two days ago. This is probably the biggest change in the history of Windows, in terms of impact on user experiences and businesses. However, from a developer’s point of view, it is not easy to say. On one hand, it is huge because one binary runs everywhere, on the other, the technology is not entirely new. Coding for Universal Application Platform(UAP) is not much different than coding for Windows or Windows Phone. So the transition should be very smooth if you’ve ever done anything on Windows or Window Phone, coded for Silverlight or WPF. Well, that’s when you have a good understanding of the new development tools.

Along with the release of Windows 10 Technical Preview SDK, 14 videos titled Developer’s Guide to Windows 10 Preview was posted on Microsoft’s Channel 9 site. It’s Jerry Nixon and Andy Wigley talk about the new SDK and what’s new in UAP. If you have any interest in developing for Windows 10, I strongly recommend you to watch those videos to get a big picture of what’s going to come in the summer. BUILD conference starts April 29th, the official SDK will likely come out then, but it is important to get some feelings now. There are a total of 14 videos, I don’t think anyone have the time or patience to watch them all. So I did it for you. No, I just watch some of them, but enough to give an overview what it is about, before you spend time on it.

The Convergence Story

They talked about the story of convergence in Windows 10(please see Fig.1). There are a few checkpoints: At the time of Windows 7, Windows Phone 7.5, and Xbox 360, those devices are running on totally different operating systems. The first step of convergence happens during Windows 8, Windows Phone 8 and Xbox One, when NT kernel was adopted by all devices, but leave the UI and user experience separated. Windows 8.1 and Windows Phone 8.1 moved things a little closer: Developer can create one project and target both Windows and Windows Phone, but you still build two binaries, and summit them to two different market places, user has to buy them separately. With Windows 10, it comes to full convergence.

 

UAP

Fig.2 UAP

Fig.2 UAP

UAP – Universal App Platform is the single most important concept brought in with Windows 10. Under Windows 10, you no longer target specific type of devices (PC, phone, Xbox, HoloLens, etc.), you develop for UAP. It a big change, but it is also great news. You create one code, build one binary, publish it to one app store. I don’t know how you feel, to me, it’s a dream to both developers and users.

Extension SDK

 

Fig. 3 Extension SDK

Fig. 3 Extension SDK

 

UAP is great, but there are many things cannot be unified: A PC doesn’t make calls, a phone doesn’t have game controllers. The solution is Extension SDK. You can reference those extension module from Visual Studio. Also the SDK provide a way to check device capabilities at run time. Fig. 4 shows an example of that.

 

Fig. 4 Check Capability at Runtime

Fig. 4 Check Capability at Runtime

Another thing is you don’t need #if anymore. Well, the purpose of #if is to build more than one binaries on one code base, since you need only one binary, there is no need for #if anymore. It is still supported though. We now check device capabilities at run time.

The SplitView

This is first of two important new controls introduced in Windows 10. What is SplitView? This is SplitView (see Fig. 5).

Fig. 5 SplitView

Fig. 5 SplitView

 

It is a new style of menu that is more adaptable to the screen size changes, and work well for both touch and mouse. It is a control that can be constructed in XAML like other controls. Interesting thing to mention is the menu button can be positioned at the bottom (for phones and tablets), or at top (for PCs).

The RelativePanel

This is another important new control in Windows 10. It’s more important than you think. RelatvePanel is a panel, you use it for layout. It solves the problem that programmer has to anticipate screen size and orientation, and reorganize controls accordingly. Instead, you put controls inside a RelativePanel, define relationship between them. You are done. It won’t solve all layout problems, but most of them. See Fig. 6.

 

Fig. 6 RelativePanel

Fig. 6 RelativePanel

 

Now you have it. You probably want to watch some details on these videos: #2 Hello World, #4 Extenstion SDK, #5 SplitView, #8 RelativePanel. As to the other ones, it depends on your time.

Remember this is tech preview, things may change a little at end of April, but I don’t see much change then. I would expect the SDK will include Windows Holographic and HoloLens though.

 

-Update: Just found this video also on Channel 9: A First Look at Building Windows 10 Universal Apps. This is Kevin Gallo from Microsoft talking about Windows 10 UAP at MWC 2015. It’s important because there is live demo of how to adjust UI to suit different screen sizes. Also there are demos of SplitView and RelativePanel which I found better than the afore mentioned videos. You can go right to 18:05~37:00 (the whole video is long):

18:05 – 20:15 UAP
20:16 – 25:40 Adaptive UX, How UI change to adapt to screen sizes
29:28 – 37:00 Live Demo of Adaptive UI, SplitView and RelativePanel.

I want also post an interesting screenshot here. The picture is blurry but the content is good.

 

Yes, HoloLens is there!

Yes, HoloLens is there!