June 25, 2017

How Big Is HoloLens’ Field of View?



I have said this too many times, but there are still a lot of dumb asses in media do have a clue.

47″ TV: 19 x 11 degree
HoloLens: 30 X 18 degree

The above screenshot from Microsoft’s recent concept video shows just that. I know that’s a mock up, but it agrees to my estimation perfectly.



HoloLens SDK Is Expected to Release Soon!



As a number of people who have followed my blog noticed that I am writing a HoloLens programming series. I am ready to bring in another post with detailed programming technique and a working example based on my recent research on HoloLens programming. Just before I do so, I checked with the some great people that I have access to, just making sure that I am on the right direction, and that I am not misleading my readers. I got some information that is beyond exiting. I can’t contain my excitement. How reliable is my source? Well, I rarely say things that I am not sure of. Talking is cheap, lets get to is.

(We’re) working hard to release development tools and are looking forward to sharing information on the SDK for Microsoft HoloLens soon. The best way to ensure you receive developer updates and holographic APIs when available is to join the Windows Insider Program and install the Windows 10 Technical Preview. You can also start by building Windows universal apps and ramping up on Unity.

If you are a developer, you don’t need me to say anything more. This a what everyone is waiting for. I wonder why Microsoft didn’t make it available earlier.

For now, I will hold off my HoloLens programming series. Lets talk about it more when the new SDK is released. I think the date should be in line with Dev version of HoloLens hardware release, which is within two months.



HoloLens Programming in C# and DirectX – A Closer Look


Edit: The content of this is still valid, but some of the practice may not be ideal, please here for details.

We are not ready for the code yet. Before I show you how I do C# and DirectX, I want to do two things in one post: 1. Show you all the options we have for programming DirectX in C#. 2. Give you a little introduction about what the hell DirectX is. Lets get to it.

What are the options?

1. Unity + C# script.
As I said in the previous posts, this is not an option for HoloLens apps. Because Unity is for games only. You can still do apps with Unity, but it is going to be resource intensive, as you may expect for a game. If you have large amount of 3D visuals, like in a flight simulation, or the one like Case Western Reserve University does with HoloLens, you are still better off with Unity, but for light 3D rendering, mostly business logic and communication or things like a normal app does, you better write an app instead of a game. You need to look for something other than Unity, or any other game engines.

It would be ideal if Unity or another game engine manage rendering in the way like an app does, that would great for application development, but that’s not the case now. If the app itself has very dynamic objects in the scene, they are moving all the time, and requires fastest response possible from the user, then you have to go with game, no matter it is called a game or not.

2. C++/C# UWP app with DirectX in CoreWindow
CoreWindow is a class that represents the utter most application window of a UWP app. There are two project templates in Visual Studio 2015: DirectX 11 App and DirectX 12 App (New Project -> Visual C++ -> Windows -> Universal ->). Either of those two template generates bare bone code that is based on CoreWindow.

The templates titled “App”, but they are actually games. There is a game loop on app level. Visual updates are done at a possible frame rate. The DirectX rendering target is the app’s CoreWindow. The two templates are C++ projects, as I will show you in next post, it is very much possible to code DirectX in C++ and C# for rest parts, but this one is not our interest, because again, it is a game, not an app.

Let me try one more time to make it clear about the difference between an app and a game from a developer’s point of view. A game tries to use as much CPU/GPU as possible to satisfy a game’s need for frame rate. That’s why it runs on game loop to update visuals, instead of let the system handle them. Lets say, beside display and interaction, your app have a lot of other task to do, if your application is trying every 20 ms to update visuals, what time is left for everything else? I just found it’s not the ideal way of programming.

3. C++/C# DirectX with SwapChainPanel
SwapChainPanel is a xaml control designed to make DirectX work inside of the xaml controls. This is much like XNA and Silverlight interop in the past. It achieve the best of both world. Usually you put SwapChainPanel as the main scene, with other controls work as overlay. SwapChainPanel deal with DirectX rendering. However, the design of SwapChainPanel is still game oriented. You may use SwapChainPanel as islands among other controls, but the general guideline from Microsoft is SwapChainPanel targeting game development.

By the way, there was a SwapChainBackgroundPanel, which is replace by SwapChainPanel.

4. C++/C# DirectX with SurfaceImageSource
SurfaceImageSource works like general ImageSource for XAML controls. Many XAML controls can have ImageSource, SurfaceImageSource works in similar way, the difference is SurfaceImageSource is a surface for DirectX renderings. The resource of than rendering can be used as a image source for many different controls. In the next post, I will show you how to do the rendering part in C++, and everything else C#, like what you do in any UWP apps.

5. C++/C# DirectX with VirtualSurfaceImageSource
VirtualSurfaceImageSource is the same as SurfaceImageSource, except the former is for DirectX rendering of unlimited size, usually larger than the screen size. This is, I think, the best fit for HoloLen applications, because your visuals are likely spreading in the space instead of a small window. In the post after next one, I will show you a working example of that.

So, what the hell is DirectX any way?

DirectX is the ultimate 3D rendering APIs provide by all modern Windows OSes. Whatever 3D engines you know that work on Windows, work on top of DirectX. Being Unity, XNA, Unreal Engine, or any others, there is no exception, they are on top of DirectX.

DirectX is a collection of APIs that do not only 3D, also 2D, Text, Texture, shaders, that’s the X in DirectX. They are:

* Direct3D – for 3D objects
* Direct2D – for 2D overlays
* DriectWrite – for text
* Untilities: DirectXMath, XAudio, XInput

That’s not all, the following are parts of the DirectX subsystem:

* DXGI – Microsoft DirectX Graphics Infrastructure, deals with display devices directly. Because 3D drawing need to be faster than usual, that’s why programmer are required to deal with hardware directly. That’s the interface for it.
* HLSL – High Level Shading Language for DirectX. You write shading in this special language, with extension .hlsl, Visual Studio will compile it to binary.

In DiretX programming, you hear one term a lot, but it doesn’t make sense to most people, that’s Swap Chain. Forget about “swap” or “chain”, think of it as a display buffer, you write on the buffer instead of the monitor itself, the video card will take care of it. The reason it is called a chain because there are more than one buffers, one after another to improve performance. Why swap, well, you don’t need to know. Swap chain has nothing to do with DirectX, it is part of display technology existed long time ago.

Why DirectX programming is hard?

There are two reasons, and two only: 1. the designer of DirectX decided to use stupid names for their APIs, like D3D11CreateDevice(). Fyi, take this one for example, D3D11 means Direct3D version 11. They could just call it CreateDeive(), but they decided to make our lives harder. 2. We don’t do 3D that much. But I promise you, it is just a matter of familiarity. When you see the term “Swap Chain” or the ugly function names the 20th time, they start making sense to you.

That’s it. Next up is an example of SurfaceImageSource, then there is another sample code of VirtualSurfaceImageSource.



HoloLens Programming in C# and DirectX – The Big Picture



We need to be clear about this first: what is developing a HoloLens app? HoloLens is a computer running on Windows 10, by definition, every UWP application should be able to run on HoloLens. So if you write an app for Windows 10 UWP, and publish it to Windows Store. It should run on HoloLens, the way it runs on Windows desktop computers. However, that’s not what I mean by developing for HoloLens. Developing for HoloLens means your app takes advantage of HoloLens 3D mixed reality display, and optionally utilize spatial mapping, hand gestures and voice input. In this post and a few others after this, I only explore 3D display, in the context of HoloLens reality mode, not within a floating/pinned window as for a normal UWP app.

In multiple occasions, Microsoft had hinted Unity with C# script would the way build HoloLens apps. I don’t disagree. Unity is a high level game engine, especially for 3D/2D screen rendering. It is in the same position as XNA was before. Because XNA is no long supported in Windows 10. Unity is probably the best replacement. Unity is recommended for game development in general, not just for HoloLens. Even though you can do DirectX directly, but Unity would make many complicated task much easier otherwise. Also, the Unity Store has tons of stuff for asset creation, some of them are free.

Unity runtime is written in C++, so your C# code for Unity is called C# script, meaning they are not compiled to .Net code. On the low level, Unity is based on DirectX. It is a high level wrapper of DirectX, but a lot more than just a wrapper. The compilation of a Unity project results in a Visual Studio project. There you can deploy, run and debug on HoloLens. To get start with Unity game development, I recommend Adam Tuliper’s post on MSDN: Unity : Developing Your First Game with Unity and C#.

It is clear that if you want to develop a HoloLens 3D Holographic game, or simulation app, like fly simulation, or educational holographic 3D illustration, you can safely choose Unity with C# scripting. Then use Visual Studio to deploy and test on HoloLens.

That’s all good… but, there is a problem.

Unity is a game engine, it is NOT an app engine. The same way XNA is a game engine, not an app engine. I have been asking this question: can you write an app on Unity? Now it is clear, the answer is NO. Of course you can call a fly simulator an app instead of a game, but a game and an app are intrinsically different. Allow me to explain.

A game requires fast real time response and fast real time visual refresh. It sacrifice everything to satisfy the above too requirements. That’s why a game is always running on it own separately loop, called the game loop. An app, on the other hand, is supposed to be on idle until there is input from a user. An app tries to use as less resource as possible. An app is running under system message loop, it is message based. The system take care of visual update.

If you are developing an app instead of a game, you don’t want your application in a busy loop. You don’t want your app to use as much resource as possible to gain frame rate. You want your app to be friendly to other apps.

So, if you develop an app, Unity is out of picture, but you still want to show 3D holograms. Well, that’s the point of HoloLens apps, right? Now what would you do? The answer is DirectX. As far as my research goes, there is no other way currently.

Write an app using DirectX? That must be hard, you say. In fact, it is not as hard as you think. If your application is mainly application logics, with light 3D visuals, like AutoCAD, it is not that hard. It all depends on how complex your 3D models are. I had no previous DirectX experience, by reading and researching for a couple of weeks, I can write simple 3D renderings now. I will present you those DirectX terms and basics to get you started.

Now it comes the real question: Do I have write my app in C++ to use DirectX? Microsoft says yes. There is also a C# wrapper for DirectX called SharpDX, but I found it doesn’t make things much easier. Also, you put your future to an uncertainty by using a open source package.

During the past week or two, I have worked out a way to separate DirectX and your app code: you put your DirectX code in a Windows Runtime Component project, in C++. Your main UWP app code in a another C# project. That way, you can have the freedom to write your big chunk of code in C#, but only a small portion of rendering code in C++. So far, I found it’s best solution for a HoloLens app structure. In the next post I will show you how that works. Also give you working code for download.

Before ending this post, I want to explain why I am doing that. For two reasons: I like to write C# code more than C++. Indeed, I have far more experience in C++ than in C#. I also have good understanding of COM programming, as the server and consumer, but I don’t like writing a UWP app in C++. It just doesn’t seem natural. I believe many would agree. The second reason is most of my examples are in C#. If you have written Windows or Windows Phone apps before, there is a good chance you were using C#. There is zero chance you did it with C++.



HoloLens Programming in C# and DirectX – Intermission



While writing this programming HoloLens series, an extremely relevant, extremely informative, and extremely on topic video popup. I embedded video here, I think it is very very helpful for developer.

Main points from the video:

1. Battery life: 2hrs heavy renders, 5.5h light use.
2. FOV: 15″ monitor half arm length from eyes. Why? Cost and battery life.
3. A number of sensors (I think GPS is very likely).
4. Transparency is adjustable.
5. For complex visual, need to utilize a server to do the rendering, then push the rendering result to HoloLens.
6. Server can be used for multiple wearers to share the same scene.
7. Connection: WIFI, Bluetooth.
8. Unity is the main program engine.
9. It’s a like a laptop in term of CPU power. (Think of Surface 3.)

I will continue this series later.



HoloLens Programming in C# and DirectX, without Unity



As a developer myself, I am exploring how to get started to develop HoloLens applications for the up coming device. The information has been limited from a developer’s perspective. Currently, all sources pointed to Unity. However, that’s doesn’t answer all the questions. Actually there are far more questions unanswered comparing to that it answers.

I have been researching for the past a couple months. Along the way writing this blog, I have gained a comfortable amount understanding about HoloLens from both a user’s perspective and a developer’s perspective. Thought I would share my research results now. This journey consists of a few posts: I will first give a big picture of HoloLens programming: options and realities, then there will be two or more tutorial like postd about how to develop 3D application with working examples and code to download. Keep in mind, I don’t have a device with me. The code has some assumptions. But based on my understanding, if you trust me, my assumptions are usually close to the reality.

I will answer the questions like: Why use Unity; Why Unity is not the answer for everything; What to do if Unity can’t be used. I will show you the difference between an app and a game. I will show you how to write a DirectX application in C# without using a wrapper like SharpDX.

This is just a setup post, the rest is coming up in the next a few days.