June 25, 2017

HoloLens Programming in C# and DirectX – A Closer Look


Edit: The content of this is still valid, but some of the practice may not be ideal, please here for details.

We are not ready for the code yet. Before I show you how I do C# and DirectX, I want to do two things in one post: 1. Show you all the options we have for programming DirectX in C#. 2. Give you a little introduction about what the hell DirectX is. Lets get to it.

What are the options?

1. Unity + C# script.
As I said in the previous posts, this is not an option for HoloLens apps. Because Unity is for games only. You can still do apps with Unity, but it is going to be resource intensive, as you may expect for a game. If you have large amount of 3D visuals, like in a flight simulation, or the one like Case Western Reserve University does with HoloLens, you are still better off with Unity, but for light 3D rendering, mostly business logic and communication or things like a normal app does, you better write an app instead of a game. You need to look for something other than Unity, or any other game engines.

It would be ideal if Unity or another game engine manage rendering in the way like an app does, that would great for application development, but that’s not the case now. If the app itself has very dynamic objects in the scene, they are moving all the time, and requires fastest response possible from the user, then you have to go with game, no matter it is called a game or not.

2. C++/C# UWP app with DirectX in CoreWindow
CoreWindow is a class that represents the utter most application window of a UWP app. There are two project templates in Visual Studio 2015: DirectX 11 App and DirectX 12 App (New Project -> Visual C++ -> Windows -> Universal ->). Either of those two template generates bare bone code that is based on CoreWindow.

The templates titled “App”, but they are actually games. There is a game loop on app level. Visual updates are done at a possible frame rate. The DirectX rendering target is the app’s CoreWindow. The two templates are C++ projects, as I will show you in next post, it is very much possible to code DirectX in C++ and C# for rest parts, but this one is not our interest, because again, it is a game, not an app.

Let me try one more time to make it clear about the difference between an app and a game from a developer’s point of view. A game tries to use as much CPU/GPU as possible to satisfy a game’s need for frame rate. That’s why it runs on game loop to update visuals, instead of let the system handle them. Lets say, beside display and interaction, your app have a lot of other task to do, if your application is trying every 20 ms to update visuals, what time is left for everything else? I just found it’s not the ideal way of programming.

3. C++/C# DirectX with SwapChainPanel
SwapChainPanel is a xaml control designed to make DirectX work inside of the xaml controls. This is much like XNA and Silverlight interop in the past. It achieve the best of both world. Usually you put SwapChainPanel as the main scene, with other controls work as overlay. SwapChainPanel deal with DirectX rendering. However, the design of SwapChainPanel is still game oriented. You may use SwapChainPanel as islands among other controls, but the general guideline from Microsoft is SwapChainPanel targeting game development.

By the way, there was a SwapChainBackgroundPanel, which is replace by SwapChainPanel.

4. C++/C# DirectX with SurfaceImageSource
SurfaceImageSource works like general ImageSource for XAML controls. Many XAML controls can have ImageSource, SurfaceImageSource works in similar way, the difference is SurfaceImageSource is a surface for DirectX renderings. The resource of than rendering can be used as a image source for many different controls. In the next post, I will show you how to do the rendering part in C++, and everything else C#, like what you do in any UWP apps.

5. C++/C# DirectX with VirtualSurfaceImageSource
VirtualSurfaceImageSource is the same as SurfaceImageSource, except the former is for DirectX rendering of unlimited size, usually larger than the screen size. This is, I think, the best fit for HoloLen applications, because your visuals are likely spreading in the space instead of a small window. In the post after next one, I will show you a working example of that.

So, what the hell is DirectX any way?

DirectX is the ultimate 3D rendering APIs provide by all modern Windows OSes. Whatever 3D engines you know that work on Windows, work on top of DirectX. Being Unity, XNA, Unreal Engine, or any others, there is no exception, they are on top of DirectX.

DirectX is a collection of APIs that do not only 3D, also 2D, Text, Texture, shaders, that’s the X in DirectX. They are:

* Direct3D – for 3D objects
* Direct2D – for 2D overlays
* DriectWrite – for text
* Untilities: DirectXMath, XAudio, XInput

That’s not all, the following are parts of the DirectX subsystem:

* DXGI – Microsoft DirectX Graphics Infrastructure, deals with display devices directly. Because 3D drawing need to be faster than usual, that’s why programmer are required to deal with hardware directly. That’s the interface for it.
* HLSL – High Level Shading Language for DirectX. You write shading in this special language, with extension .hlsl, Visual Studio will compile it to binary.

In DiretX programming, you hear one term a lot, but it doesn’t make sense to most people, that’s Swap Chain. Forget about “swap” or “chain”, think of it as a display buffer, you write on the buffer instead of the monitor itself, the video card will take care of it. The reason it is called a chain because there are more than one buffers, one after another to improve performance. Why swap, well, you don’t need to know. Swap chain has nothing to do with DirectX, it is part of display technology existed long time ago.

Why DirectX programming is hard?

There are two reasons, and two only: 1. the designer of DirectX decided to use stupid names for their APIs, like D3D11CreateDevice(). Fyi, take this one for example, D3D11 means Direct3D version 11. They could just call it CreateDeive(), but they decided to make our lives harder. 2. We don’t do 3D that much. But I promise you, it is just a matter of familiarity. When you see the term “Swap Chain” or the ugly function names the 20th time, they start making sense to you.

That’s it. Next up is an example of SurfaceImageSource, then there is another sample code of VirtualSurfaceImageSource.


  • Thom

    You are saying that Unity + C# “is not an option”. Even though I do understand your reasoning, do you mean it’s going to be just impossible, or not recommended? I am planning on creating an app for the Hololens with which you can use physical controls to animate a few light 3D models. Would this be possible in Unity, you think? I have quite some experience in Unity, and shamefully not in the other options you listed.

    • Thom

      Excuse me – I just read your ‘The Big Picture’ post, which kinda already answered me question :).