July 1, 2022

HoloLens Programming in C# and DirectX – The Big Picture



We need to be clear about this first: what is developing a HoloLens app? HoloLens is a computer running on Windows 10, by definition, every UWP application should be able to run on HoloLens. So if you write an app for Windows 10 UWP, and publish it to Windows Store. It should run on HoloLens, the way it runs on Windows desktop computers. However, that’s not what I mean by developing for HoloLens. Developing for HoloLens means your app takes advantage of HoloLens 3D mixed reality display, and optionally utilize spatial mapping, hand gestures and voice input. In this post and a few others after this, I only explore 3D display, in the context of HoloLens reality mode, not within a floating/pinned window as for a normal UWP app.

In multiple occasions, Microsoft had hinted Unity with C# script would the way build HoloLens apps. I don’t disagree. Unity is a high level game engine, especially for 3D/2D screen rendering. It is in the same position as XNA was before. Because XNA is no long supported in Windows 10. Unity is probably the best replacement. Unity is recommended for game development in general, not just for HoloLens. Even though you can do DirectX directly, but Unity would make many complicated task much easier otherwise. Also, the Unity Store has tons of stuff for asset creation, some of them are free.

Unity runtime is written in C++, so your C# code for Unity is called C# script, meaning they are not compiled to .Net code. On the low level, Unity is based on DirectX. It is a high level wrapper of DirectX, but a lot more than just a wrapper. The compilation of a Unity project results in a Visual Studio project. There you can deploy, run and debug on HoloLens. To get start with Unity game development, I recommend Adam Tuliper’s post on MSDN: Unity : Developing Your First Game with Unity and C#.

It is clear that if you want to develop a HoloLens 3D Holographic game, or simulation app, like fly simulation, or educational holographic 3D illustration, you can safely choose Unity with C# scripting. Then use Visual Studio to deploy and test on HoloLens.

That’s all good… but, there is a problem.

Unity is a game engine, it is NOT an app engine. The same way XNA is a game engine, not an app engine. I have been asking this question: can you write an app on Unity? Now it is clear, the answer is NO. Of course you can call a fly simulator an app instead of a game, but a game and an app are intrinsically different. Allow me to explain.

A game requires fast real time response and fast real time visual refresh. It sacrifice everything to satisfy the above too requirements. That’s why a game is always running on it own separately loop, called the game loop. An app, on the other hand, is supposed to be on idle until there is input from a user. An app tries to use as less resource as possible. An app is running under system message loop, it is message based. The system take care of visual update.

If you are developing an app instead of a game, you don’t want your application in a busy loop. You don’t want your app to use as much resource as possible to gain frame rate. You want your app to be friendly to other apps.

So, if you develop an app, Unity is out of picture, but you still want to show 3D holograms. Well, that’s the point of HoloLens apps, right? Now what would you do? The answer is DirectX. As far as my research goes, there is no other way currently.

Write an app using DirectX? That must be hard, you say. In fact, it is not as hard as you think. If your application is mainly application logics, with light 3D visuals, like AutoCAD, it is not that hard. It all depends on how complex your 3D models are. I had no previous DirectX experience, by reading and researching for a couple of weeks, I can write simple 3D renderings now. I will present you those DirectX terms and basics to get you started.

Now it comes the real question: Do I have write my app in C++ to use DirectX? Microsoft says yes. There is also a C# wrapper for DirectX called SharpDX, but I found it doesn’t make things much easier. Also, you put your future to an uncertainty by using a open source package.

During the past week or two, I have worked out a way to separate DirectX and your app code: you put your DirectX code in a Windows Runtime Component project, in C++. Your main UWP app code in a another C# project. That way, you can have the freedom to write your big chunk of code in C#, but only a small portion of rendering code in C++. So far, I found it’s best solution for a HoloLens app structure. In the next post I will show you how that works. Also give you working code for download.

Before ending this post, I want to explain why I am doing that. For two reasons: I like to write C# code more than C++. Indeed, I have far more experience in C++ than in C#. I also have good understanding of COM programming, as the server and consumer, but I don’t like writing a UWP app in C++. It just doesn’t seem natural. I believe many would agree. The second reason is most of my examples are in C#. If you have written Windows or Windows Phone apps before, there is a good chance you were using C#. There is zero chance you did it with C++.