July 1, 2022

Sidekick: Another Project Involving HoloLens with NASA

PHOTO DATE: 05-06-15 LOCATION: Ellington Field - NASA C-9 Aircraft SUBJECT: Reduced Gravity Office's JPL research flights. Flight 3 PHOTOGRAPHER(S): BILL STAFFORD

PHOTO DATE: 05-06-15
LOCATION: Ellington Field – NASA C-9 Aircraft
SUBJECT: Reduced Gravity Office’s JPL research flights. Flight 3


This news is bigger than you think, that’s why I rush to post it.

NASA and Microsoft are on another project beside OnSight, called SideKick. See how HoloLens helps astronauts through SideKick project:

Sidekick has two modes of operation. The first is “Remote Expert Mode,” which uses Skype, part of Microsoft, to allow a ground operator to see what a crew member sees, provide real-time guidance, and draw annotations into the crew member’s environment to coach him or her through a task. Until now, crew members have relied on written and voice instructions when performing complex repair tasks or experiments.

The second mode is “Procedure Mode,” which augments standalone procedures with animated holographic illustrations displayed on top of the objects with which the crew is interacting. This capability could lessen the amount of training that future crews will require and could be an invaluable resource for missions deep into our solar system, where communication delays complicate difficult operations.

Brilliant. Now here is the news:

A pair of the devices is scheduled to launch on SpaceX’s seventh commercial resupply mission to the station on June 28.

A second set of devices to be delivered on a future mission will test and verify Sidekick functionality with network connectivity to test the Remote Expert Mode.

Sidekick also will be used and evaluated during the NASA Extreme Environment Mission Operations (NEEMO) 20 expedition set to begin July 21 when a group of astronauts and engineers live in the world’s only undersea research station, Aquarius, for two weeks. The Aquarius habitat and its surroundings provide a convincing analog for space exploration.

Those are test missions for SideKick, and,

NASA expects astronauts on the station will first use Sidekick by the end of the year.

This one sounds like a real thing. Can we expect HoloLens to be released by the end of the year? That’s why I am writing this post.

By the way, they are okey with the FOV?

Oh, NASA also posted a video.



Cambridgeshire Constabulary Is Soon to Begin Using Microsoft’s HoloLens


This news is interesting in many levels.

News comes from Computing that UK’s Cambridgeshire police will SOON start using HoloLens. You heard it right, the title is direct copy from the post. The implementation is in collaboration with a consultancy firm Black Marble, which is likely a Microsoft value added distributor or partner.

A Microsoft spokesperson later told Computing that, while it’s unlikely that members of the Cambridgeshire force will go out on the beat wearing the slightly cumbersome devices in the near future, the technology could well be used in the recreation of crime scenes.

I don’t know how, but sounds like an interesting idea, and a big deal for police forces, but I think HoloLens can do more than recreating crime scenes.  It’s good to know that police decide to use it before it is even released.

The word “soon” is interesting. I would assume, next year is not soon, right?

Interestingly, the same constabulary is also currently using Microsoft Band. This is how they use it:

Finally, wearable device Microsoft Band is also now being utilised in a unique way, in a trial period in Cambridgeshire. If the user’s heart rate goes over a certain level, or if it is removed from an officer’s wrist, it triggers an alarm and officers will scramble to support their colleague.







Why Is There No HoloLens SDK?



This post is long over due. I have had it in the backlog for a while, now I can’t wait anymore.


A Bit of History…

Since the day HoloLens went public, which was January 21, 2015, the developer community have been discussing the possibility of HoloLens SDK. People posted on  Channel 9 of the up coming Windows Holographic APIs from Windows 10 build 9926. They have discovered that there are many holographic related namespaces, even holographic controls (in C++):


The good people from NUI World had discovered namespaces like the following (in C#):


I had explored Windows 10 debug symbols myself. All those made us to believe that Windows Holographic Platform is well defined from the beginning. There were three distinct namespaces: Human, World, and Mirage, which represent the whole picture of the mixed reality. Please check my early post here.


Then It Happened…

We understand at the beginning that HoloLens is one of many Windows 10 devices. It is said from January 21 that all Windows 10 universal apps will run on HoloLens. We also understand HoloLens is a special Windows 10 device, maybe more special than a phone or a tablet. We understand again, for special device families, there is something called Extension SDK to deal with the specialties of certain features that are not implemented on all devices. Naturally, we have guessed this must be the case for HoloLens.

General consensus is there should a set of special APIs for Hololens, or more broadly for Windows Holographic Platform. Even today, you go to HoloLens official website, click Developers on top, then scroll to the bottom to Q&A section, expend “When will the SDK available?”, the answer shows,

We look forward to sharing information on the SDK for Microsoft HoloLens soon. The best way to ensure you receive developer updates and holographic APIs when available is to join the Windows Insider Program.

As it turns out this is false.

Yesterday, Alex Kipman (you don’t know who he is ?!) tweeted in response to #HoloLensDevs,

He actually said the same thing long before yesterday. He had made the clear at Build 2015, there is no special APIs that is specific for HoloLens, therefore SDK won’t be needed. Please check the embedded video at end of the post (@2:20).

Interesting thing is: Out of curiosity, I checked the latest Windows 10 debug symbol packages, well, the latest is still very old, it corresponds to Build 10074, the pdbs that contain afore-mentioned namespaces no longer exist.  For example there use to be,


and many other hologram related pdbs, now only a few suspected ones are left, like,


In any case, this is the very much a confirmation, that HoloLens APIs were planned, and then removed.


To Understand the Fact…

Knowing the fact there will be no HoloLens API, or HoloLens SDK, or Windows Holographic SDK in general is one thing, understanding it is another.

It’s kind of strange why there is no need for HoloLens SDK at all? The way I understand this is: HoloLens is no different than any other Windows 10 devices. It is a PC indeed, by any definition. The only difference, apart from worn on head and look nothing like a computer, is: it is a computer with a monitor, a keyboard, a mouse, touch interface, voice interface all built-in. As it happened, that’s a special monitor, it shows 2D objects and 3D objects, and they are called holograms. That’s all, why do we need any specially SDK at all?

One possible implication is developers will not have access to the spatial mapping data. This can be either a good thing or bad thing. Good thing because the OS hide the complexity, so developers wouldn’t have to worry about the contours of the environment. It’s real hard to write good games or apps if you consider that. Bad thing because sometimes, the environment matters, if you develop a virtual pet, you don’t want it to walk in the air right? This is unknown to me.

When you think of OEMs, who are developing VR devices or AR devices other than HoloLens. They need to write drivers in Windows 10 for their devices, there might be special DDKs for them, which is generally not what we are worrying about.

But wait, how about mixed reality video capturing. That was the thing appeared in the old debug symbols, but disappeared now. Alex had said the function is there (@ the end of the video). Will it be exposed to developer as an API, or to the end user through the OS, or both? We don’t have answer yet.



Microsoft Is Working to Implement Holo-Video Recording For HoloLens

Tech In Short: Surface 3




Xbox One Latest Update: Anything to Do with HoloLens?




Microsoft had released a minor update for Xbox One on June 5th. Microsoft have said on its Xbox website that the update only include a few fixes. But also added one thing:

10-bit HD High Efficiency Video Coding (HEVC) platform support added

Also, yesterday evening, Forbes reported that Microsoft’s Larry Hyrb had tweeted that there is new update to Xbox One, that may change the interface to prepare for the future.

Your Xbox One will get a small update today. No new features. Just a few behind the scenes changes to prepare for the future.

I couldn’t find that tweet, so not sure if that’s the same thing as the first one above or different one. There is no log for that yet on Xbox One support page. Forbes said it might be something relates to UI change. That’s pure speculation, I don’t really buy that theory, because it makes no sense to me to change UI knowing Windows 10 will kick in within two months.

I suspect the above two (if they are not the same) are both related to E3, but how? We know E3 is all about games. The main stage will be set for new titles, a lot of snake peeks and demo plays. However, I think E3 wouldn’t be complete without stream between Xbox One and PCs.  In the context of HEVC, Microsoft mentioned video streaming like Netflix. If that’s just improvement over video streaming, it won’t make it to the news at all.  I am not sure if HEVC is newly added or 10-bit HEVC is newly added. I hope for the first. HEVC is said to have doubled the compression ratio over MPEG and MP4. That’s pretty big.

You already know I am hoping for game streaming, that’s totally different thing than video streaming because the former doesn’t tolerant delays. HEVC will certainly make video transmission much faster, but if the codec process is too slow, it can cause overhead too.

The bigger question is: will it make it to HoloLens? Why do I ask this question? Because 1. For HoloLens, streaming makes a night-or-day difference. 2. A new codec means streaming through WIFI would be possible.

I will be disappointed if this won’t happen in E3.

Field Of Vision: A Short Story

Windows 10 SKUs: The Most Concise List

HoloLens at E3 – What to Expect



HoloLens: Recommended Reading – Stream Xbox One Games to Windows 10 PC




Tom Warren posted this article: Streaming Xbox One games to a Windows 10 PC is awesome on April 30 (during the Build Conference). If you care about HoloLens, this is great new. That’s why recommend to read that article and watch the video again.

Why is this important? Imagine replacing the laptop with a HoloLens. Hook it with Xbox One and the controller. Start Xbox App in HoloLens, select a game to play. You get the game plays on Xbox, it uses Xbox CPU and GPU but HoloLens as a display. The only thing is missing is how to have a wired connection between Xbox One and a HoloLens.

However streaming alone won’t make HoloLens desirable unless HoloLens can show the game contents in 3D. Actually, most game plays already in 3D format, just need HoloLens to realize that. I feel this is very close to reality once Windows 10 is released.


HoloLens: Field of View (FOV) Collections



HoloLens at E3 – What to Expect


A week from today, on June 15th, E3 will start in Los Angeles. Exhibition dates are from 16 to 18, but press events start June 15th.

The major events we are looking at are:

Microsoft press conference – June 15 at 12:30am ET

This is traditional Xbox time slot. New games would be the main theme, but there could be more.

PC Gaming conference – June 16 at 8:00pm ET

This is the first time PC gaming is premiered in E3. Microsoft’s involvement could be the main reason, and Windows 10 is the reason behind all these.


So, how does HoloLens fit into this?

First, as Alex Kipman said during Build 2015, HoloLens is a mobile gaming device. Games developed targeting HoloLens will be relatively light weighted games. However, streaming from Xbox to PCs is sure to happen. HoloLens is a PC, right? Why would it be a problem with HoloLens? Here is the thing: Stream from Xbox to PC is through wire. Wireless streaming is still not a reality dues to the bandwidth, but what’s wrong with wired streaming from Xbox to HoloLens. Nothing wrong for a gamer. I certainly don’t mind wearing HoloLens while playing Xbox games with cable connecting the two.

If streaming like that happens, I guarantee every Xbox owners own a HoloLens within two years.

I am not a gamer, but if there is HoloLens involved, I will be.



Field Of Vision: A Short Story

* This post is filed under Fictional Reality genre.




When: April 21, 2015. Three months after HoloLens first unveiling to the public, one week before Build 2015.

Where: Lab 12. One of most secretive bunkers under the vast land of Redmond.

Characters: Terry, a cooperate executive in Redmond. Joe, a highly publicized cooperate VP who reports to Terry. Alex, an inventor and tenured researcher who is in charge of Project HoloLens.

Settings: A self-service bar with highest security possible. The lock can only be opened when Terry, Joe and Alex are present at the same time. Each of them are holding a separate 128 byte key by a special GUID generator. They all have to memorize them. The lock can only be opened with that 384 characters.


It’s 4:00 in the morning. The sky looks more grimy than it should be, and this is the 5th days of non-stopping rain. “April rain, May flowers”, as the saying goes, but you have to realize this is Redmond.

Alex drives his SUV from his office in main campus Building 24  to Lab 12. After first security check, he is allowed to park his car in front of the lab. Terry and Joe are already waiting in front of the door. It’s a routine: three line up in order of T-J-A, and type their 128 byte password, they have to all finish within 30 second.

The room is tinny and simple. Terry sits down. Joe sits down, so does Alex.

Terry:  “What the hell is that little light flashing over the table?”

Joe: “That is Alex’s new prototype HoloLens,” winking to Alex.

Terry: “Didn’t they say no electronics is allowed in this room?” Terry smashed the HoloLens into pieces with a construction hammer.

Alex: “Terry, you just destroyed my entire night of work!”

Terry: “So?”

Alex is left shaking his head.

Terry: “OK, guys. Again congrats for the show of Windows 10, particularly, the HoloLens! Now we have another show to  do next week: the Build Conference… What’s wrong with your face Alex?”

Alex: “What?”

Terry: “Joe, how does he look?”

Joe: “Miserable?”

Alex: “No, I don’t… Ah, well, I am stressed out.”

Terry: “Go ahead…”

Alex: “OK, gentlemen, we have a tough choice here. The Cherry Trail didn’t work out on HoloLens. We need a more powerful one, but that won’t be available soon enough for the Build. So, we have 500 HoloLens ready in the truck for San Francisco this morning. We can either have a small FOV with no image shifting, no nauseating, or big FOV with noticeable the other two.”

Terry: “Shit! What do you think?”

Alex: “Well, I don’t know. That’s why we are here.”

Terry: “OK. Head for small FOV, tail for nauseating.  I throw this quarter, Joe catch it in the air and pan it to the table.”

The result, as everybody already know is Head.

The meetings happen here are always short and effective, like this one. So out they went.

On the way back to his office, Alex picks up this Lumia 940XL prototype (as a gift from Joe), and calls the guy on the truck: “Where is the truck? OK then, tune down the FOV.”





Alex Kipman Channel 9 Interview @ //BUILD/2015 – Must See


Alex Kipman, the head of HoloLens was interviewed in Channel 9 at //BUILD/ yesterday. The interview was live streamed. If you missed that, now the recorded video has been pushed and you can watch it right here.

This is a must see video if you are interested in HoloLens in any way. I will just write down some random thoughts watching the video.

Alex looks very approachable. He is no-nonsense, right to the point type of guy. He is open, and sincere in what he say. There is no diplomacy, exaggeration.

There is no HoloLens specific APIs. HoloLens is one product that uses Windows Holographic. There will be more devices.

Terabytes of data going in at any moment. First processed by HPU then to CPU and GPU. HPU takes care of human and world understanding.

The architecture/construction example is impressive, engineers can go to the site and put the part to be done on the part of already done.

He said there is no difference between development regular mobile 3D application than HoloLens development. I think DriectX or Unity should something you are looking at.

In Skype with HoloLens, person wearing HoloLens is not visible. In a collaboration setup, people each other by form avatars.

Cortana appears similar to on a PC or a phone.

User has control over where the application stands, developer has control over anything inside it. Of course developer can give user control of the content, that’s up to the developers.

HoloLens contains next generation of Kinect, that is “significantly more powerful than currently shipped Kinect.

He talked about how the holographic demos was made behind the scenes. I have wrote an article speculated that mixed reality scenes are captured then streamed to the camera. That’s not the case. Actually, they have a special camera(4K, 8K) that has HoloLens components on top of it. It’s like the camera is wearing HoloLens. I was right, however, HoloLens does capture mixed really videos, with both holograms and real world in it. That’s exciting. I think there will be a lot of mixed reality hologram videos online soon.

He also said if two people in same room both wearing HoloLens, they can see the same scene. I don’t think that automatic, but it can work that way. That’s interesting, how do they sync with each other?



Watch HoloLens Presentation From //BUILD/2015 Keynote (Day1)




Many people have watched Build day 1 keynote Live stream or watched the video after the event. It’s 3 hours long, and have 3 parts: 1. Azure, Cloud (ScottGu), 2. Office, Outlook.co, 3. Windows 10: (Terry Myerson, Joe Belfiore, Alex Kipman). The last part is Alex, 25min all about HoloLens.

Now I want to share with you the 25 minute HoloLens section. Enjoy.