July 1, 2022

Sidekick: Another Project Involving HoloLens with NASA

PHOTO DATE: 05-06-15 LOCATION: Ellington Field - NASA C-9 Aircraft SUBJECT: Reduced Gravity Office's JPL research flights. Flight 3 PHOTOGRAPHER(S): BILL STAFFORD

PHOTO DATE: 05-06-15
LOCATION: Ellington Field – NASA C-9 Aircraft
SUBJECT: Reduced Gravity Office’s JPL research flights. Flight 3


This news is bigger than you think, that’s why I rush to post it.

NASA and Microsoft are on another project beside OnSight, called SideKick. See how HoloLens helps astronauts through SideKick project:

Sidekick has two modes of operation. The first is “Remote Expert Mode,” which uses Skype, part of Microsoft, to allow a ground operator to see what a crew member sees, provide real-time guidance, and draw annotations into the crew member’s environment to coach him or her through a task. Until now, crew members have relied on written and voice instructions when performing complex repair tasks or experiments.

The second mode is “Procedure Mode,” which augments standalone procedures with animated holographic illustrations displayed on top of the objects with which the crew is interacting. This capability could lessen the amount of training that future crews will require and could be an invaluable resource for missions deep into our solar system, where communication delays complicate difficult operations.

Brilliant. Now here is the news:

A pair of the devices is scheduled to launch on SpaceX’s seventh commercial resupply mission to the station on June 28.

A second set of devices to be delivered on a future mission will test and verify Sidekick functionality with network connectivity to test the Remote Expert Mode.

Sidekick also will be used and evaluated during the NASA Extreme Environment Mission Operations (NEEMO) 20 expedition set to begin July 21 when a group of astronauts and engineers live in the world’s only undersea research station, Aquarius, for two weeks. The Aquarius habitat and its surroundings provide a convincing analog for space exploration.

Those are test missions for SideKick, and,

NASA expects astronauts on the station will first use Sidekick by the end of the year.

This one sounds like a real thing. Can we expect HoloLens to be released by the end of the year? That’s why I am writing this post.

By the way, they are okey with the FOV?

Oh, NASA also posted a video.



Cambridgeshire Constabulary Is Soon to Begin Using Microsoft’s HoloLens


This news is interesting in many levels.

News comes from Computing that UK’s Cambridgeshire police will SOON start using HoloLens. You heard it right, the title is direct copy from the post. The implementation is in collaboration with a consultancy firm Black Marble, which is likely a Microsoft value added distributor or partner.

A Microsoft spokesperson later told Computing that, while it’s unlikely that members of the Cambridgeshire force will go out on the beat wearing the slightly cumbersome devices in the near future, the technology could well be used in the recreation of crime scenes.

I don’t know how, but sounds like an interesting idea, and a big deal for police forces, but I think HoloLens can do more than recreating crime scenes.  It’s good to know that police decide to use it before it is even released.

The word “soon” is interesting. I would assume, next year is not soon, right?

Interestingly, the same constabulary is also currently using Microsoft Band. This is how they use it:

Finally, wearable device Microsoft Band is also now being utilised in a unique way, in a trial period in Cambridgeshire. If the user’s heart rate goes over a certain level, or if it is removed from an officer’s wrist, it triggers an alarm and officers will scramble to support their colleague.







Why Is There No HoloLens SDK?



This post is long over due. I have had it in the backlog for a while, now I can’t wait anymore.


A Bit of History…

Since the day HoloLens went public, which was January 21, 2015, the developer community have been discussing the possibility of HoloLens SDK. People posted on  Channel 9 of the up coming Windows Holographic APIs from Windows 10 build 9926. They have discovered that there are many holographic related namespaces, even holographic controls (in C++):


The good people from NUI World had discovered namespaces like the following (in C#):


I had explored Windows 10 debug symbols myself. All those made us to believe that Windows Holographic Platform is well defined from the beginning. There were three distinct namespaces: Human, World, and Mirage, which represent the whole picture of the mixed reality. Please check my early post here.


Then It Happened…

We understand at the beginning that HoloLens is one of many Windows 10 devices. It is said from January 21 that all Windows 10 universal apps will run on HoloLens. We also understand HoloLens is a special Windows 10 device, maybe more special than a phone or a tablet. We understand again, for special device families, there is something called Extension SDK to deal with the specialties of certain features that are not implemented on all devices. Naturally, we have guessed this must be the case for HoloLens.

General consensus is there should a set of special APIs for Hololens, or more broadly for Windows Holographic Platform. Even today, you go to HoloLens official website, click Developers on top, then scroll to the bottom to Q&A section, expend “When will the SDK available?”, the answer shows,

We look forward to sharing information on the SDK for Microsoft HoloLens soon. The best way to ensure you receive developer updates and holographic APIs when available is to join the Windows Insider Program.

As it turns out this is false.

Yesterday, Alex Kipman (you don’t know who he is ?!) tweeted in response to #HoloLensDevs,

He actually said the same thing long before yesterday. He had made the clear at Build 2015, there is no special APIs that is specific for HoloLens, therefore SDK won’t be needed. Please check the embedded video at end of the post (@2:20).

Interesting thing is: Out of curiosity, I checked the latest Windows 10 debug symbol packages, well, the latest is still very old, it corresponds to Build 10074, the pdbs that contain afore-mentioned namespaces no longer exist.  For example there use to be,


and many other hologram related pdbs, now only a few suspected ones are left, like,


In any case, this is the very much a confirmation, that HoloLens APIs were planned, and then removed.


To Understand the Fact…

Knowing the fact there will be no HoloLens API, or HoloLens SDK, or Windows Holographic SDK in general is one thing, understanding it is another.

It’s kind of strange why there is no need for HoloLens SDK at all? The way I understand this is: HoloLens is no different than any other Windows 10 devices. It is a PC indeed, by any definition. The only difference, apart from worn on head and look nothing like a computer, is: it is a computer with a monitor, a keyboard, a mouse, touch interface, voice interface all built-in. As it happened, that’s a special monitor, it shows 2D objects and 3D objects, and they are called holograms. That’s all, why do we need any specially SDK at all?

One possible implication is developers will not have access to the spatial mapping data. This can be either a good thing or bad thing. Good thing because the OS hide the complexity, so developers wouldn’t have to worry about the contours of the environment. It’s real hard to write good games or apps if you consider that. Bad thing because sometimes, the environment matters, if you develop a virtual pet, you don’t want it to walk in the air right? This is unknown to me.

When you think of OEMs, who are developing VR devices or AR devices other than HoloLens. They need to write drivers in Windows 10 for their devices, there might be special DDKs for them, which is generally not what we are worrying about.

But wait, how about mixed reality video capturing. That was the thing appeared in the old debug symbols, but disappeared now. Alex had said the function is there (@ the end of the video). Will it be exposed to developer as an API, or to the end user through the OS, or both? We don’t have answer yet.



Microsoft Is Working to Implement Holo-Video Recording For HoloLens

Tech In Short: Surface 3




Xbox One Latest Update: Anything to Do with HoloLens?




Microsoft had released a minor update for Xbox One on June 5th. Microsoft have said on its Xbox website that the update only include a few fixes. But also added one thing:

10-bit HD High Efficiency Video Coding (HEVC) platform support added

Also, yesterday evening, Forbes reported that Microsoft’s Larry Hyrb had tweeted that there is new update to Xbox One, that may change the interface to prepare for the future.

Your Xbox One will get a small update today. No new features. Just a few behind the scenes changes to prepare for the future.

I couldn’t find that tweet, so not sure if that’s the same thing as the first one above or different one. There is no log for that yet on Xbox One support page. Forbes said it might be something relates to UI change. That’s pure speculation, I don’t really buy that theory, because it makes no sense to me to change UI knowing Windows 10 will kick in within two months.

I suspect the above two (if they are not the same) are both related to E3, but how? We know E3 is all about games. The main stage will be set for new titles, a lot of snake peeks and demo plays. However, I think E3 wouldn’t be complete without stream between Xbox One and PCs.  In the context of HEVC, Microsoft mentioned video streaming like Netflix. If that’s just improvement over video streaming, it won’t make it to the news at all.  I am not sure if HEVC is newly added or 10-bit HEVC is newly added. I hope for the first. HEVC is said to have doubled the compression ratio over MPEG and MP4. That’s pretty big.

You already know I am hoping for game streaming, that’s totally different thing than video streaming because the former doesn’t tolerant delays. HEVC will certainly make video transmission much faster, but if the codec process is too slow, it can cause overhead too.

The bigger question is: will it make it to HoloLens? Why do I ask this question? Because 1. For HoloLens, streaming makes a night-or-day difference. 2. A new codec means streaming through WIFI would be possible.

I will be disappointed if this won’t happen in E3.

Field Of Vision: A Short Story

Windows 10 SKUs: The Most Concise List

HoloLens at E3 – What to Expect



HoloLens: Recommended Reading – Stream Xbox One Games to Windows 10 PC




Tom Warren posted this article: Streaming Xbox One games to a Windows 10 PC is awesome on April 30 (during the Build Conference). If you care about HoloLens, this is great new. That’s why recommend to read that article and watch the video again.

Why is this important? Imagine replacing the laptop with a HoloLens. Hook it with Xbox One and the controller. Start Xbox App in HoloLens, select a game to play. You get the game plays on Xbox, it uses Xbox CPU and GPU but HoloLens as a display. The only thing is missing is how to have a wired connection between Xbox One and a HoloLens.

However streaming alone won’t make HoloLens desirable unless HoloLens can show the game contents in 3D. Actually, most game plays already in 3D format, just need HoloLens to realize that. I feel this is very close to reality once Windows 10 is released.


HoloLens: Field of View (FOV) Collections



HoloLens at E3 – What to Expect


A week from today, on June 15th, E3 will start in Los Angeles. Exhibition dates are from 16 to 18, but press events start June 15th.

The major events we are looking at are:

Microsoft press conference – June 15 at 12:30am ET

This is traditional Xbox time slot. New games would be the main theme, but there could be more.

PC Gaming conference – June 16 at 8:00pm ET

This is the first time PC gaming is premiered in E3. Microsoft’s involvement could be the main reason, and Windows 10 is the reason behind all these.


So, how does HoloLens fit into this?

First, as Alex Kipman said during Build 2015, HoloLens is a mobile gaming device. Games developed targeting HoloLens will be relatively light weighted games. However, streaming from Xbox to PCs is sure to happen. HoloLens is a PC, right? Why would it be a problem with HoloLens? Here is the thing: Stream from Xbox to PC is through wire. Wireless streaming is still not a reality dues to the bandwidth, but what’s wrong with wired streaming from Xbox to HoloLens. Nothing wrong for a gamer. I certainly don’t mind wearing HoloLens while playing Xbox games with cable connecting the two.

If streaming like that happens, I guarantee every Xbox owners own a HoloLens within two years.

I am not a gamer, but if there is HoloLens involved, I will be.



HoloLens: Field of View (FOV) Collections




The following is a list of posts of mine about HoloLens FOV. That’s the result of my research of the topic in the past. I feel it’s time to put them together, hope it gives a better picture of what the matter is. I want to quote what Bill Gates said about HoloLens a couple months ago.

Making the device so you don’t get dizzy or nauseous is really hard.

I think that’s the center of the problem everyone in AR/VR trying to solve. We had already seen how it could work in January, we wouldn’t mind to see the same happens in final products.


HoloLens FOV: Why Some Are Bigger, Others Are Smaller?

Field Of Vision: A Short Story

HoloLens: Checking with the Reality

HoloLens, Field of View, and the Fixes…

Realizing HoloLens Field of View (FOV)

HoloLens FOV – What’s Next

HoloLens – The Matter of FOV

Analyzing HoloLens Field of View (FOV) – Updated



Field Of Vision: A Short Story

* This post is filed under Fictional Reality genre.




When: April 21, 2015. Three months after HoloLens first unveiling to the public, one week before Build 2015.

Where: Lab 12. One of most secretive bunkers under the vast land of Redmond.

Characters: Terry, a cooperate executive in Redmond. Joe, a highly publicized cooperate VP who reports to Terry. Alex, an inventor and tenured researcher who is in charge of Project HoloLens.

Settings: A self-service bar with highest security possible. The lock can only be opened when Terry, Joe and Alex are present at the same time. Each of them are holding a separate 128 byte key by a special GUID generator. They all have to memorize them. The lock can only be opened with that 384 characters.


It’s 4:00 in the morning. The sky looks more grimy than it should be, and this is the 5th days of non-stopping rain. “April rain, May flowers”, as the saying goes, but you have to realize this is Redmond.

Alex drives his SUV from his office in main campus Building 24  to Lab 12. After first security check, he is allowed to park his car in front of the lab. Terry and Joe are already waiting in front of the door. It’s a routine: three line up in order of T-J-A, and type their 128 byte password, they have to all finish within 30 second.

The room is tinny and simple. Terry sits down. Joe sits down, so does Alex.

Terry:  “What the hell is that little light flashing over the table?”

Joe: “That is Alex’s new prototype HoloLens,” winking to Alex.

Terry: “Didn’t they say no electronics is allowed in this room?” Terry smashed the HoloLens into pieces with a construction hammer.

Alex: “Terry, you just destroyed my entire night of work!”

Terry: “So?”

Alex is left shaking his head.

Terry: “OK, guys. Again congrats for the show of Windows 10, particularly, the HoloLens! Now we have another show to  do next week: the Build Conference… What’s wrong with your face Alex?”

Alex: “What?”

Terry: “Joe, how does he look?”

Joe: “Miserable?”

Alex: “No, I don’t… Ah, well, I am stressed out.”

Terry: “Go ahead…”

Alex: “OK, gentlemen, we have a tough choice here. The Cherry Trail didn’t work out on HoloLens. We need a more powerful one, but that won’t be available soon enough for the Build. So, we have 500 HoloLens ready in the truck for San Francisco this morning. We can either have a small FOV with no image shifting, no nauseating, or big FOV with noticeable the other two.”

Terry: “Shit! What do you think?”

Alex: “Well, I don’t know. That’s why we are here.”

Terry: “OK. Head for small FOV, tail for nauseating.  I throw this quarter, Joe catch it in the air and pan it to the table.”

The result, as everybody already know is Head.

The meetings happen here are always short and effective, like this one. So out they went.

On the way back to his office, Alex picks up this Lumia 940XL prototype (as a gift from Joe), and calls the guy on the truck: “Where is the truck? OK then, tune down the FOV.”





HoloLens – The Matter of FOV




Reports have come out from BUILD conference focusing on concerns over HoloLens’ FOV (Field of View, or Field of Vision). I first got the message from social media, and I was in shock at first. For a moment, I thought it was the same crowd who complained about FOV when HoloLens was first announced. I understand those who complained usually based their opinion on comparison to VR head sets. There weren’t a concern because they are different technology, and targeting different use cases. Then I read a number of post online, from the same people who were shown the demo last time. I have even checked with friends. I am convinced that there is an issue to deal with, big or small.

Regarding FOV, the feelings are universal, and almost identical: it’s just too small this time around. I looked back at their previous posts, the opinion toward HoloLens FOV are mostly favorable. Most of them felt it’s not great but fine. Of course, bigger would be better.  It has never become a barrier for a great user experience. Honestly, after I have written the post “Analysis of HoloLens FOV“, I have never considered this a problem.

Many people I have read online are surprised. No one had expected this. It is certainly not because the technology is not capable, we have seen how it worked. It should get better over time not the other way around. Now many have realized the setup in two cases are different: First time they were strapped with a laptop. This time it was on a less powerful Atom CherryTrail PC, according to report. Based on that, it can be one of 3 things: 1. The light engine (generating and projecting lights) component has changed to fit in the small chassis. 2. The light engine is the same, but less powerful CPU to power it up, which limits the among of lights it can process. 3. The GPU that takes care of creating the initial image, though that image is never rendered. I am no expert, but if have to take a guess, I tend to think #3 is less likely, because the GPU does the same thing as it does on any other PC. Generally speaking, today’s GPU has no problem dealing with 1080p videos. I think #2 is most likely.

I know there must be difficult decisions behind the scene, but what makes it hard to believe is they (Microsoft) appear to take chance to see how people react. There is strong evidence of that: 1. Obviously, they don’t just find this out today. They must have known it long before today. 2. They have marketing people ask your expectation when you come in, then ask again afterwards. They are testing water. That shouldn’t have happened. If it is really like what people have wrote today, that’s too obvious, there is no need to test water like that. It’s a dangerous act they shouldn’t ever tried.  At least, it damps enthusiasm among everyone who are watching the development. The impact can be more than that.

In any case, I would expect an explanation coming out in any form. People need to be convinced there is a fix. Related question is: how would that affect the  release date. One thing is almost certain: HoloLens can not be released in this state. The reason is obvious.

The positive side is we knew it worked.




Analyzing HoloLens Field of View (FOV) – Updated


Fig. 1 Peripheral Vision

Fig. 1 Peripheral Vision


Update on June 25, 2015:

After exchange with the author from doc-ok.org, now I believe measurement of FOV from the article is indeed in line with what it is in the close-to-final HoloLens prototype.

Update on May 17, 2015:
After reading anon’s comments (in the comment section below), I headed to Doc-Ok.org, and read the article about the author’s 15min try-out of HoloLens. I want to add some updates and comments to this article. Reason #1 is this article is all based on the first prototype shown in January. As many have noticed, there is a big difference between current near-final product and the first prototype, mostly field of view. Reason #2 is even this post is out-dated, it is still of the most viewed.

I choose to keep the original post intact, only add update right here at the top, because I am not convinced that the second demo unit is the product going forward and the first one is not, if you know what I mean.

So, here is my comments to the doc-ok.org article: On the road for VR: Microsoft HoloLens at Build 2015, San Francisco.

  • It is based on the BUILD 2015 unit, no reference of the first one.
  • The article states a FOV of 30×17 degrees. At first I was surprised, but I cannot find facts that prove he’s wrong. I talked with someone else, and I generally accept his estimation in the sense of user experience level, not necessary his numbers. Now, imagine some engineers from NASA are going to wear that to manipulate MARS Rover in July, what is it going to be like? I would like to direct you to another of my post: HoloLens, Field of View, and the Fixes…
  • Regarding transparency, please read this post of mine.
  • Regarding battery life, please read Guessing HoloLens Battery Life.

Update on May 17, 2015 ends. Thanks anon for the head-up.
Field of View (FOV) is the main topic for virtual reality (VR) and augmented reality (AR). Achieving the greatest FOV is one of major measurements of technical success, especially for VR. It is also a potential concern for HoloLens. Recently, I did some research on this topic. I also dove into the limited information I have with HoloLens. By putting these together, I am trying to give you a sense of satisfaction level regarding HoloLens’ FOV. Remember this is not an assessment of the HoloLens device. The writing is based on the mere information I have about HoloLens.


About FOV

Field of View (also called Field of Vision) is the extent of the observable world that is seen at any given moment (from Wikipedia). The term is used in the context of human vision or the view angle of instruments. To understand FOV, lets look at human peripheral vision as shown in Fig. 1 (also from Wikipedia).  Peripheral vision is the part of vision surrounding the center focus. For human,  peripheral vision is divided into difference areas: a 60 degrees of inner area called near peripheral vision (shown 30 degrees in Fig. 1, which is half of the symmetrical area). Outside the near peripheral vision is a 120 degree area surrounding the near peripheral vision, called mid-peripheral vision. Outside it is about 200~220 degrees of far peripheral vision.  Generally speaking, anything inside of mid-peripheral vision is important to our vision (120 degrees), outside of it is good to have. Especially with eye tracking technology, far peripheral vision can be remedied greatly.


For HoloLens, there are two FOVs: The depth senor’s FOV, which is what the sensor sees, and the wearer’s FOV, which is what the wearer sees. 


Why Microsoft Should Cover the VR Ground?


FOV of HoloLens – (1) Depth Sensor

In HoloLens, the depth sensors are in charge of mapping the room and recognize user’s hand gestures. The FOV of the depth sensors directly affects these two things. For the first, it affects how wide a 3D space it can map. This is important for 3D space aware applications. For the second, it affects how well the main input method in HoloLens: hand gestures work. In some sense, the second is more significant than the first.

We already know HoloLens has a field of view of 120 degree by 120 degree. To understand what that means, I took a statistical median of a regular man’s anatomical figures (Don’t ask how I figure that out). Also I want to point out the height of the person doesn’t really matter for any adults, because it only concerns distance from the eye to the joint of the upper arm and the forearm), and drew the figure and areas that the 120 degree vision covers. The result is in Fig. 2.


Fig. 2 Depth Sensor FOV

Fig. 2 Depth Sensor FOV


In Fig. 2, the shaded area is the space HoloLens depth sensor can’t see. First thing we realize: if the person stands up, there is a certain 3D space close to the person’s feet cannot be seen by the depth sensors.  The area gets much smaller when sitting down. I am not sure if there is technology in place to deal with it, but I don’t think it’s something important, unless applications really need depth info of that area. More interesting thing is: when you put down your upper arm, then keep your forearm to the front horizontally, the depth sensor should be able to see your hand. That’s important because that’s likely the bottom limit for hand gesture recognition. So, basically, in order for your hands/finger movement to be recognized, the afore mentioned pose should be enough, maybe raise the hand a little bit. That’s not what some have said you have to raise your hand to the mid-air. If you are sitting at the desk, just put your hands naturally at where you are using a keyboard, your hand gestures should be seen.

Why HoloLens Is Not Kinect


FOV of HoloLens – (2) Wearer

Now we will look at what the wearer sees when he puts on the HoloLens. We are specifically talking about the view of the digital content, not including the surrounding real world objects. Information I have about this is almost nothing. Most of journalists who were given a demo on Jan. 21st stated the FOV is limited, some even said it is narrow. This doesn’t give us any information because it’s all depending on your point of view. I believe those comments are based on the comparison to FOVs in VR handsets. Eventually I decided to use Daniel Rubino’s comment under his article on Windows Central.  Reasons are: First, no one else gives a measurable information that is helpful for my purpose. Second, Daniel is a reputable writer and I can trust him for being factual, non-bias. In answer reader’s question in the same article, he said,

…field of vision, it was fine, but I would liked to have seen more. Look straight ahead at a wall, now put a rectangle in front of you that takes up about 80% of your space, and that is what it is like.

Remember this is just from his impression, we shouldn’t take his word literally. However, I feel that’s enough information to answer many questions about FOV of HoloLens. You have to do some homework as I do. I take the “80%” as 80% of digital area are visible comparing to a general 120 degree mid-peripheral vision (I have read his comment many times, and looked at many places to draw the conclusion). 80% of the 120 degree vision is about 100 degrees of FOV (not linear, trust my math).  I use Fig. 3 to show what it is like for a 80% area comparing to the whole.


Fig. 3 80% Vision Area


The pretend outer rectangle is the entire 120 degree of vision, the inner rectangle would be the 80% area. From the look of it, 100 degrees of FOV is pretty good. It depends on your purpose, for totally immersive gamming like VR does, this is probably not enough. I believe the FOVs in VR are bigger than that, but for anything else, I think it is not a concern at all.


PC + HoloLens: Would It Work?

Why HoloLens’ Most Remarkable Achievement Is Its Display