December 7, 2021

Sidekick: Another Project Involving HoloLens with NASA

PHOTO DATE: 05-06-15 LOCATION: Ellington Field - NASA C-9 Aircraft SUBJECT: Reduced Gravity Office's JPL research flights. Flight 3 PHOTOGRAPHER(S): BILL STAFFORD

PHOTO DATE: 05-06-15
LOCATION: Ellington Field – NASA C-9 Aircraft
SUBJECT: Reduced Gravity Office’s JPL research flights. Flight 3
PHOTOGRAPHER(S): BILL STAFFORD

 

This news is bigger than you think, that’s why I rush to post it.

NASA and Microsoft are on another project beside OnSight, called SideKick. See how HoloLens helps astronauts through SideKick project:

Sidekick has two modes of operation. The first is “Remote Expert Mode,” which uses Skype, part of Microsoft, to allow a ground operator to see what a crew member sees, provide real-time guidance, and draw annotations into the crew member’s environment to coach him or her through a task. Until now, crew members have relied on written and voice instructions when performing complex repair tasks or experiments.

The second mode is “Procedure Mode,” which augments standalone procedures with animated holographic illustrations displayed on top of the objects with which the crew is interacting. This capability could lessen the amount of training that future crews will require and could be an invaluable resource for missions deep into our solar system, where communication delays complicate difficult operations.

Brilliant. Now here is the news:

A pair of the devices is scheduled to launch on SpaceX’s seventh commercial resupply mission to the station on June 28.

A second set of devices to be delivered on a future mission will test and verify Sidekick functionality with network connectivity to test the Remote Expert Mode.

Sidekick also will be used and evaluated during the NASA Extreme Environment Mission Operations (NEEMO) 20 expedition set to begin July 21 when a group of astronauts and engineers live in the world’s only undersea research station, Aquarius, for two weeks. The Aquarius habitat and its surroundings provide a convincing analog for space exploration.

Those are test missions for SideKick, and,

NASA expects astronauts on the station will first use Sidekick by the end of the year.

This one sounds like a real thing. Can we expect HoloLens to be released by the end of the year? That’s why I am writing this post.

By the way, they are okey with the FOV?

Oh, NASA also posted a video.
 

 

 

Cambridgeshire Constabulary Is Soon to Begin Using Microsoft’s HoloLens

hl4

This news is interesting in many levels.

News comes from Computing that UK’s Cambridgeshire police will SOON start using HoloLens. You heard it right, the title is direct copy from the post. The implementation is in collaboration with a consultancy firm Black Marble, which is likely a Microsoft value added distributor or partner.

A Microsoft spokesperson later told Computing that, while it’s unlikely that members of the Cambridgeshire force will go out on the beat wearing the slightly cumbersome devices in the near future, the technology could well be used in the recreation of crime scenes.

I don’t know how, but sounds like an interesting idea, and a big deal for police forces, but I think HoloLens can do more than recreating crime scenes.  It’s good to know that police decide to use it before it is even released.

The word “soon” is interesting. I would assume, next year is not soon, right?

Interestingly, the same constabulary is also currently using Microsoft Band. This is how they use it:

Finally, wearable device Microsoft Band is also now being utilised in a unique way, in a trial period in Cambridgeshire. If the user’s heart rate goes over a certain level, or if it is removed from an officer’s wrist, it triggers an alarm and officers will scramble to support their colleague.

 

 

 

 

 

 

Why Is There No HoloLens SDK?

Gallo-blog-1-v2

 

This post is long over due. I have had it in the backlog for a while, now I can’t wait anymore.

 

A Bit of History…

Since the day HoloLens went public, which was January 21, 2015, the developer community have been discussing the possibility of HoloLens SDK. People posted on  Channel 9 of the up coming Windows Holographic APIs from Windows 10 build 9926. They have discovered that there are many holographic related namespaces, even holographic controls (in C++):

Windows::Mirage::HologramFramework::Controls::Button
Windows::Mirage::HologramFramework::Controls::ListView
Windows::Mirage::HologramFramework::Controls::TextBlock
Windows::Mirage::HologramFramework::Controls::TextBox

The good people from NUI World had discovered namespaces like the following (in C#):

Windows.Human
Windows.Human.Gestures
Windows.Human.Hands
Windows.World
Windows.World.Lighting.Prototype
Windows.World.Sensors
Windows.World.Surfaces
Windows.World.Surfaces.Prototype

I had explored Windows 10 debug symbols myself. All those made us to believe that Windows Holographic Platform is well defined from the beginning. There were three distinct namespaces: Human, World, and Mirage, which represent the whole picture of the mixed reality. Please check my early post here.

 

Then It Happened…

We understand at the beginning that HoloLens is one of many Windows 10 devices. It is said from January 21 that all Windows 10 universal apps will run on HoloLens. We also understand HoloLens is a special Windows 10 device, maybe more special than a phone or a tablet. We understand again, for special device families, there is something called Extension SDK to deal with the specialties of certain features that are not implemented on all devices. Naturally, we have guessed this must be the case for HoloLens.

General consensus is there should a set of special APIs for Hololens, or more broadly for Windows Holographic Platform. Even today, you go to HoloLens official website, click Developers on top, then scroll to the bottom to Q&A section, expend “When will the SDK available?”, the answer shows,

We look forward to sharing information on the SDK for Microsoft HoloLens soon. The best way to ensure you receive developer updates and holographic APIs when available is to join the Windows Insider Program.

As it turns out this is false.

Yesterday, Alex Kipman (you don’t know who he is ?!) tweeted in response to #HoloLensDevs,


He actually said the same thing long before yesterday. He had made the clear at Build 2015, there is no special APIs that is specific for HoloLens, therefore SDK won’t be needed. Please check the embedded video at end of the post (@2:20).

Interesting thing is: Out of curiosity, I checked the latest Windows 10 debug symbol packages, well, the latest is still very old, it corresponds to Build 10074, the pdbs that contain afore-mentioned namespaces no longer exist.  For example there use to be,

windows.mirage.hologramframework.pdb

and many other hologram related pdbs, now only a few suspected ones are left, like,

windows.mirage.holgramframwork.query.pdb
holoshellruntime.pdb

In any case, this is the very much a confirmation, that HoloLens APIs were planned, and then removed.

 

To Understand the Fact…

Knowing the fact there will be no HoloLens API, or HoloLens SDK, or Windows Holographic SDK in general is one thing, understanding it is another.

It’s kind of strange why there is no need for HoloLens SDK at all? The way I understand this is: HoloLens is no different than any other Windows 10 devices. It is a PC indeed, by any definition. The only difference, apart from worn on head and look nothing like a computer, is: it is a computer with a monitor, a keyboard, a mouse, touch interface, voice interface all built-in. As it happened, that’s a special monitor, it shows 2D objects and 3D objects, and they are called holograms. That’s all, why do we need any specially SDK at all?

One possible implication is developers will not have access to the spatial mapping data. This can be either a good thing or bad thing. Good thing because the OS hide the complexity, so developers wouldn’t have to worry about the contours of the environment. It’s real hard to write good games or apps if you consider that. Bad thing because sometimes, the environment matters, if you develop a virtual pet, you don’t want it to walk in the air right? This is unknown to me.

When you think of OEMs, who are developing VR devices or AR devices other than HoloLens. They need to write drivers in Windows 10 for their devices, there might be special DDKs for them, which is generally not what we are worrying about.

But wait, how about mixed reality video capturing. That was the thing appeared in the old debug symbols, but disappeared now. Alex had said the function is there (@ the end of the video). Will it be exposed to developer as an API, or to the end user through the OS, or both? We don’t have answer yet.

 

 

Microsoft Is Working to Implement Holo-Video Recording For HoloLens

Tech In Short: Surface 3