CastleHUB and CastleOS Review – Part 3 – Voice Control Test

This post is part of a series of my thoughts on CastleOS and the CastleHUB. If you haven’t already, you should check out my initial thoughts here and Part 2 here. In this installment, I’ll be setting up and experimenting with the voice control features advertised by CastleOS. The following is my opinion, and Your Mileage May Vary.

Setting up the Kinect

During the Kickstarter campaign I purchased a secondhand Kinect for Xbox One (Kinect v2) off eBay, and also purchased the official Kinect to PC Adapter from Amazon for it. Despite it being listed on the Compatible Devices page, at the time of writing this CastleOS doesn’t support the Kinect v2 sensor. It’s been well over twelve months, with constant promises from CastleOS that the Kinect v2 support is “weeks” away.

I got a secondhand Kinect for Xbox360 (Kinect v1) off eBay, and plugged that into the CastleHUB. You can configure the Kinect from the web version of CastleOS. However it seems the version of the Kinect Service that ships with the CastleHUB is broken. I had to plug my keyboard and monitor back into the CastleHUB and download the latest CastleOS Kinect Service from the CastleOS homepage, to get voice commands working.

 

Android Wear

One of the promises of the CastleHUB Kickstarter Campaign was

It supports voice control on additional platforms like Android and Android Wear. The future of home intelligence is powered by your voice!

— CastleHUB Kickstarter Campaign

So, how is the voice control on Android and Android wear? I put it to the test on my Samsung Galaxy Note 4, and Moto 360 v2 smart watch. The results? Nothing. There is no support for CastleOS via voice control on either Android, or Android Wear. Infact, it’s only recently been asked on the CastleOS forums here.

 

How it Compares

I think it’s important to debunk a couple of claims made or implied by CastleOS.

The voice recognition from CastleOS is limited to only controlling and running commands CastleOS knows of. Other voice control platforms, like Alexa and Google Now allow third-party developers to create new voice control actions.

Of course, you can teach CastleOS new commands. However, if CastleOS was the “most flexible” voice control available, it would need the following:

1. The ability for developers/programmers to teach CastleOS new voice controls programmatically.If you want to teach CastleOS ten new commands, you need to add ten new items to the settings in CastleOS. There’s no way for a program to mass install new voice commands to CastleOS.
2. The ability for CastleOS to accept natural language.Voice commands in CastleOS are very very rigid. You need to know exactly how to talk to CastleOS. Saying Computer, turn off the lights will work, but asking Computer, please turn off the lights won’t. CastleOS is looking for specific phrases, as opposed to processing what you’ve said.
3. The ability for custom commands to accept variables. Probably the best example of how CastleOS is not the most flexible, is its inability to accept variables in your speech. You can’t create a voice command to add items to a shopping list. Computer, add <Milk, Bread, Butter> to my shopping list won’t work. You’d need to add a custom script and voice command for every possible item you want to add to your shopping list.There’s no reason CastleOS can’t implement such a thing.The Kinect plugin for Homeseer comes with that feature as standard.

 

Last but certainly not least, if you use a Kinect or Android Wear smartwatch for voice control, you’ll be able to control media apps with your voice! No more fumbling for remotes, just speak out loud to your TV!

— CastleHUB Kickstarter Campaign

I’ve already established that Android voice control for CastleOS isn’t supported. It may come in the next few years, but right now it doesn’t exist. So that’s easy to dismiss.

However, the main implication of this claim is voice control of media apps. The CastleHUB Kickstarter Page also shows a photo of a TV with the Windows 10 Netflix App open, being controlled by CastelOS.

This is where the marketing strategy used by CastleOS gets a bit dodgy. Yes, the CastleHUB can run the Windows 10 Netflix App. Yes, you can open the Windows 10 Netflix App using your voice, if you create a custom C# script. However, you cannot control Netflix using your voice. You can’t tell CastleOS to pause/play your movie on Netflix. You can’t ask your CastleHUB to play Star Wars on Netflix. This is because the Netflix Windows 10 Application requires you to use a mouse. Keyboard commands won’t work, which means programming actions against Netflix becomes really really hard. Showing this picture, right after mentioning the ability to control media applications with voice is very very misleading.

See those other logos there? Plex, Kodi etc. They also imply media control with CastleOS. At the time of writing this, the “media” functionality of CastleOS is “coming soon”. Its been coming soon ever since the Kickstarter campaign launched. The Kinect add-on for CastleOS has the ability to issue Play/Pause commands to Plex clients. However, you still can’t tell CastleOS to play a movie or TV show. I guess we’re still “fumbling for our remotes” for the time being.

With Amazon recently releasing their API for developers, and other more open protocols starting to use them, I’m sure it won’t be long until we see software like OpenHAB or Home Assistant supporting commands like Alexa, play Modern Family on Plex.

 

 

Testing Voice Control

So the question on everyones mind, how well does the “most powerful and flexible voice control” work in real life? I decided to put it to the test.

First of all I think it’s important to note that CastleOS will never be anywhere near as good as a cloud voice recognition system like Google Now, or Alexa on the Amazon Echo. The advantage to this is there’s never an “always listening” device sending your conversations to the cloud.

Here’s how I’m going to run these tests.

1. I’ve chosen some phrases and commands that CastleOS natively supports from their Wiki.
2. For the tests, I’m using the only supported device at the time of writing, the Kinect for Xbox 360.
3. I’ll be using the default hotword, which is enabled in all CastleOS installations; computer.
4. The Kinect is plugged into the CastleHUB, which is also where CastleOS is running.
5. My installation is set to use English/Australia as the input language, as I have an Australian accent.
6. I’ll be using the default threshold settings shipped with the CastleHUB.
7. My speakers will be set to my Sonos Play:1, which is connected via the same Wifi network in my apartment.
8. The Kinect is in my bedroom, which is a carpeted room. There’s little to no echos in this small room.
9. I’ll repeat the command 10 times, per distance, and approximately measure how long it takes for a command to be accepted.

With this information, we should be able to get the average of how well the Kinect Sensor understands me, and how long it takes for CastleOS to issue the commands across my network.

 

 

 

 

 

 

 

 

The Bottom Line

So after all the tests, how does it stack up? Remembering these tests were done in an ideal room, with no background noise.

 

80 Centimeters

2.5 Meters

 

First I should explain some interesting things that you can probably see in the results.

For changing my lamp colour, all my tests at 80 centimeters failed. Not because CastleOS didn’t know what to do, it just didn’t do it. The command was accepted, but the light colour just didn’t change. I was going to call this a bug and write off the test, but then at 2.5 meters, without me changing anything the colour changed! It then failed for a few other times for the same reason. Clearly something is up here. However, from a users point of view (ie my girlfriend or someone else trying to change the colour) the voice control just isn’t working.

There’s also the scene control setting. I’m using a command which is listed on the CastleOS Wiki, and also posted by Chris on the CastleOS forums, which should allow us to set the scene using the phrase I was using. On every test, CastleOS did not recognise what I was saying. I’m guessing the phrase listed on the CastleOS website isn’t supported. However, as its listed as a supported phrase, and didn’t work I felt it important to highlight.

Now the numbers above seem pretty bleak. For arguments sake let’s take out the two tests above, as clearly something wasn’t right there, and see what the numbers give us now.

80 Centimeters

2.5 Meters

The success rate at 80 centimeters jumps up to an impressive 96%. However the more important test at 2.5 meters is basically the same result. At 2.5 meters (or 8 feet if you’re in the US), in an ideal room with no background noise, CastleOS only responded correctly to my command half of the time. This distance is less than half of the maximum distance CastleOS recommends, which is just over 8m (35 feet).

 

The most important test

Amongst Home Automation Hobbyists there’s a term I see thrown around a bit, which is the WAF (Wife Approval Factor). I think this also applies to anyone else in your home. Children, guests, perhaps even your pets. If your Home Automation system affects any of those, you need to make sure it’s rock solid. The last thing you want to be doing on a weekend is trying to find out why a light isn’t turning on anymore.

I’m pretty lucky to have a girlfriend that is also excited to have our apartment automated. I’m really into the automation side of things. I don’t subscribe to the belief that a device with an “app” is automated. Unfortunately with today’s technology we’re not completely there. Turning off the lights still requires some kind of manual action. Whether it be a switch, or voice command.

We are offering three rewards that allow you to donate this technology to the disabled. We will work with government agencies to find those in need to donate the CastleOS Hub and related gear to.

— CastleHUB Kickstarter Campaign

With the promise of voice control to turn off the lights with CastleOS, I decided to show a real-world example of CastleOS in action. How easy and reliable is it to turn off the lights using your voice at night? For a week I put it to the test. Some basic rules, as my last round of tests.

1. I’m using the Kinect for Xbox 360.
2. I’ll be using the default hotword, which is enabled in all CastleOS installations; computer.
3. The Kinect is plugged into the CastleHUB, which is also where CastleOS is running.
4. My installation is set to use English/Australia as the input language, as I have an Australian accent.
5. I’ll be using the default threshold settings shipped with the CastleHUB.
6. My speakers will be set to my Sonos Play:1, which is connected via the same Wifi network in my apartment.
7. The Kinect is in my bedroom, which is a carpeted room. There’s little to no echos in this small room.
8. I’ll be in the same position in bed, attemping to turn the lights off with the command Computer, lights.
9. Generally I don’t use the CastleHUB daily. Its usually tucked away in cupboard just running. I won’t be touching the CastleHUB for at least 2 hours before bedtime, so it is up to the CastleHUB to make sure it stays up and ready for my command.
10. I’ll measure how many times I have to repeat the command for it to be accepted. If after 5 attempts the lights still won’t turn off, I’ll resort to using the CastleOS App on my Phone.

 

For three days, I could not get CastleOS to turn my lights off within 5 attempts. That’s me looking like I’m talking to a wall in front of my girlfriend, before I give up and reach for my phone to turn the lights off. Only two out of seven days did CastleOS turn off my lights first attempt.

Now this is a major first world problem. I know. However, when your product (or at least the Kickstarter Campaign) was being sold for use by elderly and disabled people, the fact I had to reach for my phone is a major concern. Why wasn’t CastleOS listening? Why was the accuracy so bad?

Perhaps the Kinect v2 will make it slightly more accurate. I’m not holding my breath though (I could be holding it for another year!).

Next Up

The next stop on my review of the CastleHUB will be Scripting followed by testing the Media Playback ability of the CastleHUB. Be sure to follow me or subscribe on your favourite social media.