Home Assistant VM: Part 2 - App Setup and Conclusion

Z-Wave Migration

Okay, when I last discussed the Home Assistant VM, I had just completed getting the Aoetec Z-Wave stick working. Now I’m going to cover the installation of the home assistant software itself. There are numerous blogs and posts out there about how this is done.

I wasn’t sure if I wanted to cover this part at all. Stepping through Z-Wave inclusion / exclusion isn’t interesting, nor practically useful for other people. I do think that basic setup is useful, and I did encounter at least one interesting issue with door locks during this setup. I don’t think that a fully featured walkthrough is useful here. The truth is that any Home Assistant install is very much dependent upon what components are being put in the network and what the scope of the home automation network is. I intend to cover just the basics I used to get the Home Assistant application running, and then to get the z-wave network working.

First, I wanted to get Home Assistant working in a docker container. There is a good page on this in the Home Assistant project’s documentation [1]. This looked differently than what I was running before.

It took me a minute to remember that an issue I had run into last time I did this was that I wanted to run Hass.IO not Home Assistant. Hass.IO is a native install for Raspberry Pis, and includes extra features, like add-on modules and dashboards. I believe these can all be made in Home Assistant as well, but the configuration is decidedly different.

This time, I decided I wanted to run a supported docker install. The Hass.IO docker doesn’t appear to be officially supported by the project, but just some good developer’s attempt to make a Hass.IO docker look like it would if it were run natively on a raspberry pi. Being unsupported like that, I haven’t updated it in years. I want to make sure I can keep an update path this time.

Something that happens in infrastructure in general is the need to keep software up to date and patched. Mostly for security, but also new features and performance improvements. Long running projects try to keep an upgrade path for several years. An easy example is the Ubuntu LTS (standing for Long Term Support) which lasts for 5 years, but their non-LTS releases are only supported for 9 months [2]. In these cases, it pays to update every 6 months or so.

In my professional life, I try to target an update cycle of 12 months. If the software is ever too far behind, it loses the ability to be updated. Most updates will create standard update points. Think something like Windows 10’s service packs. All of the previous updates are rolled up into one massive update that will bring the system to a particular point. That way all future updates can assume this point has been achieved, and the need to update from any random other point up to present isn’t necessary. This has been working for a decade or so in Linux and Windows.

I have been pulled into so many projects where the database is too old to update anymore, and there is no choice but to deploy a new database, perform a migration or build an ETL (Extract, Transform, Load) from the old database to the new. This is a project that takes a lot of time and provides minimal value from the end client point of view. It usually needs to be done because of security issues rather than features. Don’t lose the update path, always put in that time to keep things up to date. The cost of these major migrations in time and effort vastly outweighs any update issues. It is also easier to account for the time in smaller chunks rather than bigger migration plans.

Anyways, this relates to Home Assistant because I want to keep an officially supported update path. They do have a docker container maintained by the developers [3]. That helps because it will automatically pull the latest version and take care of migrations. Plus, this gives me an excuse to work more with docker. My experience is a little light here. I have deployed several containers, and even developed in them. I have not had any cause to build an actual installation though, so I will enjoy working through it.

My first docker decision was to go with a docker-compose file. This isn’t because the base docker doesn’t work, it does, but more because I very much prefer being explicit on all of my options and methods for running any software. Docker-compose YAML files are great at being explicit on how they are to be run. Plus they can be shoved into a git repo really easily. They act as documentation on how to run a particular container as well as providing a log of changes made to the way it has been run that can be saved and worked with. My first docker compose YAML file was not complicated.

version: '3'
services:
    homeassistant:
        container_name: home-assistant
        image: homeassistant/home-assistant:stable
        volumes:
            - /home/nrweaver/home-assistant/config:/config
        environment:
            - TZ=America/Chicago
        restart: always
        devices:
            - /dev/ttyACM0:/dev/ttyACM0
            - /dev/ttyUSB0:/dev/ttyUSB0
        network_mode: host

This is exactly like the suggestions on the docker page. I added my own devices for the Z-Wave USB stick and the ZigBee USB stick. It worked just great. I didn’t have any issues getting Home Assistant running, or even starting the Z-Wave network rebuild.

This is the part where I am going to skip over a lot of things. I have 109 Z-Wave outlets on the Wink network. I had to go and exclude each one, and include them into the new Home Assistant Z-Wave network. What I ended up doing was actually grabbing a fiber USB cable I had around [4], combining it with a powered USB hub [5] and moving the Aeotec Z-Wave stick around the house so it would be closer to the actual devices I was adding to the Z-Wave network. I guess that answers the question about whether the Z-Wave stick works with hubs at all.

I think device inclusion is something ZigBee does better than Z-Wave. It appears the mesh network in ZigBee actually helps handle the new device inclusion. If I am being honest, I believe the new device inclusion in Z-Wave does not use the mesh network. I think it only uses direct communication between the Z-Wave stick and the device. I am not sure if that is part of the Z-Wave spec or not, so I don’t want to call it out directly, it could just be the specific Z-Wave implementation that Home Assistant uses. It took a couple days to actually hit all the outlets.

Next up was the smart locks. They added just fine, but the user codes became an issue. There is no native user code management tool in Home Assistant. After a bit of searching I found two options, first was one that was simple, and allowed individual management of each lock’s user codes [6]. The second appears to try and maintain user codes between locks [7]. For example, Joe’s lock code is 1234, and needs to be set on every lock in the house.

These are essentially the two ways of viewing user codes. First is lock centric, I look at a particular lock and set it’s codes. The second is user centric, I assign a code to a user, then look to which locks it needs to be set on.

I was the VP of Engineering who oversaw the development of the user code management tool at my last company. We needed to implement the user-centric version of user code management. I saw firsthand how difficult it is to maintain codes between locks. It took us darn near six months with at least one engineer full time (max was 3 in the first two months) on it to get it working well. I am skeptical that without a full time engineer on it, an open source project could achieve user code management between locks. Individual lock code management is much easier and simpler to work with.

In the lock-centric project it mentions the reason for its creation were the same issues we always dealt with in the user-centric project: keeping everything in sync with changes to the code, making sure each lock got all changes, deleting codes on every lock. Most issues are around lock code set/delete failures, where it partially succeeded, like 2 out of 3 locks got updates but the 3rd didn’t. I went with the lock-centric project, without the Home Assistant project itself taking this on, I just don’t trust the issues will be worked through on user-centric.

I expected this to be rather simple and easy, and I was very wrong. From the beginning setting lock codes worked, but deleting them did not. I spent a lot of time working through the issue I was seeing. At first I thought it was related to the lock code manager system. It was not. I eventually worked my way to a few GitHub issues about my locks [8] [9] [10]. Turns out, my old Schlage BE469 Touchscreen Deadbolts have a more fundamental issue with Home Assistant’s native Z-Wave implementation than I had thought.

Due to limitations and a lack of standardization of the actual lock user code protocol itself, the manufacturers themselves were left defining exactly what command will clear a lock user code. Since, there isn’t really a clear delete user code command built into the Z-Wave lock protocol, manufacturers use the set user code command with a special code meaning “don’t actually set this code, clear any code from this lock user code slot”. Many of them (Kwikset is the biggest I know of) went with setting a lock user code of ASCII 0000. (ASCII table for those that need a refresh like me [11]).

This isn’t the only complication though, the number of digits in a code is variable, so it could be 000000. This involves setting a string in the user code set command to 0x3030303000 (for 0000) or 0x30303030303000 (for 000000). The trailing 2 zeroes representing NULL or colloquially “end of string”.

Now this is where the lack of standardization hurts. Schlage chose to allow a lock code of 0000 which for obvious reasons above isn’t actually possible on Kwikset locks. The Schlage delete code involves a set of all NULLs or 0x000000000000. The issue comes from the fact that only one NULL is actually necessary to end a string. For example the string for “Hello” looks like 0x48656C6C6F00. However, to string processing all of the following are equivalent:

0x48656C6C6F003031
0x48656C6C6F004856
0x48656C6C6F000000
0x48656C6C6F009983

That is the core issue, for lower level string processing, and the x86 ISA actually has native string processing instructions, it doesn’t care after the first NULL is encountered. This means that when passing a string of 0x00000000 only the first one is guaranteed to be copied by those string commands into the set user code message. Then the Schlage lock sees something like 0x00323232, which is not the same as 0x00000000 for the clear command (Schlage should have realized this), so it errors out and leaves the old code in place. This is a cluster. Honestly I’m torn between thinking Schlage did a good job by not reserving the lock code of 0000 and thinking they did a bad one of not actually processing NULL strings well. They also picked the worst method for dealing with this based on programming history.

It took a while to deduce the issue because others are aware of it, have worked on it, and tried to work around this issue. Whenever one of these codes is set, the lock user code management software will often send a user code set command for a random 4 digit code. That way at least the code that isn’t supposed to work, won’t anymore.

There is a minor chance that a code collision will occur, and most smart locks don’t allow the same code to be set on two different user code slots (there are typically 30 slots for user codes, but I’ve seen as high as 250). In that case the old code will typically stay. I think that the thought process is a random code is better than leaving a known code in place. That is probably true.

Also, user code set commands are not high reliability. My most remote lock has about a 70% success rate. Since the user code set can fail, very often the old user code still worked. This is compounded by the fact that for security reasons Schlage doesn’t actually return what code is set in a user code slot, only that it exists.

It never set well with me that there are codes on my locks, that definitely work, and that I don’t know. I wanted no lock code, not a random unknown one. Seeing in the Z-Wave logs that there was a code there just bothered me.

Alright, the solution to this issue is actually to abandon the native implementation of Z-Wave in Home Assistant. Home Assistant uses a different open source project for Z-Wave, Open Z-Wave [12] (that’s the name of the project). It appears that the native implementation is on version ~1.4. It also looks like they have been doing a major refactor called Open Z-Wave Beta which is on version 1.6. There are a fair number of updates here, but I’d say the biggest one for my purposes is that they are moving Z-Wave management into its own service.

That means that Open Z-Wave Beta and Home Assistant are no longer in the same environment. This allows Z-Wave management to be controlled separately than Home Assistant. For example, when I was working through the lock user code issue outlined above, I had to restart Home Assistant a lot to register configuration changes. That caused the Z-Wave network to be re-initialized, which takes 5-10 minutes with a large Z-Wave network like mine. Having these in separate services allowed more flexibility without constantly restarting the Z-Wave network.

Ultimately, this is a better plan for the future, and a lot of IoT systems are moving to something more like this for distributed management with a central queueing system for maintaining the various connections. In Home Assistant’s case they use MQTT [13]. There is an implementation I have used called Eclipse Mosquito MQTT [14]. As I was researching how to do this on my Home Assistant VM, I encountered a good guide for scripting this all up in docker compose. I had most of this already, but it is pretty clear what is being done [15]. My current docker compose file looks like:

version: '3'
services:
    mosquitto:
        container_name: mosquitto
        image: eclipse-mosquitto:latest
        restart: always
        ports:
            - "1883:1883"
            - "9001:9001"
        volumes:
            - /home/nrweaver/mosquitto/config/mosquitto.conf:/mosquitto/config/mosquitto.conf
            - /home/nrweaver/mosquitto/data:/mosquitto/data
            - /home/nrweaver/mosquitto/log:/mosquitto/log
        deploy:
            resources:
                limits:
                    memory: "768M"
    postgres:
        container_name: postgres-ha
        restart: always
        image: postgres
        ports:
            - "5432:5432"
        volumes:
            - /home/nrweaver/postgres/data/:/var/lib/postgresql/data/
        env_file:
            - /home/nrweaver/postgres/env/env
  # zigbee2mqtt:
  #     container_name: zigbee
  #     depends_on:
  #         - mosquitto
  #     image: koenkk/zigbee2mqtt
  #     volumes:
  #         - /home/nrweaver/zigbee2mqtt/data:/app/data
  #         - /run/udev:/run/udev:ro
  #     devices:
  #         - /dev/ttyUSB0:/dev/ttyUSB0
  #     restart: always
  #     privileged: true
  #     environment:
  #         - TZ=America/Chicago
  #     deploy:
  #         resources:
  #             limits:
  #                 memory: "768M"
    ozwd:
        container_name: openzwave
        depends_on:
            - mosquitto
        image: openzwave/ozwdaemon:latest
        ports:
            - "1983:1983"
        restart: always
        security_opt:
            - seccomp:unconfined
        volumes:
            - /home/nrweaver/ozwd/config:/opt/ozw/config
        devices:
            - /dev/ttyACM0:/dev/ttyACM0
        environment:
            MQTT_SERVER: "ha-mqtt1.internal"
            USB_PATH: /dev/ttyACM0
            OZW_NETWORK_KEY: "<THIS IS A SECRET>"
        deploy:
            resources:
                limits:
                    memory: "768M"
    homeassistant:
        container_name: home-assistant
        depends_on:
            - mosquitto
            - postgres
        image: homeassistant/home-assistant:stable
        volumes:
            - /home/nrweaver/home-assistant/config:/config
        environment:
            - TZ=America/Chicago
        restart: always
     #  devices:
     #      - /dev/ttyACM0:/dev/ttyACM0
     #      - /dev/ttyUSB0:/dev/ttyUSB0
        network_mode: host
        logging:
            driver: "json-file"
            options:
                max-size: "10m"
                max-file: "5"

Briefly, the mosquito service pulls from eclipse-mosquito’s latest docker container and gets it running. It is resource-limited to 756MB of memory.

Next is PostgreSQL. The guide covered how to get Home Assistant using Postgres and I think that is a good idea since I am branching out into a more complicated IoT setup, I wanted to also have a better database backing all of these changes.

Then I have the zigbee2mqtt defined, but commented out. Zigbee2mqtt didn’t recognize my ZigBee USB stick correctly. I did not do a lot of searching for compatibility on this one, I had it for work reasons. I think I will need to get a compatible USB stick if I want to add ZigBee. For right now though, I don’t actually need it, so I am ignoring it.

Next I have the ozwd which stands for Open Z-Wave Daemon. This is a dockerized version of the Open Z-Wave Beta native service. I prefer to keep things in containers if I can. It is also resource-limited to 756MB.

I keep these resource-limited because even if z-wave is down, I want to be able to control the lights. In fact, that is probably more important than all of the z-wave network, which doesn’t have as much daily use. Last is the Home Assistant container. I have limited its logging, this was something I was doing while debugging the lock code issue since I ended up with a ton of log files.

A couple things of note. First, I issued multiple DNS names to this same Home Assistant VM because of this reconfiguration. ha-mqtt1.internal now exists. I did this because MQTT is now a service in my infrastructure. If I want to ever deploy it to its own VM, I can do so, switch the IP address ha-mqtt1.internal resolves to, and I don’t need to make any further configuration changes. Second, I kept the old device configuration around on Home Assistant because I like to preserve what I was thinking back then. This probably best belongs in a git repository, but I didn’t have a lot of changes here, I didn’t think it warranted that level of infrastructure just yet. Perhaps I’ll set up a git management VM at some point and put all of these config files in there, but not yet.

Once I completed the setup of these docker containers, all of my lock user code issues went away. The open Z-Wave daemon logs even show the code slot as Available meaning empty, exactly like I wanted. For my locks at least, this fixes the issue I encountered.

Docker stats of all the running containers

I didn’t need to go and re-add all of the outlets to the z-wave network. Basic node information is stored on the Z-Wave USB stick. However, I did lose all my names. I had 109 outlets all named some variation of <brand> switch <number>. I did have to go through the house and re-figure out what they all were, but that is done now. After finishing the migration of the locks I migrated the door sensors, and finally the garage door. The lock code management software works as well without all that need to correct for lower level issues.

With that, and actually syncing in my Phillips hue hubs, I have only one more thing I wanted to do to complete my migration. I deactivated my Wink account. That was more satisfying than saving $5/month should have been. I think it got mixed in with the excitement of actually succeeding on the overall configuration and migration.

Voice Assistants

There is only one more thing I want to mention under the Home Assistant VM guise. I have used Amazon’s Alexa service since they first came out. Way back then there was very limited voice support for Smart Home. One of my previous projects was a python application that could spawn a bunch of uPnP listeners that pretended to be Belkin WeMo devices so I could run any random code based off of the first gen WeMo protocol. It was awesome back then.

However, as time has gone on, I think Alexa is doing too much extra stuff and not focusing on its core. It’s quite clear that Amazon’s voice processing is behind Google’s. It has been basically from the beginning of Google Home. Google’s Home devices are better at answering questions, maintaining context, and generally recognizing voices. The straw that broke the camel’s back is that Alexa flat out refuses to recognize my partner’s voice. I have witnessed her screaming at Alexa before because it doesn’t recognize the activation phrase or understand what she is saying.

Then I saw the Amazon Sidewalk features [16] hit some blogs I follow. Essentially Amazon wants to connect all the Alexas and devices in a neighborhood. That will enable other people to have access to my network for Amazon’s purposes. Hard pass. There is zero chance Amazon doesn’t have an issue here. I have been in plenty of company meetings about this stuff. There will be a breach, someone will have their network hacked, and who knows what happens from there.

I have already replaced all of my Amazon Alexa devices with Google Home devices save one, and it’s replacement is on the way. I want my voice assistant to be good at hearing voices, understanding what I want, and issuing commands. If Amazon wants public Wi-Fi, lobby for it, instead of trying to make me subsidize their company goals. I don’t want my voice assistants to be about helping Amazon’s bottom line (beyond the purchase), or whatever fiction they are presenting as its actual purpose.