(The following blog post is written by Erick Daniszewski, a Senior Software Engineer at Vapor IO and maintainer of the Synse opensource project)
At Vapor IO, we have been working hard on a hardware and software solution for the edge. At the core of our software solution is Synse: an open platform to monitor and control physical and virtual devices.
That one-line description of Synse is purposefully imprecise; it doesn’t describe what makes a device physical or virtual, nor does it define “device”. This vagueness is intentional. One of the primary design goals of Synse has been for it to be pluggable so that it can interface with any device. This allows Synse to unify telemetry and control of any number of disparate devices, including sensors (such as temperature, humidity, airflow, etc.), environmental (such as fan speeds, cooling system controls, etc.), security (such as video feeds, door locks, etc.), and accessibility (lights, LED indicators, etc.). All of this through a single JSON HTTP API.
By exposing all devices over the same HTTP interface, we abstract away the actual device-level protocol, meaning that an SNMP device looks the same as an IPMI device which looks the same as an RS-485 device. The pluggable nature of Synse coupled with its open-source status means that you can create and share your own plugin, or contribute to existing ones.
Why build an open telemetry platform?
Other telemetry platforms exist in one form or another, so why build our own? In a lot of ways, comparing Synse to other telemetry platforms is a bit of an apple-to-oranges comparison largely because of the control aspect of Synse Server.
When Vapor was first starting out, we spent a lot of time thinking about the complexities of the data center and what could be done to minimize them in software. A few high level items stood out:
- IPMI is frequently used for server control. While this protocol has served us well, it is not necessarily user-friendly and not all boards implement the IPMI spec in the same way, so behavior can change between different server models.
- Many systems operate independent of one another. In your home, you don’t need to tell your AC unit when your refrigerator (or other heat-generating appliance) is running. When you put hundreds or thousands of heat generating appliances in a room, the HVAC costs will be extreme. By integrating system control with environmental monitoring, the overall system efficiency will go up and costs will go down.
- There generally isn’t great environmental monitoring. Our hardware solution, the Chamber, has nearly 100 sensors built in to get a detailed picture of the operational environment.
We needed something to address all of these points, and thus Synse was born. It simplifies and unifies lower-level protocols, allows for manual control of systems, and exposes environmental telemetry which can be used for monitoring or automated system control/optimization.
As we started looking more towards the edge, these same problems existed, and in many cases were amplified. Edge data centers are not be staffed, so they will need to support lights-out operation, remote management, and remote monitoring. Sending a technician to an edge data center is not always cost effective and it takes time. Being able to perform maintenance remotely and having disaster recovery built-in is a must. Synse provides the uniform, easy-to-use API that enables all of this.
How does it work?
There are two primary components to the Synse platform: Synse Server, and Synse Plugins.
The plugins are where device interactions actually happen. A plugin should generally provide support for a single protocol, like SNMP, IPMI, RS-485, etc., however it is not limited to do so. To keep plugins as flexible and generalizable as possible, there are few restrictions on what the plugin can do or how it can be implemented. This makes it easy to support new plugins and to create custom plugins, even if they use a subset of some protocol or require multiple protocols.
Naturally, the backend logic implementing the device protocol will vary by plugin, but there are some common actions that all plugins will perform:
- Read: get a reading from a device (if it supports it)
- Write: write data to a device (if it supports it)
- Metadata: get info about a device (if there is any)
The Synse Plugin SDK (written in Golang) provides a convenient framework around all of the common plugin functionality. It handles device routing, device metainfo, write transaction generation and management, and communication with Synse Server. This means that a plugin only needs to define its read and write behavior. Each plugin exposes all of this information to Synse Server via an internal gRPC API.
Synse Server provides the HTTP API for Synse and acts as a front-end/router for the plugins. When a request comes, such as a read or write, Synse Server will figure out which device is being referenced, figure out which plugin manages the device, validate the request as much as it can, and then pass the request along to the plugin to fulfill.
The Synse API supports many actions ranging from simple connectivity checking, to device enumeration, and from providing device metadata to reading from and writing to that device. As an example, from a simple deployment with Synse Server and an emulator plugin, we can `/scan` to enumerate the available devices:
$ curl localhost:5000/synse/v2/scan
{
"racks": [
{
"id": "rack-1",
"boards": [
{
"id": "vec",
"devices": [
{
"id": "0fe8f06229aa9a01ef6032d1ddaf18a5",
"info": "Synse Temperature Sensor 3",
"type": "temperature"
},
{
"id": "12835beffd3e6c603aa4dd92127707b5",
"info": "Synse Fan",
"type": "fan"
},
{
"id": "12ea5644d052c6bf1bca3c9864fd8a44",
"info": "Synse LED",
"type": "led"
},
{
"id": "34c226b1afadaae5f172a4e1763fd1a6",
"info": "Synse Humidity Sensor",
"type": "humidity"
},
{
"id": "3ee84834c79c5a124d858e237e81e186",
"info": "Synse Temperature Sensor 2",
"type": "temperature"
},
{
"id": "45ffe8f7f7a2b0ae970b687abd06f9e6",
"info": "Synse Temperature Sensor 1",
"type": "temperature"
},
{
"id": "8f7ac60be5c8a3815ce89753de138edf",
"info": "Synse Temperature Sensor 5",
"type": "temperature"
},
{
"id": "bcf0618c50bff9121cb10d141d66f46f",
"info": "Synse backup LED",
"type": "led"
},
{
"id": "df6a06d6e28da8aab0c25ee41688fd1c",
"info": "Synse Airflow Sensor",
"type": "airflow"
},
{
"id": "e385de0e2b5d16af5e34167d479fc766",
"info": "Synse Pressure Sensor 1",
"type": "pressure"
},
{
"id": "f441d97b2f6545ef3001a688489e820a",
"info": "Synse Temperature Sensor 4",
"type": "temperature"
},
{
"id":" f838b2d6afceb01e7a2634893f6f935c",
"info": "Synse Pressure Sensor 2",
"type": "pressure"
}
]
}
]
}
]
}
Synse plugins are able to support both serial and parallel protocols. In both cases, the plugin architecture attempts to minimize blocking from long read or write times on a single device, allowing it to read from/write to the other devices it manages more expediently. This is done in part by having separate read and write loops. The read loop will read from devices continuously (with a configurable interval) and update internal plugin state with those readings, which will be passed along to Synse Server when it requests reading data. The write loop will take write transactions off of the write queue, populated by incoming write requests from Synse Server, and fulfill them, updating the transaction state as it goes.
For more on how plugins work, see the SDK Architecture documentation.
All components of Synse are containerized, making them easy to deploy and manage either locally or with your preferred orchestration system, such as Kubernetes. Synse Server is stateless, so if it goes down on one node, a new instance can be restarted and pick up right where the old instance left off, only needing to rebuild its caches using plugin provided info. Plugins do contain some state, such as the current reading values, the transaction cache, and a queue of write events. Since all of those are time bound (e.g. they will expire or be consumed within a short period of time), only a minimal amount of state is lost if the plugin goes down, all of which can be rebuilt or replaced later.
What’s next?
In the near term, our primary focus is to ensure that Synse is stable and well-tested. We use it daily and find it to be reliable. As we continue to integrate more components with Synse and continue to build out our deployments and architecture, we want to ensure this stability continues.
Looking further out, we have identified some changes at a higher-level that we think would benefit Synse and make it even more flexible and useful moving into the future. None of these features/changes are set in stone, nor is there any timeline associated with them. They are listed here to give an idea of the direction Synse is moving towards.
- Moving to “tag”-based routing: this would allow arbitrary tags to be associated with a device, not just the required rack/board/device identifiers.
- Allowing bulk actions/filtering based on tag groups.
- Development and support of additional plugins.
In addition to those higher-level features, we also plan on supporting more devices. These devices range from new physical sensors, such as air particulate meters, to infrastructure devices, such as additional power and cooling integration, to networking equipment, such as network switches.
As more plugins and devices are added, the Synse platform becomes an increasingly valuable tool in the datacenter industry. It enables complex workflows, such as workload migration based on network latency and congestion, developing machine learning models for hardware health in various environmental conditions, or role-based action sequences triggered by technician identity.
See the Synse GitHub page: https://github.com/vapor-ware/synse, to get Synse and see it in action. The page includes links to the GitHub repos for all Synse components in the Synse ecosystem as well as a tutorial on getting started with it. The design and development of Synse is primarily based on functional requirements and use cases. As the need for a new feature arises, it gets added to the pipeline. However, being an open source project, your voice counts — so if you’d like to see new features in Synse, let us know!
