Bump version to 0.9.41

It wasn't the validator duplicating, but the update duplicating, so
avoid that happening properly this time.

.gitignore

@@ -1,3 +1,8 @@
 *.pyc
 *.tmp
 *.swp
+# Ignore build artifacts
+debian/pvc-*/
+debian/*.log
+debian/*.substvars
+debian/files

.version

@@ -0,0 +1 @@
+0.9.41

CHANGELOG.md

@@ -0,0 +1,302 @@
+## PVC Changelog
+
+###### [v0.9.41](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.41)
+
+  * Fixes a bad conditional check in IPMI verification
+  * Implements per-network MTU configuration; NOTE: Requires new keys in pvcnoded.yaml (`bridge_mtu`) and Ansible group_vars (`pvc_bridge_mtu`)
+
+###### [v0.9.40](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.40)
+
+  * [Docs] Documentation updates for new Changelog file
+  * [Node Daemon] Fixes bug with schema updates
+
+###### [v0.9.39](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.39)
+
+  * [Documentation] Update several documentation sections
+  * [API Daemon/CLI Client] Add negate flag for VM option limits (node, tag, state)
+  * [Build] Add linting check to build-and-deploy.sh
+
+###### [v0.9.38](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.38)
+
+  * [All] Significantly improve storage benchmark format and reporting
+
+###### [v0.9.37](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.37)
+
+  * [All] Adds support for configurable OSD DB size ratios
+  * [Node Daemon] Fixes bugs with OSD creation
+  * [Node Daemon] Fixes exception bugs in CephInstance
+  * [CLI Client] Adjusts descriptions around Ceph OSDs
+  * [Node Daemon] Fixes ordering of pvc-flush unit
+  * [Node Daemon] Fixes bugs in fence handling and libvirt keepalive
+  * [Node Daemon] Simplifies locking for and speeds up VM migrations
+  * [Node Daemon] Fixes bugs in queue get timeouts
+  * [API Daemon] Adjusts benchmark test jobs configuration and naming
+
+###### [v0.9.36](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.36)
+
+  * [Node Daemon] Fixes a bug during early cleanup
+  * [All] Adds support for OSD database/WAL block devices to improve Ceph performance; NOTE: Applies only to new OSDs
+
+###### [v0.9.35](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.35)
+
+  * [Node Daemon] Fixes several bugs and crashes in node daemon
+  * [General] Updates linting rules for newer Flake8 linter
+  * [Daemons/CLI client] Adds VM network and disk hot attach/detach support; NOTE: Changes the default behaviour of `pvc vm network add`/`remove` and `pvc vm volume add`/`remove`
+  * [API Daemon] Adds checks for pool size when resizing volumes
+  * [API Daemon] Adds checks for RAM and vCPU sizes when defining or modifying VMs
+
+###### [v0.9.34](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.34)
+
+  * [Provisioner] Adds support for filesystem arguments containing =
+  * [CLI Client] Fixes bug with pvc provisioner status output formatting
+  * [Node Daemon] Fixes minor typo in startup message
+
+###### [v0.9.33](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.33)
+
+  * [Node Daemon] A major refactoring of the node daemon
+  * [CLI Client] Fixes output errors if a node has no provisioner data
+  * [Packages] Fixes issues with including __pycache__ directories in .deb files
+
+###### [v0.9.32](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.32)
+
+  * [CLI Client] Fixes some incorrect colours in network lists
+  * [Documentation] Adds documentation screenshots of CLI client
+  * [Node Daemon] Fixes a bug if VM stats gathering fails
+
+###### [v0.9.31](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.31)
+
+  * [Packages] Cleans up obsolete Suggests lines
+  * [Node Daemon] Adjusts log text of VM migrations to show the correct source node
+  * [API Daemon] Adjusts the OVA importer to support floppy RASD types for compatability
+  * [API Daemon] Ensures that volume resize commands without a suffix get B appended
+  * [API Daemon] Removes the explicit setting of image-features in PVC; defaulting to the limited set has been moved to the ceph.conf configuration on nodes via PVC Ansible
+
+###### [v0.9.30](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.30)
+
+  * [Node Daemon] Fixes bug with schema validation
+
+###### [v0.9.29](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.29)
+
+  * [Node Daemon] Corrects numerous bugs with node logging framework
+
+###### [v0.9.28](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.28)
+
+  * [CLI Client] Revamp confirmation options for "vm modify" command
+
+###### [v0.9.27](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.27)
+
+  * [CLI Client] Fixes a bug with vm modify command when passed a file
+
+###### [v0.9.26](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.26)
+
+  * [Node Daemon] Corrects some bad assumptions about fencing results during hardware failures
+  * [All] Implements VM tagging functionality
+  * [All] Implements Node log access via PVC functionality
+
+###### [v0.9.25](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.25)
+
+  * [Node Daemon] Returns to Rados library calls for Ceph due to performance problems
+  * [Node Daemon] Adds a date output to keepalive messages
+  * [Daemons] Configures ZK connection logging only for persistent connections
+  * [API Provisioner] Add context manager-based chroot to Debootstrap example script
+  * [Node Daemon] Fixes a bug where shutdown daemon state was overwritten
+
+###### [v0.9.24](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.24)
+
+  * [Node Daemon] Removes Rados module polling of Ceph cluster and returns to command-based polling for timeout purposes, and removes some flaky return statements
+  * [Node Daemon] Removes flaky Zookeeper connection renewals that caused problems
+  * [CLI Client] Allow raw lists of clusters from `pvc cluster list`
+  * [API Daemon] Fixes several issues when getting VM data without stats
+  * [API Daemon] Fixes issues with removing VMs while disks are still in use (failed provisioning, etc.)
+
+###### [v0.9.23](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.23)
+
+  * [Daemons] Fixes a critical overwriting bug in zkhandler when schema paths are not yet valid
+  * [Node Daemon] Ensures the daemon mode is updated on every startup (fixes the side effect of the above bug in 0.9.22)
+
+###### [v0.9.22](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.22)
+
+  * [API Daemon] Drastically improves performance when getting large lists (e.g. VMs)
+  * [Daemons] Adds profiler functions for use in debug mode
+  * [Daemons] Improves reliability of ZK locking
+  * [Daemons] Adds the new logo in ASCII form to the Daemon startup message
+  * [Node Daemon] Fixes bug where VMs would sometimes not stop
+  * [Node Daemon] Code cleanups in various classes
+  * [Node Daemon] Fixes a bug when reading node schema data
+  * [All] Adds node PVC version information to the list output
+  * [CLI Client] Improves the style and formatting of list output including a new header line
+  * [API Worker] Fixes a bug that prevented the storage benchmark job from running
+
+###### [v0.9.21](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.21)
+
+  * [API Daemon] Ensures VMs stop before removing them
+  * [Node Daemon] Fixes a bug with VM shutdowns not timing out
+  * [Documentation] Adds information about georedundancy caveats
+  * [All] Adds support for SR-IOV NICs (hostdev and macvtap) and surrounding documentation
+  * [Node Daemon] Fixes a bug where shutdown aborted migrations unexpectedly
+  * [Node Daemon] Fixes a bug where the migration method was not updated realtime
+  * [Node Daemon] Adjusts the Patroni commands to remove reference to Zookeeper path
+  * [CLI Client] Adjusts several help messages and fixes some typos
+  * [CLI Client] Converts the CLI client to a proper Python module
+  * [API Daemon] Improves VM list performance
+  * [API Daemon] Adjusts VM list matching critera (only matches against the UUID if it's a full UUID)
+  * [API Worker] Fixes incompatibility between Deb 10 and 11 in launching Celery worker
+  * [API Daemon] Corrects several bugs with initialization command
+  * [Documentation] Adds a shiny new logo and revamps introduction text
+
+###### [v0.9.20](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.20)
+
+  * [Daemons] Implemented a Zookeeper schema handler and version 0 schema
+  * [Daemons] Completes major refactoring of codebase to make use of the schema handler
+  * [Daemons] Adds support for dynamic chema changges and "hot reloading" of pvcnoded processes
+  * [Daemons] Adds a functional testing script for verifying operation against a test cluster
+  * [Daemons, CLI] Fixes several minor bugs found by the above script
+  * [Daemons, CLI] Add support for Debian 11 "Bullseye"
+
+###### [v0.9.19](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.19)
+
+  * [CLI] Corrects some flawed conditionals
+  * [API] Disables SQLAlchemy modification tracking functionality (not used by us)
+  * [Daemons] Implements new zkhandler module for improved reliability and reusability
+  * [Daemons] Refactors some code to use new zkhandler module
+  * [API, CLI] Adds support for "none" migration selector (uses cluster default instead)
+  * [Daemons] Moves some configuration keys to new /config tree
+  * [Node Daemon] Increases initial lock timeout for VM migrations to avoid out-of-sync potential
+  * [Provisioner] Support storing and using textual cluster network labels ("upstream", "storage", "cluster") in templates
+  * [API] Avoid duplicating existing node states
+
+###### [v0.9.18](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.18)
+
+  * Adds VM rename functionality to API and CLI client
+
+###### [v0.9.17](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.17)
+
+  * [CLI] Fixes bugs in log follow output
+
+###### [v0.9.16](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.16)
+
+  * Improves some CLI help messages
+  * Skips empty local cluster in CLI
+  * Adjusts how confirmations happen during VM modify restarts
+  * Fixes bug around corrupted VM log files
+  * Fixes bug around subprocess pipe exceptions
+
+###### [v0.9.15](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.15)
+
+  * [CLI] Adds additional verification (--yes) to several VM management commands
+  * [CLI] Adds a method to override --yes/confirmation requirements via envvar (PVC_UNSAFE)
+  * [CLI] Adds description fields to PVC clusters in CLI
+
+###### [v0.9.14](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.14)
+
+  * Fixes bugs around cloned volume provisioning
+  * Fixes some minor visual bugs
+  * Minor license update (from GPL3+ to GPL3)
+  * Adds qemu-guest-agent support to provisioner-created VMs by default
+
+###### [v0.9.13](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.13)
+
+  * Adds nicer startup messages for daemons
+  * Adds additional API field for stored_bytes to pool stats
+  * Fixes sorting issues with snapshot lists
+  * Fixes missing increment/decrement of snapshot_count on volumes
+  * Fixes bad calls in pool element API endpoints
+  * Fixes inconsistent bytes_tohuman behaviour in daemons
+  * Adds validation and maximum volume size on creation (must be smaller than the pool free space)
+
+###### [v0.9.12](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.12)
+
+  * Fixes a bug in the pvcnoded service unit file causing a Zookeeper startup race condition
+
+###### [v0.9.11](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.11)
+
+  * Documentation updates
+  * Adds VNC information to VM info
+  * Goes back to external Ceph commands for disk usage
+
+###### [v0.9.10](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.10)
+
+  * Moves OSD stats uploading to primary, eliminating reporting failures while hosts are down
+  * Documentation updates
+  * Significantly improves RBD locking behaviour in several situations, eliminating cold-cluster start issues and failed VM boot-ups after crashes
+  * Fixes some timeout delays with fencing
+  * Fixes bug in validating YAML provisioner userdata
+
+###### [v0.9.9](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.9)
+
+  * Adds documentation updates
+  * Removes single-element list stripping and fixes surrounding bugs
+  * Adds additional fields to some API endpoints for ease of parsing by clients
+  * Fixes bugs with network configuration
+
+###### [v0.9.8](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.8)
+
+  * Adds support for cluster backup/restore
+  * Moves location of `init` command in CLI to make room for the above
+  * Cleans up some invalid help messages from the API
+
+###### [v0.9.7](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.7)
+
+  * Fixes bug with provisioner system template modifications
+
+###### [v0.9.6](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.6)
+
+  * Fixes bug with migrations
+
+###### [v0.9.5](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.5)
+
+  * Fixes bug with line count in log follow
+  * Fixes bug with disk stat output being None
+  * Adds short pretty health output
+  * Documentation updates
+
+###### [v0.9.4](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.4)
+
+  * Fixes major bug in OVA parser
+
+###### [v0.9.3](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.3)
+
+  * Fixes bugs with image & OVA upload parsing
+
+###### [v0.9.2](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.2)
+
+  * Major linting of the codebase with flake8; adds linting tools
+  * Implements CLI-based modification of VM vCPUs, memory, networks, and disks without directly editing XML
+  * Fixes bug where `pvc vm log -f` would show all 1000 lines before starting
+  * Fixes bug in default provisioner libvirt schema (`drive` -> `driver` typo)
+
+###### [v0.9.1](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.1)
+
+  * Added per-VM migration method feature
+  * Fixed bug with provisioner system template listing
+
+###### [v0.9.0](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.0)
+
+Numerous small improvements and bugfixes. This release is suitable for general use and is pre-release-quality software.
+
+This release introduces an updated version scheme; all future stable releases until 1.0.0 is ready will be made under this 0.9.z naming. This does not represent semantic versioning and all changes (feature, improvement, or bugfix) will be considered for inclusion in this release train.
+
+###### [v0.8](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.8)
+
+Numerous improvements and bugfixes. This release is suitable for general use and is pre-release-quality software.
+
+###### [v0.7](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.7)
+
+Numerous improvements and bugfixes, revamped documentation. This release is suitable for general use and is beta-quality software.
+
+###### [v0.6](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.6)
+
+Numerous improvements and bugfixes, full implementation of the provisioner, full implementation of the API CLI client (versus direct CLI client). This release is suitable for general use and is beta-quality software.
+
+###### [v0.5](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.5)
+
+First public release; fully implements the VM, network, and storage managers, the HTTP API, and the pvc-ansible framework for deploying and bootstrapping a cluster. This release is suitable for general use, though it is still alpha-quality software and should be expected to change significantly until 1.0 is released.
+
+###### [v0.4](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.4)
+
+Full implementation of virtual management and virtual networking functionality. Partial implementation of storage functionality.
+
+###### [v0.3](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.3)
+
+Basic implementation of virtual management functionality.
+

LICENSE

@@ -672,8 +672,3 @@ may consider it more useful to permit linking proprietary applications with
 the library.  If this is what you want to do, use the GNU Lesser General
 Public License instead of this License.  But first, please read
 <https://www.gnu.org/licenses/why-not-lgpl.html>.
-
-----
-
-Logo contains elements Copyright Anjara Begue via <http://www.vecteezy.com>
-which are released under a Creative Commons Attribution license.

README.md

@@ -1,178 +1,58 @@
-# PVC - The Parallel Virtual Cluster system
-
 <p align="center">
-<img alt="Logo banner" src="https://git.bonifacelabs.ca/uploads/-/system/project/avatar/135/pvc_logo.png"/>
+<img alt="Logo banner" src="docs/images/pvc_logo_black.png"/>
 <br/><br/>
 <a href="https://github.com/parallelvirtualcluster/pvc"><img alt="License" src="https://img.shields.io/github/license/parallelvirtualcluster/pvc"/></a>
 <a href="https://github.com/parallelvirtualcluster/pvc/releases"><img alt="Release" src="https://img.shields.io/github/release-pre/parallelvirtualcluster/pvc"/></a>
 <a href="https://parallelvirtualcluster.readthedocs.io/en/latest/?badge=latest"><img alt="Documentation Status" src="https://readthedocs.org/projects/parallelvirtualcluster/badge/?version=latest"/></a>
 </p>
 
-**NOTICE FOR GITHUB**: This repository is a read-only mirror of the PVC repositories from my personal GitLab instance. Pull requests submitted here will not be merged. Issues submitted here will however be treated as authoritative.
+## What is PVC?
 
-PVC is a KVM+Ceph+Zookeeper-based, Free Software, scalable, redundant, self-healing, and self-managing private cloud solution designed with administrator simplicity in mind. It is built from the ground-up to be redundant at the host layer, allowing the cluster to gracefully handle the loss of nodes or their components, both due to hardware failure or due to maintenance. It is able to scale from a minimum of 3 nodes up to 12 or more nodes, while retaining performance and flexibility, allowing the administrator to build a small cluster today and grow it as needed.
+PVC is a Linux KVM-based hyperconverged infrastructure (HCI) virtualization cluster solution that is fully Free Software, scalable, redundant, self-healing, self-managing, and designed for administrator simplicity. It is an alternative to other HCI solutions such as Harvester, Nutanix, and VMWare, as well as to other common virtualization stacks such as ProxMox and OpenStack.
+
+PVC is a complete HCI solution, built from well-known and well-trusted Free Software tools, to assist an administrator in creating and managing a cluster of servers to run virtual machines, as well as self-managing several important aspects including storage failover, node failure and recovery, virtual machine failure and recovery, and network plumbing. It is designed to act consistently, reliably, and unobtrusively, letting the administrator concentrate on more important things.
+
+PVC is highly scalable. From a minimum (production) node count of 3, up to 12 or more, and supporting many dozens of VMs, PVC scales along with your workload and requirements. Deploy a cluster once and grow it as your needs expand.
+
+As a consequence of its features, PVC makes administrating very high-uptime VMs extremely easy, featuring VM live migration, built-in always-enabled shared storage with transparent multi-node replication, and consistent network plumbing throughout the cluster. Nodes can also be seamlessly removed from or added to service, with zero VM downtime, to facilitate maintenance, upgrades, or other work.
+
+PVC also features an optional, fully customizable VM provisioning framework, designed to automate and simplify VM deployments using custom provisioning profiles, scripts, and CloudInit userdata API support.
+
+Installation of PVC is accomplished by two main components: a [Node installer ISO](https://github.com/parallelvirtualcluster/pvc-installer) which creates on-demand installer ISOs, and an [Ansible role framework](https://github.com/parallelvirtualcluster/pvc-ansible) to configure, bootstrap, and administrate the nodes. Once up, the cluster is managed via an HTTP REST API, accessible via a Python Click CLI client or WebUI.
+
+Just give it physical servers, and it will run your VMs without you having to think about it, all in just an hour or two of setup time.
+
+
+## What is it based on?
+
+The core node and API daemons, as well as the CLI API client, are written in Python 3 and are fully Free Software (GNU GPL v3). In addition to these, PVC makes use of the following software tools to provide a holistic hyperconverged infrastructure solution:
+
+  * Debian GNU/Linux as the base OS.
+  * Linux KVM, QEMU, and Libvirt for VM management.
+  * Linux `ip`, FRRouting, NFTables, DNSMasq, and PowerDNS for network management.
+  * Ceph for storage management.
+  * Apache Zookeeper for the primary cluster state database.
+  * Patroni PostgreSQL manager for the secondary relation databases (DNS aggregation, Provisioner configuration).
 
-The major goal of PVC is to be administrator friendly, providing the power of Enterprise-grade private clouds like OpenStack, Nutanix, and VMWare to homelabbers, SMBs, and small ISPs, without the cost or complexity. It believes in picking the best tool for a job and abstracting it behind the cluster as a whole, freeing the administrator from the boring and time-consuming task of selecting the best component, and letting them get on with the things that really matter. Administration can be done from a simple CLI or via a RESTful API capable of building full-featured web frontends or additional applications, taking a self-documenting approach to keep the administrator learning curvet as low as possible. Setup is easy and straightforward with an [ISO-based node installer](https://git.bonifacelabs.ca/parallelvirtualcluster/pvc-installer) and [Ansible role framework](https://git.bonifacelabs.ca/parallelvirtualcluster/pvc-ansible) designed to get a cluster up and running as quickly as possible. Build your cloud in an hour, grow it as you need, and never worry about it: just add physical servers.
 
 ## Getting Started
 
-To get started with PVC, please see the [About](https://parallelvirtualcluster.readthedocs.io/en/latest/about/) page for general information about the project, and the [Getting Started](https://parallelvirtualcluster.readthedocs.io/en/latest/getting-started/) page for details on configuring your cluster.
+To get started with PVC, please see the [About](https://parallelvirtualcluster.readthedocs.io/en/latest/about/) page for general information about the project, and the [Getting Started](https://parallelvirtualcluster.readthedocs.io/en/latest/getting-started/) page for details on configuring your first cluster.
+
 
 ## Changelog
 
-#### v0.9.20
+View the changelog in [CHANGELOG.md](https://github.com/parallelvirtualcluster/pvc/blob/master/CHANGELOG.md).
 
-  * [Daemons] Implemented a Zookeeper schema handler and version 0 schema
-  * [Daemons] Completes major refactoring of codebase to make use of the schema handler
-  * [Daemons] Adds support for dynamic chema changges and "hot reloading" of pvcnoded processes
-  * [Daemons] Adds a functional testing script for verifying operation against a test cluster
-  * [Daemons, CLI] Fixes several minor bugs found by the above script
-  * [Daemons, CLI] Add support for Debian 11 "Bullseye"
 
-#### v0.9.19
+## Screenshots
 
-  * [CLI] Corrects some flawed conditionals
-  * [API] Disables SQLAlchemy modification tracking functionality (not used by us)
-  * [Daemons] Implements new zkhandler module for improved reliability and reusability
-  * [Daemons] Refactors some code to use new zkhandler module
-  * [API, CLI] Adds support for "none" migration selector (uses cluster default instead)
-  * [Daemons] Moves some configuration keys to new /config tree
-  * [Node Daemon] Increases initial lock timeout for VM migrations to avoid out-of-sync potential
-  * [Provisioner] Support storing and using textual cluster network labels ("upstream", "storage", "cluster") in templates
-  * [API] Avoid duplicating existing node states
+While PVC's API and internals aren't very screenshot-worthy, here is some example output of the CLI tool.
 
-#### v0.9.18
+<p><img alt="Node listing" src="docs/images/pvc-nodes.png"/><br/><i>Listing the nodes in a cluster</i></p>
 
-  * Adds VM rename functionality to API and CLI client
+<p><img alt="Network listing" src="docs/images/pvc-networks.png"/><br/><i>Listing the networks in a cluster, showing 3 bridged and 1 IPv4-only managed networks</i></p>
 
-#### v0.9.17
-
-  * [CLI] Fixes bugs in log follow output
-
-#### v0.9.16
-
-  * Improves some CLI help messages
-  * Skips empty local cluster in CLI
-  * Adjusts how confirmations happen during VM modify restarts
-  * Fixes bug around corrupted VM log files
-  * Fixes bug around subprocess pipe exceptions
-
-#### v0.9.15
-
-  * [CLI] Adds additional verification (--yes) to several VM management commands
-  * [CLI] Adds a method to override --yes/confirmation requirements via envvar (PVC_UNSAFE)
-  * [CLI] Adds description fields to PVC clusters in CLI
-
-#### v0.9.14
-
-  * Fixes bugs around cloned volume provisioning
-  * Fixes some minor visual bugs
-  * Minor license update (from GPL3+ to GPL3)
-  * Adds qemu-guest-agent support to provisioner-created VMs by default
-
-#### v0.9.13
-
-  * Adds nicer startup messages for daemons
-  * Adds additional API field for stored_bytes to pool stats
-  * Fixes sorting issues with snapshot lists
-  * Fixes missing increment/decrement of snapshot_count on volumes
-  * Fixes bad calls in pool element API endpoints
-  * Fixes inconsistent bytes_tohuman behaviour in daemons
-  * Adds validation and maximum volume size on creation (must be smaller than the pool free space)
-
-#### v0.9.12
-
-  * Fixes a bug in the pvcnoded service unit file causing a Zookeeper startup race condition
-
-#### v0.9.11
-
-  * Documentation updates
-  * Adds VNC information to VM info
-  * Goes back to external Ceph commands for disk usage
-
-#### v0.9.10
-
-  * Moves OSD stats uploading to primary, eliminating reporting failures while hosts are down
-  * Documentation updates
-  * Significantly improves RBD locking behaviour in several situations, eliminating cold-cluster start issues and failed VM boot-ups after crashes
-  * Fixes some timeout delays with fencing
-  * Fixes bug in validating YAML provisioner userdata
-
-#### v0.9.9
-
-  * Adds documentation updates
-  * Removes single-element list stripping and fixes surrounding bugs
-  * Adds additional fields to some API endpoints for ease of parsing by clients
-  * Fixes bugs with network configuration
-
-#### v0.9.8
-
-  * Adds support for cluster backup/restore
-  * Moves location of `init` command in CLI to make room for the above
-  * Cleans up some invalid help messages from the API
-
-#### v0.9.7
-
-  * Fixes bug with provisioner system template modifications
-
-#### v0.9.6
-
-  * Fixes bug with migrations
-
-#### v0.9.5
-
-  * Fixes bug with line count in log follow
-  * Fixes bug with disk stat output being None
-  * Adds short pretty health output
-  * Documentation updates
-
-#### v0.9.4
-
-  * Fixes major bug in OVA parser
-
-#### v0.9.3
-
-  * Fixes bugs with image & OVA upload parsing
-
-#### v0.9.2
-
-  * Major linting of the codebase with flake8; adds linting tools
-  * Implements CLI-based modification of VM vCPUs, memory, networks, and disks without directly editing XML
-  * Fixes bug where `pvc vm log -f` would show all 1000 lines before starting
-  * Fixes bug in default provisioner libvirt schema (`drive` -> `driver` typo)
-
-#### v0.9.1
-
-  * Added per-VM migration method feature
-  * Fixed bug with provisioner system template listing
-
-#### v0.9.0
-
-Numerous small improvements and bugfixes. This release is suitable for general use and is pre-release-quality software.
-
-This release introduces an updated version scheme; all future stable releases until 1.0.0 is ready will be made under this 0.9.z naming. This does not represent semantic versioning and all changes (feature, improvement, or bugfix) will be considered for inclusion in this release train.
-
-#### v0.8
-
-Numerous improvements and bugfixes. This release is suitable for general use and is pre-release-quality software.
-
-#### v0.7
-
-Numerous improvements and bugfixes, revamped documentation. This release is suitable for general use and is beta-quality software.
-
-#### v0.6
-
-Numerous improvements and bugfixes, full implementation of the provisioner, full implementation of the API CLI client (versus direct CLI client). This release is suitable for general use and is beta-quality software.
-
-#### v0.5
-
-First public release; fully implements the VM, network, and storage managers, the HTTP API, and the pvc-ansible framework for deploying and bootstrapping a cluster. This release is suitable for general use, though it is still alpha-quality software and should be expected to change significantly until 1.0 is released.
-
-#### v0.4
-
-Full implementation of virtual management and virtual networking functionality. Partial implementation of storage functionality.
-
-#### v0.3
-
-Basic implementation of virtual management functionality.
+<p><img alt="VM listing and migration" src="docs/images/pvc-migration.png"/><br/><i>Listing a limited set of VMs and migrating one with status updates</i></p>
 
+<p><img alt="Node logs" src="docs/images/pvc-nodelog.png"/><br/><i>Viewing the logs of a node (keepalives and VM [un]migration)</i></p>

api-daemon/migrations/versions/5c2109dbbeae_pvc_version_0_9_37.py

@@ -0,0 +1,28 @@
+"""PVC version 0.9.37
+
+Revision ID: 5c2109dbbeae
+Revises: bae4d5a77c74
+Create Date: 2021-10-02 00:47:29.693579
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = '5c2109dbbeae'
+down_revision = 'bae4d5a77c74'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('storage_benchmarks', sa.Column('test_format', sa.Integer(), server_default='0', nullable=False))
+    # ### end Alembic commands ###
+
+
+def downgrade():
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('storage_benchmarks', 'test_format')
+    # ### end Alembic commands ###

api-daemon/provisioner/examples/debootstrap_script.py

@@ -34,6 +34,29 @@
 # with that.
 
 import os
+from contextlib import contextmanager
+
+
+# Create a chroot context manager
+# This can be used later in the script to chroot to the destination directory
+# for instance to run commands within the target.
+@contextmanager
+def chroot_target(destination):
+    try:
+        real_root = os.open("/", os.O_RDONLY)
+        os.chroot(destination)
+        fake_root = os.open("/", os.O_RDONLY)
+        os.fchdir(fake_root)
+        yield
+    finally:
+        os.fchdir(real_root)
+        os.chroot(".")
+        os.fchdir(real_root)
+        os.close(fake_root)
+        os.close(real_root)
+        del fake_root
+        del real_root
+
 
 # Installation function - performs a debootstrap install of a Debian system
 # Note that the only arguments are keyword arguments.
@@ -193,40 +216,25 @@ GRUB_DISABLE_LINUX_UUID=false
         fh.write(data)
 
     # Chroot, do some in-root tasks, then exit the chroot
-    # EXITING THE CHROOT IS VERY IMPORTANT OR THE FOLLOWING STAGES OF THE PROVISIONER
-    # WILL FAIL IN UNEXPECTED WAYS! Keep this in mind when using chroot in your scripts.
-    real_root = os.open("/", os.O_RDONLY)
-    os.chroot(temporary_directory)
-    fake_root = os.open("/", os.O_RDONLY)
-    os.fchdir(fake_root)
-
-    # Install and update GRUB
-    os.system(
-        "grub-install --force /dev/rbd/{}/{}_{}".format(root_disk['pool'], vm_name, root_disk['disk_id'])
-    )
-    os.system(
-        "update-grub"
-    )
-    # Set a really dumb root password [TEMPORARY]
-    os.system(
-        "echo root:test123 | chpasswd"
-    )
-    # Enable cloud-init target on (first) boot
-    # NOTE: Your user-data should handle this and disable it once done, or things get messy.
-    #       That cloud-init won't run without this hack seems like a bug... but even the official
-    #       Debian cloud images are affected, so who knows.
-    os.system(
-        "systemctl enable cloud-init.target"
-    )
-
-    # Restore our original root/exit the chroot
-    # EXITING THE CHROOT IS VERY IMPORTANT OR THE FOLLOWING STAGES OF THE PROVISIONER
-    # WILL FAIL IN UNEXPECTED WAYS! Keep this in mind when using chroot in your scripts.
-    os.fchdir(real_root)
-    os.chroot(".")
-    os.fchdir(real_root)
-    os.close(fake_root)
-    os.close(real_root)
+    with chroot_target(temporary_directory):
+        # Install and update GRUB
+        os.system(
+            "grub-install --force /dev/rbd/{}/{}_{}".format(root_disk['pool'], vm_name, root_disk['disk_id'])
+        )
+        os.system(
+            "update-grub"
+        )
+        # Set a really dumb root password [TEMPORARY]
+        os.system(
+            "echo root:test123 | chpasswd"
+        )
+        # Enable cloud-init target on (first) boot
+        # NOTE: Your user-data should handle this and disable it once done, or things get messy.
+        #       That cloud-init won't run without this hack seems like a bug... but even the official
+        #       Debian cloud images are affected, so who knows.
+        os.system(
+            "systemctl enable cloud-init.target"
+        )
 
     # Unmount the bound devfs
     os.system(
@@ -235,8 +243,4 @@ GRUB_DISABLE_LINUX_UUID=false
         )
     )
 
-    # Clean up file handles so paths can be unmounted
-    del fake_root
-    del real_root
-
     # Everything else is done via cloud-init user-data

api-daemon/provisioner/examples/dummy_script.py

@@ -29,7 +29,7 @@
 # This script will run under root privileges as the provisioner does. Be careful
 # with that.
 
-# Installation function - performs a debootstrap install of a Debian system
+# Installation function - performs no actions then returns
 # Note that the only arguments are keyword arguments.
 def install(**kwargs):
     # The provisioner has already mounted the disks on kwargs['temporary_directory'].

api-daemon/pvcapid-worker.service

@@ -9,7 +9,7 @@ Type = simple
 WorkingDirectory = /usr/share/pvc
 Environment = PYTHONUNBUFFERED=true
 Environment = PVC_CONFIG_FILE=/etc/pvc/pvcapid.yaml
-ExecStart = /usr/bin/celery --app pvcapid.flaskapi.celery worker --concurrency 1 --loglevel INFO
+ExecStart = /usr/share/pvc/pvcapid-worker.sh
 Restart = on-failure
 
 [Install]

api-daemon/pvcapid-worker.sh

@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+
+# pvcapid-worker.py - API Celery worker daemon startup stub
+# Part of the Parallel Virtual Cluster (PVC) system
+#
+#    Copyright (C) 2018-2021 Joshua M. Boniface <joshua@boniface.me>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, version 3.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+#
+###############################################################################
+
+CELERY_BIN="$( which celery )"
+
+# This absolute hackery is needed because Celery got the bright idea to change how their
+# app arguments work in a non-backwards-compatible way with Celery 5.
+case "$( cat /etc/debian_version )" in
+    10.*)
+        CELERY_ARGS="worker --app pvcapid.flaskapi.celery --concurrency 1 --loglevel INFO"
+    ;;
+    11.*)
+        CELERY_ARGS="--app pvcapid.flaskapi.celery worker --concurrency 1 --loglevel INFO"
+    ;;
+    *)
+        echo "Invalid Debian version found!"
+        exit 1
+    ;;
+esac
+
+${CELERY_BIN} ${CELERY_ARGS}
+exit $?

api-daemon/pvcapid/Daemon.py

@@ -25,7 +25,7 @@ import yaml
 from distutils.util import strtobool as dustrtobool
 
 # Daemon version
-version = '0.9.20'
+version = '0.9.41'
 
 # API version
 API_VERSION = 1.0
@@ -117,21 +117,21 @@ def entrypoint():
 
     # Print our startup messages
     print('')
-    print('|--------------------------------------------------|')
-    print('|           ########  ##     ##  ######            |')
-    print('|           ##     ## ##     ## ##    ##           |')
-    print('|           ##     ## ##     ## ##                 |')
-    print('|           ########  ##     ## ##                 |')
-    print('|           ##         ##   ##  ##                 |')
-    print('|           ##          ## ##   ##    ##           |')
-    print('|           ##           ###     ######            |')
-    print('|--------------------------------------------------|')
-    print('| Parallel Virtual Cluster API daemon v{0: <11} |'.format(version))
-    print('| API version: v{0: <34} |'.format(API_VERSION))
-    print('| Listen: {0: <40} |'.format('{}:{}'.format(config['listen_address'], config['listen_port'])))
-    print('| SSL: {0: <43} |'.format(str(config['ssl_enabled'])))
-    print('| Authentication: {0: <32} |'.format(str(config['auth_enabled'])))
-    print('|--------------------------------------------------|')
+    print('|----------------------------------------------------------|')
+    print('|                                                          |')
+    print('|           ███████████ ▜█▙      ▟█▛ █████ █ █ █           |')
+    print('|                    ██  ▜█▙    ▟█▛  ██                    |')
+    print('|           ███████████   ▜█▙  ▟█▛   ██                    |')
+    print('|           ██             ▜█▙▟█▛    ███████████           |')
+    print('|                                                          |')
+    print('|----------------------------------------------------------|')
+    print('| Parallel Virtual Cluster API daemon v{0: <19} |'.format(version))
+    print('| Debug: {0: <49} |'.format(str(config['debug'])))
+    print('| API version: v{0: <42} |'.format(API_VERSION))
+    print('| Listen: {0: <48} |'.format('{}:{}'.format(config['listen_address'], config['listen_port'])))
+    print('| SSL: {0: <51} |'.format(str(config['ssl_enabled'])))
+    print('| Authentication: {0: <40} |'.format(str(config['auth_enabled'])))
+    print('|----------------------------------------------------------|')
     print('')
 
     pvc_api.app.run(config['listen_address'], config['listen_port'], threaded=True, ssl_context=context)

api-daemon/pvcapid/benchmark.py

@@ -22,9 +22,11 @@
 import psycopg2
 import psycopg2.extras
 
+from json import loads, dumps
+
 from pvcapid.Daemon import config
 
-from daemon_lib.zkhandler import ZKConnection
+from daemon_lib.zkhandler import ZKHandler
 
 import daemon_lib.common as pvc_common
 import daemon_lib.ceph as pvc_ceph
@@ -37,12 +39,12 @@ class BenchmarkError(Exception):
     """
     An exception that results from the Benchmark job.
     """
-    def __init__(self, message, cur_time=None, db_conn=None, db_cur=None, zkhandler=None):
+    def __init__(self, message, job_name=None, db_conn=None, db_cur=None, zkhandler=None):
         self.message = message
-        if cur_time is not None:
+        if job_name is not None:
             # Clean up our dangling result
             query = "DELETE FROM storage_benchmarks WHERE job = %s;"
-            args = (cur_time,)
+            args = (job_name,)
             db_cur.execute(query, args)
             db_conn.commit()
             # Close the database connections cleanly
@@ -93,7 +95,14 @@ def list_benchmarks(job=None):
         benchmark_data = dict()
         benchmark_data['id'] = benchmark['id']
         benchmark_data['job'] = benchmark['job']
-        benchmark_data['benchmark_result'] = benchmark['result']
+        benchmark_data['test_format'] = benchmark['test_format']
+        if benchmark['result'] == 'Running':
+            benchmark_data['benchmark_result'] = 'Running'
+        else:
+            try:
+                benchmark_data['benchmark_result'] = loads(benchmark['result'])
+            except Exception:
+                benchmark_data['benchmark_result'] = {}
         # Append the new data to our actual output structure
         data.append(benchmark_data)
     close_database(conn, cur)
@@ -103,19 +112,16 @@ def list_benchmarks(job=None):
         return {'message': 'No benchmark found.'}, 404
 
 
-@ZKConnection(config)
-def run_benchmark(self, zkhandler, pool):
+def run_benchmark(self, pool):
     # Runtime imports
     import time
-    import json
     from datetime import datetime
 
+    # Define the current test format
+    TEST_FORMAT = 1
+
     time.sleep(2)
 
-    cur_time = datetime.now().isoformat(timespec='seconds')
-
-    print("Starting storage benchmark '{}' on pool '{}'".format(cur_time, pool))
-
     # Phase 0 - connect to databases
     try:
         db_conn, db_cur = open_database(config)
@@ -123,14 +129,27 @@ def run_benchmark(self, zkhandler, pool):
         print('FATAL - failed to connect to Postgres')
         raise Exception
 
-    print("Storing running status for job '{}' in database".format(cur_time))
     try:
-        query = "INSERT INTO storage_benchmarks (job, result) VALUES (%s, %s);"
-        args = (cur_time, "Running",)
+        zkhandler = ZKHandler(config)
+        zkhandler.connect()
+    except Exception:
+        print('FATAL - failed to connect to Zookeeper')
+        raise Exception
+
+    cur_time = datetime.now().isoformat(timespec='seconds')
+    cur_primary = zkhandler.read('base.config.primary_node')
+    job_name = '{}_{}'.format(cur_time, cur_primary)
+
+    print("Starting storage benchmark '{}' on pool '{}'".format(job_name, pool))
+
+    print("Storing running status for job '{}' in database".format(job_name))
+    try:
+        query = "INSERT INTO storage_benchmarks (job, test_format, result) VALUES (%s, %s, %s);"
+        args = (job_name, TEST_FORMAT, "Running",)
         db_cur.execute(query, args)
         db_conn.commit()
     except Exception as e:
-        raise BenchmarkError("Failed to store running status: {}".format(e), cur_time=cur_time, db_conn=db_conn, db_cur=db_cur, zkhandler=zkhandler)
+        raise BenchmarkError("Failed to store running status: {}".format(e), job_name=job_name, db_conn=db_conn, db_cur=db_cur, zkhandler=zkhandler)
 
     # Phase 1 - volume preparation
     self.update_state(state='RUNNING', meta={'current': 1, 'total': 3, 'status': 'Creating benchmark volume'})
@@ -141,7 +160,7 @@ def run_benchmark(self, zkhandler, pool):
     # Create the RBD volume
     retcode, retmsg = pvc_ceph.add_volume(zkhandler, pool, volume, "8G")
     if not retcode:
-        raise BenchmarkError('Failed to create volume "{}": {}'.format(volume, retmsg), cur_time=cur_time, db_conn=db_conn, db_cur=db_cur, zkhandler=zkhandler)
+        raise BenchmarkError('Failed to create volume "{}": {}'.format(volume, retmsg), job_name=job_name, db_conn=db_conn, db_cur=db_cur, zkhandler=zkhandler)
     else:
         print(retmsg)
 
@@ -163,266 +182,87 @@ def run_benchmark(self, zkhandler, pool):
     test_matrix = {
         'seq_read': {
             'direction': 'read',
+            'iodepth': '64',
             'bs': '4M',
             'rw': 'read'
         },
         'seq_write': {
             'direction': 'write',
+            'iodepth': '64',
             'bs': '4M',
             'rw': 'write'
         },
         'rand_read_4M': {
             'direction': 'read',
+            'iodepth': '64',
             'bs': '4M',
             'rw': 'randread'
         },
         'rand_write_4M': {
             'direction': 'write',
+            'iodepth': '64',
             'bs': '4M',
             'rw': 'randwrite'
         },
-        'rand_read_256K': {
-            'direction': 'read',
-            'bs': '256K',
-            'rw': 'randread'
-        },
-        'rand_write_256K': {
-            'direction': 'write',
-            'bs': '256K',
-            'rw': 'randwrite'
-        },
         'rand_read_4K': {
             'direction': 'read',
+            'iodepth': '64',
             'bs': '4K',
             'rw': 'randread'
         },
         'rand_write_4K': {
             'direction': 'write',
+            'iodepth': '64',
             'bs': '4K',
             'rw': 'randwrite'
-        }
+        },
+        'rand_read_4K_lowdepth': {
+            'direction': 'read',
+            'iodepth': '1',
+            'bs': '4K',
+            'rw': 'randread'
+        },
+        'rand_write_4K_lowdepth': {
+            'direction': 'write',
+            'iodepth': '1',
+            'bs': '4K',
+            'rw': 'randwrite'
+        },
     }
-    parsed_results = dict()
+
+    results = dict()
     for test in test_matrix:
         print("Running test '{}'".format(test))
         fio_cmd = """
             fio \
-                --output-format=terse \
-                --terse-version=5 \
+                --name={test} \
                 --ioengine=rbd \
                 --pool={pool} \
                 --rbdname={volume} \
+                --output-format=json \
                 --direct=1 \
                 --randrepeat=1 \
-                --iodepth=64 \
-                --size=8G \
-                --name={test} \
+                --numjobs=1 \
+                --time_based \
+                --runtime=75 \
+                --group_reporting \
+                --iodepth={iodepth} \
                 --bs={bs} \
                 --readwrite={rw}
         """.format(
+            test=test,
             pool=pool,
             volume=volume,
-            test=test,
+            iodepth=test_matrix[test]['iodepth'],
             bs=test_matrix[test]['bs'],
             rw=test_matrix[test]['rw'])
 
+        print("Running fio job: {}".format(' '.join(fio_cmd.split())))
         retcode, stdout, stderr = pvc_common.run_os_command(fio_cmd)
         if retcode:
-            raise BenchmarkError("Failed to run fio test: {}".format(stderr), cur_time=cur_time, db_conn=db_conn, db_cur=db_cur, zkhandler=zkhandler)
+            raise BenchmarkError("Failed to run fio test: {}".format(stderr), job_name=job_name, db_conn=db_conn, db_cur=db_cur, zkhandler=zkhandler)
 
-        # Parse the terse results to avoid storing tons of junk
-        # Reference: https://fio.readthedocs.io/en/latest/fio_doc.html#terse-output
-        # This is written out broken up because the man page didn't bother to do this, and I'm putting it here for posterity.
-        # Example Read test (line breaks to match man ref):
-        #    I 5;fio-3.12;test;0;0; (5) [0, 1, 2, 3, 4]
-        #    R 8388608;2966268;724;2828; (4) [5, 6, 7, 8]
-        #      0;0;0.000000;0.000000; (4) [9, 10, 11, 12]
-        #      0;0;0.000000;0.000000; (4) [13, 14, 15, 16]
-        #      0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0; (20) [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,33, 34, 35, 36]
-        #      0;0;0.000000;0.000000; (4) [37, 38, 39, 40]
-        #      2842624;3153920;100.000000%;2967142.400000;127226.797479;5; (6) [41, 42, 43, 44, 45, 46]
-        #      694;770;724.400000;31.061230;5; (5) [47, 48, 49, 50, 51]
-        #    W 0;0;0;0; (4) [52, 53, 54, 55]
-        #      0;0;0.000000;0.000000; (4) [56, 57, 58, 59]
-        #      0;0;0.000000;0.000000; (4) [60, 61, 62, 63]
-        #      0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0; (20) [64, 65, 66, 67, 68. 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83]
-        #      0;0;0.000000;0.000000; (4) [84, 85, 86, 87]
-        #      0;0;0.000000%;0.000000;0.000000;0; (6) [88, 89, 90, 91, 92, 93]
-        #      0;0;0.000000;0.000000;0; (5) [94, 95, 96, 97, 98]
-        #    T 0;0;0;0; (4) [99, 100, 101, 102]
-        #      0;0;0.000000;0.000000; (4) [103, 104, 105, 106]
-        #      0;0;0.000000;0.000000; (4) [107, 108, 109, 110]
-        #      0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0; (20) [111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130]
-        #      0;0;0.000000;0.000000; (4) [131, 132, 133, 134]
-        #      0;0;0.000000%;0.000000;0.000000;0; (6) [135, 136, 137, 138, 139, 140]
-        #      0;0;0.000000;0.000000;0; (5) [141, 142, 143, 144, 145]
-        #    C 0.495225%;0.000000%;2083;0;13; (5) [146, 147, 148, 149, 150]
-        #    D 0.1%;0.1%;0.2%;0.4%;0.8%;1.6%;96.9%; (7) [151, 152, 153, 154, 155, 156, 157]
-        #    U 0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%; (10) [158, 159, 160, 161, 162, 163, 164, 165, 166, 167]
-        #    M 0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%; (12) [168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178. 179]
-        #    B dm-0;0;110;0;0;0;4;4;0.15%; (9) [180, 181, 182, 183, 184, 185, 186, 187, 188]
-        #      slaves;0;118;0;28;0;23;0;0.00%; (9) [189, 190, 191, 192, 193, 194, 195, 196, 197]
-        #      sde;0;118;0;28;0;23;0;0.00% (9) [198, 199, 200, 201, 202, 203, 204, 205, 206]
-        # Example Write test:
-        #    I 5;fio-3.12;test;0;0; (5)
-        #    R 0;0;0;0; (4)
-        #      0;0;0.000000;0.000000; (4)
-        #      0;0;0.000000;0.000000; (4)
-        #      0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0; (20)
-        #      0;0;0.000000;0.000000; (4)
-        #      0;0;0.000000%;0.000000;0.000000;0; (6)
-        #      0;0;0.000000;0.000000;0; (5)
-        #    W 8388608;1137438;277;7375; (4)
-        #      0;0;0.000000;0.000000; (4)
-        #      0;0;0.000000;0.000000; (4)
-        #      0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0; (20)
-        #      0;0;0.000000;0.000000; (4)
-        #      704512;1400832;99.029573%;1126400.000000;175720.860374;14; (6)
-        #      172;342;275.000000;42.900601;14; (5)
-        #    T 0;0;0;0; (4)
-        #      0;0;0.000000;0.000000; (4)
-        #      0;0;0.000000;0.000000; (4)
-        #      0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0;0%=0; (20)
-        #      0;0;0.000000;0.000000; (4)
-        #      0;0;0.000000%;0.000000;0.000000;0; (6)
-        #      0;0;0.000000;0.000000;0; (5)
-        #    C 12.950909%;1.912124%;746;0;95883; (5)
-        #    D 0.1%;0.1%;0.2%;0.4%;0.8%;1.6%;96.9%; (7)
-        #    U 0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%; (10)
-        #    M 0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%; (12)
-        #    B dm-0;0;196;0;0;0;12;12;0.16%; (9)
-        #      slaves;0;207;0;95;0;39;16;0.21%; (9)
-        #      sde;0;207;0;95;0;39;16;0.21% (9)
-        results = stdout.split(';')
-        if test_matrix[test]['direction'] == 'read':
-            # Stats
-            #         5:   Total IO (KiB)
-            #         6:   bandwidth (KiB/sec)
-            #         7:   IOPS
-            #         8:   runtime (msec)
-            # Total latency
-            #         37:  min
-            #         38:  max
-            #         39:  mean
-            #         40:  stdev
-            # Bandwidth
-            #         41:  min
-            #         42:  max
-            #         44:  mean
-            #         45:  stdev
-            #         46:  # samples
-            # IOPS
-            #         47:  min
-            #         48:  max
-            #         49:  mean
-            #         50:  stdev
-            #         51:  # samples
-            # CPU
-            #         146: user
-            #         147: system
-            #         148: ctx switches
-            #         149: maj faults
-            #         150: min faults
-            parsed_results[test] = {
-                "overall": {
-                    "iosize": results[5],
-                    "bandwidth": results[6],
-                    "iops": results[7],
-                    "runtime": results[8]
-                },
-                "latency": {
-                    "min": results[37],
-                    "max": results[38],
-                    "mean": results[39],
-                    "stdev": results[40]
-                },
-                "bandwidth": {
-                    "min": results[41],
-                    "max": results[42],
-                    "mean": results[44],
-                    "stdev": results[45],
-                    "numsamples": results[46],
-                },
-                "iops": {
-                    "min": results[47],
-                    "max": results[48],
-                    "mean": results[49],
-                    "stdev": results[50],
-                    "numsamples": results[51]
-                },
-                "cpu": {
-                    "user": results[146],
-                    "system": results[147],
-                    "ctxsw": results[148],
-                    "majfault": results[149],
-                    "minfault": results[150]
-                }
-            }
-
-        if test_matrix[test]['direction'] == 'write':
-            # Stats
-            #         52:  Total IO (KiB)
-            #         53:  bandwidth (KiB/sec)
-            #         54:  IOPS
-            #         55:  runtime (msec)
-            # Total latency
-            #         84:  min
-            #         85:  max
-            #         86:  mean
-            #         87:  stdev
-            # Bandwidth
-            #         88:  min
-            #         89:  max
-            #         91:  mean
-            #         92:  stdev
-            #         93:  # samples
-            # IOPS
-            #         94:  min
-            #         95:  max
-            #         96:  mean
-            #         97:  stdev
-            #         98:  # samples
-            # CPU
-            #         146: user
-            #         147: system
-            #         148: ctx switches
-            #         149: maj faults
-            #         150: min faults
-            parsed_results[test] = {
-                "overall": {
-                    "iosize": results[52],
-                    "bandwidth": results[53],
-                    "iops": results[54],
-                    "runtime": results[55]
-                },
-                "latency": {
-                    "min": results[84],
-                    "max": results[85],
-                    "mean": results[86],
-                    "stdev": results[87]
-                },
-                "bandwidth": {
-                    "min": results[88],
-                    "max": results[89],
-                    "mean": results[91],
-                    "stdev": results[92],
-                    "numsamples": results[93],
-                },
-                "iops": {
-                    "min": results[94],
-                    "max": results[95],
-                    "mean": results[96],
-                    "stdev": results[97],
-                    "numsamples": results[98]
-                },
-                "cpu": {
-                    "user": results[146],
-                    "system": results[147],
-                    "ctxsw": results[148],
-                    "majfault": results[149],
-                    "minfault": results[150]
-                }
-            }
+        results[test] = loads(stdout)
 
     # Phase 3 - cleanup
     self.update_state(state='RUNNING', meta={'current': 3, 'total': 3, 'status': 'Cleaning up and storing results'})
@@ -431,18 +271,21 @@ def run_benchmark(self, zkhandler, pool):
     # Remove the RBD volume
     retcode, retmsg = pvc_ceph.remove_volume(zkhandler, pool, volume)
     if not retcode:
-        raise BenchmarkError('Failed to remove volume "{}": {}'.format(volume, retmsg), cur_time=cur_time, db_conn=db_conn, db_cur=db_cur, zkhandler=zkhandler)
+        raise BenchmarkError('Failed to remove volume "{}": {}'.format(volume, retmsg), job_name=job_name, db_conn=db_conn, db_cur=db_cur, zkhandler=zkhandler)
     else:
         print(retmsg)
 
-    print("Storing result of tests for job '{}' in database".format(cur_time))
+    print("Storing result of tests for job '{}' in database".format(job_name))
     try:
         query = "UPDATE storage_benchmarks SET result = %s WHERE job = %s;"
-        args = (json.dumps(parsed_results), cur_time)
+        args = (dumps(results), job_name)
         db_cur.execute(query, args)
         db_conn.commit()
     except Exception as e:
-        raise BenchmarkError("Failed to store test results: {}".format(e), cur_time=cur_time, db_conn=db_conn, db_cur=db_cur, zkhandler=zkhandler)
+        raise BenchmarkError("Failed to store test results: {}".format(e), job_name=job_name, db_conn=db_conn, db_cur=db_cur, zkhandler=zkhandler)
 
     close_database(db_conn, db_cur)
+    zkhandler.disconnect()
+    del zkhandler
+
     return {'status': "Storage benchmark '{}' completed successfully.", 'current': 3, 'total': 3}

api-daemon/pvcapid/flaskapi.py

@@ -299,13 +299,12 @@ class API_Initialize(Resource):
           400:
             description: Bad request
         """
-        if reqargs.get('overwrite', False):
+        if reqargs.get('overwrite', 'False') == 'True':
             overwrite_flag = True
-
-        if api_helper.initialize_cluster(overwrite=overwrite_flag):
-            return {"message": "Successfully initialized a new PVC cluster"}, 200
         else:
-            return {"message": "PVC cluster already initialized"}, 400
+            overwrite_flag = False
+
+        return api_helper.initialize_cluster(overwrite=overwrite_flag)
 
 
 api.add_resource(API_Initialize, '/initialize')
@@ -536,6 +535,9 @@ class API_Node_Root(Resource):
                 domain_state:
                   type: string
                   description: The current domain (VM) state
+                pvc_version:
+                  type: string
+                  description: The current running PVC node daemon version
                 cpu_count:
                   type: integer
                   description: The number of available CPU cores
@@ -590,7 +592,7 @@ class API_Node_Root(Resource):
             name: limit
             type: string
             required: false
-            description: A search limit; fuzzy by default, use ^/$ to force exact matches
+            description: A search limit in the name, tags, or an exact UUID; fuzzy by default, use ^/$ to force exact matches
           - in: query
             name: daemon_state
             type: string
@@ -832,6 +834,52 @@ class API_Node_DomainState(Resource):
 api.add_resource(API_Node_DomainState, '/node/<node>/domain-state')
 
 
+# /node/<node</log
+class API_Node_Log(Resource):
+    @RequestParser([
+        {'name': 'lines'}
+    ])
+    @Authenticator
+    def get(self, node, reqargs):
+        """
+        Return the recent logs of {node}
+        ---
+        tags:
+          - node
+        parameters:
+          - in: query
+            name: lines
+            type: integer
+            required: false
+            description: The number of lines to retrieve
+        responses:
+          200:
+            description: OK
+            schema:
+              type: object
+              id: NodeLog
+              properties:
+                name:
+                  type: string
+                  description: The name of the Node
+                data:
+                  type: string
+                  description: The recent log text
+          404:
+            description: Node not found
+            schema:
+              type: object
+              id: Message
+        """
+        return api_helper.node_log(
+            node,
+            reqargs.get('lines', None)
+        )
+
+
+api.add_resource(API_Node_Log, '/node/<node>/log')
+
+
 ##########################################################
 # Client API - VM
 ##########################################################
@@ -842,6 +890,8 @@ class API_VM_Root(Resource):
         {'name': 'limit'},
         {'name': 'node'},
         {'name': 'state'},
+        {'name': 'tag'},
+        {'name': 'negate'},
     ])
     @Authenticator
     def get(self, reqargs):
@@ -890,6 +940,22 @@ class API_VM_Root(Resource):
                 migration_method:
                   type: string
                   description: The preferred migration method (live, shutdown, none)
+                tags:
+                  type: array
+                  description: The tag(s) of the VM
+                  items:
+                    type: object
+                    id: VMTag
+                    properties:
+                      name:
+                        type: string
+                        description: The name of the tag
+                      type:
+                        type: string
+                        description: The type of the tag (user, system)
+                      protected:
+                        type: boolean
+                        description: Whether the tag is protected or not
                 description:
                   type: string
                   description: The description of the VM
@@ -1074,7 +1140,7 @@ class API_VM_Root(Resource):
             name: limit
             type: string
             required: false
-            description: A name search limit; fuzzy by default, use ^/$ to force exact matches
+            description: A search limit in the name, tags, or an exact UUID; fuzzy by default, use ^/$ to force exact matches
           - in: query
             name: node
             type: string
@@ -1085,6 +1151,16 @@ class API_VM_Root(Resource):
             type: string
             required: false
             description: Limit list to VMs in this state
+          - in: query
+            name: tag
+            type: string
+            required: false
+            description: Limit list to VMs with this tag
+          - in: query
+            name: negate
+            type: boolean
+            required: false
+            description: Negate the specified node, state, or tag limit(s)
         responses:
           200:
             description: OK
@@ -1094,9 +1170,11 @@ class API_VM_Root(Resource):
                 $ref: '#/definitions/vm'
         """
         return api_helper.vm_list(
-            reqargs.get('node', None),
-            reqargs.get('state', None),
-            reqargs.get('limit', None)
+            node=reqargs.get('node', None),
+            state=reqargs.get('state', None),
+            tag=reqargs.get('tag', None),
+            limit=reqargs.get('limit', None),
+            negate=bool(strtobool(reqargs.get('negate', 'False'))),
         )
 
     @RequestParser([
@@ -1105,6 +1183,8 @@ class API_VM_Root(Resource):
         {'name': 'selector', 'choices': ('mem', 'vcpus', 'load', 'vms', 'none'), 'helptext': "A valid selector must be specified"},
         {'name': 'autostart'},
         {'name': 'migration_method', 'choices': ('live', 'shutdown', 'none'), 'helptext': "A valid migration_method must be specified"},
+        {'name': 'user_tags', 'action': 'append'},
+        {'name': 'protected_tags', 'action': 'append'},
         {'name': 'xml', 'required': True, 'helptext': "A Libvirt XML document must be specified"},
     ])
     @Authenticator
@@ -1156,6 +1236,20 @@ class API_VM_Root(Resource):
               - live
               - shutdown
               - none
+          - in: query
+            name: user_tags
+            type: array
+            required: false
+            description: The user tag(s) of the VM
+            items:
+              type: string
+          - in: query
+            name: protected_tags
+            type: array
+            required: false
+            description: The protected user tag(s) of the VM
+            items:
+              type: string
         responses:
           200:
             description: OK
@@ -1168,13 +1262,22 @@ class API_VM_Root(Resource):
               type: object
               id: Message
         """
+        user_tags = reqargs.get('user_tags', None)
+        if user_tags is None:
+            user_tags = []
+        protected_tags = reqargs.get('protected_tags', None)
+        if protected_tags is None:
+            protected_tags = []
+
         return api_helper.vm_define(
             reqargs.get('xml'),
             reqargs.get('node', None),
             reqargs.get('limit', None),
             reqargs.get('selector', 'none'),
             bool(strtobool(reqargs.get('autostart', 'false'))),
-            reqargs.get('migration_method', 'none')
+            reqargs.get('migration_method', 'none'),
+            user_tags,
+            protected_tags
         )
 
 
@@ -1201,7 +1304,7 @@ class API_VM_Element(Resource):
               type: object
               id: Message
         """
-        return api_helper.vm_list(None, None, vm, is_fuzzy=False)
+        return api_helper.vm_list(node=None, state=None, tag=None, limit=vm, is_fuzzy=False, negate=False)
 
     @RequestParser([
         {'name': 'limit'},
@@ -1209,6 +1312,8 @@ class API_VM_Element(Resource):
         {'name': 'selector', 'choices': ('mem', 'vcpus', 'load', 'vms', 'none'), 'helptext': "A valid selector must be specified"},
         {'name': 'autostart'},
         {'name': 'migration_method', 'choices': ('live', 'shutdown', 'none'), 'helptext': "A valid migration_method must be specified"},
+        {'name': 'user_tags', 'action': 'append'},
+        {'name': 'protected_tags', 'action': 'append'},
         {'name': 'xml', 'required': True, 'helptext': "A Libvirt XML document must be specified"},
     ])
     @Authenticator
@@ -1263,6 +1368,20 @@ class API_VM_Element(Resource):
               - live
               - shutdown
               - none
+          - in: query
+            name: user_tags
+            type: array
+            required: false
+            description: The user tag(s) of the VM
+            items:
+              type: string
+          - in: query
+            name: protected_tags
+            type: array
+            required: false
+            description: The protected user tag(s) of the VM
+            items:
+              type: string
         responses:
           200:
             description: OK
@@ -1275,13 +1394,22 @@ class API_VM_Element(Resource):
               type: object
               id: Message
         """
+        user_tags = reqargs.get('user_tags', None)
+        if user_tags is None:
+            user_tags = []
+        protected_tags = reqargs.get('protected_tags', None)
+        if protected_tags is None:
+            protected_tags = []
+
         return api_helper.vm_define(
             reqargs.get('xml'),
             reqargs.get('node', None),
             reqargs.get('limit', None),
             reqargs.get('selector', 'none'),
             bool(strtobool(reqargs.get('autostart', 'false'))),
-            reqargs.get('migration_method', 'none')
+            reqargs.get('migration_method', 'none'),
+            user_tags,
+            protected_tags
         )
 
     @RequestParser([
@@ -1399,7 +1527,7 @@ class API_VM_Metadata(Resource):
                   type: string
                   description: The preferred migration method (live, shutdown, none)
           404:
-            description: Not found
+            description: VM not found
             schema:
               type: object
               id: Message
@@ -1467,6 +1595,11 @@ class API_VM_Metadata(Resource):
             schema:
               type: object
               id: Message
+          404:
+            description: VM not found
+            schema:
+              type: object
+              id: Message
         """
         return api_helper.update_vm_meta(
             vm,
@@ -1481,6 +1614,99 @@ class API_VM_Metadata(Resource):
 api.add_resource(API_VM_Metadata, '/vm/<vm>/meta')
 
 
+# /vm/<vm>/tags
+class API_VM_Tags(Resource):
+    @Authenticator
+    def get(self, vm):
+        """
+        Return the tags of {vm}
+        ---
+        tags:
+          - vm
+        responses:
+          200:
+            description: OK
+            schema:
+              type: object
+              id: VMTags
+              properties:
+                name:
+                  type: string
+                  description: The name of the VM
+                tags:
+                  type: array
+                  description: The tag(s) of the VM
+                  items:
+                    type: object
+                    id: VMTag
+          404:
+            description: VM not found
+            schema:
+              type: object
+              id: Message
+        """
+        return api_helper.get_vm_tags(vm)
+
+    @RequestParser([
+        {'name': 'action', 'choices': ('add', 'remove'), 'helptext': "A valid action must be specified"},
+        {'name': 'tag'},
+        {'name': 'protected'}
+    ])
+    @Authenticator
+    def post(self, vm, reqargs):
+        """
+        Set the tags of {vm}
+        ---
+        tags:
+          - vm
+        parameters:
+          - in: query
+            name: action
+            type: string
+            required: true
+            description: The action to perform with the tag
+            enum:
+              - add
+              - remove
+          - in: query
+            name: tag
+            type: string
+            required: true
+            description: The text value of the tag
+          - in: query
+            name: protected
+            type: boolean
+            required: false
+            default: false
+            description: Set the protected state of the tag
+        responses:
+          200:
+            description: OK
+            schema:
+              type: object
+              id: Message
+          400:
+            description: Bad request
+            schema:
+              type: object
+              id: Message
+          404:
+            description: VM not found
+            schema:
+              type: object
+              id: Message
+        """
+        return api_helper.update_vm_tag(
+            vm,
+            reqargs.get('action'),
+            reqargs.get('tag'),
+            reqargs.get('protected', False)
+        )
+
+
+api.add_resource(API_VM_Tags, '/vm/<vm>/tags')
+
+
 # /vm/<vm</state
 class API_VM_State(Resource):
     @Authenticator
@@ -1784,6 +2010,72 @@ class API_VM_Rename(Resource):
 api.add_resource(API_VM_Rename, '/vm/<vm>/rename')
 
 
+# /vm/<vm>/device
+class API_VM_Device(Resource):
+    @RequestParser([
+        {'name': 'xml', 'required': True, 'helptext': "A Libvirt XML device document must be specified"},
+    ])
+    @Authenticator
+    def post(self, vm, reqargs):
+        """
+        Hot-attach device XML to {vm}
+        ---
+        tags:
+          - vm
+        parameters:
+          - in: query
+            name: xml
+            type: string
+            required: true
+            description: The raw Libvirt XML definition of the device to attach
+        responses:
+          200:
+            description: OK
+            schema:
+              type: object
+              id: Message
+          400:
+            description: Bad request
+            schema:
+              type: object
+              id: Message
+        """
+        return api_helper.vm_attach_device(vm, reqargs.get('xml', None))
+
+    @RequestParser([
+        {'name': 'xml', 'required': True, 'helptext': "A Libvirt XML device document must be specified"},
+    ])
+    @Authenticator
+    def delete(self, vm, reqargs):
+        """
+        Hot-detach device XML to {vm}
+        ---
+        tags:
+          - vm
+        parameters:
+          - in: query
+            name: xml
+            type: string
+            required: true
+            description: The raw Libvirt XML definition of the device to detach
+        responses:
+          200:
+            description: OK
+            schema:
+              type: object
+              id: Message
+          400:
+            description: Bad request
+            schema:
+              type: object
+              id: Message
+        """
+        return api_helper.vm_detach_device(vm, reqargs.get('xml', None))
+
+
+api.add_resource(API_VM_Device, '/vm/<vm>/device')
+
+
 ##########################################################
 # Client API - Network
 ##########################################################
@@ -1817,6 +2109,9 @@ class API_Network_Root(Resource):
                   enum:
                     - managed
                     - bridged
+                mtu:
+                  type: integer
+                  description: The MTU of the network, if set; empty otherwise
                 domain:
                   type: string
                   description: The DNS domain of the network ("managed" networks only)
@@ -1877,6 +2172,7 @@ class API_Network_Root(Resource):
         {'name': 'vni', 'required': True},
         {'name': 'description', 'required': True},
         {'name': 'nettype', 'choices': ('managed', 'bridged'), 'helptext': 'A valid nettype must be specified', 'required': True},
+        {'name': 'mtu'},
         {'name': 'domain'},
         {'name': 'name_servers'},
         {'name': 'ip4_network'},
@@ -1913,6 +2209,10 @@ class API_Network_Root(Resource):
             enum:
               - managed
               - bridged
+          - in: query
+            name: mtu
+            type: integer
+            description: The MTU of the network; defaults to the underlying interface MTU if not set
           - in: query
             name: domain
             type: string
@@ -1969,6 +2269,7 @@ class API_Network_Root(Resource):
             reqargs.get('vni', None),
             reqargs.get('description', None),
             reqargs.get('nettype', None),
+            reqargs.get('mtu', ''),
             reqargs.get('domain', None),
             name_servers,
             reqargs.get('ip4_network', None),
@@ -2009,6 +2310,7 @@ class API_Network_Element(Resource):
     @RequestParser([
         {'name': 'description', 'required': True},
         {'name': 'nettype', 'choices': ('managed', 'bridged'), 'helptext': 'A valid nettype must be specified', 'required': True},
+        {'name': 'mtu'},
         {'name': 'domain'},
         {'name': 'name_servers'},
         {'name': 'ip4_network'},
@@ -2040,6 +2342,10 @@ class API_Network_Element(Resource):
             enum:
               - managed
               - bridged
+          - in: query
+            name: mtu
+            type: integer
+            description: The MTU of the network; defaults to the underlying interface MTU if not set
           - in: query
             name: domain
             type: string
@@ -2096,6 +2402,7 @@ class API_Network_Element(Resource):
             reqargs.get('vni', None),
             reqargs.get('description', None),
             reqargs.get('nettype', None),
+            reqargs.get('mtu', ''),
             reqargs.get('domain', None),
             name_servers,
             reqargs.get('ip4_network', None),
@@ -2109,6 +2416,7 @@ class API_Network_Element(Resource):
 
     @RequestParser([
         {'name': 'description'},
+        {'name': 'mtu'},
         {'name': 'domain'},
         {'name': 'name_servers'},
         {'name': 'ip4_network'},
@@ -2132,6 +2440,10 @@ class API_Network_Element(Resource):
             name: description
             type: string
             description: The description of the network
+          - in: query
+            name: mtu
+            type: integer
+            description: The MTU of the network
           - in: query
             name: domain
             type: string
@@ -2192,6 +2504,7 @@ class API_Network_Element(Resource):
         return api_helper.net_modify(
             vni,
             reqargs.get('description', None),
+            reqargs.get('mtu', None),
             reqargs.get('domain', None),
             name_servers,
             reqargs.get('ip4_network', None),
@@ -2719,6 +3032,301 @@ class API_Network_ACL_Element(Resource):
 api.add_resource(API_Network_ACL_Element, '/network/<vni>/acl/<description>')
 
 
+##########################################################
+# Client API - SR-IOV
+##########################################################
+
+# /sriov
+class API_SRIOV_Root(Resource):
+    @Authenticator
+    def get(self):
+        pass
+
+
+api.add_resource(API_SRIOV_Root, '/sriov')
+
+
+# /sriov/pf
+class API_SRIOV_PF_Root(Resource):
+    @RequestParser([
+        {'name': 'node', 'required': True, 'helptext': "A valid node must be specified."},
+    ])
+    @Authenticator
+    def get(self, reqargs):
+        """
+        Return a list of SR-IOV PFs on a given node
+        ---
+        tags:
+          - network / sriov
+        responses:
+          200:
+            description: OK
+            schema:
+              type: object
+              id: sriov_pf
+              properties:
+                phy:
+                  type: string
+                  description: The name of the SR-IOV PF device
+                mtu:
+                  type: string
+                  description: The MTU of the SR-IOV PF device
+                vfs:
+                  type: list
+                  items:
+                    type: string
+                    description: The PHY name of a VF of this PF
+        """
+        return api_helper.sriov_pf_list(reqargs.get('node'))
+
+
+api.add_resource(API_SRIOV_PF_Root, '/sriov/pf')
+
+
+# /sriov/pf/<node>
+class API_SRIOV_PF_Node(Resource):
+    @Authenticator
+    def get(self, node):
+        """
+        Return a list of SR-IOV PFs on node {node}
+        ---
+        tags:
+          - network / sriov
+        responses:
+          200:
+            description: OK
+            schema:
+              $ref: '#/definitions/sriov_pf'
+        """
+        return api_helper.sriov_pf_list(node)
+
+
+api.add_resource(API_SRIOV_PF_Node, '/sriov/pf/<node>')
+
+
+# /sriov/vf
+class API_SRIOV_VF_Root(Resource):
+    @RequestParser([
+        {'name': 'node', 'required': True, 'helptext': "A valid node must be specified."},
+        {'name': 'pf', 'required': False, 'helptext': "A PF parent may be specified."},
+    ])
+    @Authenticator
+    def get(self, reqargs):
+        """
+        Return a list of SR-IOV VFs on a given node, optionally limited to those in the specified PF
+        ---
+        tags:
+          - network / sriov
+        responses:
+          200:
+            description: OK
+            schema:
+              type: object
+              id: sriov_vf
+              properties:
+                phy:
+                  type: string
+                  description: The name of the SR-IOV VF device
+                pf:
+                  type: string
+                  description: The name of the SR-IOV PF parent of this VF device
+                mtu:
+                  type: integer
+                  description: The current MTU of the VF device
+                mac:
+                  type: string
+                  description: The current MAC address of the VF device
+                config:
+                  type: object
+                  id: sriov_vf_config
+                  properties:
+                    vlan_id:
+                      type: string
+                      description: The tagged vLAN ID of the SR-IOV VF device
+                    vlan_qos:
+                      type: string
+                      description: The QOS group of the tagged vLAN
+                    tx_rate_min:
+                      type: string
+                      description: The minimum TX rate of the SR-IOV VF device
+                    tx_rate_max:
+                      type: string
+                      description: The maximum TX rate of the SR-IOV VF device
+                    spoof_check:
+                      type: boolean
+                      description: Whether device spoof checking is enabled or disabled
+                    link_state:
+                      type: string
+                      description: The current SR-IOV VF link state (either enabled, disabled, or auto)
+                    trust:
+                      type: boolean
+                      description: Whether guest device trust is enabled or disabled
+                    query_rss:
+                      type: boolean
+                      description: Whether VF RSS querying is enabled or disabled
+                usage:
+                  type: object
+                  id: sriov_vf_usage
+                  properties:
+                    used:
+                      type: boolean
+                      description: Whether the SR-IOV VF is currently used by a VM or not
+                    domain:
+                      type: boolean
+                      description: The UUID of the domain the SR-IOV VF is currently used by
+        """
+        return api_helper.sriov_vf_list(reqargs.get('node'), reqargs.get('pf', None))
+
+
+api.add_resource(API_SRIOV_VF_Root, '/sriov/vf')
+
+
+# /sriov/vf/<node>
+class API_SRIOV_VF_Node(Resource):
+    @RequestParser([
+        {'name': 'pf', 'required': False, 'helptext': "A PF parent may be specified."},
+    ])
+    @Authenticator
+    def get(self, node, reqargs):
+        """
+        Return a list of SR-IOV VFs on node {node}, optionally limited to those in the specified PF
+        ---
+        tags:
+          - network / sriov
+        responses:
+          200:
+            description: OK
+            schema:
+              $ref: '#/definitions/sriov_vf'
+        """
+        return api_helper.sriov_vf_list(node, reqargs.get('pf', None))
+
+
+api.add_resource(API_SRIOV_VF_Node, '/sriov/vf/<node>')
+
+
+# /sriov/vf/<node>/<vf>
+class API_SRIOV_VF_Element(Resource):
+    @Authenticator
+    def get(self, node, vf):
+        """
+        Return information about {vf} on {node}
+        ---
+        tags:
+          - network / sriov
+        responses:
+          200:
+            description: OK
+            schema:
+              $ref: '#/definitions/sriov_vf'
+          404:
+            description: Not found
+            schema:
+              type: object
+              id: Message
+        """
+        vf_list = list()
+        full_vf_list, _ = api_helper.sriov_vf_list(node)
+        for vf_element in full_vf_list:
+            if vf_element['phy'] == vf:
+                vf_list.append(vf_element)
+
+        if len(vf_list) == 1:
+            return vf_list, 200
+        else:
+            return {'message': "No VF '{}' found on node '{}'".format(vf, node)}, 404
+
+    @RequestParser([
+        {'name': 'vlan_id'},
+        {'name': 'vlan_qos'},
+        {'name': 'tx_rate_min'},
+        {'name': 'tx_rate_max'},
+        {'name': 'link_state', 'choices': ('auto', 'enable', 'disable'), 'helptext': "A valid state must be specified"},
+        {'name': 'spoof_check'},
+        {'name': 'trust'},
+        {'name': 'query_rss'},
+    ])
+    @Authenticator
+    def put(self, node, vf, reqargs):
+        """
+        Set the configuration of {vf} on {node}
+        ---
+        tags:
+          - network / sriov
+        parameters:
+          - in: query
+            name: vlan_id
+            type: integer
+            required: false
+            description: The vLAN ID for vLAN tagging (0 is disabled)
+          - in: query
+            name: vlan_qos
+            type: integer
+            required: false
+            description: The vLAN QOS priority (0 is disabled)
+          - in: query
+            name: tx_rate_min
+            type: integer
+            required: false
+            description: The minimum TX rate (0 is disabled)
+          - in: query
+            name: tx_rate_max
+            type: integer
+            required: false
+            description: The maximum TX rate (0 is disabled)
+          - in: query
+            name: link_state
+            type: string
+            required: false
+            description: The administrative link state
+            enum:
+              - auto
+              - enable
+              - disable
+          - in: query
+            name: spoof_check
+            type: boolean
+            required: false
+            description: Enable or disable spoof checking
+          - in: query
+            name: trust
+            type: boolean
+            required: false
+            description: Enable or disable VF user trust
+          - in: query
+            name: query_rss
+            type: boolean
+            required: false
+            description: Enable or disable query RSS support
+        responses:
+          200:
+            description: OK
+            schema:
+              type: object
+              id: Message
+          400:
+            description: Bad request
+            schema:
+              type: object
+              id: Message
+        """
+        return api_helper.update_sriov_vf_config(
+            node,
+            vf,
+            reqargs.get('vlan_id', None),
+            reqargs.get('vlan_qos', None),
+            reqargs.get('tx_rate_min', None),
+            reqargs.get('tx_rate_max', None),
+            reqargs.get('link_state', None),
+            reqargs.get('spoof_check', None),
+            reqargs.get('trust', None),
+            reqargs.get('query_rss', None),
+        )
+
+
+api.add_resource(API_SRIOV_VF_Element, '/sriov/vf/<node>/<vf>')
+
+
 ##########################################################
 # Client API - Storage
 ##########################################################
@@ -2840,8 +3448,12 @@ class API_Storage_Ceph_Benchmark(Resource):
                 job:
                   type: string
                   description: The job name (an ISO date) of the test result
+                test_format:
+                  type: integer
+                  description: The PVC benchmark format of the results
                 benchmark_result:
                   type: object
+                  description: A format 0 test result
                   properties:
                     test_name:
                       type: object
@@ -3019,6 +3631,52 @@ class API_Storage_Ceph_Option(Resource):
 api.add_resource(API_Storage_Ceph_Option, '/storage/ceph/option')
 
 
+# /storage/ceph/osddb
+class API_Storage_Ceph_OSDDB_Root(Resource):
+    @RequestParser([
+        {'name': 'node', 'required': True, 'helptext': "A valid node must be specified."},
+        {'name': 'device', 'required': True, 'helptext': "A valid device must be specified."},
+    ])
+    @Authenticator
+    def post(self, reqargs):
+        """
+        Add a Ceph OSD database volume group to the cluster
+        Note: This task may take up to 30s to complete and return
+        ---
+        tags:
+          - storage / ceph
+        parameters:
+          - in: query
+            name: node
+            type: string
+            required: true
+            description: The PVC node to create the OSD DB volume group on
+          - in: query
+            name: device
+            type: string
+            required: true
+            description: The block device (e.g. "/dev/sdb", "/dev/disk/by-path/...", etc.) to create the OSD DB volume group on
+        responses:
+          200:
+            description: OK
+            schema:
+              type: object
+              id: Message
+          400:
+            description: Bad request
+            schema:
+              type: object
+              id: Message
+        """
+        return api_helper.ceph_osd_db_vg_add(
+            reqargs.get('node', None),
+            reqargs.get('device', None)
+        )
+
+
+api.add_resource(API_Storage_Ceph_OSDDB_Root, '/storage/ceph/osddb')
+
+
 # /storage/ceph/osd
 class API_Storage_Ceph_OSD_Root(Resource):
     @RequestParser([
@@ -3039,6 +3697,12 @@ class API_Storage_Ceph_OSD_Root(Resource):
                 id:
                   type: string (containing integer)
                   description: The Ceph ID of the OSD
+                device:
+                  type: string
+                  description: The OSD data block device
+                db_device:
+                  type: string
+                  description: The OSD database/WAL block device (logical volume); empty if not applicable
                 stats:
                   type: object
                   properties:
@@ -3118,6 +3782,8 @@ class API_Storage_Ceph_OSD_Root(Resource):
         {'name': 'node', 'required': True, 'helptext': "A valid node must be specified."},
         {'name': 'device', 'required': True, 'helptext': "A valid device must be specified."},
         {'name': 'weight', 'required': True, 'helptext': "An OSD weight must be specified."},
+        {'name': 'ext_db', 'required': False, 'helptext': "Whether to use an external OSD DB LV device."},
+        {'name': 'ext_db_ratio', 'required': False, 'helptext': "Decimal size ratio of the external OSD DB LV device."},
     ])
     @Authenticator
     def post(self, reqargs):
@@ -3143,6 +3809,16 @@ class API_Storage_Ceph_OSD_Root(Resource):
             type: number
             required: true
             description: The Ceph CRUSH weight for the OSD
+          - in: query
+            name: ext_db
+            type: boolean
+            required: false
+            description: Whether to use an external OSD DB LV device
+          - in: query
+            name: ext_db_ratio
+            type: float
+            required: false
+            description: Decimal ratio of total OSD size for the external OSD DB LV device, default 0.05 (5%)
         responses:
           200:
             description: OK
@@ -3158,7 +3834,9 @@ class API_Storage_Ceph_OSD_Root(Resource):
         return api_helper.ceph_osd_add(
             reqargs.get('node', None),
             reqargs.get('device', None),
-            reqargs.get('weight', None)
+            reqargs.get('weight', None),
+            reqargs.get('ext_db', False),
+            float(reqargs.get('ext_db_ratio', 0.05)),
         )
 
 

api-daemon/pvcapid/helper.py

@@ -97,6 +97,7 @@ def restore_cluster(zkhandler, cluster_data_raw):
 #
 # Cluster functions
 #
+@pvc_common.Profiler(config)
 @ZKConnection(config)
 def cluster_status(zkhandler):
     """
@@ -128,6 +129,7 @@ def cluster_maintenance(zkhandler, maint_state='false'):
 #
 # Node functions
 #
+@pvc_common.Profiler(config)
 @ZKConnection(config)
 def node_list(zkhandler, limit=None, daemon_state=None, coordinator_state=None, domain_state=None, is_fuzzy=True):
     """
@@ -305,6 +307,34 @@ def node_ready(zkhandler, node, wait):
     return output, retcode
 
 
+@ZKConnection(config)
+def node_log(zkhandler, node, lines=None):
+    """
+    Return the current logs for Node.
+    """
+    # Default to 10 lines of log if not set
+    try:
+        lines = int(lines)
+    except TypeError:
+        lines = 10
+
+    retflag, retdata = pvc_node.get_node_log(zkhandler, node, lines)
+
+    if retflag:
+        retcode = 200
+        retdata = {
+            'name': node,
+            'data': retdata
+        }
+    else:
+        retcode = 400
+        retdata = {
+            'message': retdata
+        }
+
+    return retdata, retcode
+
+
 #
 # VM functions
 #
@@ -318,12 +348,13 @@ def vm_is_migrated(zkhandler, vm):
     return retdata
 
 
+@pvc_common.Profiler(config)
 @ZKConnection(config)
 def vm_state(zkhandler, vm):
     """
     Return the state of virtual machine VM.
     """
-    retflag, retdata = pvc_vm.get_list(zkhandler, None, None, vm, is_fuzzy=False)
+    retflag, retdata = pvc_vm.get_list(zkhandler, None, None, None, vm, is_fuzzy=False, negate=False)
 
     if retflag:
         if retdata:
@@ -346,12 +377,13 @@ def vm_state(zkhandler, vm):
     return retdata, retcode
 
 
+@pvc_common.Profiler(config)
 @ZKConnection(config)
 def vm_node(zkhandler, vm):
     """
     Return the current node of virtual machine VM.
     """
-    retflag, retdata = pvc_vm.get_list(zkhandler, None, None, vm, is_fuzzy=False)
+    retflag, retdata = pvc_vm.get_list(zkhandler, None, None, None, vm, is_fuzzy=False, negate=False)
 
     if retflag:
         if retdata:
@@ -403,12 +435,13 @@ def vm_console(zkhandler, vm, lines=None):
     return retdata, retcode
 
 
+@pvc_common.Profiler(config)
 @ZKConnection(config)
-def vm_list(zkhandler, node=None, state=None, limit=None, is_fuzzy=True):
+def vm_list(zkhandler, node=None, state=None, tag=None, limit=None, is_fuzzy=True, negate=False):
     """
     Return a list of VMs with limit LIMIT.
     """
-    retflag, retdata = pvc_vm.get_list(zkhandler, node, state, limit, is_fuzzy)
+    retflag, retdata = pvc_vm.get_list(zkhandler, node, state, tag, limit, is_fuzzy, negate)
 
     if retflag:
         if retdata:
@@ -428,7 +461,7 @@ def vm_list(zkhandler, node=None, state=None, limit=None, is_fuzzy=True):
 
 
 @ZKConnection(config)
-def vm_define(zkhandler, xml, node, limit, selector, autostart, migration_method):
+def vm_define(zkhandler, xml, node, limit, selector, autostart, migration_method, user_tags=[], protected_tags=[]):
     """
     Define a VM from Libvirt XML in the PVC cluster.
     """
@@ -439,7 +472,13 @@ def vm_define(zkhandler, xml, node, limit, selector, autostart, migration_method
     except Exception as e:
         return {'message': 'XML is malformed or incorrect: {}'.format(e)}, 400
 
-    retflag, retdata = pvc_vm.define_vm(zkhandler, new_cfg, node, limit, selector, autostart, migration_method, profile=None)
+    tags = list()
+    for tag in user_tags:
+        tags.append({'name': tag, 'type': 'user', 'protected': False})
+    for tag in protected_tags:
+        tags.append({'name': tag, 'type': 'user', 'protected': True})
+
+    retflag, retdata = pvc_vm.define_vm(zkhandler, new_cfg, node, limit, selector, autostart, migration_method, profile=None, tags=tags)
 
     if retflag:
         retcode = 200
@@ -452,33 +491,78 @@ def vm_define(zkhandler, xml, node, limit, selector, autostart, migration_method
     return output, retcode
 
 
+@ZKConnection(config)
+def vm_attach_device(zkhandler, vm, device_spec_xml):
+    """
+    Hot-attach a device (via XML spec) to a VM.
+    """
+    try:
+        _ = etree.fromstring(device_spec_xml)
+    except Exception as e:
+        return {'message': 'XML is malformed or incorrect: {}'.format(e)}, 400
+
+    retflag, retdata = pvc_vm.attach_vm_device(zkhandler, vm, device_spec_xml)
+
+    if retflag:
+        retcode = 200
+        output = {
+            'message': retdata.replace('\"', '\'')
+        }
+    else:
+        retcode = 400
+        output = {
+            'message': 'WARNING: Failed to perform hot attach; device will be added on next VM start/restart.'
+        }
+
+    return output, retcode
+
+
+@ZKConnection(config)
+def vm_detach_device(zkhandler, vm, device_spec_xml):
+    """
+    Hot-detach a device (via XML spec) from a VM.
+    """
+    try:
+        _ = etree.fromstring(device_spec_xml)
+    except Exception as e:
+        return {'message': 'XML is malformed or incorrect: {}'.format(e)}, 400
+
+    retflag, retdata = pvc_vm.detach_vm_device(zkhandler, vm, device_spec_xml)
+
+    if retflag:
+        retcode = 200
+        output = {
+            'message': retdata.replace('\"', '\'')
+        }
+    else:
+        retcode = 400
+        output = {
+            'message': 'WARNING: Failed to perform hot detach; device will be removed on next VM start/restart.'
+        }
+
+    return output, retcode
+
+
+@pvc_common.Profiler(config)
 @ZKConnection(config)
 def get_vm_meta(zkhandler, vm):
     """
     Get metadata of a VM.
     """
-    retflag, retdata = pvc_vm.get_list(zkhandler, None, None, vm, is_fuzzy=False)
+    dom_uuid = pvc_vm.getDomainUUID(zkhandler, vm)
+    if not dom_uuid:
+        return {"message": "VM not found."}, 404
 
-    if retflag:
-        if retdata:
-            retcode = 200
-            retdata = {
-                'name': vm,
-                'node_limit': retdata['node_limit'],
-                'node_selector': retdata['node_selector'],
-                'node_autostart': retdata['node_autostart'],
-                'migration_method': retdata['migration_method']
-            }
-        else:
-            retcode = 404
-            retdata = {
-                'message': 'VM not found.'
-            }
-    else:
-        retcode = 400
-        retdata = {
-            'message': retdata
-        }
+    domain_node_limit, domain_node_selector, domain_node_autostart, domain_migrate_method = pvc_common.getDomainMetadata(zkhandler, dom_uuid)
+
+    retcode = 200
+    retdata = {
+        'name': vm,
+        'node_limit': domain_node_limit,
+        'node_selector': domain_node_selector,
+        'node_autostart': domain_node_autostart,
+        'migration_method': domain_migrate_method
+    }
 
     return retdata, retcode
 
@@ -488,11 +572,16 @@ def update_vm_meta(zkhandler, vm, limit, selector, autostart, provisioner_profil
     """
     Update metadata of a VM.
     """
+    dom_uuid = pvc_vm.getDomainUUID(zkhandler, vm)
+    if not dom_uuid:
+        return {"message": "VM not found."}, 404
+
     if autostart is not None:
         try:
             autostart = bool(strtobool(autostart))
         except Exception:
             autostart = False
+
     retflag, retdata = pvc_vm.modify_vm_metadata(zkhandler, vm, limit, selector, autostart, provisioner_profile, migration_method)
 
     if retflag:
@@ -506,6 +595,51 @@ def update_vm_meta(zkhandler, vm, limit, selector, autostart, provisioner_profil
     return output, retcode
 
 
+@ZKConnection(config)
+def get_vm_tags(zkhandler, vm):
+    """
+    Get the tags of a VM.
+    """
+    dom_uuid = pvc_vm.getDomainUUID(zkhandler, vm)
+    if not dom_uuid:
+        return {"message": "VM not found."}, 404
+
+    tags = pvc_common.getDomainTags(zkhandler, dom_uuid)
+
+    retcode = 200
+    retdata = {
+        'name': vm,
+        'tags': tags
+    }
+
+    return retdata, retcode
+
+
+@ZKConnection(config)
+def update_vm_tag(zkhandler, vm, action, tag, protected=False):
+    """
+    Update a tag of a VM.
+    """
+    if action not in ['add', 'remove']:
+        return {"message": "Tag action must be one of 'add', 'remove'."}, 400
+
+    dom_uuid = pvc_vm.getDomainUUID(zkhandler, vm)
+    if not dom_uuid:
+        return {"message": "VM not found."}, 404
+
+    retflag, retdata = pvc_vm.modify_vm_tag(zkhandler, vm, action, tag, protected=protected)
+
+    if retflag:
+        retcode = 200
+    else:
+        retcode = 400
+
+    output = {
+        'message': retdata.replace('\"', '\'')
+    }
+    return output, retcode
+
+
 @ZKConnection(config)
 def vm_modify(zkhandler, name, restart, xml):
     """
@@ -746,7 +880,7 @@ def vm_flush_locks(zkhandler, vm):
     """
     Flush locks of a (stopped) VM.
     """
-    retflag, retdata = pvc_vm.get_list(zkhandler, None, None, vm, is_fuzzy=False)
+    retflag, retdata = pvc_vm.get_list(zkhandler, None, None, None, vm, is_fuzzy=False, negate=False)
 
     if retdata[0].get('state') not in ['stop', 'disable']:
         return {"message": "VM must be stopped to flush locks"}, 400
@@ -767,6 +901,7 @@ def vm_flush_locks(zkhandler, vm):
 #
 # Network functions
 #
+@pvc_common.Profiler(config)
 @ZKConnection(config)
 def net_list(zkhandler, limit=None, is_fuzzy=True):
     """
@@ -792,7 +927,7 @@ def net_list(zkhandler, limit=None, is_fuzzy=True):
 
 
 @ZKConnection(config)
-def net_add(zkhandler, vni, description, nettype, domain, name_servers,
+def net_add(zkhandler, vni, description, nettype, mtu, domain, name_servers,
             ip4_network, ip4_gateway, ip6_network, ip6_gateway,
             dhcp4_flag, dhcp4_start, dhcp4_end):
     """
@@ -800,7 +935,7 @@ def net_add(zkhandler, vni, description, nettype, domain, name_servers,
     """
     if dhcp4_flag:
         dhcp4_flag = bool(strtobool(dhcp4_flag))
-    retflag, retdata = pvc_network.add_network(zkhandler, vni, description, nettype, domain, name_servers,
+    retflag, retdata = pvc_network.add_network(zkhandler, vni, description, nettype, mtu, domain, name_servers,
                                                ip4_network, ip4_gateway, ip6_network, ip6_gateway,
                                                dhcp4_flag, dhcp4_start, dhcp4_end)
 
@@ -816,7 +951,7 @@ def net_add(zkhandler, vni, description, nettype, domain, name_servers,
 
 
 @ZKConnection(config)
-def net_modify(zkhandler, vni, description, domain, name_servers,
+def net_modify(zkhandler, vni, description, mtu, domain, name_servers,
                ip4_network, ip4_gateway,
                ip6_network, ip6_gateway,
                dhcp4_flag, dhcp4_start, dhcp4_end):
@@ -825,7 +960,7 @@ def net_modify(zkhandler, vni, description, domain, name_servers,
     """
     if dhcp4_flag is not None:
         dhcp4_flag = bool(strtobool(dhcp4_flag))
-    retflag, retdata = pvc_network.modify_network(zkhandler, vni, description, domain, name_servers,
+    retflag, retdata = pvc_network.modify_network(zkhandler, vni, description, mtu, domain, name_servers,
                                                   ip4_network, ip4_gateway, ip6_network, ip6_gateway,
                                                   dhcp4_flag, dhcp4_start, dhcp4_end)
 
@@ -858,6 +993,7 @@ def net_remove(zkhandler, network):
     return output, retcode
 
 
+@pvc_common.Profiler(config)
 @ZKConnection(config)
 def net_dhcp_list(zkhandler, network, limit=None, static=False):
     """
@@ -918,6 +1054,7 @@ def net_dhcp_remove(zkhandler, network, macaddress):
     return output, retcode
 
 
+@pvc_common.Profiler(config)
 @ZKConnection(config)
 def net_acl_list(zkhandler, network, limit=None, direction=None, is_fuzzy=True):
     """
@@ -978,6 +1115,82 @@ def net_acl_remove(zkhandler, network, description):
     return output, retcode
 
 
+#
+# SR-IOV functions
+#
+@pvc_common.Profiler(config)
+@ZKConnection(config)
+def sriov_pf_list(zkhandler, node):
+    """
+    List all PFs on a given node.
+    """
+    retflag, retdata = pvc_network.get_list_sriov_pf(zkhandler, node)
+
+    if retflag:
+        if retdata:
+            retcode = 200
+        else:
+            retcode = 404
+            retdata = {
+                'message': 'PF not found.'
+            }
+    else:
+        retcode = 400
+        retdata = {
+            'message': retdata
+        }
+
+    return retdata, retcode
+
+
+@pvc_common.Profiler(config)
+@ZKConnection(config)
+def sriov_vf_list(zkhandler, node, pf=None):
+    """
+    List all VFs on a given node, optionally limited to PF.
+    """
+    retflag, retdata = pvc_network.get_list_sriov_vf(zkhandler, node, pf)
+
+    if retflag:
+        retcode = 200
+    else:
+        retcode = 400
+
+    if retflag:
+        if retdata:
+            retcode = 200
+        else:
+            retcode = 404
+            retdata = {
+                'message': 'VF not found.'
+            }
+    else:
+        retcode = 400
+        retdata = {
+            'message': retdata
+        }
+
+    return retdata, retcode
+
+
+@ZKConnection(config)
+def update_sriov_vf_config(zkhandler, node, vf, vlan_id, vlan_qos, tx_rate_min, tx_rate_max, link_state, spoof_check, trust, query_rss):
+    """
+    Update configuration of a VF on NODE.
+    """
+    retflag, retdata = pvc_network.set_sriov_vf_config(zkhandler, node, vf, vlan_id, vlan_qos, tx_rate_min, tx_rate_max, link_state, spoof_check, trust, query_rss)
+
+    if retflag:
+        retcode = 200
+    else:
+        retcode = 400
+
+    output = {
+        'message': retdata.replace('\"', '\'')
+    }
+    return output, retcode
+
+
 #
 # Ceph functions
 #
@@ -1011,6 +1224,7 @@ def ceph_util(zkhandler):
     return retdata, retcode
 
 
+@pvc_common.Profiler(config)
 @ZKConnection(config)
 def ceph_osd_list(zkhandler, limit=None):
     """
@@ -1060,11 +1274,29 @@ def ceph_osd_state(zkhandler, osd):
 
 
 @ZKConnection(config)
-def ceph_osd_add(zkhandler, node, device, weight):
+def ceph_osd_db_vg_add(zkhandler, node, device):
+    """
+    Add a Ceph OSD database VG to the PVC Ceph storage cluster.
+    """
+    retflag, retdata = pvc_ceph.add_osd_db_vg(zkhandler, node, device)
+
+    if retflag:
+        retcode = 200
+    else:
+        retcode = 400
+
+    output = {
+        'message': retdata.replace('\"', '\'')
+    }
+    return output, retcode
+
+
+@ZKConnection(config)
+def ceph_osd_add(zkhandler, node, device, weight, ext_db_flag=False, ext_db_ratio=0.05):
     """
     Add a Ceph OSD to the PVC Ceph storage cluster.
     """
-    retflag, retdata = pvc_ceph.add_osd(zkhandler, node, device, weight)
+    retflag, retdata = pvc_ceph.add_osd(zkhandler, node, device, weight, ext_db_flag, ext_db_ratio)
 
     if retflag:
         retcode = 200
@@ -1167,6 +1399,7 @@ def ceph_osd_unset(zkhandler, option):
     return output, retcode
 
 
+@pvc_common.Profiler(config)
 @ZKConnection(config)
 def ceph_pool_list(zkhandler, limit=None, is_fuzzy=True):
     """
@@ -1227,6 +1460,7 @@ def ceph_pool_remove(zkhandler, name):
     return output, retcode
 
 
+@pvc_common.Profiler(config)
 @ZKConnection(config)
 def ceph_volume_list(zkhandler, pool=None, limit=None, is_fuzzy=True):
     """
@@ -1481,6 +1715,7 @@ def ceph_volume_upload(zkhandler, pool, volume, img_type):
         return output, retcode
 
 
+@pvc_common.Profiler(config)
 @ZKConnection(config)
 def ceph_volume_snapshot_list(zkhandler, pool=None, volume=None, limit=None, is_fuzzy=True):
     """

api-daemon/pvcapid/libvirt_schema.py

@@ -41,6 +41,7 @@ libvirt_header = """<domain type='kvm'>
     <bootmenu enable='yes'/>
     <boot dev='cdrom'/>
     <boot dev='hd'/>
+    <bios useserial='yes' rebootTimeout='5'/>
   </os>
   <features>
     <acpi/>

api-daemon/pvcapid/models.py

@@ -231,11 +231,13 @@ class DBStorageBenchmarks(db.Model):
 
     id = db.Column(db.Integer, primary_key=True)
     job = db.Column(db.Text, nullable=False)
+    test_format = db.Column(db.Integer, nullable=False, default=0, server_default='0')
     result = db.Column(db.Text, nullable=False)
 
-    def __init__(self, job, result):
+    def __init__(self, job, result, test_format):
         self.job = job
         self.result = result
+        self.test_format = test_format
 
     def __repr__(self):
         return '<id {}>'.format(self.id)

api-daemon/pvcapid/ova.py

@@ -414,6 +414,7 @@ class OVFParser(object):
         "5": "ide-controller",
         "6": "scsi-controller",
         "10": "ethernet-adapter",
+        "14": "floppy",
         "15": "cdrom",
         "17": "disk",
         "20": "other-storage-device",

api-daemon/pvcapid/provisioner.py

@@ -1323,6 +1323,30 @@ def create_vm(self, vm_name, vm_profile, define_vm=True, start_vm=True, script_r
         vm_architecture=system_architecture
     )
 
+    # Add disk devices
+    monitor_list = list()
+    coordinator_names = config['storage_hosts']
+    for coordinator in coordinator_names:
+        monitor_list.append("{}.{}".format(coordinator, config['storage_domain']))
+
+    ceph_storage_secret = config['ceph_storage_secret_uuid']
+
+    for volume in vm_data['volumes']:
+        vm_schema += libvirt_schema.devices_disk_header.format(
+            ceph_storage_secret=ceph_storage_secret,
+            disk_pool=volume['pool'],
+            vm_name=vm_name,
+            disk_id=volume['disk_id']
+        )
+        for monitor in monitor_list:
+            vm_schema += libvirt_schema.devices_disk_coordinator.format(
+                coordinator_name=monitor,
+                coordinator_ceph_mon_port=config['ceph_monitor_port']
+            )
+        vm_schema += libvirt_schema.devices_disk_footer
+
+    vm_schema += libvirt_schema.devices_vhostmd
+
     # Add network devices
     network_id = 0
     for network in vm_data['networks']:
@@ -1364,30 +1388,6 @@ def create_vm(self, vm_name, vm_profile, define_vm=True, start_vm=True, script_r
 
         network_id += 1
 
-    # Add disk devices
-    monitor_list = list()
-    coordinator_names = config['storage_hosts']
-    for coordinator in coordinator_names:
-        monitor_list.append("{}.{}".format(coordinator, config['storage_domain']))
-
-    ceph_storage_secret = config['ceph_storage_secret_uuid']
-
-    for volume in vm_data['volumes']:
-        vm_schema += libvirt_schema.devices_disk_header.format(
-            ceph_storage_secret=ceph_storage_secret,
-            disk_pool=volume['pool'],
-            vm_name=vm_name,
-            disk_id=volume['disk_id']
-        )
-        for monitor in monitor_list:
-            vm_schema += libvirt_schema.devices_disk_coordinator.format(
-                coordinator_name=monitor,
-                coordinator_ceph_mon_port=config['ceph_monitor_port']
-            )
-        vm_schema += libvirt_schema.devices_disk_footer
-
-    vm_schema += libvirt_schema.devices_vhostmd
-
     # Add default devices
     vm_schema += libvirt_schema.devices_default
 
@@ -1510,15 +1510,17 @@ def create_vm(self, vm_name, vm_profile, define_vm=True, start_vm=True, script_r
                 if volume.get('filesystem') is None:
                     continue
 
-                print("Creating {} filesystem on {}:\n{}".format(volume['filesystem'], dst_volume, stdout))
-
                 filesystem_args_list = list()
                 for arg in volume['filesystem_args'].split():
-                    arg_entry, arg_data = arg.split('=')
+                    arg_entry, *arg_data = arg.split('=')
+                    arg_data = '='.join(arg_data)
                     filesystem_args_list.append(arg_entry)
                     filesystem_args_list.append(arg_data)
                 filesystem_args = ' '.join(filesystem_args_list)
 
+                print("Creating {} filesystem on {}".format(volume['filesystem'], dst_volume))
+                print("Args: {}".format(filesystem_args))
+
                 # Map the RBD device
                 retcode, retmsg = pvc_ceph.map_volume(zkhandler, volume['pool'], dst_volume_name)
                 if not retcode:
@@ -1534,6 +1536,8 @@ def create_vm(self, vm_name, vm_profile, define_vm=True, start_vm=True, script_r
                     if retcode:
                         raise ProvisioningError('Failed to create {} filesystem on "{}": {}'.format(volume['filesystem'], dst_volume, stderr))
 
+                print(stdout)
+
         if is_script_install:
             # Create temporary directory
             retcode, stdout, stderr = pvc_common.run_os_command("mktemp -d")

build-and-deploy.sh

@@ -12,6 +12,9 @@ else
     SUDO="sudo"
 fi
 
+echo -n "> Linting code for errors... "
+./lint || exit
+
 HOSTS=( ${@} )
 echo "> Deploying to host(s): ${HOSTS[@]}"
 
@@ -41,7 +44,7 @@ for HOST in ${HOSTS[@]}; do
     ssh $HOST $SUDO systemctl restart pvcapid-worker &>/dev/null
     ssh $HOST $SUDO systemctl restart pvcnoded &>/dev/null
     echo "done."
-    echo -n "Waiting 15s for host to stabilize... "
-    sleep 15
+    echo -n "Waiting 30s for host to stabilize... "
+    sleep 30
     echo "done."
 done

build-deb.sh

@@ -2,6 +2,7 @@
 ver="$( head -1 debian/changelog | awk -F'[()-]' '{ print $2 }' )"
 git pull
 rm ../pvc_*
+find . -name "__pycache__" -exec rm -r {} \;
 dh_make -p pvc_${ver} --createorig --single --yes
 dpkg-buildpackage -us -uc
 dh_clean

build-unstable-deb.sh

@@ -10,9 +10,11 @@ new_ver="${base_ver}~git-$(git rev-parse --short HEAD)"
 echo ${new_ver} >&3
 # Back up the existing changelog and Daemon.py files
 tmpdir=$( mktemp -d )
-cp -a debian/changelog node-daemon/pvcnoded/Daemon.py ${tmpdir}/
+cp -a debian/changelog client-cli/setup.py ${tmpdir}/
+cp -a node-daemon/pvcnoded/Daemon.py ${tmpdir}/node-Daemon.py
+cp -a api-daemon/pvcapid/Daemon.py ${tmpdir}/api-Daemon.py
 # Replace the "base" version with the git revision version
-sed -i "s/version = '${base_ver}'/version = '${new_ver}'/" node-daemon/pvcnoded/Daemon.py
+sed -i "s/version = '${base_ver}'/version = '${new_ver}'/" node-daemon/pvcnoded/Daemon.py api-daemon/pvcapid/Daemon.py client-cli/setup.py
 sed -i "s/${base_ver}-0/${new_ver}/" debian/changelog 
 cat <<EOF > debian/changelog
 pvc (${new_ver}) unstable; urgency=medium
@@ -21,13 +23,17 @@ pvc (${new_ver}) unstable; urgency=medium
 
  -- Joshua Boniface <joshua@boniface.me>  $( date -R )
 EOF
+find . -name "__pycache__" -exec rm -r {} \;
 # Build source tarball
 dh_make -p pvc_${new_ver} --createorig --single --yes
 # Build packages
 dpkg-buildpackage -us -uc
 # Restore original changelog and Daemon.py files
 cp -a ${tmpdir}/changelog debian/changelog
-cp -a ${tmpdir}/Daemon.py node-daemon/pvcnoded/Daemon.py
+cp -a ${tmpdir}/setup.py client-cli/setup.py
+cp -a ${tmpdir}/node-Daemon.py node-daemon/pvcnoded/Daemon.py
+cp -a ${tmpdir}/api-Daemon.py api-daemon/pvcapid/Daemon.py
+
 # Clean up
 rm -r ${tmpdir}
 dh_clean

bump-version

@@ -7,7 +7,7 @@ if [[ -z ${new_version} ]]; then
     exit 1
 fi
 
-current_version="$( grep '^version = ' node-daemon/pvcnoded/Daemon.py | awk -F "'" '{ print $2 }' )"
+current_version="$( cat .version )"
 echo "${current_version} -> ${new_version}"
 
 changelog_file=$( mktemp )
@@ -15,29 +15,26 @@ echo "# Write the changelog below; comments will be ignored" >> ${changelog_file
 $EDITOR ${changelog_file}
 
 changelog="$( cat ${changelog_file} | grep -v '^#' | sed 's/^*/  */' )"
+rm ${changelog_file}
 
 sed -i "s,version = '${current_version}',version = '${new_version}'," node-daemon/pvcnoded/Daemon.py
 sed -i "s,version = '${current_version}',version = '${new_version}'," api-daemon/pvcapid/Daemon.py
+sed -i "s,version='${current_version}',version='${new_version}'," client-cli/setup.py
+echo ${new_version} > .version
 
-readme_tmpdir=$( mktemp -d )
-cp README.md ${readme_tmpdir}/
-cp docs/index.md ${readme_tmpdir}/
-pushd ${readme_tmpdir} &>/dev/null
+changelog_tmpdir=$( mktemp -d )
+cp CHANGELOG.md ${changelog_tmpdir}/
+pushd ${changelog_tmpdir} &>/dev/null
 
-echo -e "\n#### v${new_version}\n\n${changelog}" >> middle
+echo -e "\n###### [v${new_version}](https://github.com/parallelvirtualcluster/pvc/releases/tag/v${new_version})\n\n${changelog}" >> middle
 
-csplit README.md "/## Changelog/1" &>/dev/null
-cat xx00 middle xx01 > README.md
-rm xx00 xx01
-
-csplit index.md "/## Changelog/1" &>/dev/null
-cat xx00 middle xx01 > index.md
+csplit CHANGELOG.md "/## PVC Changelog/1" &>/dev/null
+cat xx00 middle xx01 > CHANGELOG.md
 rm xx00 xx01
 
 popd &>/dev/null
-mv ${readme_tmpdir}/README.md README.md
-mv ${readme_tmpdir}/index.md docs/index.md
-rm -r ${readme_tmpdir}
+mv ${changelog_tmpdir}/CHANGELOG.md CHANGELOG.md
+rm -r ${changelog_tmpdir}
 
 deb_changelog_orig="$( cat debian/changelog )"
 deb_changelog_new="pvc (${new_version}-0) unstable; urgency=high\n\n${changelog}\n\n -- $( git config --get user.name ) <$( git config --get user.email )>  $( date --rfc-email )\n"
@@ -47,7 +44,7 @@ echo -e "${deb_changelog_new}" >> ${deb_changelog_file}
 echo -e "${deb_changelog_orig}" >> ${deb_changelog_file}
 mv ${deb_changelog_file} debian/changelog
 
-git add node-daemon/pvcnoded/Daemon.py api-daemon/pvcapid/Daemon.py README.md docs/index.md debian/changelog
+git add node-daemon/pvcnoded/Daemon.py api-daemon/pvcapid/Daemon.py client-cli/setup.py debian/changelog CHANGELOG.md .version
 git commit -v
 
 echo

client-cli/pvc/cli_lib/ceph.py

@@ -19,13 +19,13 @@
 #
 ###############################################################################
 
-import json
 import math
 
+from json import dumps
 from requests_toolbelt.multipart.encoder import MultipartEncoder, MultipartEncoderMonitor
 
-import cli_lib.ansiprint as ansiprint
-from cli_lib.common import UploadProgressBar, call_api
+import pvc.cli_lib.ansiprint as ansiprint
+from pvc.cli_lib.common import UploadProgressBar, call_api
 
 #
 # Supplemental functions
@@ -81,8 +81,8 @@ def format_ops_tohuman(dataops):
     datahuman = ''
     for unit in sorted(ops_unit_matrix, key=ops_unit_matrix.get, reverse=True):
         new_ops = int(math.ceil(dataops / ops_unit_matrix[unit]))
-        # Round up if 5 or more digits
-        if new_ops > 9999:
+        # Round up if 6 or more digits
+        if new_ops > 99999:
             # We can jump down another level
             continue
         else:
@@ -149,6 +149,31 @@ def format_raw_output(status_data):
     return '\n'.join(ainformation)
 
 
+#
+# OSD DB VG functions
+#
+def ceph_osd_db_vg_add(config, node, device):
+    """
+    Add new Ceph OSD database volume group
+
+    API endpoint: POST /api/v1/storage/ceph/osddb
+    API arguments: node={node}, device={device}
+    API schema: {"message":"{data}"}
+    """
+    params = {
+        'node': node,
+        'device': device
+    }
+    response = call_api(config, 'post', '/storage/ceph/osddb', params=params)
+
+    if response.status_code == 200:
+        retstatus = True
+    else:
+        retstatus = False
+
+    return retstatus, response.json().get('message', '')
+
+
 #
 # OSD functions
 #
@@ -197,18 +222,20 @@ def ceph_osd_list(config, limit):
         return False, response.json().get('message', '')
 
 
-def ceph_osd_add(config, node, device, weight):
+def ceph_osd_add(config, node, device, weight, ext_db_flag, ext_db_ratio):
     """
     Add new Ceph OSD
 
     API endpoint: POST /api/v1/storage/ceph/osd
-    API arguments: node={node}, device={device}, weight={weight}
+    API arguments: node={node}, device={device}, weight={weight}, ext_db={ext_db_flag}, ext_db_ratio={ext_db_ratio}
     API schema: {"message":"{data}"}
     """
     params = {
         'node': node,
         'device': device,
-        'weight': weight
+        'weight': weight,
+        'ext_db': ext_db_flag,
+        'ext_db_ratio': ext_db_ratio
     }
     response = call_api(config, 'post', '/storage/ceph/osd', params=params)
 
@@ -312,13 +339,15 @@ def format_list_osd(osd_list):
     osd_list_output = []
 
     osd_id_length = 3
+    osd_node_length = 5
+    osd_device_length = 6
+    osd_db_device_length = 9
     osd_up_length = 4
     osd_in_length = 4
     osd_size_length = 5
     osd_weight_length = 3
     osd_reweight_length = 5
     osd_pgs_length = 4
-    osd_node_length = 5
     osd_used_length = 5
     osd_free_length = 6
     osd_util_length = 6
@@ -358,10 +387,21 @@ def format_list_osd(osd_list):
         if _osd_id_length > osd_id_length:
             osd_id_length = _osd_id_length
 
+        # Set the OSD node length
         _osd_node_length = len(osd_information['stats']['node']) + 1
         if _osd_node_length > osd_node_length:
             osd_node_length = _osd_node_length
 
+        # Set the OSD device length
+        _osd_device_length = len(osd_information['device']) + 1
+        if _osd_device_length > osd_device_length:
+            osd_device_length = _osd_device_length
+
+        # Set the OSD db_device length
+        _osd_db_device_length = len(osd_information['db_device']) + 1
+        if _osd_db_device_length > osd_db_device_length:
+            osd_db_device_length = _osd_db_device_length
+
         # Set the size and length
         _osd_size_length = len(str(osd_information['stats']['size'])) + 1
         if _osd_size_length > osd_size_length:
@@ -419,28 +459,47 @@ def format_list_osd(osd_list):
             osd_rddata_length = _osd_rddata_length
 
     # Format the output header
+    osd_list_output.append('{bold}{osd_header: <{osd_header_length}} {state_header: <{state_header_length}} {details_header: <{details_header_length}} {read_header: <{read_header_length}} {write_header: <{write_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        osd_header_length=osd_id_length + osd_node_length + osd_device_length + osd_db_device_length + 3,
+        state_header_length=osd_up_length + osd_in_length + 1,
+        details_header_length=osd_size_length + osd_pgs_length + osd_weight_length + osd_reweight_length + osd_used_length + osd_free_length + osd_util_length + osd_var_length + 7,
+        read_header_length=osd_rdops_length + osd_rddata_length + 1,
+        write_header_length=osd_wrops_length + osd_wrdata_length + 1,
+        osd_header='OSDs ' + ''.join(['-' for _ in range(5, osd_id_length + osd_node_length + osd_device_length + osd_db_device_length + 2)]),
+        state_header='State ' + ''.join(['-' for _ in range(6, osd_up_length + osd_in_length)]),
+        details_header='Details ' + ''.join(['-' for _ in range(8, osd_size_length + osd_pgs_length + osd_weight_length + osd_reweight_length + osd_used_length + osd_free_length + osd_util_length + osd_var_length + 6)]),
+        read_header='Read ' + ''.join(['-' for _ in range(5, osd_rdops_length + osd_rddata_length)]),
+        write_header='Write ' + ''.join(['-' for _ in range(6, osd_wrops_length + osd_wrdata_length)]))
+    )
+
     osd_list_output.append('{bold}\
 {osd_id: <{osd_id_length}} \
 {osd_node: <{osd_node_length}} \
+{osd_device: <{osd_device_length}} \
+{osd_db_device: <{osd_db_device_length}} \
 {osd_up: <{osd_up_length}} \
 {osd_in: <{osd_in_length}} \
 {osd_size: <{osd_size_length}} \
 {osd_pgs: <{osd_pgs_length}} \
 {osd_weight: <{osd_weight_length}} \
 {osd_reweight: <{osd_reweight_length}} \
-Sp: {osd_used: <{osd_used_length}} \
+{osd_used: <{osd_used_length}} \
 {osd_free: <{osd_free_length}} \
 {osd_util: <{osd_util_length}} \
 {osd_var: <{osd_var_length}} \
-Rd: {osd_rdops: <{osd_rdops_length}} \
+{osd_rdops: <{osd_rdops_length}} \
 {osd_rddata: <{osd_rddata_length}} \
-Wr: {osd_wrops: <{osd_wrops_length}} \
+{osd_wrops: <{osd_wrops_length}} \
 {osd_wrdata: <{osd_wrdata_length}} \
 {end_bold}'.format(
         bold=ansiprint.bold(),
         end_bold=ansiprint.end(),
         osd_id_length=osd_id_length,
         osd_node_length=osd_node_length,
+        osd_device_length=osd_device_length,
+        osd_db_device_length=osd_db_device_length,
         osd_up_length=osd_up_length,
         osd_in_length=osd_in_length,
         osd_size_length=osd_size_length,
@@ -457,6 +516,8 @@ Wr: {osd_wrops: <{osd_wrops_length}} \
         osd_rddata_length=osd_rddata_length,
         osd_id='ID',
         osd_node='Node',
+        osd_device='Block',
+        osd_db_device='DB Block',
         osd_up='Up',
         osd_in='In',
         osd_size='Size',
@@ -485,23 +546,29 @@ Wr: {osd_wrops: <{osd_wrops_length}} \
         osd_util = round(osd_information['stats']['utilization'], 2)
         osd_var = round(osd_information['stats']['var'], 2)
 
+        osd_db_device = osd_information['db_device']
+        if not osd_db_device:
+            osd_db_device = 'N/A'
+
         # Format the output header
         osd_list_output.append('{bold}\
 {osd_id: <{osd_id_length}} \
 {osd_node: <{osd_node_length}} \
+{osd_device: <{osd_device_length}} \
+{osd_db_device: <{osd_db_device_length}} \
 {osd_up_colour}{osd_up: <{osd_up_length}}{end_colour} \
 {osd_in_colour}{osd_in: <{osd_in_length}}{end_colour} \
 {osd_size: <{osd_size_length}} \
 {osd_pgs: <{osd_pgs_length}} \
 {osd_weight: <{osd_weight_length}} \
 {osd_reweight: <{osd_reweight_length}} \
-    {osd_used: <{osd_used_length}} \
+{osd_used: <{osd_used_length}} \
 {osd_free: <{osd_free_length}} \
 {osd_util: <{osd_util_length}} \
 {osd_var: <{osd_var_length}} \
-    {osd_rdops: <{osd_rdops_length}} \
+{osd_rdops: <{osd_rdops_length}} \
 {osd_rddata: <{osd_rddata_length}} \
-    {osd_wrops: <{osd_wrops_length}} \
+{osd_wrops: <{osd_wrops_length}} \
 {osd_wrdata: <{osd_wrdata_length}} \
 {end_bold}'.format(
             bold='',
@@ -509,6 +576,8 @@ Wr: {osd_wrops: <{osd_wrops_length}} \
             end_colour=ansiprint.end(),
             osd_id_length=osd_id_length,
             osd_node_length=osd_node_length,
+            osd_device_length=osd_device_length,
+            osd_db_device_length=osd_db_device_length,
             osd_up_length=osd_up_length,
             osd_in_length=osd_in_length,
             osd_size_length=osd_size_length,
@@ -525,6 +594,8 @@ Wr: {osd_wrops: <{osd_wrops_length}} \
             osd_rddata_length=osd_rddata_length,
             osd_id=osd_information['id'],
             osd_node=osd_information['stats']['node'],
+            osd_device=osd_information['device'],
+            osd_db_device=osd_db_device,
             osd_up_colour=osd_up_colour,
             osd_up=osd_up_flag,
             osd_in_colour=osd_in_colour,
@@ -648,7 +719,7 @@ def format_list_pool(pool_list):
     pool_name_length = 5
     pool_id_length = 3
     pool_used_length = 5
-    pool_usedpct_length = 5
+    pool_usedpct_length = 6
     pool_free_length = 5
     pool_num_objects_length = 6
     pool_num_clones_length = 7
@@ -737,19 +808,32 @@ def format_list_pool(pool_list):
             pool_read_data_length = _pool_read_data_length
 
     # Format the output header
+    pool_list_output.append('{bold}{pool_header: <{pool_header_length}} {objects_header: <{objects_header_length}} {read_header: <{read_header_length}} {write_header: <{write_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        pool_header_length=pool_id_length + pool_name_length + pool_used_length + pool_usedpct_length + pool_free_length + 4,
+        objects_header_length=pool_num_objects_length + pool_num_clones_length + pool_num_copies_length + pool_num_degraded_length + 3,
+        read_header_length=pool_read_ops_length + pool_read_data_length + 1,
+        write_header_length=pool_write_ops_length + pool_write_data_length + 1,
+        pool_header='Pools ' + ''.join(['-' for _ in range(6, pool_id_length + pool_name_length + pool_used_length + pool_usedpct_length + pool_free_length + 3)]),
+        objects_header='Objects ' + ''.join(['-' for _ in range(8, pool_num_objects_length + pool_num_clones_length + pool_num_copies_length + pool_num_degraded_length + 2)]),
+        read_header='Read ' + ''.join(['-' for _ in range(5, pool_read_ops_length + pool_read_data_length)]),
+        write_header='Write ' + ''.join(['-' for _ in range(6, pool_write_ops_length + pool_write_data_length)]))
+    )
+
     pool_list_output.append('{bold}\
 {pool_id: <{pool_id_length}} \
 {pool_name: <{pool_name_length}} \
 {pool_used: <{pool_used_length}} \
 {pool_usedpct: <{pool_usedpct_length}} \
 {pool_free: <{pool_free_length}} \
-Obj: {pool_objects: <{pool_objects_length}} \
+{pool_objects: <{pool_objects_length}} \
 {pool_clones: <{pool_clones_length}} \
 {pool_copies: <{pool_copies_length}} \
 {pool_degraded: <{pool_degraded_length}} \
-Rd: {pool_read_ops: <{pool_read_ops_length}} \
+{pool_read_ops: <{pool_read_ops_length}} \
 {pool_read_data: <{pool_read_data_length}} \
-Wr: {pool_write_ops: <{pool_write_ops_length}} \
+{pool_write_ops: <{pool_write_ops_length}} \
 {pool_write_data: <{pool_write_data_length}} \
 {end_bold}'.format(
         bold=ansiprint.bold(),
@@ -770,7 +854,7 @@ Wr: {pool_write_ops: <{pool_write_ops_length}} \
         pool_id='ID',
         pool_name='Name',
         pool_used='Used',
-        pool_usedpct='%',
+        pool_usedpct='Used%',
         pool_free='Free',
         pool_objects='Count',
         pool_clones='Clones',
@@ -790,13 +874,13 @@ Wr: {pool_write_ops: <{pool_write_ops_length}} \
 {pool_used: <{pool_used_length}} \
 {pool_usedpct: <{pool_usedpct_length}} \
 {pool_free: <{pool_free_length}} \
-     {pool_objects: <{pool_objects_length}} \
+{pool_objects: <{pool_objects_length}} \
 {pool_clones: <{pool_clones_length}} \
 {pool_copies: <{pool_copies_length}} \
 {pool_degraded: <{pool_degraded_length}} \
-    {pool_read_ops: <{pool_read_ops_length}} \
+{pool_read_ops: <{pool_read_ops_length}} \
 {pool_read_data: <{pool_read_data_length}} \
-    {pool_write_ops: <{pool_write_ops_length}} \
+{pool_write_ops: <{pool_write_ops_length}} \
 {pool_write_data: <{pool_write_data_length}} \
 {end_bold}'.format(
             bold='',
@@ -1057,6 +1141,15 @@ def format_list_volume(volume_list):
             volume_features_length = _volume_features_length
 
     # Format the output header
+    volume_list_output.append('{bold}{volume_header: <{volume_header_length}} {details_header: <{details_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        volume_header_length=volume_name_length + volume_pool_length + 1,
+        details_header_length=volume_size_length + volume_objects_length + volume_order_length + volume_format_length + volume_features_length + 4,
+        volume_header='Volumes ' + ''.join(['-' for _ in range(8, volume_name_length + volume_pool_length)]),
+        details_header='Details ' + ''.join(['-' for _ in range(8, volume_size_length + volume_objects_length + volume_order_length + volume_format_length + volume_features_length + 3)]))
+    )
+
     volume_list_output.append('{bold}\
 {volume_name: <{volume_name_length}} \
 {volume_pool: <{volume_pool_length}} \
@@ -1084,7 +1177,7 @@ def format_list_volume(volume_list):
         volume_features='Features')
     )
 
-    for volume_information in volume_list:
+    for volume_information in sorted(volume_list, key=lambda v: v['pool'] + v['name']):
         volume_list_output.append('{bold}\
 {volume_name: <{volume_name_length}} \
 {volume_pool: <{volume_pool_length}} \
@@ -1112,7 +1205,7 @@ def format_list_volume(volume_list):
             volume_features=','.join(volume_information['stats']['features']))
         )
 
-    return '\n'.join(sorted(volume_list_output))
+    return '\n'.join(volume_list_output)
 
 
 #
@@ -1263,6 +1356,13 @@ def format_list_snapshot(snapshot_list):
             snapshot_pool_length = _snapshot_pool_length
 
     # Format the output header
+    snapshot_list_output.append('{bold}{snapshot_header: <{snapshot_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        snapshot_header_length=snapshot_name_length + snapshot_volume_length + snapshot_pool_length + 2,
+        snapshot_header='Snapshots ' + ''.join(['-' for _ in range(10, snapshot_name_length + snapshot_volume_length + snapshot_pool_length + 1)]))
+    )
+
     snapshot_list_output.append('{bold}\
 {snapshot_name: <{snapshot_name_length}} \
 {snapshot_volume: <{snapshot_volume_length}} \
@@ -1278,7 +1378,7 @@ def format_list_snapshot(snapshot_list):
         snapshot_pool='Pool')
     )
 
-    for snapshot_information in snapshot_list:
+    for snapshot_information in sorted(snapshot_list, key=lambda s: s['pool'] + s['volume'] + s['snapshot']):
         snapshot_name = snapshot_information['snapshot']
         snapshot_volume = snapshot_information['volume']
         snapshot_pool = snapshot_information['pool']
@@ -1297,7 +1397,7 @@ def format_list_snapshot(snapshot_list):
             snapshot_pool=snapshot_pool)
         )
 
-    return '\n'.join(sorted(snapshot_list_output))
+    return '\n'.join(snapshot_list_output)
 
 
 #
@@ -1353,10 +1453,51 @@ def ceph_benchmark_list(config, job):
     return retvalue, retdata
 
 
+def get_benchmark_list_results_legacy(benchmark_data):
+    benchmark_bandwidth = dict()
+    benchmark_iops = dict()
+    for test in ["seq_read", "seq_write", "rand_read_4K", "rand_write_4K"]:
+        benchmark_bandwidth[test] = format_bytes_tohuman(int(benchmark_data[test]['overall']['bandwidth']) * 1024)
+        benchmark_iops[test] = format_ops_tohuman(int(benchmark_data[test]['overall']['iops']))
+
+    return benchmark_bandwidth, benchmark_iops
+
+
+def get_benchmark_list_results_json(benchmark_data):
+    benchmark_bandwidth = dict()
+    benchmark_iops = dict()
+    for test in ['seq_read', 'seq_write', 'rand_read_4K', 'rand_write_4K']:
+        benchmark_test_data = benchmark_data[test]
+        active_class = None
+        for io_class in ['read', 'write']:
+            if benchmark_test_data['jobs'][0][io_class]['io_bytes'] > 0:
+                active_class = io_class
+        if active_class is not None:
+            benchmark_bandwidth[test] = format_bytes_tohuman(int(benchmark_test_data['jobs'][0][active_class]['bw_bytes']))
+            benchmark_iops[test] = format_ops_tohuman(int(benchmark_test_data['jobs'][0][active_class]['iops']))
+
+    return benchmark_bandwidth, benchmark_iops
+
+
+def get_benchmark_list_results(benchmark_format, benchmark_data):
+    if benchmark_format == 0:
+        benchmark_bandwidth, benchmark_iops = get_benchmark_list_results_legacy(benchmark_data)
+    elif benchmark_format == 1:
+        benchmark_bandwidth, benchmark_iops = get_benchmark_list_results_json(benchmark_data)
+
+    seq_benchmark_bandwidth = "{} / {}".format(benchmark_bandwidth['seq_read'], benchmark_bandwidth['seq_write'])
+    seq_benchmark_iops = "{} / {}".format(benchmark_iops['seq_read'], benchmark_iops['seq_write'])
+    rand_benchmark_bandwidth = "{} / {}".format(benchmark_bandwidth['rand_read_4K'], benchmark_bandwidth['rand_write_4K'])
+    rand_benchmark_iops = "{} / {}".format(benchmark_iops['rand_read_4K'], benchmark_iops['rand_write_4K'])
+
+    return seq_benchmark_bandwidth, seq_benchmark_iops, rand_benchmark_bandwidth, rand_benchmark_iops
+
+
 def format_list_benchmark(config, benchmark_information):
     benchmark_list_output = []
 
     benchmark_job_length = 20
+    benchmark_format_length = 6
     benchmark_bandwidth_length = dict()
     benchmark_iops_length = dict()
 
@@ -1365,61 +1506,75 @@ def format_list_benchmark(config, benchmark_information):
         benchmark_bandwidth_length[test] = 7
         benchmark_iops_length[test] = 6
 
+    benchmark_seq_bw_length = 15
+    benchmark_seq_iops_length = 10
+    benchmark_rand_bw_length = 15
+    benchmark_rand_iops_length = 10
+
     for benchmark in benchmark_information:
         benchmark_job = benchmark['job']
+        benchmark_format = benchmark['test_format']  # noqa: F841
+
         _benchmark_job_length = len(benchmark_job)
         if _benchmark_job_length > benchmark_job_length:
             benchmark_job_length = _benchmark_job_length
 
         if benchmark['benchmark_result'] == 'Running':
             continue
-        benchmark_data = json.loads(benchmark['benchmark_result'])
 
-        benchmark_bandwidth = dict()
-        benchmark_iops = dict()
-        for test in ["seq_read", "seq_write", "rand_read_4K", "rand_write_4K"]:
-            benchmark_bandwidth[test] = format_bytes_tohuman(int(benchmark_data[test]['overall']['bandwidth']) * 1024)
-            benchmark_iops[test] = format_ops_tohuman(int(benchmark_data[test]['overall']['iops']))
+        benchmark_data = benchmark['benchmark_result']
+        seq_benchmark_bandwidth, seq_benchmark_iops, rand_benchmark_bandwidth, rand_benchmark_iops = get_benchmark_list_results(benchmark_format, benchmark_data)
 
-            _benchmark_bandwidth_length = len(benchmark_bandwidth[test]) + 1
-            if _benchmark_bandwidth_length > benchmark_bandwidth_length[test]:
-                benchmark_bandwidth_length[test] = _benchmark_bandwidth_length
+        _benchmark_seq_bw_length = len(seq_benchmark_bandwidth) + 1
+        if _benchmark_seq_bw_length > benchmark_seq_bw_length:
+            benchmark_seq_bw_length = _benchmark_seq_bw_length
 
-            _benchmark_iops_length = len(benchmark_iops[test]) + 1
-            if _benchmark_iops_length > benchmark_bandwidth_length[test]:
-                benchmark_iops_length[test] = _benchmark_iops_length
+        _benchmark_seq_iops_length = len(seq_benchmark_iops) + 1
+        if _benchmark_seq_iops_length > benchmark_seq_iops_length:
+            benchmark_seq_iops_length = _benchmark_seq_iops_length
+
+        _benchmark_rand_bw_length = len(rand_benchmark_bandwidth) + 1
+        if _benchmark_rand_bw_length > benchmark_rand_bw_length:
+            benchmark_rand_bw_length = _benchmark_rand_bw_length
+
+        _benchmark_rand_iops_length = len(rand_benchmark_iops) + 1
+        if _benchmark_rand_iops_length > benchmark_rand_iops_length:
+            benchmark_rand_iops_length = _benchmark_rand_iops_length
 
     # Format the output header line 1
     benchmark_list_output.append('{bold}\
-{benchmark_job: <{benchmark_job_length}} \
- {seq_header: <{seq_header_length}} \
-{rand_header: <{rand_header_length}} \
+{benchmark_job: <{benchmark_job_length}}   \
+{seq_header: <{seq_header_length}}  \
+{rand_header: <{rand_header_length}}\
 {end_bold}'.format(
         bold=ansiprint.bold(),
         end_bold=ansiprint.end(),
-        benchmark_job_length=benchmark_job_length,
-        seq_header_length=benchmark_bandwidth_length['seq_read'] + benchmark_bandwidth_length['seq_write'] + benchmark_iops_length['seq_read'] + benchmark_iops_length['seq_write'] + 3,
-        rand_header_length=benchmark_bandwidth_length['rand_read_4K'] + benchmark_bandwidth_length['rand_write_4K'] + benchmark_iops_length['rand_read_4K'] + benchmark_iops_length['rand_write_4K'] + 2,
-        benchmark_job='Benchmark Job',
-        seq_header='Sequential (4M blocks):',
-        rand_header='Random (4K blocks):')
+        benchmark_job_length=benchmark_job_length + benchmark_format_length + 1,
+        seq_header_length=benchmark_seq_bw_length + benchmark_seq_iops_length + 1,
+        rand_header_length=benchmark_rand_bw_length + benchmark_rand_iops_length + 1,
+        benchmark_job='Benchmarks ' + ''.join(['-' for _ in range(11, benchmark_job_length + benchmark_format_length + 2)]),
+        seq_header='Sequential (4M blocks) ' + ''.join(['-' for _ in range(23, benchmark_seq_bw_length + benchmark_seq_iops_length)]),
+        rand_header='Random (4K blocks) ' + ''.join(['-' for _ in range(19, benchmark_rand_bw_length + benchmark_rand_iops_length)]))
     )
 
     benchmark_list_output.append('{bold}\
-{benchmark_job: <{benchmark_job_length}} \
- {seq_benchmark_bandwidth: <{seq_benchmark_bandwidth_length}} \
-{seq_benchmark_iops: <{seq_benchmark_iops_length}} \
+{benchmark_job: <{benchmark_job_length}}  \
+{benchmark_format: <{benchmark_format_length}}   \
+{seq_benchmark_bandwidth: <{seq_benchmark_bandwidth_length}} \
+{seq_benchmark_iops: <{seq_benchmark_iops_length}}  \
 {rand_benchmark_bandwidth: <{rand_benchmark_bandwidth_length}} \
-{rand_benchmark_iops: <{rand_benchmark_iops_length}} \
+{rand_benchmark_iops: <{rand_benchmark_iops_length}}\
 {end_bold}'.format(
         bold=ansiprint.bold(),
         end_bold=ansiprint.end(),
         benchmark_job_length=benchmark_job_length,
-        seq_benchmark_bandwidth_length=benchmark_bandwidth_length['seq_read'] + benchmark_bandwidth_length['seq_write'] + 2,
-        seq_benchmark_iops_length=benchmark_iops_length['seq_read'] + benchmark_iops_length['seq_write'],
-        rand_benchmark_bandwidth_length=benchmark_bandwidth_length['rand_read_4K'] + benchmark_bandwidth_length['rand_write_4K'] + 1,
-        rand_benchmark_iops_length=benchmark_iops_length['rand_read_4K'] + benchmark_iops_length['rand_write_4K'],
-        benchmark_job='',
+        benchmark_format_length=benchmark_format_length,
+        seq_benchmark_bandwidth_length=benchmark_seq_bw_length,
+        seq_benchmark_iops_length=benchmark_seq_iops_length,
+        rand_benchmark_bandwidth_length=benchmark_rand_bw_length,
+        rand_benchmark_iops_length=benchmark_rand_iops_length,
+        benchmark_job='Job',
+        benchmark_format='Format',
         seq_benchmark_bandwidth='R/W Bandwith/s',
         seq_benchmark_iops='R/W IOPS',
         rand_benchmark_bandwidth='R/W Bandwith/s',
@@ -1428,6 +1583,7 @@ def format_list_benchmark(config, benchmark_information):
 
     for benchmark in benchmark_information:
         benchmark_job = benchmark['job']
+        benchmark_format = benchmark['test_format']  # noqa: F841
 
         if benchmark['benchmark_result'] == 'Running':
             seq_benchmark_bandwidth = 'Running'
@@ -1435,33 +1591,27 @@ def format_list_benchmark(config, benchmark_information):
             rand_benchmark_bandwidth = 'Running'
             rand_benchmark_iops = 'Running'
         else:
-            benchmark_bandwidth = dict()
-            benchmark_iops = dict()
-            for test in ["seq_read", "seq_write", "rand_read_4K", "rand_write_4K"]:
-                benchmark_data = json.loads(benchmark['benchmark_result'])
-                benchmark_bandwidth[test] = format_bytes_tohuman(int(benchmark_data[test]['overall']['bandwidth']) * 1024)
-                benchmark_iops[test] = format_ops_tohuman(int(benchmark_data[test]['overall']['iops']))
-
-            seq_benchmark_bandwidth = "{} / {}".format(benchmark_bandwidth['seq_read'], benchmark_bandwidth['seq_write'])
-            seq_benchmark_iops = "{} / {}".format(benchmark_iops['seq_read'], benchmark_iops['seq_write'])
-            rand_benchmark_bandwidth = "{} / {}".format(benchmark_bandwidth['rand_read_4K'], benchmark_bandwidth['rand_write_4K'])
-            rand_benchmark_iops = "{} / {}".format(benchmark_iops['rand_read_4K'], benchmark_iops['rand_write_4K'])
+            benchmark_data = benchmark['benchmark_result']
+            seq_benchmark_bandwidth, seq_benchmark_iops, rand_benchmark_bandwidth, rand_benchmark_iops = get_benchmark_list_results(benchmark_format, benchmark_data)
 
         benchmark_list_output.append('{bold}\
-{benchmark_job: <{benchmark_job_length}} \
- {seq_benchmark_bandwidth: <{seq_benchmark_bandwidth_length}} \
-{seq_benchmark_iops: <{seq_benchmark_iops_length}} \
+{benchmark_job: <{benchmark_job_length}}  \
+{benchmark_format: <{benchmark_format_length}}   \
+{seq_benchmark_bandwidth: <{seq_benchmark_bandwidth_length}} \
+{seq_benchmark_iops: <{seq_benchmark_iops_length}}  \
 {rand_benchmark_bandwidth: <{rand_benchmark_bandwidth_length}} \
-{rand_benchmark_iops: <{rand_benchmark_iops_length}} \
+{rand_benchmark_iops: <{rand_benchmark_iops_length}}\
 {end_bold}'.format(
             bold='',
             end_bold='',
             benchmark_job_length=benchmark_job_length,
-            seq_benchmark_bandwidth_length=benchmark_bandwidth_length['seq_read'] + benchmark_bandwidth_length['seq_write'] + 2,
-            seq_benchmark_iops_length=benchmark_iops_length['seq_read'] + benchmark_iops_length['seq_write'],
-            rand_benchmark_bandwidth_length=benchmark_bandwidth_length['rand_read_4K'] + benchmark_bandwidth_length['rand_write_4K'] + 1,
-            rand_benchmark_iops_length=benchmark_iops_length['rand_read_4K'] + benchmark_iops_length['rand_write_4K'],
+            benchmark_format_length=benchmark_format_length,
+            seq_benchmark_bandwidth_length=benchmark_seq_bw_length,
+            seq_benchmark_iops_length=benchmark_seq_iops_length,
+            rand_benchmark_bandwidth_length=benchmark_rand_bw_length,
+            rand_benchmark_iops_length=benchmark_rand_iops_length,
             benchmark_job=benchmark_job,
+            benchmark_format=benchmark_format,
             seq_benchmark_bandwidth=seq_benchmark_bandwidth,
             seq_benchmark_iops=seq_benchmark_iops,
             rand_benchmark_bandwidth=rand_benchmark_bandwidth,
@@ -1471,11 +1621,29 @@ def format_list_benchmark(config, benchmark_information):
     return '\n'.join(benchmark_list_output)
 
 
-def format_info_benchmark(config, benchmark_information):
-    if benchmark_information[0]['benchmark_result'] == "Running":
+def format_info_benchmark(config, oformat, benchmark_information):
+    # This matrix is a list of the possible format functions for a benchmark result
+    # It is extensable in the future should newer formats be required.
+    benchmark_matrix = {
+        0: format_info_benchmark_legacy,
+        1: format_info_benchmark_json,
+    }
+
+    benchmark_version = benchmark_information[0]['test_format']
+
+    if oformat == 'json-pretty':
+        return dumps(benchmark_information, indent=4)
+    elif oformat == 'json':
+        return dumps(benchmark_information)
+    else:
+        return benchmark_matrix[benchmark_version](config, benchmark_information[0])
+
+
+def format_info_benchmark_legacy(config, benchmark_information):
+    if benchmark_information['benchmark_result'] == "Running":
         return "Benchmark test is still running."
 
-    benchmark_details = json.loads(benchmark_information[0]['benchmark_result'])
+    benchmark_details = benchmark_information['benchmark_result']
 
     # Format a nice output; do this line-by-line then concat the elements at the end
     ainformation = []
@@ -1486,10 +1654,10 @@ def format_info_benchmark(config, benchmark_information):
         "seq_write": "Sequential Write (4M blocks)",
         "rand_read_4M": "Random Read (4M blocks)",
         "rand_write_4M": "Random Write (4M blocks)",
-        "rand_read_256K": "Random Read (256K blocks)",
-        "rand_write_256K": "Random Write (256K blocks)",
         "rand_read_4K": "Random Read (4K blocks)",
-        "rand_write_4K": "Random Write (4K blocks)"
+        "rand_write_4K": "Random Write (4K blocks)",
+        "rand_read_4K_lowdepth": "Random Read (4K blocks, single-queue)",
+        "rand_write_4K_lowdepth": "Random Write (4K blocks, single-queue)",
     }
 
     test_name_length = 30
@@ -1502,7 +1670,16 @@ def format_info_benchmark(config, benchmark_information):
     cpuutil_label_length = 11
     cpuutil_column_length = 9
 
+    # Work around old results that did not have these tests
+    if 'rand_read_4K_lowdepth' not in benchmark_details:
+        del nice_test_name_map['rand_read_4K_lowdepth']
+        del nice_test_name_map['rand_write_4K_lowdepth']
+
     for test in benchmark_details:
+        # Work around old results that had these obsolete tests
+        if test == 'rand_read_256K' or test == 'rand_write_256K':
+            continue
+
         _test_name_length = len(nice_test_name_map[test])
         if _test_name_length > test_name_length:
             test_name_length = _test_name_length
@@ -1539,6 +1716,10 @@ def format_info_benchmark(config, benchmark_information):
                 cpuutil_column_length = _element_length
 
     for test in benchmark_details:
+        # Work around old results that had these obsolete tests
+        if test == 'rand_read_256K' or test == 'rand_write_256K':
+            continue
+
         ainformation.append('')
 
         test_details = benchmark_details[test]
@@ -1745,3 +1926,269 @@ def format_info_benchmark(config, benchmark_information):
         ainformation.append('')
 
     return '\n'.join(ainformation)
+
+
+def format_info_benchmark_json(config, benchmark_information):
+    if benchmark_information['benchmark_result'] == "Running":
+        return "Benchmark test is still running."
+
+    benchmark_details = benchmark_information['benchmark_result']
+
+    # Format a nice output; do this line-by-line then concat the elements at the end
+    ainformation = []
+    ainformation.append('{}Storage Benchmark details:{}'.format(ansiprint.bold(), ansiprint.end()))
+
+    nice_test_name_map = {
+        "seq_read": "Sequential Read (4M blocks, queue depth 64)",
+        "seq_write": "Sequential Write (4M blocks, queue depth 64)",
+        "rand_read_4M": "Random Read (4M blocks, queue depth 64)",
+        "rand_write_4M": "Random Write (4M blocks queue depth 64)",
+        "rand_read_4K": "Random Read (4K blocks, queue depth 64)",
+        "rand_write_4K": "Random Write (4K blocks, queue depth 64)",
+        "rand_read_4K_lowdepth": "Random Read (4K blocks, queue depth 1)",
+        "rand_write_4K_lowdepth": "Random Write (4K blocks, queue depth 1)",
+    }
+
+    for test in benchmark_details:
+        ainformation.append('')
+
+        io_class = None
+        for _io_class in ['read', 'write']:
+            if benchmark_details[test]['jobs'][0][_io_class]['io_bytes'] > 0:
+                io_class = _io_class
+        if io_class is None:
+            continue
+
+        job_details = benchmark_details[test]['jobs'][0]
+
+        # Calculate the unified latency categories (in us)
+        latency_tree = list()
+        for field in job_details['latency_ns']:
+            bucket = str(int(field) / 1000)
+            latency_tree.append((bucket, job_details['latency_ns'][field]))
+        for field in job_details['latency_us']:
+            bucket = field
+            latency_tree.append((bucket, job_details['latency_us'][field]))
+        for field in job_details['latency_ms']:
+            # That one annoying one
+            if field == '>=2000':
+                bucket = '>=2000000'
+            else:
+                bucket = str(int(field) * 1000)
+            latency_tree.append((bucket, job_details['latency_ms'][field]))
+
+        # Find the minimum entry without a zero
+        useful_latency_tree = list()
+        for element in latency_tree:
+            if element[1] != 0:
+                useful_latency_tree.append(element)
+
+        max_rows = 9
+        if len(useful_latency_tree) > 9:
+            max_rows = len(useful_latency_tree)
+        elif len(useful_latency_tree) < 9:
+            while len(useful_latency_tree) < 9:
+                useful_latency_tree.append(('', ''))
+
+        # Format the static data
+        overall_label = [ 'Overall BW/s:',
+                          'Overall IOPS:',
+                          'Total I/O:',
+                          'Runtime (s):',
+                          'User CPU %:',
+                          'System CPU %:',
+                          'Ctx Switches:',
+                          'Major Faults:',
+                          'Minor Faults:' ]
+        while len(overall_label) < max_rows:
+            overall_label.append('')
+
+        overall_data =  [ format_bytes_tohuman(int(job_details[io_class]['bw_bytes'])),
+                          format_ops_tohuman(int(job_details[io_class]['iops'])),
+                          format_bytes_tohuman(int(job_details[io_class]['io_bytes'])),
+                          job_details['job_runtime'] / 1000,
+                          job_details['usr_cpu'],
+                          job_details['sys_cpu'],
+                          job_details['ctx'],
+                          job_details['majf'],
+                          job_details['minf'] ]
+        while len(overall_data) < max_rows:
+            overall_data.append('')
+
+        bandwidth_label = [ 'Min:',
+                            'Max:',
+                            'Mean:',
+                            'StdDev:',
+                            'Samples:',
+                            '',
+                            '',
+                            '',
+                            '' ]
+        while len(bandwidth_label) < max_rows:
+            bandwidth_label.append('')
+
+        bandwidth_data =  [ format_bytes_tohuman(int(job_details[io_class]['bw_min']) * 1024),
+                            format_bytes_tohuman(int(job_details[io_class]['bw_max']) * 1024),
+                            format_bytes_tohuman(int(job_details[io_class]['bw_mean']) * 1024),
+                            format_bytes_tohuman(int(job_details[io_class]['bw_dev']) * 1024),
+                            job_details[io_class]['bw_samples'],
+                            '',
+                            '',
+                            '',
+                            '' ]
+        while len(bandwidth_data) < max_rows:
+            bandwidth_data.append('')
+
+        iops_data = [ format_ops_tohuman(int(job_details[io_class]['iops_min'])),
+                      format_ops_tohuman(int(job_details[io_class]['iops_max'])),
+                      format_ops_tohuman(int(job_details[io_class]['iops_mean'])),
+                      format_ops_tohuman(int(job_details[io_class]['iops_stddev'])),
+                      job_details[io_class]['iops_samples'],
+                      '',
+                      '',
+                      '',
+                      '' ]
+        while len(iops_data) < max_rows:
+            iops_data.append('')
+
+        lat_data = [ int(job_details[io_class]['lat_ns']['min']) / 1000,
+                     int(job_details[io_class]['lat_ns']['max']) / 1000,
+                     int(job_details[io_class]['lat_ns']['mean']) / 1000,
+                     int(job_details[io_class]['lat_ns']['stddev']) / 1000,
+                     '',
+                     '',
+                     '',
+                     '',
+                     '' ]
+        while len(lat_data) < max_rows:
+            lat_data.append('')
+
+        # Format the dynamic buckets
+        lat_bucket_label = list()
+        lat_bucket_data = list()
+        for element in useful_latency_tree:
+            lat_bucket_label.append(element[0])
+            lat_bucket_data.append(element[1])
+
+        # Column default widths
+        overall_label_length = 0
+        overall_column_length = 0
+        bandwidth_label_length = 0
+        bandwidth_column_length = 11
+        iops_column_length = 4
+        latency_column_length = 12
+        latency_bucket_label_length = 0
+
+        # Column layout:
+        #    General    Bandwidth   IOPS      Latency   Percentiles
+        #    ---------  ----------  --------  --------  ---------------
+        #    Size       Min         Min       Min       A
+        #    BW         Max         Max       Max       B
+        #    IOPS       Mean        Mean      Mean      ...
+        #    Runtime    StdDev      StdDev    StdDev    Z
+        #    UsrCPU     Samples     Samples
+        #    SysCPU
+        #    CtxSw
+        #    MajFault
+        #    MinFault
+
+        # Set column widths
+        for item in overall_label:
+            _item_length = len(str(item))
+            if _item_length > overall_label_length:
+                overall_label_length = _item_length
+
+        for item in overall_data:
+            _item_length = len(str(item))
+            if _item_length > overall_column_length:
+                overall_column_length = _item_length
+
+        test_name_length = len(nice_test_name_map[test])
+        if test_name_length > overall_label_length + overall_column_length:
+            _diff = test_name_length - (overall_label_length + overall_column_length)
+            overall_column_length += _diff
+
+        for item in bandwidth_label:
+            _item_length = len(str(item))
+            if _item_length > bandwidth_label_length:
+                bandwidth_label_length = _item_length
+
+        for item in bandwidth_data:
+            _item_length = len(str(item))
+            if _item_length > bandwidth_column_length:
+                bandwidth_column_length = _item_length
+
+        for item in iops_data:
+            _item_length = len(str(item))
+            if _item_length > iops_column_length:
+                iops_column_length = _item_length
+
+        for item in lat_data:
+            _item_length = len(str(item))
+            if _item_length > latency_column_length:
+                latency_column_length = _item_length
+
+        for item in lat_bucket_label:
+            _item_length = len(str(item))
+            if _item_length > latency_bucket_label_length:
+                latency_bucket_label_length = _item_length
+
+        # Top row (Headers)
+        ainformation.append('{bold}\
+{overall_label: <{overall_label_length}}    \
+{bandwidth_label: <{bandwidth_label_length}} \
+{bandwidth: <{bandwidth_length}}   \
+{iops: <{iops_length}}   \
+{latency: <{latency_length}}   \
+{latency_bucket_label: <{latency_bucket_label_length}} \
+{latency_bucket} \
+{end_bold}'.format(
+            bold=ansiprint.bold(),
+            end_bold=ansiprint.end(),
+            overall_label=nice_test_name_map[test],
+            overall_label_length=overall_label_length,
+            bandwidth_label='',
+            bandwidth_label_length=bandwidth_label_length,
+            bandwidth='Bandwidth/s',
+            bandwidth_length=bandwidth_column_length,
+            iops='IOPS',
+            iops_length=iops_column_length,
+            latency='Latency (μs)',
+            latency_length=latency_column_length,
+            latency_bucket_label='Latency Buckets (μs/%)',
+            latency_bucket_label_length=latency_bucket_label_length,
+            latency_bucket='',
+        ))
+
+        for idx in range(0, max_rows):
+            # Top row (Headers)
+            ainformation.append('{bold}\
+{overall_label: >{overall_label_length}} \
+{overall: <{overall_length}}   \
+{bandwidth_label: >{bandwidth_label_length}} \
+{bandwidth: <{bandwidth_length}}   \
+{iops: <{iops_length}}   \
+{latency: <{latency_length}}   \
+{latency_bucket_label: >{latency_bucket_label_length}} \
+{latency_bucket} \
+{end_bold}'.format(
+                bold='',
+                end_bold='',
+                overall_label=overall_label[idx],
+                overall_label_length=overall_label_length,
+                overall=overall_data[idx],
+                overall_length=overall_column_length,
+                bandwidth_label=bandwidth_label[idx],
+                bandwidth_label_length=bandwidth_label_length,
+                bandwidth=bandwidth_data[idx],
+                bandwidth_length=bandwidth_column_length,
+                iops=iops_data[idx],
+                iops_length=iops_column_length,
+                latency=lat_data[idx],
+                latency_length=latency_column_length,
+                latency_bucket_label=lat_bucket_label[idx],
+                latency_bucket_label_length=latency_bucket_label_length,
+                latency_bucket=lat_bucket_data[idx],
+            ))
+
+    return '\n'.join(ainformation)

client-cli/pvc/cli_lib/cluster.py

@@ -21,8 +21,8 @@
 
 import json
 
-import cli_lib.ansiprint as ansiprint
-from cli_lib.common import call_api
+import pvc.cli_lib.ansiprint as ansiprint
+from pvc.cli_lib.common import call_api
 
 
 def initialize(config, overwrite=False):

client-cli/pvc/cli_lib/network.py

@@ -20,8 +20,8 @@
 ###############################################################################
 
 import re
-import cli_lib.ansiprint as ansiprint
-from cli_lib.common import call_api
+import pvc.cli_lib.ansiprint as ansiprint
+from pvc.cli_lib.common import call_api
 
 
 def isValidMAC(macaddr):
@@ -100,7 +100,7 @@ def net_list(config, limit):
         return False, response.json().get('message', '')
 
 
-def net_add(config, vni, description, nettype, domain, name_servers, ip4_network, ip4_gateway, ip6_network, ip6_gateway, dhcp4_flag, dhcp4_start, dhcp4_end):
+def net_add(config, vni, description, nettype, mtu, domain, name_servers, ip4_network, ip4_gateway, ip6_network, ip6_gateway, dhcp4_flag, dhcp4_start, dhcp4_end):
     """
     Add new network
 
@@ -112,6 +112,7 @@ def net_add(config, vni, description, nettype, domain, name_servers, ip4_network
         'vni': vni,
         'description': description,
         'nettype': nettype,
+        'mtu': mtu,
         'domain': domain,
         'name_servers': name_servers,
         'ip4_network': ip4_network,
@@ -132,7 +133,7 @@ def net_add(config, vni, description, nettype, domain, name_servers, ip4_network
     return retstatus, response.json().get('message', '')
 
 
-def net_modify(config, net, description, domain, name_servers, ip4_network, ip4_gateway, ip6_network, ip6_gateway, dhcp4_flag, dhcp4_start, dhcp4_end):
+def net_modify(config, net, description, mtu, domain, name_servers, ip4_network, ip4_gateway, ip6_network, ip6_gateway, dhcp4_flag, dhcp4_start, dhcp4_end):
     """
     Modify a network
 
@@ -143,6 +144,8 @@ def net_modify(config, net, description, domain, name_servers, ip4_network, ip4_
     params = dict()
     if description is not None:
         params['description'] = description
+    if mtu is not None:
+        params['mtu'] = mtu
     if domain is not None:
         params['domain'] = domain
     if name_servers is not None:
@@ -360,7 +363,6 @@ def net_acl_add(config, net, direction, description, rule, order):
 
 
 def net_acl_remove(config, net, description):
-
     """
     Remove a network ACL
 
@@ -378,27 +380,131 @@ def net_acl_remove(config, net, description):
     return retstatus, response.json().get('message', '')
 
 
+#
+# SR-IOV functions
+#
+def net_sriov_pf_list(config, node):
+    """
+    List all PFs on NODE
+
+    API endpoint: GET /api/v1/sriov/pf/<node>
+    API arguments: node={node}
+    API schema: [{json_data_object},{json_data_object},etc.]
+    """
+    response = call_api(config, 'get', '/sriov/pf/{}'.format(node))
+
+    if response.status_code == 200:
+        return True, response.json()
+    else:
+        return False, response.json().get('message', '')
+
+
+def net_sriov_vf_set(config, node, vf, vlan_id, vlan_qos, tx_rate_min, tx_rate_max, link_state, spoof_check, trust, query_rss):
+    """
+    Mdoify configuration of a SR-IOV VF
+
+    API endpoint: PUT /api/v1/sriov/vf/<node>/<vf>
+    API arguments: vlan_id={vlan_id}, vlan_qos={vlan_qos}, tx_rate_min={tx_rate_min}, tx_rate_max={tx_rate_max},
+                   link_state={link_state}, spoof_check={spoof_check}, trust={trust}, query_rss={query_rss}
+    API schema: {"message": "{data}"}
+    """
+    params = dict()
+
+    # Update any params that we've sent
+    if vlan_id is not None:
+        params['vlan_id'] = vlan_id
+
+    if vlan_qos is not None:
+        params['vlan_qos'] = vlan_qos
+
+    if tx_rate_min is not None:
+        params['tx_rate_min'] = tx_rate_min
+
+    if tx_rate_max is not None:
+        params['tx_rate_max'] = tx_rate_max
+
+    if link_state is not None:
+        params['link_state'] = link_state
+
+    if spoof_check is not None:
+        params['spoof_check'] = spoof_check
+
+    if trust is not None:
+        params['trust'] = trust
+
+    if query_rss is not None:
+        params['query_rss'] = query_rss
+
+    # Write the new configuration to the API
+    response = call_api(config, 'put', '/sriov/vf/{node}/{vf}'.format(node=node, vf=vf), params=params)
+
+    if response.status_code == 200:
+        retstatus = True
+    else:
+        retstatus = False
+
+    return retstatus, response.json().get('message', '')
+
+
+def net_sriov_vf_list(config, node, pf=None):
+    """
+    List all VFs on NODE, optionally limited by PF
+
+    API endpoint: GET /api/v1/sriov/vf/<node>
+    API arguments: node={node}, pf={pf}
+    API schema: [{json_data_object},{json_data_object},etc.]
+    """
+    params = dict()
+    params['pf'] = pf
+
+    response = call_api(config, 'get', '/sriov/vf/{}'.format(node), params=params)
+
+    if response.status_code == 200:
+        return True, response.json()
+    else:
+        return False, response.json().get('message', '')
+
+
+def net_sriov_vf_info(config, node, vf):
+    """
+    Get info about VF on NODE
+
+    API endpoint: GET /api/v1/sriov/vf/<node>/<vf>
+    API arguments:
+    API schema: [{json_data_object}]
+    """
+    response = call_api(config, 'get', '/sriov/vf/{}/{}'.format(node, vf))
+
+    if response.status_code == 200:
+        if isinstance(response.json(), list) and len(response.json()) != 1:
+            # No exact match; return not found
+            return False, "VF not found."
+        else:
+            # Return a single instance if the response is a list
+            if isinstance(response.json(), list):
+                return True, response.json()[0]
+            # This shouldn't happen, but is here just in case
+            else:
+                return True, response.json()
+    else:
+        return False, response.json().get('message', '')
+
+
 #
 # Output display functions
 #
-def getOutputColours(network_information):
-    if network_information['ip6']['network'] != "None":
-        v6_flag_colour = ansiprint.green()
+def getColour(value):
+    if value in ["False", "None"]:
+        return ansiprint.blue()
     else:
-        v6_flag_colour = ansiprint.blue()
-    if network_information['ip4']['network'] != "None":
-        v4_flag_colour = ansiprint.green()
-    else:
-        v4_flag_colour = ansiprint.blue()
+        return ansiprint.green()
 
-    if network_information['ip6']['dhcp_flag'] == "True":
-        dhcp6_flag_colour = ansiprint.green()
-    else:
-        dhcp6_flag_colour = ansiprint.blue()
-    if network_information['ip4']['dhcp_flag'] == "True":
-        dhcp4_flag_colour = ansiprint.green()
-    else:
-        dhcp4_flag_colour = ansiprint.blue()
+
+def getOutputColours(network_information):
+    v6_flag_colour = getColour(network_information['ip6']['network'])
+    v4_flag_colour = getColour(network_information['ip4']['network'])
+    dhcp6_flag_colour = getColour(network_information['ip6']['dhcp_flag'])
+    dhcp4_flag_colour = getColour(network_information['ip4']['dhcp_flag'])
 
     return v6_flag_colour, v4_flag_colour, dhcp6_flag_colour, dhcp4_flag_colour
 
@@ -416,6 +522,7 @@ def format_info(config, network_information, long_output):
     # Basic information
     ainformation.append('{}VNI:{}            {}'.format(ansiprint.purple(), ansiprint.end(), network_information['vni']))
     ainformation.append('{}Type:{}           {}'.format(ansiprint.purple(), ansiprint.end(), network_information['type']))
+    ainformation.append('{}MTU:{}            {}'.format(ansiprint.purple(), ansiprint.end(), network_information['mtu']))
     ainformation.append('{}Description:{}    {}'.format(ansiprint.purple(), ansiprint.end(), network_information['description']))
     if network_information['type'] == 'managed':
         ainformation.append('{}Domain:{}         {}'.format(ansiprint.purple(), ansiprint.end(), network_information['domain']))
@@ -472,6 +579,7 @@ def format_list(config, network_list):
     net_vni_length = 5
     net_description_length = 12
     net_nettype_length = 8
+    net_mtu_length = 4
     net_domain_length = 6
     net_v6_flag_length = 6
     net_dhcp6_flag_length = 7
@@ -486,16 +594,29 @@ def format_list(config, network_list):
         _net_description_length = len(network_information['description']) + 1
         if _net_description_length > net_description_length:
             net_description_length = _net_description_length
+        # mtu column
+        _net_mtu_length = len(str(network_information['mtu'])) + 1
+        if _net_mtu_length > net_mtu_length:
+            net_mtu_length = _net_mtu_length
         # domain column
         _net_domain_length = len(network_information['domain']) + 1
         if _net_domain_length > net_domain_length:
             net_domain_length = _net_domain_length
 
     # Format the string (header)
+    network_list_output.append('{bold}{networks_header: <{networks_header_length}} {config_header: <{config_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        networks_header_length=net_vni_length + net_description_length + 1,
+        config_header_length=net_nettype_length + net_mtu_length + net_domain_length + net_v6_flag_length + net_dhcp6_flag_length + net_v4_flag_length + net_dhcp4_flag_length + 7,
+        networks_header='Networks ' + ''.join(['-' for _ in range(9, net_vni_length + net_description_length)]),
+        config_header='Config ' + ''.join(['-' for _ in range(7, net_nettype_length + net_mtu_length + net_domain_length + net_v6_flag_length + net_dhcp6_flag_length + net_v4_flag_length + net_dhcp4_flag_length + 6)]))
+    )
     network_list_output.append('{bold}\
 {net_vni: <{net_vni_length}} \
 {net_description: <{net_description_length}} \
 {net_nettype: <{net_nettype_length}} \
+{net_mtu: <{net_mtu_length}} \
 {net_domain: <{net_domain_length}}  \
 {net_v6_flag: <{net_v6_flag_length}} \
 {net_dhcp6_flag: <{net_dhcp6_flag_length}} \
@@ -507,6 +628,7 @@ def format_list(config, network_list):
         net_vni_length=net_vni_length,
         net_description_length=net_description_length,
         net_nettype_length=net_nettype_length,
+        net_mtu_length=net_mtu_length,
         net_domain_length=net_domain_length,
         net_v6_flag_length=net_v6_flag_length,
         net_dhcp6_flag_length=net_dhcp6_flag_length,
@@ -515,6 +637,7 @@ def format_list(config, network_list):
         net_vni='VNI',
         net_description='Description',
         net_nettype='Type',
+        net_mtu='MTU',
         net_domain='Domain',
         net_v6_flag='IPv6',
         net_dhcp6_flag='DHCPv6',
@@ -522,7 +645,7 @@ def format_list(config, network_list):
         net_dhcp4_flag='DHCPv4')
     )
 
-    for network_information in network_list:
+    for network_information in sorted(network_list, key=lambda n: int(n['vni'])):
         v6_flag_colour, v4_flag_colour, dhcp6_flag_colour, dhcp4_flag_colour = getOutputColours(network_information)
         if network_information['ip4']['network'] != "None":
             v4_flag = 'True'
@@ -538,6 +661,7 @@ def format_list(config, network_list):
 {net_vni: <{net_vni_length}} \
 {net_description: <{net_description_length}} \
 {net_nettype: <{net_nettype_length}} \
+{net_mtu: <{net_mtu_length}} \
 {net_domain: <{net_domain_length}}  \
 {v6_flag_colour}{net_v6_flag: <{net_v6_flag_length}}{colour_off} \
 {dhcp6_flag_colour}{net_dhcp6_flag: <{net_dhcp6_flag_length}}{colour_off} \
@@ -549,6 +673,7 @@ def format_list(config, network_list):
             net_vni_length=net_vni_length,
             net_description_length=net_description_length,
             net_nettype_length=net_nettype_length,
+            net_mtu_length=net_mtu_length,
             net_domain_length=net_domain_length,
             net_v6_flag_length=net_v6_flag_length,
             net_dhcp6_flag_length=net_dhcp6_flag_length,
@@ -557,6 +682,7 @@ def format_list(config, network_list):
             net_vni=network_information['vni'],
             net_description=network_information['description'],
             net_nettype=network_information['type'],
+            net_mtu=network_information['mtu'],
             net_domain=network_information['domain'],
             net_v6_flag=v6_flag,
             v6_flag_colour=v6_flag_colour,
@@ -569,7 +695,7 @@ def format_list(config, network_list):
             colour_off=ansiprint.end())
         )
 
-    return '\n'.join(sorted(network_list_output))
+    return '\n'.join(network_list_output)
 
 
 def format_list_dhcp(dhcp_lease_list):
@@ -579,7 +705,7 @@ def format_list_dhcp(dhcp_lease_list):
     lease_hostname_length = 9
     lease_ip4_address_length = 11
     lease_mac_address_length = 13
-    lease_timestamp_length = 13
+    lease_timestamp_length = 10
     for dhcp_lease_information in dhcp_lease_list:
         # hostname column
         _lease_hostname_length = len(str(dhcp_lease_information['hostname'])) + 1
@@ -593,8 +719,19 @@ def format_list_dhcp(dhcp_lease_list):
         _lease_mac_address_length = len(str(dhcp_lease_information['mac_address'])) + 1
         if _lease_mac_address_length > lease_mac_address_length:
             lease_mac_address_length = _lease_mac_address_length
+        # timestamp column
+        _lease_timestamp_length = len(str(dhcp_lease_information['timestamp'])) + 1
+        if _lease_timestamp_length > lease_timestamp_length:
+            lease_timestamp_length = _lease_timestamp_length
 
     # Format the string (header)
+    dhcp_lease_list_output.append('{bold}{lease_header: <{lease_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        lease_header_length=lease_hostname_length + lease_ip4_address_length + lease_mac_address_length + lease_timestamp_length + 3,
+        lease_header='Leases ' + ''.join(['-' for _ in range(7, lease_hostname_length + lease_ip4_address_length + lease_mac_address_length + lease_timestamp_length + 2)]))
+    )
+
     dhcp_lease_list_output.append('{bold}\
 {lease_hostname: <{lease_hostname_length}} \
 {lease_ip4_address: <{lease_ip4_address_length}} \
@@ -613,7 +750,7 @@ def format_list_dhcp(dhcp_lease_list):
         lease_timestamp='Timestamp')
     )
 
-    for dhcp_lease_information in dhcp_lease_list:
+    for dhcp_lease_information in sorted(dhcp_lease_list, key=lambda l: l['hostname']):
         dhcp_lease_list_output.append('{bold}\
 {lease_hostname: <{lease_hostname_length}} \
 {lease_ip4_address: <{lease_ip4_address_length}} \
@@ -632,7 +769,7 @@ def format_list_dhcp(dhcp_lease_list):
             lease_timestamp=str(dhcp_lease_information['timestamp']))
         )
 
-    return '\n'.join(sorted(dhcp_lease_list_output))
+    return '\n'.join(dhcp_lease_list_output)
 
 
 def format_list_acl(acl_list):
@@ -662,6 +799,13 @@ def format_list_acl(acl_list):
             acl_rule_length = _acl_rule_length
 
     # Format the string (header)
+    acl_list_output.append('{bold}{acl_header: <{acl_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        acl_header_length=acl_direction_length + acl_order_length + acl_description_length + acl_rule_length + 3,
+        acl_header='ACLs ' + ''.join(['-' for _ in range(5, acl_direction_length + acl_order_length + acl_description_length + acl_rule_length + 2)]))
+    )
+
     acl_list_output.append('{bold}\
 {acl_direction: <{acl_direction_length}} \
 {acl_order: <{acl_order_length}} \
@@ -680,7 +824,7 @@ def format_list_acl(acl_list):
         acl_rule='Rule')
     )
 
-    for acl_information in acl_list:
+    for acl_information in sorted(acl_list, key=lambda l: l['direction'] + str(l['order'])):
         acl_list_output.append('{bold}\
 {acl_direction: <{acl_direction_length}} \
 {acl_order: <{acl_order_length}} \
@@ -699,4 +843,264 @@ def format_list_acl(acl_list):
             acl_rule=acl_information['rule'])
         )
 
-    return '\n'.join(sorted(acl_list_output))
+    return '\n'.join(acl_list_output)
+
+
+def format_list_sriov_pf(pf_list):
+    # The maximum column width of the VFs column
+    max_vfs_length = 70
+
+    # Handle when we get an empty entry
+    if not pf_list:
+        pf_list = list()
+
+    pf_list_output = []
+
+    # Determine optimal column widths
+    pf_phy_length = 6
+    pf_mtu_length = 4
+    pf_vfs_length = 4
+
+    for pf_information in pf_list:
+        # phy column
+        _pf_phy_length = len(str(pf_information['phy'])) + 1
+        if _pf_phy_length > pf_phy_length:
+            pf_phy_length = _pf_phy_length
+        # mtu column
+        _pf_mtu_length = len(str(pf_information['mtu'])) + 1
+        if _pf_mtu_length > pf_mtu_length:
+            pf_mtu_length = _pf_mtu_length
+        # vfs column
+        _pf_vfs_length = len(str(', '.join(pf_information['vfs']))) + 1
+        if _pf_vfs_length > pf_vfs_length:
+            pf_vfs_length = _pf_vfs_length
+
+    # We handle columnizing very long lists later
+    if pf_vfs_length > max_vfs_length:
+        pf_vfs_length = max_vfs_length
+
+    # Format the string (header)
+    pf_list_output.append('{bold}{pf_header: <{pf_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        pf_header_length=pf_phy_length + pf_mtu_length + pf_vfs_length + 2,
+        pf_header='PFs ' + ''.join(['-' for _ in range(4, pf_phy_length + pf_mtu_length + pf_vfs_length + 1)]))
+    )
+
+    pf_list_output.append('{bold}\
+{pf_phy: <{pf_phy_length}} \
+{pf_mtu: <{pf_mtu_length}} \
+{pf_vfs: <{pf_vfs_length}} \
+{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        pf_phy_length=pf_phy_length,
+        pf_mtu_length=pf_mtu_length,
+        pf_vfs_length=pf_vfs_length,
+        pf_phy='Device',
+        pf_mtu='MTU',
+        pf_vfs='VFs')
+    )
+
+    for pf_information in sorted(pf_list, key=lambda p: p['phy']):
+        # Figure out how to nicely columnize our list
+        nice_vfs_list = [list()]
+        vfs_lines = 0
+        cur_vfs_length = 0
+        for vfs in pf_information['vfs']:
+            vfs_len = len(vfs)
+            cur_vfs_length += vfs_len + 2  # for the comma and space
+            if cur_vfs_length > max_vfs_length:
+                cur_vfs_length = 0
+                vfs_lines += 1
+                nice_vfs_list.append(list())
+            nice_vfs_list[vfs_lines].append(vfs)
+
+        # Append the lines
+        pf_list_output.append('{bold}\
+{pf_phy: <{pf_phy_length}} \
+{pf_mtu: <{pf_mtu_length}} \
+{pf_vfs: <{pf_vfs_length}} \
+{end_bold}'.format(
+            bold='',
+            end_bold='',
+            pf_phy_length=pf_phy_length,
+            pf_mtu_length=pf_mtu_length,
+            pf_vfs_length=pf_vfs_length,
+            pf_phy=pf_information['phy'],
+            pf_mtu=pf_information['mtu'],
+            pf_vfs=', '.join(nice_vfs_list[0]))
+        )
+
+        if len(nice_vfs_list) > 1:
+            for idx in range(1, len(nice_vfs_list)):
+                pf_list_output.append('{bold}\
+{pf_phy: <{pf_phy_length}} \
+{pf_mtu: <{pf_mtu_length}} \
+{pf_vfs: <{pf_vfs_length}} \
+{end_bold}'.format(
+                    bold='',
+                    end_bold='',
+                    pf_phy_length=pf_phy_length,
+                    pf_mtu_length=pf_mtu_length,
+                    pf_vfs_length=pf_vfs_length,
+                    pf_phy='',
+                    pf_mtu='',
+                    pf_vfs=', '.join(nice_vfs_list[idx]))
+                )
+
+    return '\n'.join(pf_list_output)
+
+
+def format_list_sriov_vf(vf_list):
+    # Handle when we get an empty entry
+    if not vf_list:
+        vf_list = list()
+
+    vf_list_output = []
+
+    # Determine optimal column widths
+    vf_phy_length = 4
+    vf_pf_length = 3
+    vf_mtu_length = 4
+    vf_mac_length = 11
+    vf_used_length = 5
+    vf_domain_length = 5
+
+    for vf_information in vf_list:
+        # phy column
+        _vf_phy_length = len(str(vf_information['phy'])) + 1
+        if _vf_phy_length > vf_phy_length:
+            vf_phy_length = _vf_phy_length
+        # pf column
+        _vf_pf_length = len(str(vf_information['pf'])) + 1
+        if _vf_pf_length > vf_pf_length:
+            vf_pf_length = _vf_pf_length
+        # mtu column
+        _vf_mtu_length = len(str(vf_information['mtu'])) + 1
+        if _vf_mtu_length > vf_mtu_length:
+            vf_mtu_length = _vf_mtu_length
+        # mac column
+        _vf_mac_length = len(str(vf_information['mac'])) + 1
+        if _vf_mac_length > vf_mac_length:
+            vf_mac_length = _vf_mac_length
+        # used column
+        _vf_used_length = len(str(vf_information['usage']['used'])) + 1
+        if _vf_used_length > vf_used_length:
+            vf_used_length = _vf_used_length
+        # domain column
+        _vf_domain_length = len(str(vf_information['usage']['domain'])) + 1
+        if _vf_domain_length > vf_domain_length:
+            vf_domain_length = _vf_domain_length
+
+    # Format the string (header)
+    vf_list_output.append('{bold}{vf_header: <{vf_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        vf_header_length=vf_phy_length + vf_pf_length + vf_mtu_length + vf_mac_length + vf_used_length + vf_domain_length + 5,
+        vf_header='VFs ' + ''.join(['-' for _ in range(4, vf_phy_length + vf_pf_length + vf_mtu_length + vf_mac_length + vf_used_length + vf_domain_length + 4)]))
+    )
+
+    vf_list_output.append('{bold}\
+{vf_phy: <{vf_phy_length}} \
+{vf_pf: <{vf_pf_length}} \
+{vf_mtu: <{vf_mtu_length}} \
+{vf_mac: <{vf_mac_length}} \
+{vf_used: <{vf_used_length}} \
+{vf_domain: <{vf_domain_length}} \
+{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        vf_phy_length=vf_phy_length,
+        vf_pf_length=vf_pf_length,
+        vf_mtu_length=vf_mtu_length,
+        vf_mac_length=vf_mac_length,
+        vf_used_length=vf_used_length,
+        vf_domain_length=vf_domain_length,
+        vf_phy='Device',
+        vf_pf='PF',
+        vf_mtu='MTU',
+        vf_mac='MAC Address',
+        vf_used='Used',
+        vf_domain='Domain')
+    )
+
+    for vf_information in sorted(vf_list, key=lambda v: v['phy']):
+        vf_domain = vf_information['usage']['domain']
+        if not vf_domain:
+            vf_domain = 'N/A'
+
+        vf_list_output.append('{bold}\
+{vf_phy: <{vf_phy_length}} \
+{vf_pf: <{vf_pf_length}} \
+{vf_mtu: <{vf_mtu_length}} \
+{vf_mac: <{vf_mac_length}} \
+{vf_used: <{vf_used_length}} \
+{vf_domain: <{vf_domain_length}} \
+{end_bold}'.format(
+            bold=ansiprint.bold(),
+            end_bold=ansiprint.end(),
+            vf_phy_length=vf_phy_length,
+            vf_pf_length=vf_pf_length,
+            vf_mtu_length=vf_mtu_length,
+            vf_mac_length=vf_mac_length,
+            vf_used_length=vf_used_length,
+            vf_domain_length=vf_domain_length,
+            vf_phy=vf_information['phy'],
+            vf_pf=vf_information['pf'],
+            vf_mtu=vf_information['mtu'],
+            vf_mac=vf_information['mac'],
+            vf_used=vf_information['usage']['used'],
+            vf_domain=vf_domain)
+        )
+
+    return '\n'.join(vf_list_output)
+
+
+def format_info_sriov_vf(config, vf_information, node):
+    if not vf_information:
+        return "No VF found"
+
+    # Get information on the using VM if applicable
+    if vf_information['usage']['used'] == 'True' and vf_information['usage']['domain']:
+        vm_information = call_api(config, 'get', '/vm/{vm}'.format(vm=vf_information['usage']['domain'])).json()
+        if isinstance(vm_information, list) and len(vm_information) > 0:
+            vm_information = vm_information[0]
+        else:
+            vm_information = None
+
+    # Format a nice output: do this line-by-line then concat the elements at the end
+    ainformation = []
+    ainformation.append('{}SR-IOV VF information:{}'.format(ansiprint.bold(), ansiprint.end()))
+    ainformation.append('')
+    # Basic information
+    ainformation.append('{}PHY:{}               {}'.format(ansiprint.purple(), ansiprint.end(), vf_information['phy']))
+    ainformation.append('{}PF:{}                {} @ {}'.format(ansiprint.purple(), ansiprint.end(), vf_information['pf'], node))
+    ainformation.append('{}MTU:{}               {}'.format(ansiprint.purple(), ansiprint.end(), vf_information['mtu']))
+    ainformation.append('{}MAC Address:{}       {}'.format(ansiprint.purple(), ansiprint.end(), vf_information['mac']))
+    ainformation.append('')
+    # Configuration information
+    ainformation.append('{}vLAN ID:{}           {}'.format(ansiprint.purple(), ansiprint.end(), vf_information['config']['vlan_id']))
+    ainformation.append('{}vLAN QOS priority:{} {}'.format(ansiprint.purple(), ansiprint.end(), vf_information['config']['vlan_qos']))
+    ainformation.append('{}Minimum TX Rate:{}   {}'.format(ansiprint.purple(), ansiprint.end(), vf_information['config']['tx_rate_min']))
+    ainformation.append('{}Maximum TX Rate:{}   {}'.format(ansiprint.purple(), ansiprint.end(), vf_information['config']['tx_rate_max']))
+    ainformation.append('{}Link State:{}        {}'.format(ansiprint.purple(), ansiprint.end(), vf_information['config']['link_state']))
+    ainformation.append('{}Spoof Checking:{}    {}{}{}'.format(ansiprint.purple(), ansiprint.end(), getColour(vf_information['config']['spoof_check']), vf_information['config']['spoof_check'], ansiprint.end()))
+    ainformation.append('{}VF User Trust:{}     {}{}{}'.format(ansiprint.purple(), ansiprint.end(), getColour(vf_information['config']['trust']), vf_information['config']['trust'], ansiprint.end()))
+    ainformation.append('{}Query RSS Config:{}  {}{}{}'.format(ansiprint.purple(), ansiprint.end(), getColour(vf_information['config']['query_rss']), vf_information['config']['query_rss'], ansiprint.end()))
+    ainformation.append('')
+    # PCIe bus information
+    ainformation.append('{}PCIe domain:{}       {}'.format(ansiprint.purple(), ansiprint.end(), vf_information['pci']['domain']))
+    ainformation.append('{}PCIe bus:{}          {}'.format(ansiprint.purple(), ansiprint.end(), vf_information['pci']['bus']))
+    ainformation.append('{}PCIe slot:{}         {}'.format(ansiprint.purple(), ansiprint.end(), vf_information['pci']['slot']))
+    ainformation.append('{}PCIe function:{}     {}'.format(ansiprint.purple(), ansiprint.end(), vf_information['pci']['function']))
+    ainformation.append('')
+    # Usage information
+    ainformation.append('{}VF Used:{}           {}{}{}'.format(ansiprint.purple(), ansiprint.end(), getColour(vf_information['usage']['used']), vf_information['usage']['used'], ansiprint.end()))
+    if vf_information['usage']['used'] == 'True' and vm_information is not None:
+        ainformation.append('{}Using Domain:{}      {} ({}) ({}{}{})'.format(ansiprint.purple(), ansiprint.end(), vf_information['usage']['domain'], vm_information['name'], getColour(vm_information['state']), vm_information['state'], ansiprint.end()))
+    else:
+        ainformation.append('{}Using Domain:{}      N/A'.format(ansiprint.purple(), ansiprint.end()))
+
+    # Join it all together
+    return '\n'.join(ainformation)

client-cli/pvc/cli_lib/node.py

@@ -19,8 +19,10 @@
 #
 ###############################################################################
 
-import cli_lib.ansiprint as ansiprint
-from cli_lib.common import call_api
+import time
+
+import pvc.cli_lib.ansiprint as ansiprint
+from pvc.cli_lib.common import call_api
 
 
 #
@@ -69,6 +71,89 @@ def node_domain_state(config, node, action, wait):
     return retstatus, response.json().get('message', '')
 
 
+def view_node_log(config, node, lines=100):
+    """
+    Return node log lines from the API (and display them in a pager in the main CLI)
+
+    API endpoint: GET /node/{node}/log
+    API arguments: lines={lines}
+    API schema: {"name":"{node}","data":"{node_log}"}
+    """
+    params = {
+        'lines': lines
+    }
+    response = call_api(config, 'get', '/node/{node}/log'.format(node=node), params=params)
+
+    if response.status_code != 200:
+        return False, response.json().get('message', '')
+
+    node_log = response.json()['data']
+
+    # Shrink the log buffer to length lines
+    shrunk_log = node_log.split('\n')[-lines:]
+    loglines = '\n'.join(shrunk_log)
+
+    return True, loglines
+
+
+def follow_node_log(config, node, lines=10):
+    """
+    Return and follow node log lines from the API
+
+    API endpoint: GET /node/{node}/log
+    API arguments: lines={lines}
+    API schema: {"name":"{nodename}","data":"{node_log}"}
+    """
+    # We always grab 200 to match the follow call, but only _show_ `lines` number
+    params = {
+        'lines': 200
+    }
+    response = call_api(config, 'get', '/node/{node}/log'.format(node=node), params=params)
+
+    if response.status_code != 200:
+        return False, response.json().get('message', '')
+
+    # Shrink the log buffer to length lines
+    node_log = response.json()['data']
+    shrunk_log = node_log.split('\n')[-int(lines):]
+    loglines = '\n'.join(shrunk_log)
+
+    # Print the initial data and begin following
+    print(loglines, end='')
+    print('\n', end='')
+
+    while True:
+        # Grab the next line set (200 is a reasonable number of lines per half-second; any more are skipped)
+        try:
+            params = {
+                'lines': 200
+            }
+            response = call_api(config, 'get', '/node/{node}/log'.format(node=node), params=params)
+            new_node_log = response.json()['data']
+        except Exception:
+            break
+        # Split the new and old log strings into constitutent lines
+        old_node_loglines = node_log.split('\n')
+        new_node_loglines = new_node_log.split('\n')
+
+        # Set the node log to the new log value for the next iteration
+        node_log = new_node_log
+
+        # Get the difference between the two sets of lines
+        old_node_loglines_set = set(old_node_loglines)
+        diff_node_loglines = [x for x in new_node_loglines if x not in old_node_loglines_set]
+
+        # If there's a difference, print it out
+        if len(diff_node_loglines) > 0:
+            print('\n'.join(diff_node_loglines), end='')
+            print('\n', end='')
+
+        # Wait half a second
+        time.sleep(0.5)
+
+    return True, ''
+
+
 def node_info(config, node):
     """
     Get information about node
@@ -169,6 +254,7 @@ def format_info(node_information, long_output):
     ainformation = []
     # Basic information
     ainformation.append('{}Name:{}                  {}'.format(ansiprint.purple(), ansiprint.end(), node_information['name']))
+    ainformation.append('{}PVC Version:{}           {}'.format(ansiprint.purple(), ansiprint.end(), node_information['pvc_version']))
     ainformation.append('{}Daemon State:{}          {}{}{}'.format(ansiprint.purple(), ansiprint.end(), daemon_state_colour, node_information['daemon_state'], ansiprint.end()))
     ainformation.append('{}Coordinator State:{}     {}{}{}'.format(ansiprint.purple(), ansiprint.end(), coordinator_state_colour, node_information['coordinator_state'], ansiprint.end()))
     ainformation.append('{}Domain State:{}          {}{}{}'.format(ansiprint.purple(), ansiprint.end(), domain_state_colour, node_information['domain_state'], ansiprint.end()))
@@ -204,9 +290,10 @@ def format_list(node_list, raw):
 
     # Determine optimal column widths
     node_name_length = 5
+    pvc_version_length = 8
     daemon_state_length = 7
     coordinator_state_length = 12
-    domain_state_length = 8
+    domain_state_length = 7
     domains_count_length = 4
     cpu_count_length = 6
     load_length = 5
@@ -220,6 +307,10 @@ def format_list(node_list, raw):
         _node_name_length = len(node_information['name']) + 1
         if _node_name_length > node_name_length:
             node_name_length = _node_name_length
+        # node_pvc_version column
+        _pvc_version_length = len(node_information.get('pvc_version', 'N/A')) + 1
+        if _pvc_version_length > pvc_version_length:
+            pvc_version_length = _pvc_version_length
         # daemon_state column
         _daemon_state_length = len(node_information['daemon_state']) + 1
         if _daemon_state_length > daemon_state_length:
@@ -268,11 +359,27 @@ def format_list(node_list, raw):
 
     # Format the string (header)
     node_list_output.append(
-        '{bold}{node_name: <{node_name_length}} \
-St: {daemon_state_colour}{node_daemon_state: <{daemon_state_length}}{end_colour} {coordinator_state_colour}{node_coordinator_state: <{coordinator_state_length}}{end_colour} {domain_state_colour}{node_domain_state: <{domain_state_length}}{end_colour} \
-Res: {node_domains_count: <{domains_count_length}} {node_cpu_count: <{cpu_count_length}} {node_load: <{load_length}} \
-Mem (M): {node_mem_total: <{mem_total_length}} {node_mem_used: <{mem_used_length}} {node_mem_free: <{mem_free_length}} {node_mem_allocated: <{mem_alloc_length}} {node_mem_provisioned: <{mem_prov_length}}{end_bold}'.format(
+        '{bold}{node_header: <{node_header_length}} {state_header: <{state_header_length}} {resource_header: <{resource_header_length}} {memory_header: <{memory_header_length}}{end_bold}'.format(
+            node_header_length=node_name_length + pvc_version_length + 1,
+            state_header_length=daemon_state_length + coordinator_state_length + domain_state_length + 2,
+            resource_header_length=domains_count_length + cpu_count_length + load_length + 2,
+            memory_header_length=mem_total_length + mem_used_length + mem_free_length + mem_alloc_length + mem_prov_length + 4,
+            bold=ansiprint.bold(),
+            end_bold=ansiprint.end(),
+            node_header='Nodes ' + ''.join(['-' for _ in range(6, node_name_length + pvc_version_length)]),
+            state_header='States ' + ''.join(['-' for _ in range(7, daemon_state_length + coordinator_state_length + domain_state_length + 1)]),
+            resource_header='Resources ' + ''.join(['-' for _ in range(10, domains_count_length + cpu_count_length + load_length + 1)]),
+            memory_header='Memory (M) ' + ''.join(['-' for _ in range(11, mem_total_length + mem_used_length + mem_free_length + mem_alloc_length + mem_prov_length + 3)])
+        )
+    )
+
+    node_list_output.append(
+        '{bold}{node_name: <{node_name_length}} {node_pvc_version: <{pvc_version_length}} \
+{daemon_state_colour}{node_daemon_state: <{daemon_state_length}}{end_colour} {coordinator_state_colour}{node_coordinator_state: <{coordinator_state_length}}{end_colour} {domain_state_colour}{node_domain_state: <{domain_state_length}}{end_colour} \
+{node_domains_count: <{domains_count_length}} {node_cpu_count: <{cpu_count_length}} {node_load: <{load_length}} \
+{node_mem_total: <{mem_total_length}} {node_mem_used: <{mem_used_length}} {node_mem_free: <{mem_free_length}} {node_mem_allocated: <{mem_alloc_length}} {node_mem_provisioned: <{mem_prov_length}}{end_bold}'.format(
             node_name_length=node_name_length,
+            pvc_version_length=pvc_version_length,
             daemon_state_length=daemon_state_length,
             coordinator_state_length=coordinator_state_length,
             domain_state_length=domain_state_length,
@@ -291,6 +398,7 @@ Mem (M): {node_mem_total: <{mem_total_length}} {node_mem_used: <{mem_used_length
             domain_state_colour='',
             end_colour='',
             node_name='Name',
+            node_pvc_version='Version',
             node_daemon_state='Daemon',
             node_coordinator_state='Coordinator',
             node_domain_state='Domain',
@@ -306,14 +414,15 @@ Mem (M): {node_mem_total: <{mem_total_length}} {node_mem_used: <{mem_used_length
     )
 
     # Format the string (elements)
-    for node_information in node_list:
+    for node_information in sorted(node_list, key=lambda n: n['name']):
         daemon_state_colour, coordinator_state_colour, domain_state_colour, mem_allocated_colour, mem_provisioned_colour = getOutputColours(node_information)
         node_list_output.append(
-            '{bold}{node_name: <{node_name_length}} \
-    {daemon_state_colour}{node_daemon_state: <{daemon_state_length}}{end_colour} {coordinator_state_colour}{node_coordinator_state: <{coordinator_state_length}}{end_colour} {domain_state_colour}{node_domain_state: <{domain_state_length}}{end_colour} \
-     {node_domains_count: <{domains_count_length}} {node_cpu_count: <{cpu_count_length}} {node_load: <{load_length}} \
-         {node_mem_total: <{mem_total_length}} {node_mem_used: <{mem_used_length}} {node_mem_free: <{mem_free_length}} {mem_allocated_colour}{node_mem_allocated: <{mem_alloc_length}}{end_colour} {mem_provisioned_colour}{node_mem_provisioned: <{mem_prov_length}}{end_colour}{end_bold}'.format(
+            '{bold}{node_name: <{node_name_length}} {node_pvc_version: <{pvc_version_length}} \
+{daemon_state_colour}{node_daemon_state: <{daemon_state_length}}{end_colour} {coordinator_state_colour}{node_coordinator_state: <{coordinator_state_length}}{end_colour} {domain_state_colour}{node_domain_state: <{domain_state_length}}{end_colour} \
+{node_domains_count: <{domains_count_length}} {node_cpu_count: <{cpu_count_length}} {node_load: <{load_length}} \
+{node_mem_total: <{mem_total_length}} {node_mem_used: <{mem_used_length}} {node_mem_free: <{mem_free_length}} {mem_allocated_colour}{node_mem_allocated: <{mem_alloc_length}}{end_colour} {mem_provisioned_colour}{node_mem_provisioned: <{mem_prov_length}}{end_colour}{end_bold}'.format(
                 node_name_length=node_name_length,
+                pvc_version_length=pvc_version_length,
                 daemon_state_length=daemon_state_length,
                 coordinator_state_length=coordinator_state_length,
                 domain_state_length=domain_state_length,
@@ -334,6 +443,7 @@ Mem (M): {node_mem_total: <{mem_total_length}} {node_mem_used: <{mem_used_length
                 mem_provisioned_colour=mem_allocated_colour,
                 end_colour=ansiprint.end(),
                 node_name=node_information['name'],
+                node_pvc_version=node_information.get('pvc_version', 'N/A'),
                 node_daemon_state=node_information['daemon_state'],
                 node_coordinator_state=node_information['coordinator_state'],
                 node_domain_state=node_information['domain_state'],
@@ -348,4 +458,4 @@ Mem (M): {node_mem_total: <{mem_total_length}} {node_mem_used: <{mem_used_length
             )
         )
 
-    return '\n'.join(sorted(node_list_output))
+    return '\n'.join(node_list_output)

client-cli/pvc/cli_lib/provisioner.py

@@ -20,9 +20,10 @@
 ###############################################################################
 
 from requests_toolbelt.multipart.encoder import MultipartEncoder, MultipartEncoderMonitor
+from ast import literal_eval
 
-import cli_lib.ansiprint as ansiprint
-from cli_lib.common import UploadProgressBar, call_api
+import pvc.cli_lib.ansiprint as ansiprint
+from pvc.cli_lib.common import UploadProgressBar, call_api
 
 
 #
@@ -707,7 +708,11 @@ def task_status(config, task_id=None, is_watching=False):
         # Format the Celery data into a more useful data structure
         task_data = list()
         for task_type in ['active', 'reserved', 'scheduled']:
-            type_data = task_data_raw[task_type]
+            try:
+                type_data = task_data_raw[task_type]
+            except Exception:
+                type_data = None
+
             if not type_data:
                 type_data = dict()
             for task_host in type_data:
@@ -719,10 +724,10 @@ def task_status(config, task_id=None, is_watching=False):
                         task['type'] = task_type
                     task['worker'] = task_host
                     task['id'] = task_job.get('id')
-                    task_args = task_job.get('args')
+                    task_args = literal_eval(task_job.get('args'))
                     task['vm_name'] = task_args[0]
                     task['vm_profile'] = task_args[1]
-                    task_kwargs = task_job.get('kwargs')
+                    task_kwargs = literal_eval(task_job.get('kwargs'))
                     task['vm_define'] = str(bool(task_kwargs['define_vm']))
                     task['vm_start'] = str(bool(task_kwargs['start_vm']))
                     task_data.append(task)
@@ -753,22 +758,16 @@ def format_list_template(template_data, template_type=None):
         normalized_template_data = template_data
 
     if 'system' in template_types:
-        ainformation.append('System templates:')
-        ainformation.append('')
         ainformation.append(format_list_template_system(normalized_template_data['system_templates']))
         if len(template_types) > 1:
             ainformation.append('')
 
     if 'network' in template_types:
-        ainformation.append('Network templates:')
-        ainformation.append('')
         ainformation.append(format_list_template_network(normalized_template_data['network_templates']))
         if len(template_types) > 1:
             ainformation.append('')
 
     if 'storage' in template_types:
-        ainformation.append('Storage templates:')
-        ainformation.append('')
         ainformation.append(format_list_template_storage(normalized_template_data['storage_templates']))
 
     return '\n'.join(ainformation)
@@ -781,13 +780,13 @@ def format_list_template_system(template_data):
     template_list_output = []
 
     # Determine optimal column widths
-    template_name_length = 5
-    template_id_length = 3
+    template_name_length = 15
+    template_id_length = 5
     template_vcpu_length = 6
-    template_vram_length = 10
+    template_vram_length = 9
     template_serial_length = 7
     template_vnc_length = 4
-    template_vnc_bind_length = 10
+    template_vnc_bind_length = 9
     template_node_limit_length = 6
     template_node_selector_length = 9
     template_node_autostart_length = 10
@@ -840,16 +839,33 @@ def format_list_template_system(template_data):
             template_migration_method_length = _template_migration_method_length
 
     # Format the string (header)
-    template_list_output_header = '{bold}{template_name: <{template_name_length}} {template_id: <{template_id_length}} \
+    template_list_output.append('{bold}{template_header: <{template_header_length}} {resources_header: <{resources_header_length}} {consoles_header: <{consoles_header_length}} {metadata_header: <{metadata_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        template_header_length=template_name_length + template_id_length + 1,
+        resources_header_length=template_vcpu_length + template_vram_length + 1,
+        consoles_header_length=template_serial_length + template_vnc_length + template_vnc_bind_length + 2,
+        metadata_header_length=template_node_limit_length + template_node_selector_length + template_node_autostart_length + template_migration_method_length + 3,
+        template_header='System Templates ' + ''.join(['-' for _ in range(17, template_name_length + template_id_length)]),
+        resources_header='Resources ' + ''.join(['-' for _ in range(10, template_vcpu_length + template_vram_length)]),
+        consoles_header='Consoles ' + ''.join(['-' for _ in range(9, template_serial_length + template_vnc_length + template_vnc_bind_length + 1)]),
+        metadata_header='Metadata ' + ''.join(['-' for _ in range(9, template_node_limit_length + template_node_selector_length + template_node_autostart_length + template_migration_method_length + 2)]))
+    )
+
+    template_list_output.append('{bold}{template_name: <{template_name_length}} {template_id: <{template_id_length}} \
 {template_vcpu: <{template_vcpu_length}} \
 {template_vram: <{template_vram_length}} \
-Console: {template_serial: <{template_serial_length}} \
+{template_serial: <{template_serial_length}} \
 {template_vnc: <{template_vnc_length}} \
 {template_vnc_bind: <{template_vnc_bind_length}} \
-Meta: {template_node_limit: <{template_node_limit_length}} \
+{template_node_limit: <{template_node_limit_length}} \
 {template_node_selector: <{template_node_selector_length}} \
 {template_node_autostart: <{template_node_autostart_length}} \
 {template_migration_method: <{template_migration_method_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        template_state_colour='',
+        end_colour='',
         template_name_length=template_name_length,
         template_id_length=template_id_length,
         template_vcpu_length=template_vcpu_length,
@@ -861,14 +877,10 @@ Meta: {template_node_limit: <{template_node_limit_length}} \
         template_node_selector_length=template_node_selector_length,
         template_node_autostart_length=template_node_autostart_length,
         template_migration_method_length=template_migration_method_length,
-        bold=ansiprint.bold(),
-        end_bold=ansiprint.end(),
-        template_state_colour='',
-        end_colour='',
         template_name='Name',
         template_id='ID',
         template_vcpu='vCPUs',
-        template_vram='vRAM [MB]',
+        template_vram='vRAM [M]',
         template_serial='Serial',
         template_vnc='VNC',
         template_vnc_bind='VNC bind',
@@ -876,6 +888,7 @@ Meta: {template_node_limit: <{template_node_limit_length}} \
         template_node_selector='Selector',
         template_node_autostart='Autostart',
         template_migration_method='Migration')
+    )
 
     # Format the string (elements)
     for template in sorted(template_data, key=lambda i: i.get('name', None)):
@@ -883,10 +896,10 @@ Meta: {template_node_limit: <{template_node_limit_length}} \
             '{bold}{template_name: <{template_name_length}} {template_id: <{template_id_length}} \
 {template_vcpu: <{template_vcpu_length}} \
 {template_vram: <{template_vram_length}} \
-         {template_serial: <{template_serial_length}} \
+{template_serial: <{template_serial_length}} \
 {template_vnc: <{template_vnc_length}} \
 {template_vnc_bind: <{template_vnc_bind_length}} \
-      {template_node_limit: <{template_node_limit_length}} \
+{template_node_limit: <{template_node_limit_length}} \
 {template_node_selector: <{template_node_selector_length}} \
 {template_node_autostart: <{template_node_autostart_length}} \
 {template_migration_method: <{template_migration_method_length}}{end_bold}'.format(
@@ -917,9 +930,7 @@ Meta: {template_node_limit: <{template_node_limit_length}} \
             )
         )
 
-    return '\n'.join([template_list_output_header] + template_list_output)
-
-    return True, ''
+    return '\n'.join(template_list_output)
 
 
 def format_list_template_network(template_template):
@@ -929,8 +940,8 @@ def format_list_template_network(template_template):
     template_list_output = []
 
     # Determine optimal column widths
-    template_name_length = 5
-    template_id_length = 3
+    template_name_length = 18
+    template_id_length = 5
     template_mac_template_length = 13
     template_networks_length = 10
 
@@ -960,7 +971,16 @@ def format_list_template_network(template_template):
             template_networks_length = _template_networks_length
 
     # Format the string (header)
-    template_list_output_header = '{bold}{template_name: <{template_name_length}} {template_id: <{template_id_length}} \
+    template_list_output.append('{bold}{template_header: <{template_header_length}} {details_header: <{details_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        template_header_length=template_name_length + template_id_length + 1,
+        details_header_length=template_mac_template_length + template_networks_length + 1,
+        template_header='Network Templates ' + ''.join(['-' for _ in range(18, template_name_length + template_id_length)]),
+        details_header='Details ' + ''.join(['-' for _ in range(8, template_mac_template_length + template_networks_length)]))
+    )
+
+    template_list_output.append('{bold}{template_name: <{template_name_length}} {template_id: <{template_id_length}} \
 {template_mac_template: <{template_mac_template_length}} \
 {template_networks: <{template_networks_length}}{end_bold}'.format(
         template_name_length=template_name_length,
@@ -973,6 +993,7 @@ def format_list_template_network(template_template):
         template_id='ID',
         template_mac_template='MAC template',
         template_networks='Network VNIs')
+    )
 
     # Format the string (elements)
     for template in sorted(template_template, key=lambda i: i.get('name', None)):
@@ -993,7 +1014,7 @@ def format_list_template_network(template_template):
             )
         )
 
-    return '\n'.join([template_list_output_header] + template_list_output)
+    return '\n'.join(template_list_output)
 
 
 def format_list_template_storage(template_template):
@@ -1003,12 +1024,12 @@ def format_list_template_storage(template_template):
     template_list_output = []
 
     # Determine optimal column widths
-    template_name_length = 5
-    template_id_length = 3
+    template_name_length = 18
+    template_id_length = 5
     template_disk_id_length = 8
-    template_disk_pool_length = 8
+    template_disk_pool_length = 5
     template_disk_source_length = 14
-    template_disk_size_length = 10
+    template_disk_size_length = 9
     template_disk_filesystem_length = 11
     template_disk_fsargs_length = 10
     template_disk_mountpoint_length = 10
@@ -1054,7 +1075,16 @@ def format_list_template_storage(template_template):
                 template_disk_mountpoint_length = _template_disk_mountpoint_length
 
     # Format the string (header)
-    template_list_output_header = '{bold}{template_name: <{template_name_length}} {template_id: <{template_id_length}} \
+    template_list_output.append('{bold}{template_header: <{template_header_length}} {details_header: <{details_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        template_header_length=template_name_length + template_id_length + 1,
+        details_header_length=template_disk_id_length + template_disk_pool_length + template_disk_source_length + template_disk_size_length + template_disk_filesystem_length + template_disk_fsargs_length + template_disk_mountpoint_length + 7,
+        template_header='Storage Templates ' + ''.join(['-' for _ in range(18, template_name_length + template_id_length)]),
+        details_header='Details ' + ''.join(['-' for _ in range(8, template_disk_id_length + template_disk_pool_length + template_disk_source_length + template_disk_size_length + template_disk_filesystem_length + template_disk_fsargs_length + template_disk_mountpoint_length + 6)]))
+    )
+
+    template_list_output.append('{bold}{template_name: <{template_name_length}} {template_id: <{template_id_length}} \
 {template_disk_id: <{template_disk_id_length}} \
 {template_disk_pool: <{template_disk_pool_length}} \
 {template_disk_source: <{template_disk_source_length}} \
@@ -1078,10 +1108,11 @@ def format_list_template_storage(template_template):
         template_disk_id='Disk ID',
         template_disk_pool='Pool',
         template_disk_source='Source Volume',
-        template_disk_size='Size [GB]',
+        template_disk_size='Size [G]',
         template_disk_filesystem='Filesystem',
         template_disk_fsargs='Arguments',
         template_disk_mountpoint='Mountpoint')
+    )
 
     # Format the string (elements)
     for template in sorted(template_template, key=lambda i: i.get('name', None)):
@@ -1128,7 +1159,7 @@ def format_list_template_storage(template_template):
                 )
             )
 
-    return '\n'.join([template_list_output_header] + template_list_output)
+    return '\n'.join(template_list_output)
 
 
 def format_list_userdata(userdata_data, lines=None):
@@ -1138,8 +1169,9 @@ def format_list_userdata(userdata_data, lines=None):
     userdata_list_output = []
 
     # Determine optimal column widths
-    userdata_name_length = 5
-    userdata_id_length = 3
+    userdata_name_length = 12
+    userdata_id_length = 5
+    userdata_document_length = 92 - userdata_name_length - userdata_id_length
 
     for userdata in userdata_data:
         # userdata_name column
@@ -1152,7 +1184,14 @@ def format_list_userdata(userdata_data, lines=None):
             userdata_id_length = _userdata_id_length
 
     # Format the string (header)
-    userdata_list_output_header = '{bold}{userdata_name: <{userdata_name_length}} {userdata_id: <{userdata_id_length}} \
+    userdata_list_output.append('{bold}{userdata_header: <{userdata_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        userdata_header_length=userdata_name_length + userdata_id_length + userdata_document_length + 2,
+        userdata_header='Userdata ' + ''.join(['-' for _ in range(9, userdata_name_length + userdata_id_length + userdata_document_length + 1)]))
+    )
+
+    userdata_list_output.append('{bold}{userdata_name: <{userdata_name_length}} {userdata_id: <{userdata_id_length}} \
 {userdata_data}{end_bold}'.format(
         userdata_name_length=userdata_name_length,
         userdata_id_length=userdata_id_length,
@@ -1161,6 +1200,7 @@ def format_list_userdata(userdata_data, lines=None):
         userdata_name='Name',
         userdata_id='ID',
         userdata_data='Document')
+    )
 
     # Format the string (elements)
     for data in sorted(userdata_data, key=lambda i: i.get('name', None)):
@@ -1202,7 +1242,7 @@ def format_list_userdata(userdata_data, lines=None):
                 )
             )
 
-    return '\n'.join([userdata_list_output_header] + userdata_list_output)
+    return '\n'.join(userdata_list_output)
 
 
 def format_list_script(script_data, lines=None):
@@ -1212,8 +1252,9 @@ def format_list_script(script_data, lines=None):
     script_list_output = []
 
     # Determine optimal column widths
-    script_name_length = 5
-    script_id_length = 3
+    script_name_length = 12
+    script_id_length = 5
+    script_data_length = 92 - script_name_length - script_id_length
 
     for script in script_data:
         # script_name column
@@ -1226,7 +1267,14 @@ def format_list_script(script_data, lines=None):
             script_id_length = _script_id_length
 
     # Format the string (header)
-    script_list_output_header = '{bold}{script_name: <{script_name_length}} {script_id: <{script_id_length}} \
+    script_list_output.append('{bold}{script_header: <{script_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        script_header_length=script_name_length + script_id_length + script_data_length + 2,
+        script_header='Script ' + ''.join(['-' for _ in range(7, script_name_length + script_id_length + script_data_length + 1)]))
+    )
+
+    script_list_output.append('{bold}{script_name: <{script_name_length}} {script_id: <{script_id_length}} \
 {script_data}{end_bold}'.format(
         script_name_length=script_name_length,
         script_id_length=script_id_length,
@@ -1235,6 +1283,7 @@ def format_list_script(script_data, lines=None):
         script_name='Name',
         script_id='ID',
         script_data='Script')
+    )
 
     # Format the string (elements)
     for script in sorted(script_data, key=lambda i: i.get('name', None)):
@@ -1276,7 +1325,7 @@ def format_list_script(script_data, lines=None):
                 )
             )
 
-    return '\n'.join([script_list_output_header] + script_list_output)
+    return '\n'.join(script_list_output)
 
 
 def format_list_ova(ova_data):
@@ -1286,8 +1335,8 @@ def format_list_ova(ova_data):
     ova_list_output = []
 
     # Determine optimal column widths
-    ova_name_length = 5
-    ova_id_length = 3
+    ova_name_length = 18
+    ova_id_length = 5
     ova_disk_id_length = 8
     ova_disk_size_length = 10
     ova_disk_pool_length = 5
@@ -1327,7 +1376,16 @@ def format_list_ova(ova_data):
                 ova_disk_volume_name_length = _ova_disk_volume_name_length
 
     # Format the string (header)
-    ova_list_output_header = '{bold}{ova_name: <{ova_name_length}} {ova_id: <{ova_id_length}} \
+    ova_list_output.append('{bold}{ova_header: <{ova_header_length}} {details_header: <{details_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        ova_header_length=ova_name_length + ova_id_length + 1,
+        details_header_length=ova_disk_id_length + ova_disk_size_length + ova_disk_pool_length + ova_disk_volume_format_length + ova_disk_volume_name_length + 4,
+        ova_header='OVAs ' + ''.join(['-' for _ in range(5, ova_name_length + ova_id_length)]),
+        details_header='Details ' + ''.join(['-' for _ in range(8, ova_disk_id_length + ova_disk_size_length + ova_disk_pool_length + ova_disk_volume_format_length + ova_disk_volume_name_length + 3)]))
+    )
+
+    ova_list_output.append('{bold}{ova_name: <{ova_name_length}} {ova_id: <{ova_id_length}} \
 {ova_disk_id: <{ova_disk_id_length}} \
 {ova_disk_size: <{ova_disk_size_length}} \
 {ova_disk_pool: <{ova_disk_pool_length}} \
@@ -1349,6 +1407,7 @@ def format_list_ova(ova_data):
         ova_disk_pool='Pool',
         ova_disk_volume_format='Format',
         ova_disk_volume_name='Source Volume')
+    )
 
     # Format the string (elements)
     for ova in sorted(ova_data, key=lambda i: i.get('name', None)):
@@ -1389,7 +1448,7 @@ def format_list_ova(ova_data):
                 )
             )
 
-    return '\n'.join([ova_list_output_header] + ova_list_output)
+    return '\n'.join(ova_list_output)
 
 
 def format_list_profile(profile_data):
@@ -1409,8 +1468,8 @@ def format_list_profile(profile_data):
     profile_list_output = []
 
     # Determine optimal column widths
-    profile_name_length = 5
-    profile_id_length = 3
+    profile_name_length = 18
+    profile_id_length = 5
     profile_source_length = 7
 
     profile_system_template_length = 7
@@ -1418,6 +1477,7 @@ def format_list_profile(profile_data):
     profile_storage_template_length = 8
     profile_userdata_length = 9
     profile_script_length = 7
+    profile_arguments_length = 18
 
     for profile in profile_data:
         # profile_name column
@@ -1454,11 +1514,22 @@ def format_list_profile(profile_data):
             profile_script_length = _profile_script_length
 
     # Format the string (header)
-    profile_list_output_header = '{bold}{profile_name: <{profile_name_length}} {profile_id: <{profile_id_length}} {profile_source: <{profile_source_length}} \
-Templates: {profile_system_template: <{profile_system_template_length}} \
+    profile_list_output.append('{bold}{profile_header: <{profile_header_length}} {templates_header: <{templates_header_length}} {data_header: <{data_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        profile_header_length=profile_name_length + profile_id_length + profile_source_length + 2,
+        templates_header_length=profile_system_template_length + profile_network_template_length + profile_storage_template_length + 2,
+        data_header_length=profile_userdata_length + profile_script_length + profile_arguments_length + 2,
+        profile_header='Profiles ' + ''.join(['-' for _ in range(9, profile_name_length + profile_id_length + profile_source_length + 1)]),
+        templates_header='Templates ' + ''.join(['-' for _ in range(10, profile_system_template_length + profile_network_template_length + profile_storage_template_length + 1)]),
+        data_header='Data ' + ''.join(['-' for _ in range(5, profile_userdata_length + profile_script_length + profile_arguments_length + 1)]))
+    )
+
+    profile_list_output.append('{bold}{profile_name: <{profile_name_length}} {profile_id: <{profile_id_length}} {profile_source: <{profile_source_length}} \
+{profile_system_template: <{profile_system_template_length}} \
 {profile_network_template: <{profile_network_template_length}} \
 {profile_storage_template: <{profile_storage_template_length}} \
-Data: {profile_userdata: <{profile_userdata_length}} \
+{profile_userdata: <{profile_userdata_length}} \
 {profile_script: <{profile_script_length}} \
 {profile_arguments}{end_bold}'.format(
         profile_name_length=profile_name_length,
@@ -1480,15 +1551,19 @@ Data: {profile_userdata: <{profile_userdata_length}} \
         profile_userdata='Userdata',
         profile_script='Script',
         profile_arguments='Script Arguments')
+    )
 
     # Format the string (elements)
     for profile in sorted(profile_data, key=lambda i: i.get('name', None)):
+        arguments_list = ', '.join(profile['arguments'])
+        if not arguments_list:
+            arguments_list = 'N/A'
         profile_list_output.append(
             '{bold}{profile_name: <{profile_name_length}} {profile_id: <{profile_id_length}} {profile_source: <{profile_source_length}} \
-           {profile_system_template: <{profile_system_template_length}} \
+{profile_system_template: <{profile_system_template_length}} \
 {profile_network_template: <{profile_network_template_length}} \
 {profile_storage_template: <{profile_storage_template_length}} \
-      {profile_userdata: <{profile_userdata_length}} \
+{profile_userdata: <{profile_userdata_length}} \
 {profile_script: <{profile_script_length}} \
 {profile_arguments}{end_bold}'.format(
                 profile_name_length=profile_name_length,
@@ -1509,11 +1584,11 @@ Data: {profile_userdata: <{profile_userdata_length}} \
                 profile_storage_template=profile['storage_template'],
                 profile_userdata=profile['userdata'],
                 profile_script=profile['script'],
-                profile_arguments=', '.join(profile['arguments'])
+                profile_arguments=arguments_list,
             )
         )
 
-    return '\n'.join([profile_list_output_header] + profile_list_output)
+    return '\n'.join(profile_list_output)
 
 
 def format_list_task(task_data):
@@ -1522,17 +1597,21 @@ def format_list_task(task_data):
     # Determine optimal column widths
     task_id_length = 7
     task_type_length = 7
+    task_worker_length = 7
     task_vm_name_length = 5
     task_vm_profile_length = 8
     task_vm_define_length = 8
     task_vm_start_length = 7
-    task_worker_length = 8
 
     for task in task_data:
         # task_id column
         _task_id_length = len(str(task['id'])) + 1
         if _task_id_length > task_id_length:
             task_id_length = _task_id_length
+        # task_worker column
+        _task_worker_length = len(str(task['worker'])) + 1
+        if _task_worker_length > task_worker_length:
+            task_worker_length = _task_worker_length
         # task_type column
         _task_type_length = len(str(task['type'])) + 1
         if _task_type_length > task_type_length:
@@ -1553,15 +1632,20 @@ def format_list_task(task_data):
         _task_vm_start_length = len(str(task['vm_start'])) + 1
         if _task_vm_start_length > task_vm_start_length:
             task_vm_start_length = _task_vm_start_length
-        # task_worker column
-        _task_worker_length = len(str(task['worker'])) + 1
-        if _task_worker_length > task_worker_length:
-            task_worker_length = _task_worker_length
 
     # Format the string (header)
-    task_list_output_header = '{bold}{task_id: <{task_id_length}} {task_type: <{task_type_length}} \
+    task_list_output.append('{bold}{task_header: <{task_header_length}} {vms_header: <{vms_header_length}}{end_bold}'.format(
+        bold=ansiprint.bold(),
+        end_bold=ansiprint.end(),
+        task_header_length=task_id_length + task_type_length + task_worker_length + 2,
+        vms_header_length=task_vm_name_length + task_vm_profile_length + task_vm_define_length + task_vm_start_length + 3,
+        task_header='Tasks ' + ''.join(['-' for _ in range(6, task_id_length + task_type_length + task_worker_length + 1)]),
+        vms_header='VM Details ' + ''.join(['-' for _ in range(11, task_vm_name_length + task_vm_profile_length + task_vm_define_length + task_vm_start_length + 2)]))
+    )
+
+    task_list_output.append('{bold}{task_id: <{task_id_length}} {task_type: <{task_type_length}} \
 {task_worker: <{task_worker_length}} \
-VM: {task_vm_name: <{task_vm_name_length}} \
+{task_vm_name: <{task_vm_name_length}} \
 {task_vm_profile: <{task_vm_profile_length}} \
 {task_vm_define: <{task_vm_define_length}} \
 {task_vm_start: <{task_vm_start_length}}{end_bold}'.format(
@@ -1581,13 +1665,14 @@ VM: {task_vm_name: <{task_vm_name_length}} \
         task_vm_profile='Profile',
         task_vm_define='Define?',
         task_vm_start='Start?')
+    )
 
     # Format the string (elements)
     for task in sorted(task_data, key=lambda i: i.get('type', None)):
         task_list_output.append(
             '{bold}{task_id: <{task_id_length}} {task_type: <{task_type_length}} \
 {task_worker: <{task_worker_length}} \
-    {task_vm_name: <{task_vm_name_length}} \
+{task_vm_name: <{task_vm_name_length}} \
 {task_vm_profile: <{task_vm_profile_length}} \
 {task_vm_define: <{task_vm_define_length}} \
 {task_vm_start: <{task_vm_start_length}}{end_bold}'.format(
@@ -1610,4 +1695,4 @@ VM: {task_vm_name: <{task_vm_name_length}} \
             )
         )
 
-    return '\n'.join([task_list_output_header] + task_list_output)
+    return '\n'.join(task_list_output)

client-cli/pvc/cli_lib/vm.py

@@ -22,8 +22,8 @@
 import time
 import re
 
-import cli_lib.ansiprint as ansiprint
-from cli_lib.common import call_api, format_bytes, format_metric
+import pvc.cli_lib.ansiprint as ansiprint
+from pvc.cli_lib.common import call_api, format_bytes, format_metric
 
 
 #
@@ -54,12 +54,12 @@ def vm_info(config, vm):
         return False, response.json().get('message', '')
 
 
-def vm_list(config, limit, target_node, target_state):
+def vm_list(config, limit, target_node, target_state, target_tag, negate):
     """
     Get list information about VMs (limited by {limit}, {target_node}, or {target_state})
 
     API endpoint: GET /api/v1/vm
-    API arguments: limit={limit}, node={target_node}, state={target_state}
+    API arguments: limit={limit}, node={target_node}, state={target_state}, tag={target_tag}, negate={negate}
     API schema: [{json_data_object},{json_data_object},etc.]
     """
     params = dict()
@@ -69,6 +69,9 @@ def vm_list(config, limit, target_node, target_state):
         params['node'] = target_node
     if target_state:
         params['state'] = target_state
+    if target_tag:
+        params['tag'] = target_tag
+    params['negate'] = negate
 
     response = call_api(config, 'get', '/vm', params=params)
 
@@ -78,12 +81,12 @@ def vm_list(config, limit, target_node, target_state):
         return False, response.json().get('message', '')
 
 
-def vm_define(config, xml, node, node_limit, node_selector, node_autostart, migration_method):
+def vm_define(config, xml, node, node_limit, node_selector, node_autostart, migration_method, user_tags, protected_tags):
     """
     Define a new VM on the cluster
 
     API endpoint: POST /vm
-    API arguments: xml={xml}, node={node}, limit={node_limit}, selector={node_selector}, autostart={node_autostart}, migration_method={migration_method}
+    API arguments: xml={xml}, node={node}, limit={node_limit}, selector={node_selector}, autostart={node_autostart}, migration_method={migration_method}, user_tags={user_tags}, protected_tags={protected_tags}
     API schema: {"message":"{data}"}
     """
     params = {
@@ -91,7 +94,9 @@ def vm_define(config, xml, node, node_limit, node_selector, node_autostart, migr
         'limit': node_limit,
         'selector': node_selector,
         'autostart': node_autostart,
-        'migration_method': migration_method
+        'migration_method': migration_method,
+        'user_tags': user_tags,
+        'protected_tags': protected_tags
     }
     data = {
         'xml': xml
@@ -130,6 +135,48 @@ def vm_modify(config, vm, xml, restart):
     return retstatus, response.json().get('message', '')
 
 
+def vm_device_attach(config, vm, xml):
+    """
+    Attach a device to a VM
+
+    API endpoint: POST /vm/{vm}/device
+    API arguments: xml={xml}
+    API schema: {"message":"{data}"}
+    """
+    data = {
+        'xml': xml
+    }
+    response = call_api(config, 'post', '/vm/{vm}/device'.format(vm=vm), data=data)
+
+    if response.status_code == 200:
+        retstatus = True
+    else:
+        retstatus = False
+
+    return retstatus, response.json().get('message', '')
+
+
+def vm_device_detach(config, vm, xml):
+    """
+    Detach a device from a VM
+
+    API endpoint: DELETE /vm/{vm}/device
+    API arguments: xml={xml}
+    API schema: {"message":"{data}"}
+    """
+    data = {
+        'xml': xml
+    }
+    response = call_api(config, 'delete', '/vm/{vm}/device'.format(vm=vm), data=data)
+
+    if response.status_code == 200:
+        retstatus = True
+    else:
+        retstatus = False
+
+    return retstatus, response.json().get('message', '')
+
+
 def vm_rename(config, vm, new_name):
     """
     Rename VM to new name
@@ -155,7 +202,7 @@ def vm_metadata(config, vm, node_limit, node_selector, node_autostart, migration
     """
     Modify PVC metadata of a VM
 
-    API endpoint: GET /vm/{vm}/meta,  POST /vm/{vm}/meta
+    API endpoint: POST /vm/{vm}/meta
     API arguments: limit={node_limit}, selector={node_selector}, autostart={node_autostart}, migration_method={migration_method} profile={provisioner_profile}
     API schema: {"message":"{data}"}
     """
@@ -188,6 +235,119 @@ def vm_metadata(config, vm, node_limit, node_selector, node_autostart, migration
     return retstatus, response.json().get('message', '')
 
 
+def vm_tags_get(config, vm):
+    """
+    Get PVC tags of a VM
+
+    API endpoint: GET /vm/{vm}/tags
+    API arguments:
+    API schema: {{"name": "{name}", "type": "{type}"},...}
+    """
+
+    response = call_api(config, 'get', '/vm/{vm}/tags'.format(vm=vm))
+
+    if response.status_code == 200:
+        retstatus = True
+        retdata = response.json()
+    else:
+        retstatus = False
+        retdata = response.json().get('message', '')
+
+    return retstatus, retdata
+
+
+def vm_tag_set(config, vm, action, tag, protected=False):
+    """
+    Modify PVC tags of a VM
+
+    API endpoint: POST /vm/{vm}/tags
+    API arguments: action={action}, tag={tag}, protected={protected}
+    API schema: {"message":"{data}"}
+    """
+
+    params = {
+        'action': action,
+        'tag': tag,
+        'protected': protected
+    }
+
+    # Update the tags
+    response = call_api(config, 'post', '/vm/{vm}/tags'.format(vm=vm), params=params)
+
+    if response.status_code == 200:
+        retstatus = True
+    else:
+        retstatus = False
+
+    return retstatus, response.json().get('message', '')
+
+
+def format_vm_tags(config, name, tags):
+    """
+    Format the output of a tags dictionary in a nice table
+    """
+    if len(tags) < 1:
+        return "No tags found."
+
+    output_list = []
+
+    name_length = 5
+    _name_length = len(name) + 1
+    if _name_length > name_length:
+        name_length = _name_length
+
+    tags_name_length = 4
+    tags_type_length = 5
+    tags_protected_length = 10
+    for tag in tags:
+        _tags_name_length = len(tag['name']) + 1
+        if _tags_name_length > tags_name_length:
+            tags_name_length = _tags_name_length
+
+        _tags_type_length = len(tag['type']) + 1
+        if _tags_type_length > tags_type_length:
+            tags_type_length = _tags_type_length
+
+        _tags_protected_length = len(str(tag['protected'])) + 1
+        if _tags_protected_length > tags_protected_length:
+            tags_protected_length = _tags_protected_length
+
+    output_list.append(
+        '{bold}{tags_name: <{tags_name_length}}  \
+{tags_type: <{tags_type_length}}  \
+{tags_protected: <{tags_protected_length}}{end_bold}'.format(
+            name_length=name_length,
+            tags_name_length=tags_name_length,
+            tags_type_length=tags_type_length,
+            tags_protected_length=tags_protected_length,
+            bold=ansiprint.bold(),
+            end_bold=ansiprint.end(),
+            tags_name='Name',
+            tags_type='Type',
+            tags_protected='Protected'
+        )
+    )
+
+    for tag in sorted(tags, key=lambda t: t['name']):
+        output_list.append(
+            '{bold}{tags_name: <{tags_name_length}}  \
+{tags_type: <{tags_type_length}}  \
+{tags_protected: <{tags_protected_length}}{end_bold}'.format(
+                name_length=name_length,
+                tags_type_length=tags_type_length,
+                tags_name_length=tags_name_length,
+                tags_protected_length=tags_protected_length,
+                bold='',
+                end_bold='',
+                tags_name=tag['name'],
+                tags_type=tag['type'],
+                tags_protected=str(tag['protected'])
+            )
+        )
+
+    return '\n'.join(output_list)
+
+
 def vm_remove(config, vm, delete_disks=False):
     """
     Remove a VM
@@ -501,30 +661,34 @@ def format_vm_memory(config, name, memory):
     return '\n'.join(output_list)
 
 
-def vm_networks_add(config, vm, network, macaddr, model, restart):
+def vm_networks_add(config, vm, network, macaddr, model, sriov, sriov_mode, live, restart):
     """
     Add a new network to the VM
 
     Calls vm_info to get the VM XML.
 
     Calls vm_modify to set the VM XML.
+
+    Calls vm_device_attach if live to hot-attach the device.
     """
     from lxml.objectify import fromstring
     from lxml.etree import tostring
     from random import randint
-    import cli_lib.network as pvc_network
+    import pvc.cli_lib.network as pvc_network
 
-    # Verify that the provided network is valid
-    retcode, retdata = pvc_network.net_info(config, network)
-    if not retcode:
-        # Ignore the three special networks
-        if network not in ['upstream', 'cluster', 'storage']:
-            return False, "Network {} is not present in the cluster.".format(network)
+    # Verify that the provided network is valid (not in SR-IOV mode)
+    if not sriov:
+        retcode, retdata = pvc_network.net_info(config, network)
+        if not retcode:
+            # Ignore the three special networks
+            if network not in ['upstream', 'cluster', 'storage']:
+                return False, "Network {} is not present in the cluster.".format(network)
 
-    if network in ['upstream', 'cluster', 'storage']:
-        br_prefix = 'br'
-    else:
-        br_prefix = 'vmbr'
+        # Set the bridge prefix
+        if network in ['upstream', 'cluster', 'storage']:
+            br_prefix = 'br'
+        else:
+            br_prefix = 'vmbr'
 
     status, domain_information = vm_info(config, vm)
     if not status:
@@ -551,24 +715,74 @@ def vm_networks_add(config, vm, network, macaddr, model, restart):
             octetC=random_octet_C
         )
 
-    device_string = '<interface type="bridge"><mac address="{macaddr}"/><source bridge="{bridge}"/><model type="{model}"/></interface>'.format(
-        macaddr=macaddr,
-        bridge="{}{}".format(br_prefix, network),
-        model=model
-    )
+    # Add an SR-IOV network
+    if sriov:
+        valid, sriov_vf_information = pvc_network.net_sriov_vf_info(config, domain_information['node'], network)
+        if not valid:
+            return False, 'Specified SR-IOV VF "{}" does not exist on VM node "{}".'.format(network, domain_information['node'])
+
+        # Add a hostdev (direct PCIe) SR-IOV network
+        if sriov_mode == 'hostdev':
+            bus_address = 'domain="0x{pci_domain}" bus="0x{pci_bus}" slot="0x{pci_slot}" function="0x{pci_function}"'.format(
+                pci_domain=sriov_vf_information['pci']['domain'],
+                pci_bus=sriov_vf_information['pci']['bus'],
+                pci_slot=sriov_vf_information['pci']['slot'],
+                pci_function=sriov_vf_information['pci']['function'],
+            )
+            device_string = '<interface type="hostdev" managed="yes"><mac address="{macaddr}"/><source><address type="pci" {bus_address}/></source><sriov_device>{network}</sriov_device></interface>'.format(
+                macaddr=macaddr,
+                bus_address=bus_address,
+                network=network
+            )
+        # Add a macvtap SR-IOV network
+        elif sriov_mode == 'macvtap':
+            device_string = '<interface type="direct"><mac address="{macaddr}"/><source dev="{network}" mode="passthrough"/><model type="{model}"/></interface>'.format(
+                macaddr=macaddr,
+                network=network,
+                model=model
+            )
+        else:
+            return False, "ERROR: Invalid SR-IOV mode specified."
+    # Add a normal bridged PVC network
+    else:
+        device_string = '<interface type="bridge"><mac address="{macaddr}"/><source bridge="{bridge}"/><model type="{model}"/></interface>'.format(
+            macaddr=macaddr,
+            bridge="{}{}".format(br_prefix, network),
+            model=model
+        )
+
     device_xml = fromstring(device_string)
 
-    last_interface = None
     all_interfaces = parsed_xml.devices.find('interface')
     if all_interfaces is None:
         all_interfaces = []
     for interface in all_interfaces:
-        last_interface = re.match(r'[vm]*br([0-9a-z]+)', interface.source.attrib.get('bridge')).group(1)
-        if last_interface == network:
-            return False, 'Network {} is already configured for VM {}.'.format(network, vm)
-    if last_interface is not None:
-        for interface in parsed_xml.devices.find('interface'):
-            if last_interface == re.match(r'[vm]*br([0-9a-z]+)', interface.source.attrib.get('bridge')).group(1):
+        if sriov:
+            if sriov_mode == 'hostdev':
+                if interface.attrib.get('type') == 'hostdev':
+                    interface_address = 'domain="{pci_domain}" bus="{pci_bus}" slot="{pci_slot}" function="{pci_function}"'.format(
+                        pci_domain=interface.source.address.attrib.get('domain'),
+                        pci_bus=interface.source.address.attrib.get('bus'),
+                        pci_slot=interface.source.address.attrib.get('slot'),
+                        pci_function=interface.source.address.attrib.get('function')
+                    )
+                    if interface_address == bus_address:
+                        return False, 'Network "{}" is already configured for VM "{}".'.format(network, vm)
+            elif sriov_mode == 'macvtap':
+                if interface.attrib.get('type') == 'direct':
+                    interface_dev = interface.source.attrib.get('dev')
+                    if interface_dev == network:
+                        return False, 'Network "{}" is already configured for VM "{}".'.format(network, vm)
+        else:
+            if interface.attrib.get('type') == 'bridge':
+                interface_vni = re.match(r'[vm]*br([0-9a-z]+)', interface.source.attrib.get('bridge')).group(1)
+                if interface_vni == network:
+                    return False, 'Network "{}" is already configured for VM "{}".'.format(network, vm)
+
+    # Add the interface at the end of the list (or, right above emulator)
+    if len(all_interfaces) > 0:
+        for idx, interface in enumerate(parsed_xml.devices.find('interface')):
+            if idx == len(all_interfaces) - 1:
                 interface.addnext(device_xml)
     else:
         parsed_xml.devices.find('emulator').addprevious(device_xml)
@@ -578,16 +792,36 @@ def vm_networks_add(config, vm, network, macaddr, model, restart):
     except Exception:
         return False, 'ERROR: Failed to dump XML data.'
 
-    return vm_modify(config, vm, new_xml, restart)
+    modify_retcode, modify_retmsg = vm_modify(config, vm, new_xml, restart)
+
+    if not modify_retcode:
+        return modify_retcode, modify_retmsg
+
+    if live:
+        attach_retcode, attach_retmsg = vm_device_attach(config, vm, device_string)
+
+        if not attach_retcode:
+            retcode = attach_retcode
+            retmsg = attach_retmsg
+        else:
+            retcode = attach_retcode
+            retmsg = "Network '{}' successfully added to VM config and hot attached to running VM.".format(network)
+    else:
+        retcode = modify_retcode
+        retmsg = modify_retmsg
+
+    return retcode, retmsg
 
 
-def vm_networks_remove(config, vm, network, restart):
+def vm_networks_remove(config, vm, network, sriov, live, restart):
     """
     Remove a network to the VM
 
     Calls vm_info to get the VM XML.
 
     Calls vm_modify to set the VM XML.
+
+    Calls vm_device_detach to hot-remove the device.
     """
     from lxml.objectify import fromstring
     from lxml.etree import tostring
@@ -605,17 +839,55 @@ def vm_networks_remove(config, vm, network, restart):
     except Exception:
         return False, 'ERROR: Failed to parse XML data.'
 
+    changed = False
+    device_string = None
     for interface in parsed_xml.devices.find('interface'):
-        if_vni = re.match(r'[vm]*br([0-9a-z]+)', interface.source.attrib.get('bridge')).group(1)
-        if network == if_vni:
-            interface.getparent().remove(interface)
+        if sriov:
+            if interface.attrib.get('type') == 'hostdev':
+                if_dev = str(interface.sriov_device)
+                if network == if_dev:
+                    interface.getparent().remove(interface)
+                    changed = True
+            elif interface.attrib.get('type') == 'direct':
+                if_dev = str(interface.source.attrib.get('dev'))
+                if network == if_dev:
+                    interface.getparent().remove(interface)
+                    changed = True
+        else:
+            if_vni = re.match(r'[vm]*br([0-9a-z]+)', interface.source.attrib.get('bridge')).group(1)
+            if network == if_vni:
+                interface.getparent().remove(interface)
+                changed = True
+        if changed:
+            device_string = tostring(interface)
 
-    try:
-        new_xml = tostring(parsed_xml, pretty_print=True)
-    except Exception:
-        return False, 'ERROR: Failed to dump XML data.'
+    if changed:
+        try:
+            new_xml = tostring(parsed_xml, pretty_print=True)
+        except Exception:
+            return False, 'ERROR: Failed to dump XML data.'
+    else:
+        return False, 'ERROR: Network "{}" does not exist on VM.'.format(network)
 
-    return vm_modify(config, vm, new_xml, restart)
+    modify_retcode, modify_retmsg = vm_modify(config, vm, new_xml, restart)
+
+    if not modify_retcode:
+        return modify_retcode, modify_retmsg
+
+    if live and device_string:
+        detach_retcode, detach_retmsg = vm_device_detach(config, vm, device_string)
+
+        if not detach_retcode:
+            retcode = detach_retcode
+            retmsg = detach_retmsg
+        else:
+            retcode = detach_retcode
+            retmsg = "Network '{}' successfully removed from VM config and hot detached from running VM.".format(network)
+    else:
+        retcode = modify_retcode
+        retmsg = modify_retmsg
+
+    return retcode, retmsg
 
 
 def vm_networks_get(config, vm):
@@ -645,7 +917,14 @@ def vm_networks_get(config, vm):
     for interface in parsed_xml.devices.find('interface'):
         mac_address = interface.mac.attrib.get('address')
         model = interface.model.attrib.get('type')
-        network = re.match(r'[vm]*br([0-9a-z]+)', interface.source.attrib.get('bridge')).group(1)
+        interface_type = interface.attrib.get('type')
+        if interface_type == 'bridge':
+            network = re.search(r'[vm]*br([0-9a-z]+)', interface.source.attrib.get('bridge')).group(1)
+        elif interface_type == 'direct':
+            network = 'macvtap:{}'.format(interface.source.attrib.get('dev'))
+        elif interface_type == 'hostdev':
+            network = 'hostdev:{}'.format(interface.source.attrib.get('dev'))
+
         network_data.append((network, mac_address, model))
 
     return True, network_data
@@ -721,7 +1000,7 @@ def format_vm_networks(config, name, networks):
     return '\n'.join(output_list)
 
 
-def vm_volumes_add(config, vm, volume, disk_id, bus, disk_type, restart):
+def vm_volumes_add(config, vm, volume, disk_id, bus, disk_type, live, restart):
     """
     Add a new volume to the VM
 
@@ -732,7 +1011,7 @@ def vm_volumes_add(config, vm, volume, disk_id, bus, disk_type, restart):
     from lxml.objectify import fromstring
     from lxml.etree import tostring
     from copy import deepcopy
-    import cli_lib.ceph as pvc_ceph
+    import pvc.cli_lib.ceph as pvc_ceph
 
     if disk_type == 'rbd':
         # Verify that the provided volume is valid
@@ -809,6 +1088,7 @@ def vm_volumes_add(config, vm, volume, disk_id, bus, disk_type, restart):
         new_disk_details.source.set('name', volume)
     elif disk_type == 'file':
         new_disk_details.source.set('file', volume)
+    device_xml = new_disk_details
 
     all_disks = parsed_xml.devices.find('disk')
     if all_disks is None:
@@ -816,18 +1096,42 @@ def vm_volumes_add(config, vm, volume, disk_id, bus, disk_type, restart):
     for disk in all_disks:
         last_disk = disk
 
-    if last_disk is None:
-        parsed_xml.devices.find('emulator').addprevious(new_disk_details)
+    # Add the disk at the end of the list (or, right above emulator)
+    if len(all_disks) > 0:
+        for idx, disk in enumerate(parsed_xml.devices.find('disk')):
+            if idx == len(all_disks) - 1:
+                disk.addnext(device_xml)
+    else:
+        parsed_xml.devices.find('emulator').addprevious(device_xml)
 
     try:
         new_xml = tostring(parsed_xml, pretty_print=True)
     except Exception:
         return False, 'ERROR: Failed to dump XML data.'
 
-    return vm_modify(config, vm, new_xml, restart)
+    modify_retcode, modify_retmsg = vm_modify(config, vm, new_xml, restart)
+
+    if not modify_retcode:
+        return modify_retcode, modify_retmsg
+
+    if live:
+        device_string = tostring(device_xml)
+        attach_retcode, attach_retmsg = vm_device_attach(config, vm, device_string)
+
+        if not attach_retcode:
+            retcode = attach_retcode
+            retmsg = attach_retmsg
+        else:
+            retcode = attach_retcode
+            retmsg = "Volume '{}/{}' successfully added to VM config and hot attached to running VM.".format(vpool, vname)
+    else:
+        retcode = modify_retcode
+        retmsg = modify_retmsg
+
+    return retcode, retmsg
 
 
-def vm_volumes_remove(config, vm, volume, restart):
+def vm_volumes_remove(config, vm, volume, live, restart):
     """
     Remove a volume to the VM
 
@@ -844,26 +1148,51 @@ def vm_volumes_remove(config, vm, volume, restart):
 
     xml = domain_information.get('xml', None)
     if xml is None:
-        return False, "VM does not have a valid XML doccument."
+        return False, "VM does not have a valid XML document."
 
     try:
         parsed_xml = fromstring(xml)
     except Exception:
         return False, 'ERROR: Failed to parse XML data.'
 
+    changed = False
+    device_string = None
     for disk in parsed_xml.devices.find('disk'):
         disk_name = disk.source.attrib.get('name')
         if not disk_name:
             disk_name = disk.source.attrib.get('file')
         if volume == disk_name:
+            device_string = tostring(disk)
             disk.getparent().remove(disk)
+            changed = True
 
-    try:
-        new_xml = tostring(parsed_xml, pretty_print=True)
-    except Exception:
-        return False, 'ERROR: Failed to dump XML data.'
+    if changed:
+        try:
+            new_xml = tostring(parsed_xml, pretty_print=True)
+        except Exception:
+            return False, 'ERROR: Failed to dump XML data.'
+    else:
+        return False, 'ERROR: Volume "{}" does not exist on VM.'.format(volume)
 
-    return vm_modify(config, vm, new_xml, restart)
+    modify_retcode, modify_retmsg = vm_modify(config, vm, new_xml, restart)
+
+    if not modify_retcode:
+        return modify_retcode, modify_retmsg
+
+    if live and device_string:
+        detach_retcode, detach_retmsg = vm_device_detach(config, vm, device_string)
+
+        if not detach_retcode:
+            retcode = detach_retcode
+            retmsg = detach_retmsg
+        else:
+            retcode = detach_retcode
+            retmsg = "Volume '{}' successfully removed from VM config and hot detached from running VM.".format(volume)
+    else:
+        retcode = modify_retcode
+        retmsg = modify_retmsg
+
+    return retcode, retmsg
 
 
 def vm_volumes_get(config, vm):
@@ -1023,9 +1352,9 @@ def follow_console_log(config, vm, lines=10):
     API arguments: lines={lines}
     API schema: {"name":"{vmname}","data":"{console_log}"}
     """
-    # We always grab 500 to match the follow call, but only _show_ `lines` number
+    # We always grab 200 to match the follow call, but only _show_ `lines` number
     params = {
-        'lines': 500
+        'lines': 200
     }
     response = call_api(config, 'get', '/vm/{vm}/console'.format(vm=vm), params=params)
 
@@ -1041,10 +1370,10 @@ def follow_console_log(config, vm, lines=10):
     print(loglines, end='')
 
     while True:
-        # Grab the next line set (500 is a reasonable number of lines per second; any more are skipped)
+        # Grab the next line set (200 is a reasonable number of lines per half-second; any more are skipped)
         try:
             params = {
-                'lines': 500
+                'lines': 200
             }
             response = call_api(config, 'get', '/vm/{vm}/console'.format(vm=vm), params=params)
             new_console_log = response.json()['data']
@@ -1053,8 +1382,10 @@ def follow_console_log(config, vm, lines=10):
         # Split the new and old log strings into constitutent lines
         old_console_loglines = console_log.split('\n')
         new_console_loglines = new_console_log.split('\n')
+
         # Set the console log to the new log value for the next iteration
         console_log = new_console_log
+
         # Remove the lines from the old log until we hit the first line of the new log; this
         # ensures that the old log is a string that we can remove from the new log entirely
         for index, line in enumerate(old_console_loglines, start=0):
@@ -1069,8 +1400,8 @@ def follow_console_log(config, vm, lines=10):
         # If there's a difference, print it out
         if diff_console_log:
             print(diff_console_log, end='')
-        # Wait a second
-        time.sleep(1)
+        # Wait half a second
+        time.sleep(0.5)
 
     return True, ''
 
@@ -1084,8 +1415,8 @@ def format_info(config, domain_information, long_output):
     ainformation.append('{}Virtual machine information:{}'.format(ansiprint.bold(), ansiprint.end()))
     ainformation.append('')
     # Basic information
-    ainformation.append('{}UUID:{}               {}'.format(ansiprint.purple(), ansiprint.end(), domain_information['uuid']))
     ainformation.append('{}Name:{}               {}'.format(ansiprint.purple(), ansiprint.end(), domain_information['name']))
+    ainformation.append('{}UUID:{}               {}'.format(ansiprint.purple(), ansiprint.end(), domain_information['uuid']))
     ainformation.append('{}Description:{}        {}'.format(ansiprint.purple(), ansiprint.end(), domain_information['description']))
     ainformation.append('{}Profile:{}            {}'.format(ansiprint.purple(), ansiprint.end(), domain_information['profile']))
     ainformation.append('{}Memory (M):{}         {}'.format(ansiprint.purple(), ansiprint.end(), domain_information['memory']))
@@ -1173,19 +1504,64 @@ def format_info(config, domain_information, long_output):
     ainformation.append('{}Autostart:{}          {}'.format(ansiprint.purple(), ansiprint.end(), formatted_node_autostart))
     ainformation.append('{}Migration Method:{}   {}'.format(ansiprint.purple(), ansiprint.end(), formatted_migration_method))
 
+    # Tag list
+    tags_name_length = 5
+    tags_type_length = 5
+    tags_protected_length = 10
+    for tag in domain_information['tags']:
+        _tags_name_length = len(tag['name']) + 1
+        if _tags_name_length > tags_name_length:
+            tags_name_length = _tags_name_length
+
+        _tags_type_length = len(tag['type']) + 1
+        if _tags_type_length > tags_type_length:
+            tags_type_length = _tags_type_length
+
+        _tags_protected_length = len(str(tag['protected'])) + 1
+        if _tags_protected_length > tags_protected_length:
+            tags_protected_length = _tags_protected_length
+
+    if len(domain_information['tags']) > 0:
+        ainformation.append('')
+        ainformation.append('{purple}Tags:{end}               {bold}{tags_name: <{tags_name_length}} {tags_type: <{tags_type_length}} {tags_protected: <{tags_protected_length}}{end}'.format(
+            purple=ansiprint.purple(),
+            bold=ansiprint.bold(),
+            end=ansiprint.end(),
+            tags_name_length=tags_name_length,
+            tags_type_length=tags_type_length,
+            tags_protected_length=tags_protected_length,
+            tags_name='Name',
+            tags_type='Type',
+            tags_protected='Protected'
+        ))
+
+        for tag in sorted(domain_information['tags'], key=lambda t: t['type'] + t['name']):
+            ainformation.append('                    {tags_name: <{tags_name_length}} {tags_type: <{tags_type_length}} {tags_protected: <{tags_protected_length}}'.format(
+                tags_name_length=tags_name_length,
+                tags_type_length=tags_type_length,
+                tags_protected_length=tags_protected_length,
+                tags_name=tag['name'],
+                tags_type=tag['type'],
+                tags_protected=str(tag['protected'])
+            ))
+    else:
+        ainformation.append('')
+        ainformation.append('{purple}Tags:{end}               N/A'.format(
+            purple=ansiprint.purple(),
+            bold=ansiprint.bold(),
+            end=ansiprint.end(),
+        ))
+
     # Network list
     net_list = []
+    cluster_net_list = call_api(config, 'get', '/network').json()
     for net in domain_information['networks']:
-        # Split out just the numerical (VNI) part of the brXXXX name
-        net_vnis = re.findall(r'\d+', net['source'])
-        if net_vnis:
-            net_vni = net_vnis[0]
-        else:
-            net_vni = re.sub('br', '', net['source'])
-
-        response = call_api(config, 'get', '/network/{net}'.format(net=net_vni))
-        if response.status_code != 200 and net_vni not in ['cluster', 'storage', 'upstream']:
-            net_list.append(ansiprint.red() + net_vni + ansiprint.end() + ' [invalid]')
+        net_vni = net['vni']
+        if net_vni not in ['cluster', 'storage', 'upstream'] and not re.match(r'^macvtap:.*', net_vni) and not re.match(r'^hostdev:.*', net_vni):
+            if int(net_vni) not in [net['vni'] for net in cluster_net_list]:
+                net_list.append(ansiprint.red() + net_vni + ansiprint.end() + ' [invalid]')
+            else:
+                net_list.append(net_vni)
         else:
             net_list.append(net_vni)
 
@@ -1213,17 +1589,31 @@ def format_info(config, domain_information, long_output):
                 width=name_length
             ))
         ainformation.append('')
-        ainformation.append('{}Interfaces:{}   {}ID  Type    Source     Model    MAC                 Data (r/w)   Packets (r/w)   Errors (r/w){}'.format(ansiprint.purple(), ansiprint.end(), ansiprint.bold(), ansiprint.end()))
+        ainformation.append('{}Interfaces:{}   {}ID  Type     Source       Model    MAC                 Data (r/w)   Packets (r/w)   Errors (r/w){}'.format(ansiprint.purple(), ansiprint.end(), ansiprint.bold(), ansiprint.end()))
         for net in domain_information['networks']:
-            ainformation.append('              {0: <3} {1: <7} {2: <10} {3: <8} {4: <18}  {5: <12} {6: <15} {7: <12}'.format(
+            net_type = net['type']
+            net_source = net['source']
+            net_mac = net['mac']
+            if net_type in ['direct', 'hostdev']:
+                net_model = 'N/A'
+                net_bytes = 'N/A'
+                net_packets = 'N/A'
+                net_errors = 'N/A'
+            elif net_type in ['bridge']:
+                net_model = net['model']
+                net_bytes = '/'.join([str(format_bytes(net.get('rd_bytes', 0))), str(format_bytes(net.get('wr_bytes', 0)))])
+                net_packets = '/'.join([str(format_metric(net.get('rd_packets', 0))), str(format_metric(net.get('wr_packets', 0)))])
+                net_errors = '/'.join([str(format_metric(net.get('rd_errors', 0))), str(format_metric(net.get('wr_errors', 0)))])
+
+            ainformation.append('              {0: <3} {1: <8} {2: <12} {3: <8} {4: <18}  {5: <12} {6: <15} {7: <12}'.format(
                 domain_information['networks'].index(net),
-                net['type'],
-                net['source'],
-                net['model'],
-                net['mac'],
-                '/'.join([str(format_bytes(net.get('rd_bytes', 0))), str(format_bytes(net.get('wr_bytes', 0)))]),
-                '/'.join([str(format_metric(net.get('rd_packets', 0))), str(format_metric(net.get('wr_packets', 0)))]),
-                '/'.join([str(format_metric(net.get('rd_errors', 0))), str(format_metric(net.get('wr_errors', 0)))]),
+                net_type,
+                net_source,
+                net_model,
+                net_mac,
+                net_bytes,
+                net_packets,
+                net_errors
             ))
         # Controller list
         ainformation.append('')
@@ -1242,15 +1632,17 @@ def format_list(config, vm_list, raw):
         # Network list
         net_list = []
         for net in domain_information['networks']:
-            # Split out just the numerical (VNI) part of the brXXXX name
-            net_vnis = re.findall(r'\d+', net['source'])
-            if net_vnis:
-                net_vni = net_vnis[0]
-            else:
-                net_vni = re.sub('br', '', net['source'])
-            net_list.append(net_vni)
+            net_list.append(net['vni'])
         return net_list
 
+    # Function to get tag names and returna  nicer list
+    def getNiceTagName(domain_information):
+        # Tag list
+        tag_list = []
+        for tag in sorted(domain_information['tags'], key=lambda t: t['type'] + t['name']):
+            tag_list.append(tag['name'])
+        return tag_list
+
     # Handle raw mode since it just lists the names
     if raw:
         ainformation = list()
@@ -1263,15 +1655,16 @@ def format_list(config, vm_list, raw):
     # Determine optimal column widths
     # Dynamic columns: node_name, node, migrated
     vm_name_length = 5
-    vm_uuid_length = 37
     vm_state_length = 6
+    vm_tags_length = 5
     vm_nets_length = 9
     vm_ram_length = 8
     vm_vcpu_length = 6
     vm_node_length = 8
-    vm_migrated_length = 10
+    vm_migrated_length = 9
     for domain_information in vm_list:
         net_list = getNiceNetID(domain_information)
+        tag_list = getNiceTagName(domain_information)
         # vm_name column
         _vm_name_length = len(domain_information['name']) + 1
         if _vm_name_length > vm_name_length:
@@ -1280,6 +1673,10 @@ def format_list(config, vm_list, raw):
         _vm_state_length = len(domain_information['state']) + 1
         if _vm_state_length > vm_state_length:
             vm_state_length = _vm_state_length
+        # vm_tags column
+        _vm_tags_length = len(','.join(tag_list)) + 1
+        if _vm_tags_length > vm_tags_length:
+            vm_tags_length = _vm_tags_length
         # vm_nets column
         _vm_nets_length = len(','.join(net_list)) + 1
         if _vm_nets_length > vm_nets_length:
@@ -1295,15 +1692,29 @@ def format_list(config, vm_list, raw):
 
     # Format the string (header)
     vm_list_output.append(
-        '{bold}{vm_name: <{vm_name_length}} {vm_uuid: <{vm_uuid_length}} \
+        '{bold}{vm_header: <{vm_header_length}} {resource_header: <{resource_header_length}} {node_header: <{node_header_length}}{end_bold}'.format(
+            vm_header_length=vm_name_length + vm_state_length + vm_tags_length + 2,
+            resource_header_length=vm_nets_length + vm_ram_length + vm_vcpu_length + 2,
+            node_header_length=vm_node_length + vm_migrated_length + 1,
+            bold=ansiprint.bold(),
+            end_bold=ansiprint.end(),
+            vm_header='VMs ' + ''.join(['-' for _ in range(4, vm_name_length + vm_state_length + vm_tags_length + 1)]),
+            resource_header='Resources ' + ''.join(['-' for _ in range(10, vm_nets_length + vm_ram_length + vm_vcpu_length + 1)]),
+            node_header='Node ' + ''.join(['-' for _ in range(5, vm_node_length + vm_migrated_length)])
+        )
+    )
+
+    vm_list_output.append(
+        '{bold}{vm_name: <{vm_name_length}} \
 {vm_state_colour}{vm_state: <{vm_state_length}}{end_colour} \
+{vm_tags: <{vm_tags_length}} \
 {vm_networks: <{vm_nets_length}} \
 {vm_memory: <{vm_ram_length}} {vm_vcpu: <{vm_vcpu_length}} \
 {vm_node: <{vm_node_length}} \
 {vm_migrated: <{vm_migrated_length}}{end_bold}'.format(
             vm_name_length=vm_name_length,
-            vm_uuid_length=vm_uuid_length,
             vm_state_length=vm_state_length,
+            vm_tags_length=vm_tags_length,
             vm_nets_length=vm_nets_length,
             vm_ram_length=vm_ram_length,
             vm_vcpu_length=vm_vcpu_length,
@@ -1314,20 +1725,21 @@ def format_list(config, vm_list, raw):
             vm_state_colour='',
             end_colour='',
             vm_name='Name',
-            vm_uuid='UUID',
             vm_state='State',
+            vm_tags='Tags',
             vm_networks='Networks',
             vm_memory='RAM (M)',
             vm_vcpu='vCPUs',
-            vm_node='Node',
+            vm_node='Current',
             vm_migrated='Migrated'
         )
     )
 
-    # Keep track of nets we found to be valid to cut down on duplicate API hits
-    valid_net_list = []
+    # Get a list of cluster networks for validity comparisons
+    cluster_net_list = call_api(config, 'get', '/network').json()
+
     # Format the string (elements)
-    for domain_information in vm_list:
+    for domain_information in sorted(vm_list, key=lambda v: v['name']):
         if domain_information['state'] == 'start':
             vm_state_colour = ansiprint.green()
         elif domain_information['state'] == 'restart':
@@ -1342,29 +1754,27 @@ def format_list(config, vm_list, raw):
             vm_state_colour = ansiprint.blue()
 
         # Handle colouring for an invalid network config
-        raw_net_list = getNiceNetID(domain_information)
-        net_list = []
+        net_list = getNiceNetID(domain_information)
+        tag_list = getNiceTagName(domain_information)
+        if len(tag_list) < 1:
+            tag_list = ['N/A']
         vm_net_colour = ''
-        for net_vni in raw_net_list:
-            if net_vni not in valid_net_list:
-                response = call_api(config, 'get', '/network/{net}'.format(net=net_vni))
-                if response.status_code != 200 and net_vni not in ['cluster', 'storage', 'upstream']:
+        for net_vni in net_list:
+            if net_vni not in ['cluster', 'storage', 'upstream'] and not re.match(r'^macvtap:.*', net_vni) and not re.match(r'^hostdev:.*', net_vni):
+                if int(net_vni) not in [net['vni'] for net in cluster_net_list]:
                     vm_net_colour = ansiprint.red()
-                else:
-                    valid_net_list.append(net_vni)
-
-            net_list.append(net_vni)
 
         vm_list_output.append(
-            '{bold}{vm_name: <{vm_name_length}} {vm_uuid: <{vm_uuid_length}} \
+            '{bold}{vm_name: <{vm_name_length}} \
 {vm_state_colour}{vm_state: <{vm_state_length}}{end_colour} \
+{vm_tags: <{vm_tags_length}} \
 {vm_net_colour}{vm_networks: <{vm_nets_length}}{end_colour} \
 {vm_memory: <{vm_ram_length}} {vm_vcpu: <{vm_vcpu_length}} \
 {vm_node: <{vm_node_length}} \
 {vm_migrated: <{vm_migrated_length}}{end_bold}'.format(
                 vm_name_length=vm_name_length,
-                vm_uuid_length=vm_uuid_length,
                 vm_state_length=vm_state_length,
+                vm_tags_length=vm_tags_length,
                 vm_nets_length=vm_nets_length,
                 vm_ram_length=vm_ram_length,
                 vm_vcpu_length=vm_vcpu_length,
@@ -1375,8 +1785,8 @@ def format_list(config, vm_list, raw):
                 vm_state_colour=vm_state_colour,
                 end_colour=ansiprint.end(),
                 vm_name=domain_information['name'],
-                vm_uuid=domain_information['uuid'],
                 vm_state=domain_information['state'],
+                vm_tags=','.join(tag_list),
                 vm_net_colour=vm_net_colour,
                 vm_networks=','.join(net_list),
                 vm_memory=domain_information['memory'],
@@ -1386,4 +1796,4 @@ def format_list(config, vm_list, raw):
             )
         )
 
-    return '\n'.join(sorted(vm_list_output))
+    return '\n'.join(vm_list_output)

client-cli/setup.py

@@ -0,0 +1,20 @@
+from setuptools import setup
+
+setup(
+    name='pvc',
+    version='0.9.41',
+    packages=['pvc', 'pvc.cli_lib'],
+    install_requires=[
+        'Click',
+        'PyYAML',
+        'lxml',
+        'colorama',
+        'requests',
+        'requests-toolbelt'
+    ],
+    entry_points={
+        'console_scripts': [
+            'pvc = pvc.pvc:cli',
+        ],
+    },
+)

daemon-common/ceph.py

@@ -25,6 +25,8 @@ import json
 import time
 import math
 
+from concurrent.futures import ThreadPoolExecutor
+
 import daemon_lib.vm as vm
 import daemon_lib.common as common
 
@@ -178,20 +180,63 @@ def getClusterOSDList(zkhandler):
 
 
 def getOSDInformation(zkhandler, osd_id):
+    # Get the devices
+    osd_device = zkhandler.read(('osd.device', osd_id))
+    osd_db_device = zkhandler.read(('osd.db_device', osd_id))
     # Parse the stats data
     osd_stats_raw = zkhandler.read(('osd.stats', osd_id))
     osd_stats = dict(json.loads(osd_stats_raw))
 
     osd_information = {
         'id': osd_id,
-        'stats': osd_stats
+        'device': osd_device,
+        'db_device': osd_db_device,
+        'stats': osd_stats,
     }
     return osd_information
 
 
+# OSD DB VG actions use the /cmd/ceph pipe
+# These actions must occur on the specific node they reference
+def add_osd_db_vg(zkhandler, node, device):
+    # Verify the target node exists
+    if not common.verifyNode(zkhandler, node):
+        return False, 'ERROR: No node named "{}" is present in the cluster.'.format(node)
+
+    # Tell the cluster to create a new OSD for the host
+    add_osd_db_vg_string = 'db_vg_add {},{}'.format(node, device)
+    zkhandler.write([
+        ('base.cmd.ceph', add_osd_db_vg_string)
+    ])
+    # Wait 1/2 second for the cluster to get the message and start working
+    time.sleep(0.5)
+    # Acquire a read lock, so we get the return exclusively
+    with zkhandler.readlock('base.cmd.ceph'):
+        try:
+            result = zkhandler.read('base.cmd.ceph').split()[0]
+            if result == 'success-db_vg_add':
+                message = 'Created new OSD database VG at "{}" on node "{}".'.format(device, node)
+                success = True
+            else:
+                message = 'ERROR: Failed to create new OSD database VG; check node logs for details.'
+                success = False
+        except Exception:
+            message = 'ERROR: Command ignored by node.'
+            success = False
+
+    # Acquire a write lock to ensure things go smoothly
+    with zkhandler.writelock('base.cmd.ceph'):
+        time.sleep(0.5)
+        zkhandler.write([
+            ('base.cmd.ceph', '')
+        ])
+
+    return success, message
+
+
 # OSD addition and removal uses the /cmd/ceph pipe
 # These actions must occur on the specific node they reference
-def add_osd(zkhandler, node, device, weight):
+def add_osd(zkhandler, node, device, weight, ext_db_flag=False, ext_db_ratio=0.05):
     # Verify the target node exists
     if not common.verifyNode(zkhandler, node):
         return False, 'ERROR: No node named "{}" is present in the cluster.'.format(node)
@@ -202,7 +247,7 @@ def add_osd(zkhandler, node, device, weight):
         return False, 'ERROR: Block device "{}" on node "{}" is used by OSD "{}"'.format(device, node, block_osd)
 
     # Tell the cluster to create a new OSD for the host
-    add_osd_string = 'osd_add {},{},{}'.format(node, device, weight)
+    add_osd_string = 'osd_add {},{},{},{},{}'.format(node, device, weight, ext_db_flag, ext_db_ratio)
     zkhandler.write([
         ('base.cmd.ceph', add_osd_string)
     ])
@@ -413,24 +458,36 @@ def remove_pool(zkhandler, name):
 
 
 def get_list_pool(zkhandler, limit, is_fuzzy=True):
-    pool_list = []
     full_pool_list = zkhandler.children('base.pool')
 
     if limit:
         if not is_fuzzy:
             limit = '^' + limit + '$'
 
+    get_pool_info = dict()
     for pool in full_pool_list:
+        is_limit_match = False
         if limit:
             try:
                 if re.match(limit, pool):
-                    pool_list.append(getPoolInformation(zkhandler, pool))
+                    is_limit_match = True
             except Exception as e:
                 return False, 'Regex Error: {}'.format(e)
         else:
-            pool_list.append(getPoolInformation(zkhandler, pool))
+            is_limit_match = True
 
-    return True, sorted(pool_list, key=lambda x: int(x['stats']['id']))
+        get_pool_info[pool] = True if is_limit_match else False
+
+    pool_execute_list = [pool for pool in full_pool_list if get_pool_info[pool]]
+    pool_data_list = list()
+    with ThreadPoolExecutor(max_workers=32, thread_name_prefix='pool_list') as executor:
+        futures = []
+        for pool in pool_execute_list:
+            futures.append(executor.submit(getPoolInformation, zkhandler, pool))
+        for future in futures:
+            pool_data_list.append(future.result())
+
+    return True, sorted(pool_data_list, key=lambda x: int(x['stats']['id']))
 
 
 #
@@ -466,18 +523,18 @@ def getVolumeInformation(zkhandler, pool, volume):
 
 
 def add_volume(zkhandler, pool, name, size):
+    # Add 'B' if the volume is in bytes
+    if re.match(r'^[0-9]+$', size):
+        size = '{}B'.format(size)
+
     # 1. Verify the size of the volume
     pool_information = getPoolInformation(zkhandler, pool)
     size_bytes = format_bytes_fromhuman(size)
     if size_bytes >= int(pool_information['stats']['free_bytes']):
         return False, 'ERROR: Requested volume size is greater than the available free space in the pool'
 
-    # Add 'B' if the volume is in bytes
-    if re.match(r'^[0-9]+$', size):
-        size = '{}B'.format(size)
-
     # 2. Create the volume
-    retcode, stdout, stderr = common.run_os_command('rbd create --size {} --image-feature layering,exclusive-lock {}/{}'.format(size, pool, name))
+    retcode, stdout, stderr = common.run_os_command('rbd create --size {} {}/{}'.format(size, pool, name))
     if retcode:
         return False, 'ERROR: Failed to create RBD volume "{}": {}'.format(name, stderr)
 
@@ -522,12 +579,22 @@ def resize_volume(zkhandler, pool, name, size):
     if not verifyVolume(zkhandler, pool, name):
         return False, 'ERROR: No volume with name "{}" is present in pool "{}".'.format(name, pool)
 
-    # 1. Resize the volume
+    # Add 'B' if the volume is in bytes
+    if re.match(r'^[0-9]+$', size):
+        size = '{}B'.format(size)
+
+    # 1. Verify the size of the volume
+    pool_information = getPoolInformation(zkhandler, pool)
+    size_bytes = format_bytes_fromhuman(size)
+    if size_bytes >= int(pool_information['stats']['free_bytes']):
+        return False, 'ERROR: Requested volume size is greater than the available free space in the pool'
+
+    # 2. Resize the volume
     retcode, stdout, stderr = common.run_os_command('rbd resize --size {} {}/{}'.format(size, pool, name))
     if retcode:
         return False, 'ERROR: Failed to resize RBD volume "{}" to size "{}" in pool "{}": {}'.format(name, size, pool, stderr)
 
-    # 2a. Determine the node running this VM if applicable
+    # 3a. Determine the node running this VM if applicable
     active_node = None
     volume_vm_name = name.split('_')[0]
     retcode, vm_info = vm.get_info(zkhandler, volume_vm_name)
@@ -537,7 +604,7 @@ def resize_volume(zkhandler, pool, name, size):
             if disk['name'] == '{}/{}'.format(pool, name):
                 active_node = vm_info['node']
                 volume_id = disk['dev']
-    # 2b. Perform a live resize in libvirt if the VM is running
+    # 3b. Perform a live resize in libvirt if the VM is running
     if active_node is not None and vm_info.get('state', '') == 'start':
         import libvirt
         # Run the libvirt command against the target host
@@ -551,11 +618,11 @@ def resize_volume(zkhandler, pool, name, size):
         except Exception:
             pass
 
-    # 2. Get volume stats
+    # 4. Get volume stats
     retcode, stdout, stderr = common.run_os_command('rbd info --format json {}/{}'.format(pool, name))
     volstats = stdout
 
-    # 3. Add the new volume to Zookeeper
+    # 5. Update the volume in Zookeeper
     zkhandler.write([
         (('volume', f'{pool}/{name}'), ''),
         (('volume.stats', f'{pool}/{name}'), volstats),
@@ -652,7 +719,6 @@ def unmap_volume(zkhandler, pool, name):
 
 
 def get_list_volume(zkhandler, pool, limit, is_fuzzy=True):
-    volume_list = []
     if pool and not verifyPool(zkhandler, pool):
         return False, 'ERROR: No pool with name "{}" is present in the cluster.'.format(pool)
 
@@ -668,18 +734,36 @@ def get_list_volume(zkhandler, pool, limit, is_fuzzy=True):
             if not re.match(r'.*\$', limit):
                 limit = limit + '.*'
 
+    get_volume_info = dict()
     for volume in full_volume_list:
         pool_name, volume_name = volume.split('/')
+        is_limit_match = False
+
+        # Check on limit
         if limit:
+            # Try to match the limit against the volume name
             try:
                 if re.match(limit, volume_name):
-                    volume_list.append(getVolumeInformation(zkhandler, pool_name, volume_name))
+                    is_limit_match = True
             except Exception as e:
                 return False, 'Regex Error: {}'.format(e)
         else:
-            volume_list.append(getVolumeInformation(zkhandler, pool_name, volume_name))
+            is_limit_match = True
 
-    return True, sorted(volume_list, key=lambda x: str(x['name']))
+        get_volume_info[volume] = True if is_limit_match else False
+
+    # Obtain our volume data in a thread pool
+    volume_execute_list = [volume for volume in full_volume_list if get_volume_info[volume]]
+    volume_data_list = list()
+    with ThreadPoolExecutor(max_workers=32, thread_name_prefix='volume_list') as executor:
+        futures = []
+        for volume in volume_execute_list:
+            pool_name, volume_name = volume.split('/')
+            futures.append(executor.submit(getVolumeInformation, zkhandler, pool_name, volume_name))
+        for future in futures:
+            volume_data_list.append(future.result())
+
+    return True, sorted(volume_data_list, key=lambda x: str(x['name']))
 
 
 #

daemon-common/cluster.py

@@ -60,7 +60,7 @@ def getClusterInformation(zkhandler):
     retcode, node_list = pvc_node.get_list(zkhandler, None)
 
     # Get vm information object list
-    retcode, vm_list = pvc_vm.get_list(zkhandler, None, None, None)
+    retcode, vm_list = pvc_vm.get_list(zkhandler, None, None, None, None)
 
     # Get network information object list
     retcode, network_list = pvc_network.get_list(zkhandler, None, None)
@@ -267,11 +267,15 @@ def cluster_initialize(zkhandler, overwrite=False):
         return False, 'ERROR: Cluster contains data and overwrite not set.'
 
     if overwrite:
-        # Delete the existing keys; ignore any errors
-        status = zkhandler.delete(zkhandler.schema.keys('base'), recursive=True)
+        # Delete the existing keys
+        for key in zkhandler.schema.keys('base'):
+            if key == 'root':
+                # Don't delete the root key
+                continue
 
-        if not status:
-            return False, 'ERROR: Failed to delete data in cluster; running nodes perhaps?'
+            status = zkhandler.delete('base.{}'.format(key), recursive=True)
+            if not status:
+                return False, 'ERROR: Failed to delete data in cluster; running nodes perhaps?'
 
     # Create the root keys
     zkhandler.schema.apply(zkhandler)

daemon-common/common.py

@@ -26,9 +26,63 @@ import subprocess
 import signal
 from json import loads
 from re import match as re_match
+from re import split as re_split
 from distutils.util import strtobool
 from threading import Thread
 from shlex import split as shlex_split
+from functools import wraps
+
+
+###############################################################################
+# Performance Profiler decorator
+###############################################################################
+
+# Get performance statistics on a function or class
+class Profiler(object):
+    def __init__(self, config):
+        self.is_debug = config['debug']
+        self.pvc_logdir = '/var/log/pvc'
+
+    def __call__(self, function):
+        if not callable(function):
+            return
+
+        if not self.is_debug:
+            return function
+
+        @wraps(function)
+        def profiler_wrapper(*args, **kwargs):
+            import cProfile
+            import pstats
+            from os import path, makedirs
+            from datetime import datetime
+
+            if not path.exists(self.pvc_logdir):
+                print('Profiler: Requested profiling of {} but no log dir present; printing instead.'.format(str(function.__name__)))
+                log_result = False
+            else:
+                log_result = True
+                profiler_logdir = '{}/profiler'.format(self.pvc_logdir)
+                if not path.exists(profiler_logdir):
+                    makedirs(profiler_logdir)
+
+            pr = cProfile.Profile()
+            pr.enable()
+
+            ret = function(*args, **kwargs)
+
+            pr.disable()
+            stats = pstats.Stats(pr)
+            stats.sort_stats(pstats.SortKey.TIME)
+
+            if log_result:
+                stats.dump_stats(filename='{}/{}_{}.log'.format(profiler_logdir, str(function.__name__), str(datetime.now()).replace(' ', '_')))
+            else:
+                print('Profiler stats for function {} at {}:'.format(str(function.__name__), str(datetime.now())))
+                stats.print_stats()
+
+            return ret
+        return profiler_wrapper
 
 
 ###############################################################################
@@ -252,6 +306,50 @@ def getDomainDiskList(zkhandler, dom_uuid):
     return disk_list
 
 
+#
+# Get a list of domain tags
+#
+def getDomainTags(zkhandler, dom_uuid):
+    """
+    Get a list of tags for domain dom_uuid
+
+    The UUID must be validated before calling this function!
+    """
+    tags = list()
+
+    for tag in zkhandler.children(('domain.meta.tags', dom_uuid)):
+        tag_type = zkhandler.read(('domain.meta.tags', dom_uuid, 'tag.type', tag))
+        protected = bool(strtobool(zkhandler.read(('domain.meta.tags', dom_uuid, 'tag.protected', tag))))
+        tags.append({'name': tag, 'type': tag_type, 'protected': protected})
+
+    return tags
+
+
+#
+# Get a set of domain metadata
+#
+def getDomainMetadata(zkhandler, dom_uuid):
+    """
+    Get the domain metadata for domain dom_uuid
+
+    The UUID must be validated before calling this function!
+    """
+    domain_node_limit = zkhandler.read(('domain.meta.node_limit', dom_uuid))
+    domain_node_selector = zkhandler.read(('domain.meta.node_selector', dom_uuid))
+    domain_node_autostart = zkhandler.read(('domain.meta.autostart', dom_uuid))
+    domain_migration_method = zkhandler.read(('domain.meta.migrate_method', dom_uuid))
+
+    if not domain_node_limit:
+        domain_node_limit = None
+    else:
+        domain_node_limit = domain_node_limit.split(',')
+
+    if not domain_node_autostart:
+        domain_node_autostart = None
+
+    return domain_node_limit, domain_node_selector, domain_node_autostart, domain_migration_method
+
+
 #
 # Get domain information from XML
 #
@@ -265,48 +363,26 @@ def getInformationFromXML(zkhandler, uuid):
     domain_lastnode = zkhandler.read(('domain.last_node', uuid))
     domain_failedreason = zkhandler.read(('domain.failed_reason', uuid))
 
-    try:
-        domain_node_limit = zkhandler.read(('domain.meta.node_limit', uuid))
-    except Exception:
-        domain_node_limit = None
-    try:
-        domain_node_selector = zkhandler.read(('domain.meta.node_selector', uuid))
-    except Exception:
-        domain_node_selector = None
-    try:
-        domain_node_autostart = zkhandler.read(('domain.meta.autostart', uuid))
-    except Exception:
-        domain_node_autostart = None
-    try:
-        domain_migration_method = zkhandler.read(('domain.meta.migrate_method', uuid))
-    except Exception:
-        domain_migration_method = None
+    domain_node_limit, domain_node_selector, domain_node_autostart, domain_migration_method = getDomainMetadata(zkhandler, uuid)
+    domain_tags = getDomainTags(zkhandler, uuid)
+    domain_profile = zkhandler.read(('domain.profile', uuid))
 
-    if not domain_node_limit:
-        domain_node_limit = None
-    else:
-        domain_node_limit = domain_node_limit.split(',')
-
-    if not domain_node_autostart:
-        domain_node_autostart = None
-
-    try:
-        domain_profile = zkhandler.read(('domain.profile', uuid))
-    except Exception:
-        domain_profile = None
-
-    try:
-        domain_vnc = zkhandler.read(('domain.console.vnc', uuid))
+    domain_vnc = zkhandler.read(('domain.console.vnc', uuid))
+    if domain_vnc:
         domain_vnc_listen, domain_vnc_port = domain_vnc.split(':')
-    except Exception:
+    else:
         domain_vnc_listen = 'None'
         domain_vnc_port = 'None'
 
     parsed_xml = getDomainXML(zkhandler, uuid)
 
-    try:
-        stats_data = loads(zkhandler.read(('domain.stats', uuid)))
-    except Exception:
+    stats_data = zkhandler.read(('domain.stats', uuid))
+    if stats_data is not None:
+        try:
+            stats_data = loads(stats_data)
+        except Exception:
+            stats_data = {}
+    else:
         stats_data = {}
 
     domain_uuid, domain_name, domain_description, domain_memory, domain_vcpu, domain_vcputopo = getDomainMainDetails(parsed_xml)
@@ -335,6 +411,7 @@ def getInformationFromXML(zkhandler, uuid):
         'node_selector': domain_node_selector,
         'node_autostart': bool(strtobool(domain_node_autostart)),
         'migration_method': domain_migration_method,
+        'tags': domain_tags,
         'description': domain_description,
         'profile': domain_profile,
         'memory': int(domain_memory),
@@ -372,23 +449,28 @@ def getDomainNetworks(parsed_xml, stats_data):
                 net_type = device.attrib.get('type')
             except Exception:
                 net_type = None
+
             try:
                 net_mac = device.mac.attrib.get('address')
             except Exception:
                 net_mac = None
+
             try:
                 net_bridge = device.source.attrib.get(net_type)
             except Exception:
                 net_bridge = None
+
             try:
                 net_model = device.model.attrib.get('type')
             except Exception:
                 net_model = None
+
             try:
                 net_stats_list = [x for x in stats_data.get('net_stats', []) if x.get('bridge') == net_bridge]
                 net_stats = net_stats_list[0]
             except Exception:
                 net_stats = {}
+
             net_rd_bytes = net_stats.get('rd_bytes', 0)
             net_rd_packets = net_stats.get('rd_packets', 0)
             net_rd_errors = net_stats.get('rd_errors', 0)
@@ -397,9 +479,19 @@ def getDomainNetworks(parsed_xml, stats_data):
             net_wr_packets = net_stats.get('wr_packets', 0)
             net_wr_errors = net_stats.get('wr_errors', 0)
             net_wr_drops = net_stats.get('wr_drops', 0)
+
+            if net_type == 'direct':
+                net_vni = 'macvtap:' + device.source.attrib.get('dev')
+                net_bridge = device.source.attrib.get('dev')
+            elif net_type == 'hostdev':
+                net_vni = 'hostdev:' + str(device.sriov_device)
+                net_bridge = str(device.sriov_device)
+            else:
+                net_vni = re_match(r'[vm]*br([0-9a-z]+)', net_bridge).group(1)
+
             net_obj = {
                 'type': net_type,
-                'vni': re_match(r'[vm]*br([0-9a-z]+)', net_bridge).group(1),
+                'vni': net_vni,
                 'mac': net_mac,
                 'source': net_bridge,
                 'model': net_model,
@@ -681,3 +773,25 @@ def removeIPAddress(ipaddr, cidrnetmask, dev):
             dev
         )
     )
+
+
+#
+# Sort a set of interface names (e.g. ens1f1v10)
+#
+def sortInterfaceNames(interface_names):
+    # We can't handle non-list inputs
+    if not isinstance(interface_names, list):
+        return interface_names
+
+    def atoi(text):
+        return int(text) if text.isdigit() else text
+
+    def natural_keys(text):
+        """
+        alist.sort(key=natural_keys) sorts in human order
+        http://nedbatchelder.com/blog/200712/human_sorting.html
+        (See Toothy's implementation in the comments)
+        """
+        return [atoi(c) for c in re_split(r'(\d+)', text)]
+
+    return sorted(interface_names, key=natural_keys)

daemon-common/log.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 
-# log.py - Output (stdout + logfile) functions
+# log.py - PVC daemon logger functions
 # Part of the Parallel Virtual Cluster (PVC) system
 #
 #    Copyright (C) 2018-2021 Joshua M. Boniface <joshua@boniface.me>
@@ -19,7 +19,13 @@
 #
 ###############################################################################
 
-import datetime
+from collections import deque
+from threading import Thread
+from queue import Queue
+from datetime import datetime
+from time import sleep
+
+from daemon_lib.zkhandler import ZKHandler
 
 
 class Logger(object):
@@ -77,17 +83,39 @@ class Logger(object):
         self.last_colour = ''
         self.last_prompt = ''
 
+        if self.config['zookeeper_logging']:
+            self.zookeeper_queue = Queue()
+            self.zookeeper_logger = ZookeeperLogger(self.config, self.zookeeper_queue)
+            self.zookeeper_logger.start()
+
     # Provide a hup function to close and reopen the writer
     def hup(self):
         self.writer.close()
         self.writer = open(self.logfile, 'a', buffering=0)
 
+    # Provide a termination function so all messages are flushed before terminating the main daemon
+    def terminate(self):
+        if self.config['file_logging']:
+            self.writer.close()
+        if self.config['zookeeper_logging']:
+            self.out("Waiting 15s for Zookeeper message queue to drain", state='s')
+
+            tick_count = 0
+            while not self.zookeeper_queue.empty():
+                sleep(0.5)
+                tick_count += 1
+                if tick_count > 30:
+                    break
+
+            self.zookeeper_logger.stop()
+            self.zookeeper_logger.join()
+
     # Output function
     def out(self, message, state=None, prefix=''):
 
         # Get the date
         if self.config['log_dates']:
-            date = '{} - '.format(datetime.datetime.now().strftime('%Y/%m/%d %H:%M:%S.%f'))
+            date = '{} '.format(datetime.now().strftime('%Y/%m/%d %H:%M:%S.%f'))
         else:
             date = ''
 
@@ -123,6 +151,98 @@ class Logger(object):
         if self.config['file_logging']:
             self.writer.write(message + '\n')
 
+        # Log to Zookeeper
+        if self.config['zookeeper_logging']:
+            self.zookeeper_queue.put(message)
+
         # Set last message variables
         self.last_colour = colour
         self.last_prompt = prompt
+
+
+class ZookeeperLogger(Thread):
+    """
+    Defines a threaded writer for Zookeeper locks. Threading prevents the blocking of other
+    daemon events while the records are written. They will be eventually-consistent
+    """
+    def __init__(self, config, zookeeper_queue):
+        self.config = config
+        self.node = self.config['node']
+        self.max_lines = self.config['node_log_lines']
+        self.zookeeper_queue = zookeeper_queue
+        self.connected = False
+        self.running = False
+        self.zkhandler = None
+        Thread.__init__(self, args=(), kwargs=None)
+
+    def start_zkhandler(self):
+        # We must open our own dedicated Zookeeper instance because we can't guarantee one already exists when this starts
+        if self.zkhandler is not None:
+            try:
+                self.zkhandler.disconnect()
+            except Exception:
+                pass
+
+        while True:
+            try:
+                self.zkhandler = ZKHandler(self.config, logger=None)
+                self.zkhandler.connect(persistent=True)
+                break
+            except Exception:
+                sleep(0.5)
+                continue
+
+        self.connected = True
+
+        # Ensure the root keys for this are instantiated
+        self.zkhandler.write([
+            ('base.logs', ''),
+            (('logs', self.node), '')
+        ])
+
+    def run(self):
+        while not self.connected:
+            self.start_zkhandler()
+            sleep(1)
+
+        self.running = True
+        # Get the logs that are currently in Zookeeper and populate our deque
+        raw_logs = self.zkhandler.read(('logs.messages', self.node))
+        if raw_logs is None:
+            raw_logs = ''
+        logs = deque(raw_logs.split('\n'), self.max_lines)
+        while self.running:
+            # Get a new message
+            try:
+                message = self.zookeeper_queue.get(timeout=1)
+                if not message:
+                    continue
+            except Exception:
+                continue
+
+            if not self.config['log_dates']:
+                # We want to log dates here, even if the log_dates config is not set
+                date = '{} '.format(datetime.now().strftime('%Y/%m/%d %H:%M:%S.%f'))
+            else:
+                date = ''
+            # Add the message to the deque
+            logs.append(f'{date}{message}')
+
+            tick_count = 0
+            while True:
+                try:
+                    # Write the updated messages into Zookeeper
+                    self.zkhandler.write([(('logs.messages', self.node), '\n'.join(logs))])
+                    break
+                except Exception:
+                    # The write failed (connection loss, etc.) so retry for 15 seconds
+                    sleep(0.5)
+                    tick_count += 1
+                    if tick_count > 30:
+                        break
+                    else:
+                        continue
+        return
+
+    def stop(self):
+        self.running = False

daemon-common/migrations/versions/1.json

@@ -0,0 +1 @@
+{"version": "1", "root": "", "base": {"root": "", "schema": "/schema", "schema.version": "/schema/version", "config": "/config", "config.maintenance": "/config/maintenance", "config.primary_node": "/config/primary_node", "config.primary_node.sync_lock": "/config/primary_node/sync_lock", "config.upstream_ip": "/config/upstream_ip", "config.migration_target_selector": "/config/migration_target_selector", "cmd": "/cmd", "cmd.node": "/cmd/nodes", "cmd.domain": "/cmd/domains", "cmd.ceph": "/cmd/ceph", "node": "/nodes", "domain": "/domains", "network": "/networks", "storage": "/ceph", "storage.util": "/ceph/util", "osd": "/ceph/osds", "pool": "/ceph/pools", "volume": "/ceph/volumes", "snapshot": "/ceph/snapshots"}, "node": {"name": "", "keepalive": "/keepalive", "mode": "/daemonmode", "data.active_schema": "/activeschema", "data.latest_schema": "/latestschema", "data.static": "/staticdata", "running_domains": "/runningdomains", "count.provisioned_domains": "/domainscount", "count.networks": "/networkscount", "state.daemon": "/daemonstate", "state.router": "/routerstate", "state.domain": "/domainstate", "cpu.load": "/cpuload", "vcpu.allocated": "/vcpualloc", "memory.total": "/memtotal", "memory.used": "/memused", "memory.free": "/memfree", "memory.allocated": "/memalloc", "memory.provisioned": "/memprov", "ipmi.hostname": "/ipmihostname", "ipmi.username": "/ipmiusername", "ipmi.password": "/ipmipassword", "sriov": "/sriov", "sriov.pf": "/sriov/pf", "sriov.vf": "/sriov/vf"}, "sriov_pf": {"phy": "", "mtu": "/mtu", "vfcount": "/vfcount"}, "sriov_vf": {"phy": "", "pf": "/pf", "mtu": "/mtu", "mac": "/mac", "phy_mac": "/phy_mac", "config": "/config", "config.vlan_id": "/config/vlan_id", "config.vlan_qos": "/config/vlan_qos", "config.tx_rate_min": "/config/tx_rate_min", "config.tx_rate_max": "/config/tx_rate_max", "config.spoof_check": "/config/spoof_check", "config.link_state": "/config/link_state", "config.trust": "/config/trust", "config.query_rss": "/config/query_rss", "pci": "/pci", "pci.domain": "/pci/domain", "pci.bus": "/pci/bus", "pci.slot": "/pci/slot", "pci.function": "/pci/function", "used": "/used", "used_by": "/used_by"}, "domain": {"name": "", "xml": "/xml", "state": "/state", "profile": "/profile", "stats": "/stats", "node": "/node", "last_node": "/lastnode", "failed_reason": "/failedreason", "storage.volumes": "/rbdlist", "console.log": "/consolelog", "console.vnc": "/vnc", "meta.autostart": "/node_autostart", "meta.migrate_method": "/migration_method", "meta.node_selector": "/node_selector", "meta.node_limit": "/node_limit", "migrate.sync_lock": "/migrate_sync_lock"}, "network": {"vni": "", "type": "/nettype", "rule": "/firewall_rules", "rule.in": "/firewall_rules/in", "rule.out": "/firewall_rules/out", "nameservers": "/name_servers", "domain": "/domain", "reservation": "/dhcp4_reservations", "lease": "/dhcp4_leases", "ip4.gateway": "/ip4_gateway", "ip4.network": "/ip4_network", "ip4.dhcp": "/dhcp4_flag", "ip4.dhcp_start": "/dhcp4_start", "ip4.dhcp_end": "/dhcp4_end", "ip6.gateway": "/ip6_gateway", "ip6.network": "/ip6_network", "ip6.dhcp": "/dhcp6_flag"}, "reservation": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname"}, "lease": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname", "expiry": "/expiry", "client_id": "/clientid"}, "rule": {"description": "", "rule": "/rule", "order": "/order"}, "osd": {"id": "", "node": "/node", "device": "/device", "stats": "/stats"}, "pool": {"name": "", "pgs": "/pgs", "stats": "/stats"}, "volume": {"name": "", "stats": "/stats"}, "snapshot": {"name": "", "stats": "/stats"}}

daemon-common/migrations/versions/2.json

@@ -0,0 +1 @@
+{"version": "2", "root": "", "base": {"root": "", "schema": "/schema", "schema.version": "/schema/version", "config": "/config", "config.maintenance": "/config/maintenance", "config.primary_node": "/config/primary_node", "config.primary_node.sync_lock": "/config/primary_node/sync_lock", "config.upstream_ip": "/config/upstream_ip", "config.migration_target_selector": "/config/migration_target_selector", "cmd": "/cmd", "cmd.node": "/cmd/nodes", "cmd.domain": "/cmd/domains", "cmd.ceph": "/cmd/ceph", "node": "/nodes", "domain": "/domains", "network": "/networks", "storage": "/ceph", "storage.util": "/ceph/util", "osd": "/ceph/osds", "pool": "/ceph/pools", "volume": "/ceph/volumes", "snapshot": "/ceph/snapshots"}, "node": {"name": "", "keepalive": "/keepalive", "mode": "/daemonmode", "data.active_schema": "/activeschema", "data.latest_schema": "/latestschema", "data.static": "/staticdata", "data.pvc_version": "/pvcversion", "running_domains": "/runningdomains", "count.provisioned_domains": "/domainscount", "count.networks": "/networkscount", "state.daemon": "/daemonstate", "state.router": "/routerstate", "state.domain": "/domainstate", "cpu.load": "/cpuload", "vcpu.allocated": "/vcpualloc", "memory.total": "/memtotal", "memory.used": "/memused", "memory.free": "/memfree", "memory.allocated": "/memalloc", "memory.provisioned": "/memprov", "ipmi.hostname": "/ipmihostname", "ipmi.username": "/ipmiusername", "ipmi.password": "/ipmipassword", "sriov": "/sriov", "sriov.pf": "/sriov/pf", "sriov.vf": "/sriov/vf"}, "sriov_pf": {"phy": "", "mtu": "/mtu", "vfcount": "/vfcount"}, "sriov_vf": {"phy": "", "pf": "/pf", "mtu": "/mtu", "mac": "/mac", "phy_mac": "/phy_mac", "config": "/config", "config.vlan_id": "/config/vlan_id", "config.vlan_qos": "/config/vlan_qos", "config.tx_rate_min": "/config/tx_rate_min", "config.tx_rate_max": "/config/tx_rate_max", "config.spoof_check": "/config/spoof_check", "config.link_state": "/config/link_state", "config.trust": "/config/trust", "config.query_rss": "/config/query_rss", "pci": "/pci", "pci.domain": "/pci/domain", "pci.bus": "/pci/bus", "pci.slot": "/pci/slot", "pci.function": "/pci/function", "used": "/used", "used_by": "/used_by"}, "domain": {"name": "", "xml": "/xml", "state": "/state", "profile": "/profile", "stats": "/stats", "node": "/node", "last_node": "/lastnode", "failed_reason": "/failedreason", "storage.volumes": "/rbdlist", "console.log": "/consolelog", "console.vnc": "/vnc", "meta.autostart": "/node_autostart", "meta.migrate_method": "/migration_method", "meta.node_selector": "/node_selector", "meta.node_limit": "/node_limit", "migrate.sync_lock": "/migrate_sync_lock"}, "network": {"vni": "", "type": "/nettype", "rule": "/firewall_rules", "rule.in": "/firewall_rules/in", "rule.out": "/firewall_rules/out", "nameservers": "/name_servers", "domain": "/domain", "reservation": "/dhcp4_reservations", "lease": "/dhcp4_leases", "ip4.gateway": "/ip4_gateway", "ip4.network": "/ip4_network", "ip4.dhcp": "/dhcp4_flag", "ip4.dhcp_start": "/dhcp4_start", "ip4.dhcp_end": "/dhcp4_end", "ip6.gateway": "/ip6_gateway", "ip6.network": "/ip6_network", "ip6.dhcp": "/dhcp6_flag"}, "reservation": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname"}, "lease": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname", "expiry": "/expiry", "client_id": "/clientid"}, "rule": {"description": "", "rule": "/rule", "order": "/order"}, "osd": {"id": "", "node": "/node", "device": "/device", "stats": "/stats"}, "pool": {"name": "", "pgs": "/pgs", "stats": "/stats"}, "volume": {"name": "", "stats": "/stats"}, "snapshot": {"name": "", "stats": "/stats"}}

daemon-common/migrations/versions/3.json

@@ -0,0 +1 @@
+{"version": "3", "root": "", "base": {"root": "", "schema": "/schema", "schema.version": "/schema/version", "config": "/config", "config.maintenance": "/config/maintenance", "config.primary_node": "/config/primary_node", "config.primary_node.sync_lock": "/config/primary_node/sync_lock", "config.upstream_ip": "/config/upstream_ip", "config.migration_target_selector": "/config/migration_target_selector", "cmd": "/cmd", "cmd.node": "/cmd/nodes", "cmd.domain": "/cmd/domains", "cmd.ceph": "/cmd/ceph", "node": "/nodes", "domain": "/domains", "network": "/networks", "storage": "/ceph", "storage.util": "/ceph/util", "osd": "/ceph/osds", "pool": "/ceph/pools", "volume": "/ceph/volumes", "snapshot": "/ceph/snapshots"}, "node": {"name": "", "keepalive": "/keepalive", "mode": "/daemonmode", "data.active_schema": "/activeschema", "data.latest_schema": "/latestschema", "data.static": "/staticdata", "data.pvc_version": "/pvcversion", "running_domains": "/runningdomains", "count.provisioned_domains": "/domainscount", "count.networks": "/networkscount", "state.daemon": "/daemonstate", "state.router": "/routerstate", "state.domain": "/domainstate", "cpu.load": "/cpuload", "vcpu.allocated": "/vcpualloc", "memory.total": "/memtotal", "memory.used": "/memused", "memory.free": "/memfree", "memory.allocated": "/memalloc", "memory.provisioned": "/memprov", "ipmi.hostname": "/ipmihostname", "ipmi.username": "/ipmiusername", "ipmi.password": "/ipmipassword", "sriov": "/sriov", "sriov.pf": "/sriov/pf", "sriov.vf": "/sriov/vf"}, "sriov_pf": {"phy": "", "mtu": "/mtu", "vfcount": "/vfcount"}, "sriov_vf": {"phy": "", "pf": "/pf", "mtu": "/mtu", "mac": "/mac", "phy_mac": "/phy_mac", "config": "/config", "config.vlan_id": "/config/vlan_id", "config.vlan_qos": "/config/vlan_qos", "config.tx_rate_min": "/config/tx_rate_min", "config.tx_rate_max": "/config/tx_rate_max", "config.spoof_check": "/config/spoof_check", "config.link_state": "/config/link_state", "config.trust": "/config/trust", "config.query_rss": "/config/query_rss", "pci": "/pci", "pci.domain": "/pci/domain", "pci.bus": "/pci/bus", "pci.slot": "/pci/slot", "pci.function": "/pci/function", "used": "/used", "used_by": "/used_by"}, "domain": {"name": "", "xml": "/xml", "state": "/state", "profile": "/profile", "stats": "/stats", "node": "/node", "last_node": "/lastnode", "failed_reason": "/failedreason", "storage.volumes": "/rbdlist", "console.log": "/consolelog", "console.vnc": "/vnc", "meta.autostart": "/node_autostart", "meta.migrate_method": "/migration_method", "meta.node_selector": "/node_selector", "meta.node_limit": "/node_limit", "meta.tags": "/tags", "migrate.sync_lock": "/migrate_sync_lock"}, "tag": {"name": "", "type": "/type", "protected": "/protected"}, "network": {"vni": "", "type": "/nettype", "rule": "/firewall_rules", "rule.in": "/firewall_rules/in", "rule.out": "/firewall_rules/out", "nameservers": "/name_servers", "domain": "/domain", "reservation": "/dhcp4_reservations", "lease": "/dhcp4_leases", "ip4.gateway": "/ip4_gateway", "ip4.network": "/ip4_network", "ip4.dhcp": "/dhcp4_flag", "ip4.dhcp_start": "/dhcp4_start", "ip4.dhcp_end": "/dhcp4_end", "ip6.gateway": "/ip6_gateway", "ip6.network": "/ip6_network", "ip6.dhcp": "/dhcp6_flag"}, "reservation": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname"}, "lease": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname", "expiry": "/expiry", "client_id": "/clientid"}, "rule": {"description": "", "rule": "/rule", "order": "/order"}, "osd": {"id": "", "node": "/node", "device": "/device", "stats": "/stats"}, "pool": {"name": "", "pgs": "/pgs", "stats": "/stats"}, "volume": {"name": "", "stats": "/stats"}, "snapshot": {"name": "", "stats": "/stats"}}

daemon-common/migrations/versions/4.json

@@ -0,0 +1 @@
+{"version": "4", "root": "", "base": {"root": "", "schema": "/schema", "schema.version": "/schema/version", "config": "/config", "config.maintenance": "/config/maintenance", "config.primary_node": "/config/primary_node", "config.primary_node.sync_lock": "/config/primary_node/sync_lock", "config.upstream_ip": "/config/upstream_ip", "config.migration_target_selector": "/config/migration_target_selector", "cmd": "/cmd", "cmd.node": "/cmd/nodes", "cmd.domain": "/cmd/domains", "cmd.ceph": "/cmd/ceph", "logs": "/logs", "node": "/nodes", "domain": "/domains", "network": "/networks", "storage": "/ceph", "storage.util": "/ceph/util", "osd": "/ceph/osds", "pool": "/ceph/pools", "volume": "/ceph/volumes", "snapshot": "/ceph/snapshots"}, "logs": {"node": "", "messages": "/messages"}, "node": {"name": "", "keepalive": "/keepalive", "mode": "/daemonmode", "data.active_schema": "/activeschema", "data.latest_schema": "/latestschema", "data.static": "/staticdata", "data.pvc_version": "/pvcversion", "running_domains": "/runningdomains", "count.provisioned_domains": "/domainscount", "count.networks": "/networkscount", "state.daemon": "/daemonstate", "state.router": "/routerstate", "state.domain": "/domainstate", "cpu.load": "/cpuload", "vcpu.allocated": "/vcpualloc", "memory.total": "/memtotal", "memory.used": "/memused", "memory.free": "/memfree", "memory.allocated": "/memalloc", "memory.provisioned": "/memprov", "ipmi.hostname": "/ipmihostname", "ipmi.username": "/ipmiusername", "ipmi.password": "/ipmipassword", "sriov": "/sriov", "sriov.pf": "/sriov/pf", "sriov.vf": "/sriov/vf"}, "sriov_pf": {"phy": "", "mtu": "/mtu", "vfcount": "/vfcount"}, "sriov_vf": {"phy": "", "pf": "/pf", "mtu": "/mtu", "mac": "/mac", "phy_mac": "/phy_mac", "config": "/config", "config.vlan_id": "/config/vlan_id", "config.vlan_qos": "/config/vlan_qos", "config.tx_rate_min": "/config/tx_rate_min", "config.tx_rate_max": "/config/tx_rate_max", "config.spoof_check": "/config/spoof_check", "config.link_state": "/config/link_state", "config.trust": "/config/trust", "config.query_rss": "/config/query_rss", "pci": "/pci", "pci.domain": "/pci/domain", "pci.bus": "/pci/bus", "pci.slot": "/pci/slot", "pci.function": "/pci/function", "used": "/used", "used_by": "/used_by"}, "domain": {"name": "", "xml": "/xml", "state": "/state", "profile": "/profile", "stats": "/stats", "node": "/node", "last_node": "/lastnode", "failed_reason": "/failedreason", "storage.volumes": "/rbdlist", "console.log": "/consolelog", "console.vnc": "/vnc", "meta.autostart": "/node_autostart", "meta.migrate_method": "/migration_method", "meta.node_selector": "/node_selector", "meta.node_limit": "/node_limit", "meta.tags": "/tags", "migrate.sync_lock": "/migrate_sync_lock"}, "tag": {"name": "", "type": "/type", "protected": "/protected"}, "network": {"vni": "", "type": "/nettype", "rule": "/firewall_rules", "rule.in": "/firewall_rules/in", "rule.out": "/firewall_rules/out", "nameservers": "/name_servers", "domain": "/domain", "reservation": "/dhcp4_reservations", "lease": "/dhcp4_leases", "ip4.gateway": "/ip4_gateway", "ip4.network": "/ip4_network", "ip4.dhcp": "/dhcp4_flag", "ip4.dhcp_start": "/dhcp4_start", "ip4.dhcp_end": "/dhcp4_end", "ip6.gateway": "/ip6_gateway", "ip6.network": "/ip6_network", "ip6.dhcp": "/dhcp6_flag"}, "reservation": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname"}, "lease": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname", "expiry": "/expiry", "client_id": "/clientid"}, "rule": {"description": "", "rule": "/rule", "order": "/order"}, "osd": {"id": "", "node": "/node", "device": "/device", "stats": "/stats"}, "pool": {"name": "", "pgs": "/pgs", "stats": "/stats"}, "volume": {"name": "", "stats": "/stats"}, "snapshot": {"name": "", "stats": "/stats"}}

daemon-common/migrations/versions/5.json

@@ -0,0 +1 @@
+{"version": "5", "root": "", "base": {"root": "", "schema": "/schema", "schema.version": "/schema/version", "config": "/config", "config.maintenance": "/config/maintenance", "config.primary_node": "/config/primary_node", "config.primary_node.sync_lock": "/config/primary_node/sync_lock", "config.upstream_ip": "/config/upstream_ip", "config.migration_target_selector": "/config/migration_target_selector", "cmd": "/cmd", "cmd.node": "/cmd/nodes", "cmd.domain": "/cmd/domains", "cmd.ceph": "/cmd/ceph", "logs": "/logs", "node": "/nodes", "domain": "/domains", "network": "/networks", "storage": "/ceph", "storage.util": "/ceph/util", "osd": "/ceph/osds", "pool": "/ceph/pools", "volume": "/ceph/volumes", "snapshot": "/ceph/snapshots"}, "logs": {"node": "", "messages": "/messages"}, "node": {"name": "", "keepalive": "/keepalive", "mode": "/daemonmode", "data.active_schema": "/activeschema", "data.latest_schema": "/latestschema", "data.static": "/staticdata", "data.pvc_version": "/pvcversion", "running_domains": "/runningdomains", "count.provisioned_domains": "/domainscount", "count.networks": "/networkscount", "state.daemon": "/daemonstate", "state.router": "/routerstate", "state.domain": "/domainstate", "cpu.load": "/cpuload", "vcpu.allocated": "/vcpualloc", "memory.total": "/memtotal", "memory.used": "/memused", "memory.free": "/memfree", "memory.allocated": "/memalloc", "memory.provisioned": "/memprov", "ipmi.hostname": "/ipmihostname", "ipmi.username": "/ipmiusername", "ipmi.password": "/ipmipassword", "sriov": "/sriov", "sriov.pf": "/sriov/pf", "sriov.vf": "/sriov/vf"}, "sriov_pf": {"phy": "", "mtu": "/mtu", "vfcount": "/vfcount"}, "sriov_vf": {"phy": "", "pf": "/pf", "mtu": "/mtu", "mac": "/mac", "phy_mac": "/phy_mac", "config": "/config", "config.vlan_id": "/config/vlan_id", "config.vlan_qos": "/config/vlan_qos", "config.tx_rate_min": "/config/tx_rate_min", "config.tx_rate_max": "/config/tx_rate_max", "config.spoof_check": "/config/spoof_check", "config.link_state": "/config/link_state", "config.trust": "/config/trust", "config.query_rss": "/config/query_rss", "pci": "/pci", "pci.domain": "/pci/domain", "pci.bus": "/pci/bus", "pci.slot": "/pci/slot", "pci.function": "/pci/function", "used": "/used", "used_by": "/used_by"}, "domain": {"name": "", "xml": "/xml", "state": "/state", "profile": "/profile", "stats": "/stats", "node": "/node", "last_node": "/lastnode", "failed_reason": "/failedreason", "storage.volumes": "/rbdlist", "console.log": "/consolelog", "console.vnc": "/vnc", "meta.autostart": "/node_autostart", "meta.migrate_method": "/migration_method", "meta.node_selector": "/node_selector", "meta.node_limit": "/node_limit", "meta.tags": "/tags", "migrate.sync_lock": "/migrate_sync_lock"}, "tag": {"name": "", "type": "/type", "protected": "/protected"}, "network": {"vni": "", "type": "/nettype", "rule": "/firewall_rules", "rule.in": "/firewall_rules/in", "rule.out": "/firewall_rules/out", "nameservers": "/name_servers", "domain": "/domain", "reservation": "/dhcp4_reservations", "lease": "/dhcp4_leases", "ip4.gateway": "/ip4_gateway", "ip4.network": "/ip4_network", "ip4.dhcp": "/dhcp4_flag", "ip4.dhcp_start": "/dhcp4_start", "ip4.dhcp_end": "/dhcp4_end", "ip6.gateway": "/ip6_gateway", "ip6.network": "/ip6_network", "ip6.dhcp": "/dhcp6_flag"}, "reservation": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname"}, "lease": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname", "expiry": "/expiry", "client_id": "/clientid"}, "rule": {"description": "", "rule": "/rule", "order": "/order"}, "osd": {"id": "", "node": "/node", "device": "/device", "db_device": "/db_device", "stats": "/stats"}, "pool": {"name": "", "pgs": "/pgs", "stats": "/stats"}, "volume": {"name": "", "stats": "/stats"}, "snapshot": {"name": "", "stats": "/stats"}}

daemon-common/migrations/versions/6.json

@@ -0,0 +1 @@
+{"version": "6", "root": "", "base": {"root": "", "schema": "/schema", "schema.version": "/schema/version", "config": "/config", "config.maintenance": "/config/maintenance", "config.primary_node": "/config/primary_node", "config.primary_node.sync_lock": "/config/primary_node/sync_lock", "config.upstream_ip": "/config/upstream_ip", "config.migration_target_selector": "/config/migration_target_selector", "cmd": "/cmd", "cmd.node": "/cmd/nodes", "cmd.domain": "/cmd/domains", "cmd.ceph": "/cmd/ceph", "logs": "/logs", "node": "/nodes", "domain": "/domains", "network": "/networks", "storage": "/ceph", "storage.util": "/ceph/util", "osd": "/ceph/osds", "pool": "/ceph/pools", "volume": "/ceph/volumes", "snapshot": "/ceph/snapshots"}, "logs": {"node": "", "messages": "/messages"}, "node": {"name": "", "keepalive": "/keepalive", "mode": "/daemonmode", "data.active_schema": "/activeschema", "data.latest_schema": "/latestschema", "data.static": "/staticdata", "data.pvc_version": "/pvcversion", "running_domains": "/runningdomains", "count.provisioned_domains": "/domainscount", "count.networks": "/networkscount", "state.daemon": "/daemonstate", "state.router": "/routerstate", "state.domain": "/domainstate", "cpu.load": "/cpuload", "vcpu.allocated": "/vcpualloc", "memory.total": "/memtotal", "memory.used": "/memused", "memory.free": "/memfree", "memory.allocated": "/memalloc", "memory.provisioned": "/memprov", "ipmi.hostname": "/ipmihostname", "ipmi.username": "/ipmiusername", "ipmi.password": "/ipmipassword", "sriov": "/sriov", "sriov.pf": "/sriov/pf", "sriov.vf": "/sriov/vf"}, "sriov_pf": {"phy": "", "mtu": "/mtu", "vfcount": "/vfcount"}, "sriov_vf": {"phy": "", "pf": "/pf", "mtu": "/mtu", "mac": "/mac", "phy_mac": "/phy_mac", "config": "/config", "config.vlan_id": "/config/vlan_id", "config.vlan_qos": "/config/vlan_qos", "config.tx_rate_min": "/config/tx_rate_min", "config.tx_rate_max": "/config/tx_rate_max", "config.spoof_check": "/config/spoof_check", "config.link_state": "/config/link_state", "config.trust": "/config/trust", "config.query_rss": "/config/query_rss", "pci": "/pci", "pci.domain": "/pci/domain", "pci.bus": "/pci/bus", "pci.slot": "/pci/slot", "pci.function": "/pci/function", "used": "/used", "used_by": "/used_by"}, "domain": {"name": "", "xml": "/xml", "state": "/state", "profile": "/profile", "stats": "/stats", "node": "/node", "last_node": "/lastnode", "failed_reason": "/failedreason", "storage.volumes": "/rbdlist", "console.log": "/consolelog", "console.vnc": "/vnc", "meta.autostart": "/node_autostart", "meta.migrate_method": "/migration_method", "meta.node_selector": "/node_selector", "meta.node_limit": "/node_limit", "meta.tags": "/tags", "migrate.sync_lock": "/migrate_sync_lock"}, "tag": {"name": "", "type": "/type", "protected": "/protected"}, "network": {"vni": "", "type": "/nettype", "mtu": "/mtu", "rule": "/firewall_rules", "rule.in": "/firewall_rules/in", "rule.out": "/firewall_rules/out", "nameservers": "/name_servers", "domain": "/domain", "reservation": "/dhcp4_reservations", "lease": "/dhcp4_leases", "ip4.gateway": "/ip4_gateway", "ip4.network": "/ip4_network", "ip4.dhcp": "/dhcp4_flag", "ip4.dhcp_start": "/dhcp4_start", "ip4.dhcp_end": "/dhcp4_end", "ip6.gateway": "/ip6_gateway", "ip6.network": "/ip6_network", "ip6.dhcp": "/dhcp6_flag"}, "reservation": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname"}, "lease": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname", "expiry": "/expiry", "client_id": "/clientid"}, "rule": {"description": "", "rule": "/rule", "order": "/order"}, "osd": {"id": "", "node": "/node", "device": "/device", "db_device": "/db_device", "stats": "/stats"}, "pool": {"name": "", "pgs": "/pgs", "stats": "/stats"}, "volume": {"name": "", "stats": "/stats"}, "snapshot": {"name": "", "stats": "/stats"}}

daemon-common/network.py

@@ -21,6 +21,8 @@
 
 import re
 
+import daemon_lib.common as common
+
 
 #
 # Cluster search functions
@@ -133,6 +135,7 @@ def getNetworkACLs(zkhandler, vni, _direction):
 def getNetworkInformation(zkhandler, vni):
     description = zkhandler.read(('network', vni))
     nettype = zkhandler.read(('network.type', vni))
+    mtu = zkhandler.read(('network.mtu', vni))
     domain = zkhandler.read(('network.domain', vni))
     name_servers = zkhandler.read(('network.nameservers', vni))
     ip6_network = zkhandler.read(('network.ip6.network', vni))
@@ -149,6 +152,7 @@ def getNetworkInformation(zkhandler, vni):
         'vni': int(vni),
         'description': description,
         'type': nettype,
+        'mtu': mtu,
         'domain': domain,
         'name_servers': name_servers.split(','),
         'ip6': {
@@ -233,7 +237,7 @@ def isValidIP(ipaddr):
 #
 # Direct functions
 #
-def add_network(zkhandler, vni, description, nettype,
+def add_network(zkhandler, vni, description, nettype, mtu,
                 domain, name_servers, ip4_network, ip4_gateway, ip6_network, ip6_gateway,
                 dhcp4_flag, dhcp4_start, dhcp4_end):
     # Ensure start and end DHCP ranges are set if the flag is set
@@ -265,6 +269,7 @@ def add_network(zkhandler, vni, description, nettype,
     result = zkhandler.write([
         (('network', vni), description),
         (('network.type', vni), nettype),
+        (('network.mtu', vni), mtu),
         (('network.domain', vni), domain),
         (('network.nameservers', vni), name_servers),
         (('network.ip6.network', vni), ip6_network),
@@ -288,13 +293,15 @@ def add_network(zkhandler, vni, description, nettype,
         return False, 'ERROR: Failed to add network.'
 
 
-def modify_network(zkhandler, vni, description=None, domain=None, name_servers=None,
+def modify_network(zkhandler, vni, description=None, mtu=None, domain=None, name_servers=None,
                    ip4_network=None, ip4_gateway=None, ip6_network=None, ip6_gateway=None,
                    dhcp4_flag=None, dhcp4_start=None, dhcp4_end=None):
     # Add the modified parameters to Zookeeper
     update_data = list()
     if description is not None:
         update_data.append((('network', vni), description))
+    if mtu is not None:
+        update_data.append((('network.mtu', vni), mtu))
     if domain is not None:
         update_data.append((('network.domain', vni), domain))
     if name_servers is not None:
@@ -629,3 +636,226 @@ def get_list_acl(zkhandler, network, limit, direction, is_fuzzy=True):
             acl_list.append(acl)
 
     return True, acl_list
+
+
+#
+# SR-IOV functions
+#
+# These are separate since they don't work like other network types
+#
+def getSRIOVPFInformation(zkhandler, node, pf):
+    mtu = zkhandler.read(('node.sriov.pf', node, 'sriov_pf.mtu', pf))
+
+    retcode, vf_list = get_list_sriov_vf(zkhandler, node, pf)
+    if retcode:
+        vfs = common.sortInterfaceNames([vf['phy'] for vf in vf_list if vf['pf'] == pf])
+    else:
+        vfs = []
+
+    # Construct a data structure to represent the data
+    pf_information = {
+        'phy': pf,
+        'mtu': mtu,
+        'vfs': vfs,
+    }
+    return pf_information
+
+
+def get_info_sriov_pf(zkhandler, node, pf):
+    pf_information = getSRIOVPFInformation(zkhandler, node, pf)
+    if not pf_information:
+        return False, 'ERROR: Could not get information about SR-IOV PF "{}" on node "{}"'.format(pf, node)
+
+    return True, pf_information
+
+
+def get_list_sriov_pf(zkhandler, node):
+    pf_list = list()
+    pf_phy_list = zkhandler.children(('node.sriov.pf', node))
+    for phy in pf_phy_list:
+        retcode, pf_information = get_info_sriov_pf(zkhandler, node, phy)
+        if retcode:
+            pf_list.append(pf_information)
+
+    return True, pf_list
+
+
+def getSRIOVVFInformation(zkhandler, node, vf):
+    if not zkhandler.exists(('node.sriov.vf', node, 'sriov_vf', vf)):
+        return []
+
+    pf = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.pf', vf))
+    mtu = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.mtu', vf))
+    mac = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.mac', vf))
+    vlan_id = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.config.vlan_id', vf))
+    vlan_qos = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.config.vlan_qos', vf))
+    tx_rate_min = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.config.tx_rate_min', vf))
+    tx_rate_max = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.config.tx_rate_max', vf))
+    link_state = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.config.link_state', vf))
+    spoof_check = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.config.spoof_check', vf))
+    trust = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.config.trust', vf))
+    query_rss = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.config.query_rss', vf))
+    pci_domain = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.pci.domain', vf))
+    pci_bus = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.pci.bus', vf))
+    pci_slot = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.pci.slot', vf))
+    pci_function = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.pci.function', vf))
+    used = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.used', vf))
+    used_by_domain = zkhandler.read(('node.sriov.vf', node, 'sriov_vf.used_by', vf))
+
+    vf_information = {
+        'phy': vf,
+        'pf': pf,
+        'mtu': mtu,
+        'mac': mac,
+        'config': {
+            'vlan_id': vlan_id,
+            'vlan_qos': vlan_qos,
+            'tx_rate_min': tx_rate_min,
+            'tx_rate_max': tx_rate_max,
+            'link_state': link_state,
+            'spoof_check': spoof_check,
+            'trust': trust,
+            'query_rss': query_rss,
+        },
+        'pci': {
+            'domain': pci_domain,
+            'bus': pci_bus,
+            'slot': pci_slot,
+            'function': pci_function,
+        },
+        'usage': {
+            'used': used,
+            'domain': used_by_domain,
+        }
+    }
+    return vf_information
+
+
+def get_info_sriov_vf(zkhandler, node, vf):
+    # Verify node is valid
+    valid_node = common.verifyNode(zkhandler, node)
+    if not valid_node:
+        return False, 'ERROR: Specified node "{}" is invalid.'.format(node)
+
+    vf_information = getSRIOVVFInformation(zkhandler, node, vf)
+    if not vf_information:
+        return False, 'ERROR: Could not find SR-IOV VF "{}" on node "{}"'.format(vf, node)
+
+    return True, vf_information
+
+
+def get_list_sriov_vf(zkhandler, node, pf=None):
+    # Verify node is valid
+    valid_node = common.verifyNode(zkhandler, node)
+    if not valid_node:
+        return False, 'ERROR: Specified node "{}" is invalid.'.format(node)
+
+    vf_list = list()
+    vf_phy_list = common.sortInterfaceNames(zkhandler.children(('node.sriov.vf', node)))
+    for phy in vf_phy_list:
+        retcode, vf_information = get_info_sriov_vf(zkhandler, node, phy)
+        if retcode:
+            if pf is not None:
+                if vf_information['pf'] == pf:
+                    vf_list.append(vf_information)
+            else:
+                vf_list.append(vf_information)
+
+    return True, vf_list
+
+
+def set_sriov_vf_config(zkhandler, node, vf, vlan_id=None, vlan_qos=None, tx_rate_min=None, tx_rate_max=None, link_state=None, spoof_check=None, trust=None, query_rss=None):
+    # Verify node is valid
+    valid_node = common.verifyNode(zkhandler, node)
+    if not valid_node:
+        return False, 'ERROR: Specified node "{}" is invalid.'.format(node)
+
+    # Verify VF is valid
+    vf_information = getSRIOVVFInformation(zkhandler, node, vf)
+    if not vf_information:
+        return False, 'ERROR: Could not find SR-IOV VF "{}" on node "{}".'.format(vf, node)
+
+    update_list = list()
+
+    if vlan_id is not None:
+        update_list.append((('node.sriov.vf', node, 'sriov_vf.config.vlan_id', vf), vlan_id))
+
+    if vlan_qos is not None:
+        update_list.append((('node.sriov.vf', node, 'sriov_vf.config.vlan_qos', vf), vlan_qos))
+
+    if tx_rate_min is not None:
+        update_list.append((('node.sriov.vf', node, 'sriov_vf.config.tx_rate_min', vf), tx_rate_min))
+
+    if tx_rate_max is not None:
+        update_list.append((('node.sriov.vf', node, 'sriov_vf.config.tx_rate_max', vf), tx_rate_max))
+
+    if link_state is not None:
+        update_list.append((('node.sriov.vf', node, 'sriov_vf.config.link_state', vf), link_state))
+
+    if spoof_check is not None:
+        update_list.append((('node.sriov.vf', node, 'sriov_vf.config.spoof_check', vf), spoof_check))
+
+    if trust is not None:
+        update_list.append((('node.sriov.vf', node, 'sriov_vf.config.trust', vf), trust))
+
+    if query_rss is not None:
+        update_list.append((('node.sriov.vf', node, 'sriov_vf.config.query_rss', vf), query_rss))
+
+    if len(update_list) < 1:
+        return False, 'ERROR: No changes to apply.'
+
+    result = zkhandler.write(update_list)
+    if result:
+        return True, 'Successfully modified configuration of SR-IOV VF "{}" on node "{}".'.format(vf, node)
+    else:
+        return False, 'Failed to modify configuration of SR-IOV VF "{}" on node "{}".'.format(vf, node)
+
+
+def set_sriov_vf_vm(zkhandler, vm_uuid, node, vf, vf_macaddr, vf_type):
+    # Verify node is valid
+    valid_node = common.verifyNode(zkhandler, node)
+    if not valid_node:
+        return False
+
+    # Verify VF is valid
+    vf_information = getSRIOVVFInformation(zkhandler, node, vf)
+    if not vf_information:
+        return False
+
+    update_list = [
+        (('node.sriov.vf', node, 'sriov_vf.used', vf), 'True'),
+        (('node.sriov.vf', node, 'sriov_vf.used_by', vf), vm_uuid),
+        (('node.sriov.vf', node, 'sriov_vf.mac', vf), vf_macaddr),
+    ]
+
+    # Hostdev type SR-IOV prevents the guest from live migrating
+    if vf_type == 'hostdev':
+        update_list.append(
+            (('domain.meta.migrate_method', vm_uuid), 'shutdown')
+        )
+
+    zkhandler.write(update_list)
+
+    return True
+
+
+def unset_sriov_vf_vm(zkhandler, node, vf):
+    # Verify node is valid
+    valid_node = common.verifyNode(zkhandler, node)
+    if not valid_node:
+        return False
+
+    # Verify VF is valid
+    vf_information = getSRIOVVFInformation(zkhandler, node, vf)
+    if not vf_information:
+        return False
+
+    update_list = [
+        (('node.sriov.vf', node, 'sriov_vf.used', vf), 'False'),
+        (('node.sriov.vf', node, 'sriov_vf.used_by', vf), ''),
+        (('node.sriov.vf', node, 'sriov_vf.mac', vf), zkhandler.read(('node.sriov.vf', node, 'sriov_vf.phy_mac', vf)))
+    ]
+
+    zkhandler.write(update_list)
+
+    return True

daemon-common/node.py

@@ -33,6 +33,7 @@ def getNodeInformation(zkhandler, node_name):
     node_coordinator_state = zkhandler.read(('node.state.router', node_name))
     node_domain_state = zkhandler.read(('node.state.domain', node_name))
     node_static_data = zkhandler.read(('node.data.static', node_name)).split()
+    node_pvc_version = zkhandler.read(('node.data.pvc_version', node_name))
     node_cpu_count = int(node_static_data[0])
     node_kernel = node_static_data[1]
     node_os = node_static_data[2]
@@ -53,6 +54,7 @@ def getNodeInformation(zkhandler, node_name):
         'daemon_state': node_daemon_state,
         'coordinator_state': node_coordinator_state,
         'domain_state': node_domain_state,
+        'pvc_version': node_pvc_version,
         'cpu_count': node_cpu_count,
         'kernel': node_kernel,
         'os': node_os,
@@ -180,6 +182,24 @@ def ready_node(zkhandler, node, wait=False):
     return True, retmsg
 
 
+def get_node_log(zkhandler, node, lines=2000):
+    # Verify node is valid
+    if not common.verifyNode(zkhandler, node):
+        return False, 'ERROR: No node named "{}" is present in the cluster.'.format(node)
+
+    # Get the data from ZK
+    node_log = zkhandler.read(('logs.messages', node))
+
+    if node_log is None:
+        return True, ''
+
+    # Shrink the log buffer to length lines
+    shrunk_log = node_log.split('\n')[-lines:]
+    loglines = '\n'.join(shrunk_log)
+
+    return True, loglines
+
+
 def get_info(zkhandler, node):
     # Verify node is valid
     if not common.verifyNode(zkhandler, node):

daemon-common/vm.py

@@ -24,9 +24,14 @@ import re
 import lxml.objectify
 import lxml.etree
 
+from distutils.util import strtobool
+from uuid import UUID
+from concurrent.futures import ThreadPoolExecutor
+
 import daemon_lib.common as common
 
 import daemon_lib.ceph as ceph
+from daemon_lib.network import set_sriov_vf_vm, unset_sriov_vf_vm
 
 
 #
@@ -100,14 +105,14 @@ def getDomainName(zkhandler, domain):
 # Helper functions
 #
 def change_state(zkhandler, dom_uuid, new_state):
-        lock = zkhandler.exclusivelock(('domain.state', dom_uuid))
-        with lock:
-            zkhandler.write([
-                (('domain.state', dom_uuid), new_state)
-            ])
+    lock = zkhandler.exclusivelock(('domain.state', dom_uuid))
+    with lock:
+        zkhandler.write([
+            (('domain.state', dom_uuid), new_state)
+        ])
 
-            # Wait for 1/2 second to allow state to flow to all nodes
-            time.sleep(0.5)
+        # Wait for 1/2 second to allow state to flow to all nodes
+        time.sleep(0.5)
 
 
 #
@@ -170,7 +175,7 @@ def flush_locks(zkhandler, domain):
     return success, message
 
 
-def define_vm(zkhandler, config_data, target_node, node_limit, node_selector, node_autostart, migration_method=None, profile=None, initial_state='stop'):
+def define_vm(zkhandler, config_data, target_node, node_limit, node_selector, node_autostart, migration_method=None, profile=None, tags=[], initial_state='stop'):
     # Parse the XML data
     try:
         parsed_xml = lxml.objectify.fromstring(config_data)
@@ -191,6 +196,31 @@ def define_vm(zkhandler, config_data, target_node, node_limit, node_selector, no
         if not valid_node:
             return False, 'ERROR: Specified node "{}" is invalid.'.format(target_node)
 
+    # Validate the new RAM against the current active node
+    node_total_memory = int(zkhandler.read(('node.memory.total', target_node)))
+    if int(parsed_xml.memory.text) >= node_total_memory:
+        return False, 'ERROR: VM configuration specifies more memory ({} MiB) than node "{}" has available ({} MiB).'.format(parsed_xml.memory.text, target_node, node_total_memory)
+
+    # Validate the number of vCPUs against the current active node
+    node_total_cpus = int(zkhandler.read(('node.data.static', target_node)).split()[0])
+    if (node_total_cpus - 2) <= int(parsed_xml.vcpu.text):
+        return False, 'ERROR: VM configuration specifies more vCPUs ({}) than node "{}" has available ({} minus 2).'.format(parsed_xml.vcpu.text, target_node, node_total_cpus)
+
+    # If a SR-IOV network device is being added, set its used state
+    dnetworks = common.getDomainNetworks(parsed_xml, {})
+    for network in dnetworks:
+        if network['type'] in ['direct', 'hostdev']:
+            dom_node = zkhandler.read(('domain.node', dom_uuid))
+
+            # Check if the network is already in use
+            is_used = zkhandler.read(('node.sriov.vf', dom_node, 'sriov_vf.used', network['source']))
+            if is_used == 'True':
+                used_by_name = searchClusterByUUID(zkhandler, zkhandler.read(('node.sriov.vf', dom_node, 'sriov_vf.used_by', network['source'])))
+                return False, 'ERROR: Attempted to use SR-IOV network "{}" which is already used by VM "{}" on node "{}".'.format(network['source'], used_by_name, dom_node)
+
+            # We must update the "used" section
+            set_sriov_vf_vm(zkhandler, dom_uuid, dom_node, network['source'], network['mac'], network['type'])
+
     # Obtain the RBD disk list using the common functions
     ddisks = common.getDomainDisks(parsed_xml, {})
     rbd_list = []
@@ -211,7 +241,7 @@ def define_vm(zkhandler, config_data, target_node, node_limit, node_selector, no
         formatted_rbd_list = ''
 
     # Add the new domain to Zookeeper
-    result = zkhandler.write([
+    zkhandler.write([
         (('domain', dom_uuid), dom_name),
         (('domain.xml', dom_uuid), config_data),
         (('domain.state', dom_uuid), initial_state),
@@ -227,13 +257,107 @@ def define_vm(zkhandler, config_data, target_node, node_limit, node_selector, no
         (('domain.meta.migrate_method', dom_uuid), migration_method),
         (('domain.meta.node_limit', dom_uuid), formatted_node_limit),
         (('domain.meta.node_selector', dom_uuid), node_selector),
+        (('domain.meta.tags', dom_uuid), ''),
         (('domain.migrate.sync_lock', dom_uuid), ''),
     ])
 
-    if result:
-        return True, 'Added new VM with Name "{}" and UUID "{}" to database.'.format(dom_name, dom_uuid)
-    else:
-        return False, 'ERROR: Failed to add new VM.'
+    for tag in tags:
+        tag_name = tag['name']
+        zkhandler.write([
+            (('domain.meta.tags', dom_uuid, 'tag.name', tag_name), tag['name']),
+            (('domain.meta.tags', dom_uuid, 'tag.type', tag_name), tag['type']),
+            (('domain.meta.tags', dom_uuid, 'tag.protected', tag_name), tag['protected']),
+        ])
+
+    return True, 'Added new VM with Name "{}" and UUID "{}" to database.'.format(dom_name, dom_uuid)
+
+
+def attach_vm_device(zkhandler, domain, device_spec_xml):
+    # Validate that VM exists in cluster
+    dom_uuid = getDomainUUID(zkhandler, domain)
+    if not dom_uuid:
+        return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
+
+    # Verify that the VM is in a stopped state; freeing locks is not safe otherwise
+    state = zkhandler.read(('domain.state', dom_uuid))
+    if state != 'start':
+        return False, 'ERROR: VM "{}" is not in started state; live-add unneccessary.'.format(domain)
+
+    # Tell the cluster to attach the device
+    attach_device_string = 'attach_device {} {}'.format(dom_uuid, device_spec_xml)
+    zkhandler.write([
+        ('base.cmd.domain', attach_device_string)
+    ])
+    # Wait 1/2 second for the cluster to get the message and start working
+    time.sleep(0.5)
+    # Acquire a read lock, so we get the return exclusively
+    lock = zkhandler.readlock('base.cmd.domain')
+    with lock:
+        try:
+            result = zkhandler.read('base.cmd.domain').split()[0]
+            if result == 'success-attach_device':
+                message = 'Attached device on VM "{}"'.format(domain)
+                success = True
+            else:
+                message = 'ERROR: Failed to attach device on VM "{}"; check node logs for details.'.format(domain)
+                success = False
+        except Exception:
+            message = 'ERROR: Command ignored by node.'
+            success = False
+
+    # Acquire a write lock to ensure things go smoothly
+    lock = zkhandler.writelock('base.cmd.domain')
+    with lock:
+        time.sleep(0.5)
+        zkhandler.write([
+            ('base.cmd.domain', '')
+        ])
+
+    return success, message
+
+
+def detach_vm_device(zkhandler, domain, device_spec_xml):
+    # Validate that VM exists in cluster
+    dom_uuid = getDomainUUID(zkhandler, domain)
+    if not dom_uuid:
+        return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
+
+    # Verify that the VM is in a stopped state; freeing locks is not safe otherwise
+    state = zkhandler.read(('domain.state', dom_uuid))
+    if state != 'start':
+        return False, 'ERROR: VM "{}" is not in started state; live-add unneccessary.'.format(domain)
+
+    # Tell the cluster to detach the device
+    detach_device_string = 'detach_device {} {}'.format(dom_uuid, device_spec_xml)
+    zkhandler.write([
+        ('base.cmd.domain', detach_device_string)
+    ])
+    # Wait 1/2 second for the cluster to get the message and start working
+    time.sleep(0.5)
+    # Acquire a read lock, so we get the return exclusively
+    lock = zkhandler.readlock('base.cmd.domain')
+    with lock:
+        try:
+            result = zkhandler.read('base.cmd.domain').split()[0]
+            if result == 'success-detach_device':
+                message = 'Attached device on VM "{}"'.format(domain)
+                success = True
+            else:
+                message = 'ERROR: Failed to detach device on VM "{}"; check node logs for details.'.format(domain)
+                success = False
+        except Exception:
+            message = 'ERROR: Command ignored by node.'
+            success = False
+
+    # Acquire a write lock to ensure things go smoothly
+    lock = zkhandler.writelock('base.cmd.domain')
+    with lock:
+        time.sleep(0.5)
+        zkhandler.write([
+            ('base.cmd.domain', '')
+        ])
+
+    return success, message
 
 
 def modify_vm_metadata(zkhandler, domain, node_limit, node_selector, node_autostart, provisioner_profile, migration_method):
@@ -266,6 +390,38 @@ def modify_vm_metadata(zkhandler, domain, node_limit, node_selector, node_autost
     return True, 'Successfully modified PVC metadata of VM "{}".'.format(domain)
 
 
+def modify_vm_tag(zkhandler, domain, action, tag, protected=False):
+    dom_uuid = getDomainUUID(zkhandler, domain)
+    if not dom_uuid:
+        return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
+
+    if action == 'add':
+        zkhandler.write([
+            (('domain.meta.tags', dom_uuid, 'tag.name', tag), tag),
+            (('domain.meta.tags', dom_uuid, 'tag.type', tag), 'user'),
+            (('domain.meta.tags', dom_uuid, 'tag.protected', tag), protected),
+        ])
+
+        return True, 'Successfully added tag "{}" to VM "{}".'.format(tag, domain)
+    elif action == 'remove':
+        if not zkhandler.exists(('domain.meta.tags', dom_uuid, 'tag', tag)):
+            return False, 'The tag "{}" does not exist.'.format(tag)
+
+        if zkhandler.read(('domain.meta.tags', dom_uuid, 'tag.type', tag)) != 'user':
+            return False, 'The tag "{}" is not a user tag and cannot be removed.'.format(tag)
+
+        if bool(strtobool(zkhandler.read(('domain.meta.tags', dom_uuid, 'tag.protected', tag)))):
+            return False, 'The tag "{}" is protected and cannot be removed.'.format(tag)
+
+        zkhandler.delete([
+            (('domain.meta.tags', dom_uuid, 'tag', tag))
+        ])
+
+        return True, 'Successfully removed tag "{}" from VM "{}".'.format(tag, domain)
+    else:
+        return False, 'Specified tag action is not available.'
+
+
 def modify_vm(zkhandler, domain, restart, new_vm_config):
     dom_uuid = getDomainUUID(zkhandler, domain)
     if not dom_uuid:
@@ -276,7 +432,50 @@ def modify_vm(zkhandler, domain, restart, new_vm_config):
     try:
         parsed_xml = lxml.objectify.fromstring(new_vm_config)
     except Exception:
-        return False, 'ERROR: Failed to parse XML data.'
+        return False, 'ERROR: Failed to parse new XML data.'
+
+    # Get our old network list for comparison purposes
+    old_vm_config = zkhandler.read(('domain.xml', dom_uuid))
+    old_parsed_xml = lxml.objectify.fromstring(old_vm_config)
+    old_dnetworks = common.getDomainNetworks(old_parsed_xml, {})
+
+    # Validate the new RAM against the current active node
+    node_name = zkhandler.read(('domain.node', dom_uuid))
+    node_total_memory = int(zkhandler.read(('node.memory.total', node_name)))
+    if int(parsed_xml.memory.text) >= node_total_memory:
+        return False, 'ERROR: Updated VM configuration specifies more memory ({} MiB) than node "{}" has available ({} MiB).'.format(parsed_xml.memory.text, node_name, node_total_memory)
+
+    # Validate the number of vCPUs against the current active node
+    node_total_cpus = int(zkhandler.read(('node.data.static', node_name)).split()[0])
+    if (node_total_cpus - 2) <= int(parsed_xml.vcpu.text):
+        return False, 'ERROR: Updated VM configuration specifies more vCPUs ({}) than node "{}" has available ({} minus 2).'.format(parsed_xml.vcpu.text, node_name, node_total_cpus)
+
+    # If a SR-IOV network device is being added, set its used state
+    dnetworks = common.getDomainNetworks(parsed_xml, {})
+    for network in dnetworks:
+        # Ignore networks that are already there
+        if network['source'] in [net['source'] for net in old_dnetworks]:
+            continue
+
+        if network['type'] in ['direct', 'hostdev']:
+            dom_node = zkhandler.read(('domain.node', dom_uuid))
+
+            # Check if the network is already in use
+            is_used = zkhandler.read(('node.sriov.vf', dom_node, 'sriov_vf.used', network['source']))
+            if is_used == 'True':
+                used_by_name = searchClusterByUUID(zkhandler, zkhandler.read(('node.sriov.vf', dom_node, 'sriov_vf.used_by', network['source'])))
+                return False, 'ERROR: Attempted to use SR-IOV network "{}" which is already used by VM "{}" on node "{}".'.format(network['source'], used_by_name, dom_node)
+
+            # We must update the "used" section
+            set_sriov_vf_vm(zkhandler, dom_uuid, dom_node, network['source'], network['mac'], network['type'])
+
+    # If a SR-IOV network device is being removed, unset its used state
+    for network in old_dnetworks:
+        if network['type'] in ['direct', 'hostdev']:
+            if network['mac'] not in [n['mac'] for n in dnetworks]:
+                dom_node = zkhandler.read(('domain.node', dom_uuid))
+                # We must update the "used" section
+                unset_sriov_vf_vm(zkhandler, dom_node, network['source'])
 
     # Obtain the RBD disk list using the common functions
     ddisks = common.getDomainDisks(parsed_xml, {})
@@ -355,7 +554,7 @@ def rename_vm(zkhandler, domain, new_domain):
     undefine_vm(zkhandler, dom_uuid)
 
     # Define the new VM
-    define_vm(zkhandler, vm_config_new, dom_info['node'], dom_info['node_limit'], dom_info['node_selector'], dom_info['node_autostart'], migration_method=dom_info['migration_method'], profile=dom_info['profile'], initial_state='stop')
+    define_vm(zkhandler, vm_config_new, dom_info['node'], dom_info['node_limit'], dom_info['node_selector'], dom_info['node_autostart'], migration_method=dom_info['migration_method'], profile=dom_info['profile'], tags=dom_info['tags'], initial_state='stop')
 
     # If the VM is migrated, store that
     if dom_info['migrated'] != 'no':
@@ -401,14 +600,6 @@ def remove_vm(zkhandler, domain):
     if current_vm_state != 'stop':
         change_state(zkhandler, dom_uuid, 'stop')
 
-    # Gracefully terminate the class instances
-    change_state(zkhandler, dom_uuid, 'delete')
-
-    # Delete the configurations
-    zkhandler.delete([
-        ('domain', dom_uuid)
-    ])
-
     # Wait for 1 second to allow state to flow to all nodes
     time.sleep(1)
 
@@ -417,11 +608,28 @@ def remove_vm(zkhandler, domain):
         # vmpool/vmname_volume
         try:
             disk_pool, disk_name = disk.split('/')
-            retcode, message = ceph.remove_volume(zkhandler, disk_pool, disk_name)
         except ValueError:
             continue
 
-    return True, 'Removed VM "{}" and disks from the cluster.'.format(domain)
+        retcode, message = ceph.remove_volume(zkhandler, disk_pool, disk_name)
+        if not retcode:
+            if re.match('^ERROR: No volume with name', message):
+                continue
+            else:
+                return False, message
+
+    # Gracefully terminate the class instances
+    change_state(zkhandler, dom_uuid, 'delete')
+
+    # Wait for 1/2 second to allow state to flow to all nodes
+    time.sleep(0.5)
+
+    # Delete the VM configuration from Zookeeper
+    zkhandler.delete([
+        ('domain', dom_uuid)
+    ])
+
+    return True, 'Removed VM "{}" and its disks from the cluster.'.format(domain)
 
 
 def start_vm(zkhandler, domain):
@@ -513,6 +721,38 @@ def disable_vm(zkhandler, domain):
     return True, 'Marked VM "{}" as disable.'.format(domain)
 
 
+def update_vm_sriov_nics(zkhandler, dom_uuid, source_node, target_node):
+    # Update all the SR-IOV device states on both nodes, used during migrations but called by the node-side
+    vm_config = zkhandler.read(('domain.xml', dom_uuid))
+    parsed_xml = lxml.objectify.fromstring(vm_config)
+    dnetworks = common.getDomainNetworks(parsed_xml, {})
+    retcode = True
+    retmsg = ''
+    for network in dnetworks:
+        if network['type'] in ['direct', 'hostdev']:
+            # Check if the network is already in use
+            is_used = zkhandler.read(('node.sriov.vf', target_node, 'sriov_vf.used', network['source']))
+            if is_used == 'True':
+                used_by_name = searchClusterByUUID(zkhandler, zkhandler.read(('node.sriov.vf', target_node, 'sriov_vf.used_by', network['source'])))
+                if retcode:
+                    retcode_this = False
+                    retmsg = 'Attempting to use SR-IOV network "{}" which is already used by VM "{}"'.format(network['source'], used_by_name)
+            else:
+                retcode_this = True
+
+            # We must update the "used" section
+            if retcode_this:
+                # This conditional ensure that if we failed the is_used check, we don't try to overwrite the information of a VF that belongs to another VM
+                set_sriov_vf_vm(zkhandler, dom_uuid, target_node, network['source'], network['mac'], network['type'])
+            # ... but we still want to free the old node in an case
+            unset_sriov_vf_vm(zkhandler, source_node, network['source'])
+
+            if not retcode_this:
+                retcode = retcode_this
+
+    return retcode, retmsg
+
+
 def move_vm(zkhandler, domain, target_node, wait=False, force_live=False):
     # Validate that VM exists in cluster
     dom_uuid = getDomainUUID(zkhandler, domain)
@@ -572,6 +812,9 @@ def move_vm(zkhandler, domain, target_node, wait=False, force_live=False):
         # Wait for 1/2 second for migration to start
         time.sleep(0.5)
 
+    # Update any SR-IOV NICs
+    update_vm_sriov_nics(zkhandler, dom_uuid, current_node, target_node)
+
     if wait:
         while zkhandler.read(('domain.state', dom_uuid)) == target_state:
             time.sleep(0.5)
@@ -624,6 +867,7 @@ def migrate_vm(zkhandler, domain, target_node, force_migrate, wait=False, force_
         return False, 'ERROR: Could not find a valid migration target for VM "{}".'.format(domain)
 
     # Don't overwrite an existing last_node when using force_migrate
+    real_current_node = current_node  # Used for the SR-IOV update
     if last_node and force_migrate:
         current_node = last_node
 
@@ -640,6 +884,9 @@ def migrate_vm(zkhandler, domain, target_node, force_migrate, wait=False, force_
         # Wait for 1/2 second for migration to start
         time.sleep(0.5)
 
+    # Update any SR-IOV NICs
+    update_vm_sriov_nics(zkhandler, dom_uuid, real_current_node, target_node)
+
     if wait:
         while zkhandler.read(('domain.state', dom_uuid)) == target_state:
             time.sleep(0.5)
@@ -665,6 +912,7 @@ def unmigrate_vm(zkhandler, domain, wait=False, force_live=False):
         else:
             target_state = 'migrate'
 
+    current_node = zkhandler.read(('domain.node', dom_uuid))
     target_node = zkhandler.read(('domain.last_node', dom_uuid))
 
     if target_node == '':
@@ -683,6 +931,9 @@ def unmigrate_vm(zkhandler, domain, wait=False, force_live=False):
         # Wait for 1/2 second for migration to start
         time.sleep(0.5)
 
+    # Update any SR-IOV NICs
+    update_vm_sriov_nics(zkhandler, dom_uuid, current_node, target_node)
+
     if wait:
         while zkhandler.read(('domain.state', dom_uuid)) == target_state:
             time.sleep(0.5)
@@ -698,7 +949,10 @@ def get_console_log(zkhandler, domain, lines=1000):
         return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
 
     # Get the data from ZK
-    console_log = zkhandler.read(('domain.log.console', dom_uuid))
+    console_log = zkhandler.read(('domain.console.log', dom_uuid))
+
+    if console_log is None:
+        return True, ''
 
     # Shrink the log buffer to length lines
     shrunk_log = console_log.split('\n')[-lines:]
@@ -721,7 +975,7 @@ def get_info(zkhandler, domain):
     return True, domain_information
 
 
-def get_list(zkhandler, node, state, limit, is_fuzzy=True):
+def get_list(zkhandler, node, state, tag, limit, is_fuzzy=True, negate=False):
     if node:
         # Verify node is valid
         if not common.verifyNode(zkhandler, node):
@@ -733,51 +987,96 @@ def get_list(zkhandler, node, state, limit, is_fuzzy=True):
             return False, 'VM state "{}" is not valid.'.format(state)
 
     full_vm_list = zkhandler.children('base.domain')
-    vm_list = []
 
     # Set our limit to a sensible regex
-    if limit and is_fuzzy:
+    if limit:
+        # Check if the limit is a UUID
+        is_limit_uuid = False
         try:
-            # Implcitly assume fuzzy limits
-            if not re.match(r'\^.*', limit):
-                limit = '.*' + limit
-            if not re.match(r'.*\$', limit):
-                limit = limit + '.*'
-        except Exception as e:
-            return False, 'Regex Error: {}'.format(e)
+            uuid_obj = UUID(limit, version=4)
+            limit = str(uuid_obj)
+            is_limit_uuid = True
+        except ValueError:
+            pass
 
-    # If we're limited, remove other nodes' VMs
-    vm_node = {}
-    vm_state = {}
-    for vm in full_vm_list:
-        # Check we don't match the limit
-        name = zkhandler.read(('domain', vm))
-        vm_node[vm] = zkhandler.read(('domain.node', vm))
-        vm_state[vm] = zkhandler.read(('domain.state', vm))
-        # Handle limiting
-        if limit:
+        if is_fuzzy and not is_limit_uuid:
             try:
-                if re.match(limit, vm):
-                    if not node and not state:
-                        vm_list.append(common.getInformationFromXML(zkhandler, vm))
-                    else:
-                        if vm_node[vm] == node or vm_state[vm] == state:
-                            vm_list.append(common.getInformationFromXML(zkhandler, vm))
+                # Implcitly assume fuzzy limits
+                if not re.match(r'\^.*', limit):
+                    limit = '.*' + limit
+                if not re.match(r'.*\$', limit):
+                    limit = limit + '.*'
+            except Exception as e:
+                return False, 'Regex Error: {}'.format(e)
 
+    get_vm_info = dict()
+    for vm in full_vm_list:
+        name = zkhandler.read(('domain', vm))
+        is_limit_match = False
+        is_tag_match = False
+        is_node_match = False
+        is_state_match = False
+
+        # Check on limit
+        if limit:
+            # Try to match the limit against the UUID (if applicable) and name
+            try:
+                if is_limit_uuid and re.match(limit, vm):
+                    is_limit_match = True
                 if re.match(limit, name):
-                    if not node and not state:
-                        vm_list.append(common.getInformationFromXML(zkhandler, vm))
-                    else:
-                        if vm_node[vm] == node or vm_state[vm] == state:
-                            vm_list.append(common.getInformationFromXML(zkhandler, vm))
+                    is_limit_match = True
             except Exception as e:
                 return False, 'Regex Error: {}'.format(e)
         else:
-            # Check node to avoid unneeded ZK calls
-            if not node and not state:
-                vm_list.append(common.getInformationFromXML(zkhandler, vm))
-            else:
-                if vm_node[vm] == node or vm_state[vm] == state:
-                    vm_list.append(common.getInformationFromXML(zkhandler, vm))
+            is_limit_match = True
 
-    return True, vm_list
+        if tag:
+            vm_tags = zkhandler.children(('domain.meta.tags', vm))
+            if negate and tag not in vm_tags:
+                is_tag_match = True
+            if not negate and tag in vm_tags:
+                is_tag_match = True
+        else:
+            is_tag_match = True
+
+        # Check on node
+        if node:
+            vm_node = zkhandler.read(('domain.node', vm))
+            if negate and vm_node != node:
+                is_node_match = True
+            if not negate and vm_node == node:
+                is_node_match = True
+        else:
+            is_node_match = True
+
+        # Check on state
+        if state:
+            vm_state = zkhandler.read(('domain.state', vm))
+            if negate and vm_state != state:
+                is_state_match = True
+            if not negate and vm_state == state:
+                is_state_match = True
+        else:
+            is_state_match = True
+
+        get_vm_info[vm] = True if is_limit_match and is_tag_match and is_node_match and is_state_match else False
+
+    # Obtain our VM data in a thread pool
+    # This helps parallelize the numerous Zookeeper calls a bit, within the bounds of the GIL, and
+    # should help prevent this task from becoming absurdly slow with very large numbers of VMs.
+    # The max_workers is capped at 32 to avoid creating an absurd number of threads especially if
+    # the list gets called multiple times simultaneously by the API, but still provides a noticeable
+    # speedup.
+    vm_execute_list = [vm for vm in full_vm_list if get_vm_info[vm]]
+    vm_data_list = list()
+    with ThreadPoolExecutor(max_workers=32, thread_name_prefix='vm_list') as executor:
+        futures = []
+        for vm_uuid in vm_execute_list:
+            futures.append(executor.submit(common.getInformationFromXML, zkhandler, vm_uuid))
+        for future in futures:
+            try:
+                vm_data_list.append(future.result())
+            except Exception:
+                pass
+
+    return True, vm_data_list

daemon-common/zkhandler.py

@@ -124,37 +124,29 @@ class ZKHandler(object):
     # State/connection management
     #
     def listener(self, state):
+        """
+        Listen for KazooState changes and log accordingly.
+
+        This function does not do anything except for log the state, and Kazoo handles the rest.
+        """
         if state == KazooState.CONNECTED:
-            self.log('Connection to Zookeeper started', state='o')
+            self.log('Connection to Zookeeper resumed', state='o')
         else:
-            self.log('Connection to Zookeeper lost', state='w')
-
-            while True:
-                time.sleep(0.5)
-
-                _zk_conn = KazooClient(hosts=self.coordinators)
-                try:
-                    _zk_conn.start()
-                except Exception:
-                    del _zk_conn
-                    continue
-
-                self.zk_conn = _zk_conn
-                self.zk_conn.add_listener(self.listener)
-                break
+            self.log('Connection to Zookeeper lost with state {}'.format(state), state='w')
 
     def connect(self, persistent=False):
         """
-        Start the zk_conn object and connect to the cluster, then load the current schema version
+        Start the zk_conn object and connect to the cluster
         """
         try:
             self.zk_conn.start()
             if persistent:
+                self.log('Connection to Zookeeper started', state='o')
                 self.zk_conn.add_listener(self.listener)
         except Exception as e:
             raise ZKConnectionException(self, e)
 
-    def disconnect(self):
+    def disconnect(self, persistent=False):
         """
         Stop and close the zk_conn object and disconnect from the cluster
 
@@ -162,11 +154,27 @@ class ZKHandler(object):
         """
         self.zk_conn.stop()
         self.zk_conn.close()
+        if persistent:
+            self.log('Connection to Zookeeper terminated', state='o')
 
     #
     # Schema helper actions
     #
     def get_schema_path(self, key):
+        """
+        Get the Zookeeper path for {key} from the current schema based on its format.
+
+        If {key} is a tuple of length 2, it's treated as a path plus an item instance of that path (e.g. a node, a VM, etc.).
+
+        If {key} is a tuple of length 4, it is treated as a path plus an item instance, as well as another item instance of the subpath.
+
+        If {key} is just a string, it's treated as a lone path (mostly used for the 'base' schema group.
+
+        Otherwise, returns None since this is not a valid key.
+
+        This function also handles the special case where a string that looks like an existing path (i.e. starts with '/') is passed;
+        in that case it will silently return the same path back. This was mostly a migration functionality and is deprecated.
+        """
         if isinstance(key, tuple):
             # This is a key tuple with both an ipath and an item
             if len(key) == 2:
@@ -201,6 +209,10 @@ class ZKHandler(object):
         Check if a key exists
         """
         path = self.get_schema_path(key)
+        if path is None:
+            # This path is invalid, this is likely due to missing schema entries, so return False
+            return False
+
         stat = self.zk_conn.exists(path)
         if stat:
             return True
@@ -211,13 +223,15 @@ class ZKHandler(object):
         """
         Read data from a key
         """
-        if self.exists(key):
+        try:
             path = self.get_schema_path(key)
-            data = self.zk_conn.get(path)[0].decode(self.encoding)
-        else:
-            data = None
+            if path is None:
+                # This path is invalid; this is likely due to missing schema entries, so return None
+                return None
 
-        return data
+            return self.zk_conn.get(path)[0].decode(self.encoding)
+        except NoNodeError:
+            return None
 
     def write(self, kvpairs):
         """
@@ -238,6 +252,9 @@ class ZKHandler(object):
             value = kvpair[1]
 
             path = self.get_schema_path(key)
+            if path is None:
+                # This path is invalid; this is likely due to missing schema entries, so continue
+                continue
 
             if not self.exists(key):
                 # Creating a new key
@@ -276,9 +293,9 @@ class ZKHandler(object):
             keys = [keys]
 
         for key in keys:
-            path = self.get_schema_path(key)
             if self.exists(key):
                 try:
+                    path = self.get_schema_path(key)
                     self.zk_conn.delete(path, recursive=recursive)
                 except Exception as e:
                     self.log("ZKHandler error: Failed to delete key {}: {}".format(path, e), state='e')
@@ -290,8 +307,15 @@ class ZKHandler(object):
         """
         Lists all children of a key
         """
-        path = self.get_schema_path(key)
-        return self.zk_conn.get_children(path)
+        try:
+            path = self.get_schema_path(key)
+            if path is None:
+                # This path is invalid; this is likely due to missing schema entries, so return None
+                return None
+
+            return self.zk_conn.get_children(path)
+        except NoNodeError:
+            return None
 
     def rename(self, kkpairs):
         """
@@ -322,13 +346,20 @@ class ZKHandler(object):
 
             source_key = kkpair[0]
             source_path = self.get_schema_path(source_key)
+            if source_path is None:
+                # This path is invalid; this is likely due to missing schema entries, so continue
+                continue
 
             destination_key = kkpair[1]
             destination_path = self.get_schema_path(destination_key)
+            if destination_path is None:
+                # This path is invalid; this is likely due to missing schema entries, so continue
+                continue
 
             if not self.exists(source_key):
                 self.log("ZKHander error: Source key '{}' does not exist".format(source_path), state='e')
                 return False
+
             if self.exists(destination_key):
                 self.log("ZKHander error: Destination key '{}' already exists".format(destination_path), state='e')
                 return False
@@ -359,6 +390,9 @@ class ZKHandler(object):
                 lock_id = str(uuid.uuid1())
                 lock = self.zk_conn.ReadLock(path, lock_id)
                 break
+            except NoNodeError:
+                self.log("ZKHandler warning: Failed to acquire read lock on nonexistent path {}".format(path), state='e')
+                return None
             except Exception as e:
                 if count > 5:
                     self.log("ZKHandler warning: Failed to acquire read lock after 5 tries: {}".format(e), state='e')
@@ -384,6 +418,9 @@ class ZKHandler(object):
                 lock_id = str(uuid.uuid1())
                 lock = self.zk_conn.WriteLock(path, lock_id)
                 break
+            except NoNodeError:
+                self.log("ZKHandler warning: Failed to acquire write lock on nonexistent path {}".format(path), state='e')
+                return None
             except Exception as e:
                 if count > 5:
                     self.log("ZKHandler warning: Failed to acquire write lock after 5 tries: {}".format(e), state='e')
@@ -409,6 +446,9 @@ class ZKHandler(object):
                 lock_id = str(uuid.uuid1())
                 lock = self.zk_conn.Lock(path, lock_id)
                 break
+            except NoNodeError:
+                self.log("ZKHandler warning: Failed to acquire exclusive lock on nonexistent path {}".format(path), state='e')
+                return None
             except Exception as e:
                 if count > 5:
                     self.log("ZKHandler warning: Failed to acquire exclusive lock after 5 tries: {}".format(e), state='e')
@@ -426,7 +466,7 @@ class ZKHandler(object):
 #
 class ZKSchema(object):
     # Current version
-    _version = 0
+    _version = 6
 
     # Root for doing nested keys
     _schema_root = ''
@@ -450,6 +490,7 @@ class ZKSchema(object):
             'cmd.node': f'{_schema_root}/cmd/nodes',
             'cmd.domain': f'{_schema_root}/cmd/domains',
             'cmd.ceph': f'{_schema_root}/cmd/ceph',
+            'logs': '/logs',
             'node': f'{_schema_root}/nodes',
             'domain': f'{_schema_root}/domains',
             'network': f'{_schema_root}/networks',
@@ -460,6 +501,11 @@ class ZKSchema(object):
             'volume': f'{_schema_root}/ceph/volumes',
             'snapshot': f'{_schema_root}/ceph/snapshots',
         },
+        # The schema of an individual logs entry (/logs/{node_name})
+        'logs': {
+            'node': '',  # The root key
+            'messages': '/messages',
+        },
         # The schema of an individual node entry (/nodes/{node_name})
         'node': {
             'name': '',  # The root key
@@ -468,6 +514,7 @@ class ZKSchema(object):
             'data.active_schema': '/activeschema',
             'data.latest_schema': '/latestschema',
             'data.static': '/staticdata',
+            'data.pvc_version': '/pvcversion',
             'running_domains': '/runningdomains',
             'count.provisioned_domains': '/domainscount',
             'count.networks': '/networkscount',
@@ -483,7 +530,40 @@ class ZKSchema(object):
             'memory.provisioned': '/memprov',
             'ipmi.hostname': '/ipmihostname',
             'ipmi.username': '/ipmiusername',
-            'ipmi.password': '/ipmipassword'
+            'ipmi.password': '/ipmipassword',
+            'sriov': '/sriov',
+            'sriov.pf': '/sriov/pf',
+            'sriov.vf': '/sriov/vf',
+        },
+        # The schema of an individual SR-IOV PF entry (/nodes/{node_name}/sriov/pf/{pf})
+        'sriov_pf': {
+            'phy': '',  # The root key
+            'mtu': '/mtu',
+            'vfcount': '/vfcount'
+        },
+        # The schema of an individual SR-IOV VF entry (/nodes/{node_name}/sriov/vf/{vf})
+        'sriov_vf': {
+            'phy': '',  # The root key
+            'pf': '/pf',
+            'mtu': '/mtu',
+            'mac': '/mac',
+            'phy_mac': '/phy_mac',
+            'config': '/config',
+            'config.vlan_id': '/config/vlan_id',
+            'config.vlan_qos': '/config/vlan_qos',
+            'config.tx_rate_min': '/config/tx_rate_min',
+            'config.tx_rate_max': '/config/tx_rate_max',
+            'config.spoof_check': '/config/spoof_check',
+            'config.link_state': '/config/link_state',
+            'config.trust': '/config/trust',
+            'config.query_rss': '/config/query_rss',
+            'pci': '/pci',
+            'pci.domain': '/pci/domain',
+            'pci.bus': '/pci/bus',
+            'pci.slot': '/pci/slot',
+            'pci.function': '/pci/function',
+            'used': '/used',
+            'used_by': '/used_by'
         },
         # The schema of an individual domain entry (/domains/{domain_uuid})
         'domain': {
@@ -502,12 +582,20 @@ class ZKSchema(object):
             'meta.migrate_method': '/migration_method',
             'meta.node_selector': '/node_selector',
             'meta.node_limit': '/node_limit',
+            'meta.tags': '/tags',
             'migrate.sync_lock': '/migrate_sync_lock'
         },
+        # The schema of an individual domain tag entry (/domains/{domain}/tags/{tag})
+        'tag': {
+            'name': '',  # The root key
+            'type': '/type',
+            'protected': '/protected'
+        },
         # The schema of an individual network entry (/networks/{vni})
         'network': {
             'vni': '',  # The root key
             'type': '/nettype',
+            'mtu': '/mtu',
             'rule': '/firewall_rules',
             'rule.in': '/firewall_rules/in',
             'rule.out': '/firewall_rules/out',
@@ -549,6 +637,7 @@ class ZKSchema(object):
             'id': '',  # The root key
             'node': '/node',
             'device': '/device',
+            'db_device': '/db_device',
             'stats': '/stats'
         },
         # The schema of an individual pool entry (/ceph/pools/{pool_name})
@@ -650,9 +739,16 @@ class ZKSchema(object):
             if base_path is None:
                 # This should only really happen for second-layer key types where the helper functions join them together
                 base_path = ''
+
+            if not ipath:
+                # This is a root path
+                return f'{base_path}/{item}'
+
             sub_path = self.schema.get(itype).get('.'.join(ipath))
             if sub_path is None:
-                sub_path = ''
+                # We didn't find the path we're looking for, so we don't want to do anything
+                return None
+
             return f'{base_path}/{item}{sub_path}'
 
     # Get keys of a schema location
@@ -709,6 +805,10 @@ class ZKSchema(object):
                                 if not zkhandler.zk_conn.exists(nkipath):
                                     result = False
 
+                    # One might expect child keys under node (specifically, sriov.pf and sriov.vf) to be
+                    # managed here as well, but those are created automatically every time pvcnoded starts
+                    # and thus never need to be validated or applied.
+
         # These two have several children layers that must be parsed through
         for elem in ['volume']:
             # First read all the subelements of the key class (pool layer)
@@ -782,6 +882,10 @@ class ZKSchema(object):
                                 if not zkhandler.zk_conn.exists(nkipath):
                                     zkhandler.zk_conn.create(nkipath, ''.encode(zkhandler.encoding))
 
+                    # One might expect child keys under node (specifically, sriov.pf and sriov.vf) to be
+                    # managed here as well, but those are created automatically every time pvcnoded starts
+                    # and thus never need to be validated or applied.
+
         # These two have several children layers that must be parsed through
         for elem in ['volume']:
             # First read all the subelements of the key class (pool layer)

debian/changelog

@@ -1,3 +1,189 @@
+pvc (0.9.41-0) unstable; urgency=high
+
+  * Fixes a bad conditional check in IPMI verification
+  * Implements per-network MTU configuration; NOTE: Requires new keys in pvcnoded.yaml (`bridge_mtu`) and Ansible group_vars (`pvc_bridge_mtu`)
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Sat, 09 Oct 2021 19:39:21 -0400
+
+pvc (0.9.40-0) unstable; urgency=high
+
+  * [Docs] Documentation updates for new Changelog file
+  * [Node Daemon] Fixes bug with schema updates
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Thu, 07 Oct 2021 14:42:04 -0400
+
+pvc (0.9.39-0) unstable; urgency=high
+
+  * [Documentation] Update several documentation sections
+  * [API Daemon/CLI Client] Add negate flag for VM option limits (node, tag, state)
+  * [Build] Add linting check to build-and-deploy.sh
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Thu, 07 Oct 2021 11:52:38 -0400
+
+pvc (0.9.38-0) unstable; urgency=high
+
+  * [All] Significantly improve storage benchmark format and reporting
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Sun, 03 Oct 2021 22:32:41 -0400
+
+pvc (0.9.37-0) unstable; urgency=high
+
+  * [All] Adds support for configurable OSD DB size ratios
+  * [Node Daemon] Fixes bugs with OSD creation
+  * [Node Daemon] Fixes exception bugs in CephInstance
+  * [CLI Client] Adjusts descriptions around Ceph OSDs
+  * [Node Daemon] Fixes ordering of pvc-flush unit
+  * [Node Daemon] Fixes bugs in fence handling and libvirt keepalive
+  * [Node Daemon] Simplifies locking for and speeds up VM migrations
+  * [Node Daemon] Fixes bugs in queue get timeouts
+  * [API Daemon] Adjusts benchmark test jobs configuration and naming
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Thu, 30 Sep 2021 02:02:53 -0400
+
+pvc (0.9.36-0) unstable; urgency=high
+
+  * [Node Daemon] Fixes a bug during early cleanup
+  * [All] Adds support for OSD database/WAL block devices to improve Ceph performance; NOTE: Applies only to new OSDs
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Thu, 23 Sep 2021 14:01:38 -0400
+
+pvc (0.9.35-0) unstable; urgency=high
+
+  * [Node Daemon] Fixes several bugs and crashes in node daemon
+  * [General] Updates linting rules for newer Flake8 linter
+  * [Daemons/CLI client] Adds VM network and disk hot attach/detach support; NOTE: Changes the default behaviour of `pvc vm network add`/`remove` and `pvc vm volume add`/`remove`
+  * [API Daemon] Adds checks for pool size when resizing volumes
+  * [API Daemon] Adds checks for RAM and vCPU sizes when defining or modifying VMs
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Mon, 13 Sep 2021 02:20:46 -0400
+
+pvc (0.9.34-0) unstable; urgency=high
+
+  * [Provisioner] Adds support for filesystem arguments containing =
+  * [CLI Client] Fixes bug with pvc provisioner status output formatting
+  * [Node Daemon] Fixes minor typo in startup message
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Tue, 24 Aug 2021 16:15:25 -0400
+
+pvc (0.9.33-0) unstable; urgency=high
+
+  * [Node Daemon] A major refactoring of the node daemon
+  * [CLI Client] Fixes output errors if a node has no provisioner data
+  * [Packages] Fixes issues with including __pycache__ directories in .deb files
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Sat, 21 Aug 2021 03:28:48 -0400
+
+pvc (0.9.32-0) unstable; urgency=high
+
+  * [CLI Client] Fixes some incorrect colours in network lists
+  * [Documentation] Adds documentation screenshots of CLI client
+  * [Node Daemon] Fixes a bug if VM stats gathering fails
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Thu, 19 Aug 2021 12:37:58 -0400
+
+pvc (0.9.31-0) unstable; urgency=high
+
+  * [Packages] Cleans up obsolete Suggests lines
+  * [Node Daemon] Adjusts log text of VM migrations to show the correct source node
+  * [API Daemon] Adjusts the OVA importer to support floppy RASD types for compatability
+  * [API Daemon] Ensures that volume resize commands without a suffix get B appended
+  * [API Daemon] Removes the explicit setting of image-features in PVC; defaulting to the limited set has been moved to the ceph.conf configuration on nodes via PVC Ansible
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Fri, 30 Jul 2021 12:08:12 -0400
+
+pvc (0.9.30-0) unstable; urgency=high
+
+  * [Node Daemon] Fixes bug with schema validation
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Tue, 20 Jul 2021 00:01:45 -0400
+
+pvc (0.9.29-0) unstable; urgency=high
+
+  * [Node Daemon] Corrects numerous bugs with node logging framework
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Mon, 19 Jul 2021 16:55:41 -0400
+
+pvc (0.9.28-0) unstable; urgency=high
+
+  * [CLI Client] Revamp confirmation options for "vm modify" command
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Mon, 19 Jul 2021 09:29:34 -0400
+
+pvc (0.9.27-0) unstable; urgency=high
+
+  * [CLI Client] Fixes a bug with vm modify command when passed a file
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Mon, 19 Jul 2021 00:03:40 -0400
+
+pvc (0.9.26-0) unstable; urgency=high
+
+  * [Node Daemon] Corrects some bad assumptions about fencing results during hardware failures
+  * [All] Implements VM tagging functionality
+  * [All] Implements Node log access via PVC functionality
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Sun, 18 Jul 2021 20:49:52 -0400
+
+pvc (0.9.25-0) unstable; urgency=high
+
+  * [Node Daemon] Returns to Rados library calls for Ceph due to performance problems
+  * [Node Daemon] Adds a date output to keepalive messages
+  * [Daemons] Configures ZK connection logging only for persistent connections
+  * [API Provisioner] Add context manager-based chroot to Debootstrap example script
+  * [Node Daemon] Fixes a bug where shutdown daemon state was overwritten
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Sun, 11 Jul 2021 23:19:09 -0400
+
+pvc (0.9.24-0) unstable; urgency=high
+
+  * [Node Daemon] Removes Rados module polling of Ceph cluster and returns to command-based polling for timeout purposes, and removes some flaky return statements
+  * [Node Daemon] Removes flaky Zookeeper connection renewals that caused problems
+  * [CLI Client] Allow raw lists of clusters from `pvc cluster list`
+  * [API Daemon] Fixes several issues when getting VM data without stats
+  * [API Daemon] Fixes issues with removing VMs while disks are still in use (failed provisioning, etc.)
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Fri, 09 Jul 2021 15:58:36 -0400
+
+pvc (0.9.23-0) unstable; urgency=high
+
+  * [Daemons] Fixes a critical overwriting bug in zkhandler when schema paths are not yet valid
+  * [Node Daemon] Ensures the daemon mode is updated on every startup (fixes the side effect of the above bug in 0.9.22)
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Mon, 05 Jul 2021 23:40:32 -0400
+
+pvc (0.9.22-0) unstable; urgency=high
+
+  * [API Daemon] Drastically improves performance when getting large lists (e.g. VMs)
+  * [Daemons] Adds profiler functions for use in debug mode
+  * [Daemons] Improves reliability of ZK locking
+  * [Daemons] Adds the new logo in ASCII form to the Daemon startup message
+  * [Node Daemon] Fixes bug where VMs would sometimes not stop
+  * [Node Daemon] Code cleanups in various classes
+  * [Node Daemon] Fixes a bug when reading node schema data
+  * [All] Adds node PVC version information to the list output
+  * [CLI Client] Improves the style and formatting of list output including a new header line
+  * [API Worker] Fixes a bug that prevented the storage benchmark job from running
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Mon, 05 Jul 2021 14:18:51 -0400
+
+pvc (0.9.21-0) unstable; urgency=high
+
+  * [API Daemon] Ensures VMs stop before removing them
+  * [Node Daemon] Fixes a bug with VM shutdowns not timing out
+  * [Documentation] Adds information about georedundancy caveats
+  * [All] Adds support for SR-IOV NICs (hostdev and macvtap) and surrounding documentation
+  * [Node Daemon] Fixes a bug where shutdown aborted migrations unexpectedly
+  * [Node Daemon] Fixes a bug where the migration method was not updated realtime
+  * [Node Daemon] Adjusts the Patroni commands to remove reference to Zookeeper path
+  * [CLI Client] Adjusts several help messages and fixes some typos
+  * [CLI Client] Converts the CLI client to a proper Python module
+  * [API Daemon] Improves VM list performance
+  * [API Daemon] Adjusts VM list matching critera (only matches against the UUID if it's a full UUID)
+  * [API Worker] Fixes incompatibility between Deb 10 and 11 in launching Celery worker
+  * [API Daemon] Corrects several bugs with initialization command
+  * [Documentation] Adds a shiny new logo and revamps introduction text
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Tue, 29 Jun 2021 19:21:31 -0400
+
 pvc (0.9.20-0) unstable; urgency=high
 
   * [Daemons] Implemented a Zookeeper schema handler and version 0 schema

debian/control

@@ -9,7 +9,6 @@ X-Python3-Version: >= 3.2
 Package: pvc-daemon-node
 Architecture: all
 Depends: systemd, pvc-daemon-common, python3-kazoo, python3-psutil, python3-apscheduler, python3-libvirt, python3-psycopg2, python3-dnspython, python3-yaml, python3-distutils, python3-rados, python3-gevent, ipmitool, libvirt-daemon-system, arping, vlan, bridge-utils, dnsmasq, nftables, pdns-server, pdns-backend-pgsql
-Suggests: pvc-client-api, pvc-client-cli
 Description: Parallel Virtual Cluster node daemon (Python 3)
  A KVM/Zookeeper/Ceph-based VM and private cloud manager
  .

debian/pvc-client-cli.install

@@ -1,3 +0,0 @@
-client-cli/pvc.py usr/share/pvc
-client-cli/cli_lib usr/share/pvc
-client-cli/scripts usr/share/pvc

debian/pvc-client-cli.postinst

@@ -1,4 +1,8 @@
 #!/bin/sh
 
-# Install client binary to /usr/bin via symlink
-ln -s /usr/share/pvc/pvc.py /usr/bin/pvc
+# Generate the bash completion configuration
+if [ -d /etc/bash_completion.d ]; then
+    _PVC_COMPLETE=source_bash pvc > /etc/bash_completion.d/pvc
+fi
+
+exit 0

debian/pvc-client-cli.prerm

@@ -1,4 +1,8 @@
 #!/bin/sh
 
-# Remove client binary symlink
-rm -f /usr/bin/pvc
+# Remove the bash completion
+if [ -f /etc/bash_completion.d/pvc ]; then
+    rm -f /etc/bash_completion.d/pvc
+fi
+
+exit 0

debian/pvc-daemon-api.install

@@ -5,5 +5,6 @@ api-daemon/pvcapid.sample.yaml etc/pvc
 api-daemon/pvcapid usr/share/pvc
 api-daemon/pvcapid.service lib/systemd/system
 api-daemon/pvcapid-worker.service lib/systemd/system
+api-daemon/pvcapid-worker.sh usr/share/pvc
 api-daemon/provisioner usr/share/pvc
 api-daemon/migrations usr/share/pvc

debian/pvc-daemon-api.postinst

@@ -5,11 +5,16 @@ systemctl daemon-reload
 
 # Restart the main daemon and apply database migrations (or warn on first install)
 if systemctl is-active --quiet pvcapid.service; then
-    systemctl stop pvcapid-worker.service
     systemctl stop pvcapid.service
     /usr/share/pvc/pvc-api-db-upgrade
     systemctl start pvcapid.service
+fi
+# Restart the worker daemon
+if systemctl is-active --quiet pvcapid-worker.service; then
+    systemctl stop pvcapid-worker.service
     systemctl start pvcapid-worker.service
-else
+fi
+
+if [[ ! -f /etc/pvc/pvcapid.yaml ]]; then
     echo "NOTE: The PVC client API daemon (pvcapid.service) and the PVC provisioner worker daemon (pvcapid-worker.service) have not been started; create a config file at /etc/pvc/pvcapid.yaml, then run the database configuration (/usr/share/pvc/pvc-api-db-upgrade) and start them manually."
 fi

debian/rules

@@ -1,13 +1,19 @@
 #!/usr/bin/make -f
 # See debhelper(7) (uncomment to enable)
 # output every command that modifies files on the build system.
-#export DH_VERBOSE = 1
+export DH_VERBOSE = 1
 
 %:
-	dh $@
+	dh $@ --with python3
+
+override_dh_python3:
+	cd $(CURDIR)/client-cli; pybuild --system=distutils --dest-dir=../debian/pvc-client-cli/
+	mkdir -p debian/pvc-client-cli/usr/lib/python3
+	mv debian/pvc-client-cli/usr/lib/python3*/* debian/pvc-client-cli/usr/lib/python3/
+	rm -r $(CURDIR)/client-cli/.pybuild $(CURDIR)/client-cli/pvc.egg-info
 
 override_dh_auto_clean:
-	find . -name "__pycache__" -exec rm -r {} \; || true
+	find . -name "__pycache__" -o -name ".pybuild" -exec rm -r {} \; || true
 
 # If you need to rebuild the Sphinx documentation
 # Add spinxdoc to the dh --with line

docs/about.md

@@ -19,21 +19,21 @@ This document contains information about the project itself, the software stack,
 
 ## Project Motivation
 
-Server management and system administration have changed significantly in the last decade. Computing as a resource is here, and software-defined is the norm. Gone are the days of pet servers, of tweaking configuration files by hand, and of painstakingly installing from ISO images in 52x CD-ROM drives. This is a brave new world.
+Server administration has changed significantly in recent decades. Computing-as-a-resource and software-defined infrastructure is now the norm, and the days of pet servers, painstaking manual configurations, and installing from CR-ROM ISOs is long gone. This is a brave new world.
 
-As part of this trend, the rise of IaaS (Infrastructure as a Service) has created an entirely new way for administrators and, increasingly, developers, to interact with servers. They need to be able to provision virtual machines easily and quickly, to ensure those virtual machines are reliable and consistent, and to avoid downtime wherever possible. Even in a world of containers, VMs are still important, and are not going away, so some virtual management solution is a must.
+As part of these trends, Infrastructure-as-a-Service (IaaS) has become a critical component of server administration. Administrators and developers are increasingly interfacing with their infrastructure via programmable APIs and software tools, and automation is a hard requirement. While Container infrastructure like Docker and Kubernetes has become more and more popular in this space, Virtual Machines (VMs) are still a very common feature and do not seem to be going anywhere any time soon.
 
-However, the current state of this ecosystem is lacking. At present there are 3 primary categories: the large "Stack" open-source projects, the smaller traditional "VM management" open-source projects, and the entrenched proprietary solutions.
+However, the current state of the free and open source virtualization ecosystem is lacking.
 
-At the high end of the open-source ecosystem, are the "Stacks": OpenStack, CloudStack, and their numerous "vendorware" derivatives. These are large, unwieldy projects with dozens or hundreds of pieces of software to deploy in production, and can often require a large team just to understand and manage them. They're great if you're a large enterprise, building a public cloud, or have a team to get you going. But if you just want to run a small- to medium-sized virtual cluster for your SMB or ISP, they're definitely overkill and will cause you more headaches than they will solve long-term.
+At the lower end, projects like ProxMox provide an easy way to administer small virtualization clusters, but these projects tend to lack advanced redundancy facilities that are built-in by default. While there are some new contenders in this space, such as Harvester, the options are limited and their feature-sets and tool stacks can be cumbersome or unproven.
 
-At the low end of the open source ecosystem, are what I call the "traditional tools". The biggest name in this space is ProxMox, though other, mostly defunct projects like Ganeti, tangential projects like Corosync/Pacemaker, and even traditional "I just use scripts" methods fit as well. These projects are great if you want to run a small server or homelab, but they quickly get unwieldy, though for the opposite reason from the Stacks: they're too simplistic, designed around single-host models, and when they provide redundancy at all it is often haphazard and nowhere near production-grade.
+At the higher end, very large projects like OpenStack and CloudStack provide very advanced functionality, but these project are sprawling and complicated for Administrators to use, and are very focused on large enterprise deployments, not suitable for smaller clusters and teams.
 
-Finally, the proprietary solutions like VMWare and Nutanix have entrenched themselves in the industry. They're excellent pieces of software providing just about anything you would need, but this comes at a significant cost, both in terms of money and also in software freedom and vendor lock-in. The licensing costs of Nutanix for instance can often make even enterprise-grade customers' accountants' heads spin.
+Finally, proprietary solutions dominate this space. VMWare and Nutanix are the two largest names, with these products providing functionality for both small and large clusters, but proprietary software limits both flexibility and freedom, and the costs associated with these solutions is immense.
 
-PVC seeks to bridge the gaps between these 3 categories. It is fully Free Software like the first two categories, and even more so - PVC is committed to never be "open-core" software and to never hide a single feature behind a paywall; it is able to scale from very small (1 or 3 node) clusters up to a dozen or more nodes, bridging the first two categories as effortlessly as the third does; it makes use of a hyperconverged architecture like ProxMox or Nuntanix to avoid wasting hardware resources on dedicated controller, hypervisor, and storage nodes; it is redundant at every layer from the ground-up, something that is not designed into any other free solution, and is able to tolerate the loss any single disk or entire node with barely a blip, all without administrator intervention; and finally, it is designed to be as simple to use as possible, with an Ansible-based node management framework, a RESTful API client interface, and a consistent, self-documenting CLI administration tool, allowing an administrator to create and manage their cluster quickly and simply, and then get on with more interesting things.
+PVC aims to bridge the gaps between these three categories. Like the larger FLOSS and proprietary projects, PVC can scale up to very large cluster sizes, while remaining usable even for small clusters as well. Like the smaller FLOSS and proprietary projects, PVC aims to be very simple to use, with a fully programmable API, allowing administrators to get on with more important things. Like the other FLOSS solutions, PVC is free, both as in beer and as in speech, allowing the administrator to inspect, modify, and tailor it to their needs. And finally, PVC is built from the ground-up to support host-level redundancy at every layer, rather than this being an expensive, optional, or tacked on feature.
 
-In short, it is a Free Software, scalable, redundant, self-healing, and self-managing private cloud solution designed with administrator simplicity in mind.
+In short, it is a Free Software, scalable, redundant, self-healing, and self-managing private cloud solution designed with administrator simplicity in mind. 
 
 ## Building Blocks
 
@@ -45,9 +45,9 @@ To manage cluster state, PVC uses Zookeeper. This is an Apache project designed
 
 Additional relational database functionality, specifically for the managed network DNS aggregation subsystem and the VM provisioner, is provided by the PostgreSQL database system and the Patroni management tool, which provides automatic clustering and failover for PostgreSQL database instances.
 
-Node network routing for managed networks providing EBGP VXLAN and route-learning is provided by FRRouting, a descendant project of Quaaga and GNU Zebra. Upstream routers can use this interface to learn routes to cluster networks as well.
+Node network routing for managed networks providing EBGP VXLAN and route-learning is provided by FRRouting, a descendant project of Quaaga and GNU Zebra. Upstream routers can use this interface to learn routes to cluster networks as well. PVC also makes extensive use of the standard Linux `iprouting` stack.
 
-The storage subsystem is provided by Ceph, a distributed object-based storage subsystem with extensive scalability, self-managing, and self-healing functionality. The Ceph RBD (RADOS Block Device) subsystem is used to provide VM block devices similar to traditional LVM or ZFS zvols, but in a distributed, shared-storage manner.
+The storage subsystem is provided by Ceph, a distributed object-based storage subsystem with proven stability, extensive scalability, self-managing, and self-healing functionality. The Ceph RBD (RADOS Block Device) subsystem is used to provide VM block devices similar to traditional LVM or ZFS zvols, but in a distributed, shared-storage manner.
 
 All the components are designed to be run on top of Debian GNU/Linux, specifically Debian 10.x "Buster" or 11.x "Bullseye", with the SystemD system service manager. This OS provides a stable base to run the various other subsystems while remaining truly Free Software, while SystemD provides functionality such as automatic daemon restarting and complex startup/shutdown ordering.
 
@@ -65,7 +65,7 @@ Nodes are networked together via a set of statically-configured, simple layer-2
 
  * The "upstream" network is the primary network for the nodes, and provides functions such as upstream Internet access, routing to and from the cluster nodes, and management via the API; it may be either a firewalled public or NAT'd RFC1918 network, but should never be exposed directly to the Internet. It should also contain, or be able to route to, the IPMI BMC management interfaces of the node chassis'.
  * The "cluster" network is an unrouted RFC1918 network which provides inter-node communication for managed client network traffic (VXLANs), cross-node routing, VM migration and failover, and database replication and access.
- * The "storage" network is another unrouted RFC1918 network which provides a dedicated logical and/or physical link between the nodes for storage traffic, including VM block device storage traffic, inter-OSD replication traffic, and Ceph heartbeat traffic, thus allowing it to be completely isolated from the other networks for maximum performance. This network can be optionally colocated with the "cluster" network, by specifying the same device for both, and can be further combined by specifying the same IP for both to completely collapse the "cluster" and "storage" networks. A collapsed cluster+storage configuration may be ideal to simplify management of small clusters, or a split configuration can be used to provide flexbility for large or demanding high-performance clusters - this choice is left to the administrator based on their needs.
+ * The "storage" network is another unrouted RFC1918 network which provides a dedicated logical and/or physical link between the nodes for storage traffic, including VM block device storage traffic, inter-OSD replication traffic, and Ceph heartbeat traffic, thus allowing it to be completely isolated from the other networks for maximum performance. This network can be optionally colocated with the "cluster" network, by specifying the same device for both, and can be further combined by specifying the same IP for both to completely collapse the "cluster" and "storage" networks. A collapsed cluster+storage configuration may be ideal to simplify management of small clusters, or a split configuration can be used to provide flexibility for large or demanding high-performance clusters - this choice is left to the administrator based on their needs.
  
  Within each network is a single "floating" IP address which follows the primary coordinator, providing a single interface to the cluster. Once configured, the cluster is then able to create additional networks of two kinds, "bridged" traditional vLANs and "managed" routed VXLANs, to provide network access to VMs.
 
@@ -97,7 +97,7 @@ The overall management, deployment, bootstrapping, and configuring of nodes is a
 
 The Ansible configuration and architecture manual can be found at the [Ansible manual page](/manuals/ansible).
 
-The [getting started documentation](/getting-started) provides a walkthrough of using these tools to bootstrap a new cluster.
+The [getting started documentation](/getting-started) provides a walk-through of using these tools to bootstrap a new cluster.
 
 ## Frequently Asked Questions
 
@@ -138,7 +138,7 @@ Not yet. Right now, PVC management is done exclusively with the CLI interface to
 
 #### I want feature X, does it fit with PVC?
 
-That depends on the specific feature. I will limit features to those that align with the overall goals of PVC, that is to say, to provide an easy-to-use hyperconverged virtualization system focused on redundancy. If a feature suits this goal it is likely to be considered; if it does not, it will not. PVC is rapidly approaching the completion of its 1.0 roadmap, which I consider feature-complete for the primary usecase, and future versions may expand in scope.
+That depends on the specific feature. I will limit features to those that align with the overall goals of PVC, that is to say, to provide an easy-to-use hyperconverged virtualization system focused on redundancy. If a feature suits this goal it is likely to be considered; if it does not, it will not. PVC is rapidly approaching the completion of its 1.0 road-map, which I consider feature-complete for the primary use-case, and future versions may expand in scope.
 
 ### Storage Questions
 

docs/cluster-architecture.md

@@ -1,86 +1,180 @@
 # PVC Cluster Architecture considerations
 
 - [PVC Cluster Architecture considerations](#pvc-cluster-architecture-considerations)
-  * [Node Specifications: Considering the size of nodes](#node-specifications--considering-the-size-of-nodes)
-  * [Storage Layout: Ceph and OSDs](#storage-layout--ceph-and-osds)
-  * [Physical network considerations](#physical-network-considerations)
-  * [Network Layout: Considering the required networks](#network-layout--considering-the-required-networks)
-    + [PVC system networks](#pvc-system-networks)
-        - [Upstream: Connecting the nodes to the wider world](#upstream--connecting-the-nodes-to-the-wider-world)
-        - [Cluster: Connecting the nodes with each other](#cluster--connecting-the-nodes-with-each-other)
-        - [Storage: Connecting Ceph OSD with each other](#storage--connecting-ceph-osd-with-each-other)
-    + [PVC client networks](#pvc-client-networks)
-        - [Bridged (unmanaged) Client Networks](#bridged--unmanaged--client-networks)
-        - [VXLAN (managed) Client Networks](#vxlan--managed--client-networks)
-        - [Other Client Networks](#other-client-networks)
-  * [Node Layout: Considering how nodes are laid out](#node-layout--considering-how-nodes-are-laid-out)
-    + [Node Functions: Coordinators versus Hypervisors](#node-functions--coordinators-versus-hypervisors)
-        - [Coordinators](#coordinators)
-            * [The Primary Coordinator](#the-primary-coordinator)
-        - [Hypervisors](#hypervisors)
-    + [Geographic redundancy](#geographic-redundancy)
-  * [Example Configurations](#example-configurations)
-    + [Basic 3-node cluster](#basic-3-node-cluster)
-    + [Mid-sized 8-node cluster with 3 coordinators](#mid-sized-8-node-cluster-with-3-coordinators)
-    + [Large 17-node cluster with 5 coordinators](#large-17-node-cluster-with-5-coordinators)
+  * [Node Specification](#node-specification)
+    + [n-1 Redundancy](#n-1-redundancy)
+    + [CPU](#cpu)
+    + [Memory](#memory)
+    + [Disk](#disk)
+    + [Network](#network)
+  * [PVC architecture](#pvc+architecture)
+    + [Operating System](#operating-system)
+    + [Ceph Storage Layout](#ceph-storage-layout)
+    + [Networks](#networks)
+      - [System Networks](#system+networks)
+      - [Client Networks](#client+networks)
+    + [Fencing and Recovery](#fencing-and-recovery)
+  * [Advanced Layouts](#advanced+layouts)
+    + [Coordinators versus Hypervisors](#coordinators-versus-hypervisors)
+    + [Georedundancy](#georedundancy)
+  * [Example System Diagrams](#example+system-diagrams)
+    + [Small 3-node cluster](#small-3-node-cluster)
+    + [Large 8-node cluster](#large-8-node-cluster)
 
 This document contains considerations the administrator should make when preparing for and building a PVC cluster. It is important that prospective PVC administrators read this document *thoroughly* before deploying a cluster to ensure they understand the requirements, caveats, and important details about how PVC operates.
 
-## Node Specifications: Considering the size of nodes
+## Node Specification
 
 PVC nodes, especially coordinator nodes, run a significant number of software applications in addition to the virtual machines (VMs). It is therefore extremely important to size the systems correctly for the expected workload while planning both for redundancy and future capacity. In general, taller nodes are better for performance, providing a more powerful cluster on fewer physical machines, though each workload may be different in this regard.
 
-The following table provides bare-minimum, recommended, and optimal specifications for a cluster. The bare-minimum specification would be suitable for testing or a small lab, but not for production use. The recommended specification would be suitable for a small production cluster running lightweight VMs. The optimal cluster would be the ideal for running a demanding, resource-intensive production cluster. Note that these are the minimum resources required, and actual usage will likely require more resources than those presented here - this is mostly to show the minimums for each specified configuration (i.e. testing, light production, heavy production).
+The following table provides recommended minimum specifications for each component of the cluster nodes. In general, these minimums are the lowest possible for a production-quality cluster that would provide decent performance for up to about a dozen virtual machines. Of course, further upward scaling is recommended and the specific computational and storage needs of the VM workloads should be taken into account.
 
-| Resource | Minimum | Recommended | Optimal|
-|--------------|-----------|---------------|----------|
-| CPU generation | Intel Nehalem (2008) / AMD Bulldozer (2011) | Intel Sandy Bridge (2011) / AMD Naples (2017) | Intel Haswell (2013) / AMD Rome (2019) |
-| CPU cores (per node) | 4x @1.8GHz | 8x @2.0GHz | 12x @2.2 GHz |
-| RAM (per node) | 16GB | 48GB | 64GB |
-| System disk (SSD/HDD/USB/SD/eMMC) | 1x 10GB | 2x 10GB RAID-1 | 2x 32GB RAID-1 |
-| Data disk (SSD only) | 1x 200GB | 1x 400GB | 2x 400GB |
-| Network interfaces | 1x 1Gbps | 2x 1Gbps LAG | 2x 10Gbps LAG |
-| Total CPU cores (healthy) | 12x | 24x | 36x |
-| Total CPU cores (n-1) | 8x | 16x | 24x |
-| Total RAM (healthy) | 48GB | 144GB | 192GB |
-| Total RAM (n-1) | 32GB | 96GB | 128GB |
-| Total disk space | 200GB | 400GB | 800GB |
+| Resource | Recommended Minimum |
+| -------- | --------------------|
+| CPU generation | Intel Sandy Bridge (2011) *or* AMD Naples (2017) |
+| CPU cores per node | 8 @ 2.0GHz |
+| RAM per node | 32GB |
+| System disk (SSD/HDD/USB/SD/eMMC) | 2x 100GB RAID-1 |
+| Data disk (SSD only) | 1x 400GB |
+| Network interfaces | 2x 10Gbps (LACP LAG) |
+| Remote IPMI-over-IP | Available and connected |
+| Total CPU cores (3 nodes healthy) | 24 |
+| Total CPU cores (3 nodes n-1) | 16 |
+| Total RAM (3 nodes healthy) | 96GB |
+| Total RAM (3 nodes n-1) | 64GB |
+| Total disk space (3 nodes) | 400GB |
 
-### System Disks
-
-The system disk(s) chosen are important to consider, especially for coordinators. Ideally, an SSD, or two SSDs in RAID-1/mirroring are recommended for system disks. This helps ensure optimal performance for the system (e.g. swap space) and PVC components such as databases as well as the Ceph caches.
-
-It is possible to run PVC on slower disks, for instance HDDs, USB drives, SD cards, or eMMC flash. For hypervisor-only nodes this will be acceptable; however for coordinators be advised that the performance of some aspects of the system may suffer as a result, and the longevity of the storage media must be carefully considered. RAID-1/mirroring is strongly recommended for these storage media as well, especially on coordinator nodes.
+For testing, or low-budget homelab applications, some aspects can be further tuned down, however consider the following sections carefully.
 
 ### n-1 Redundancy
 
 Care should be taken to examine the "healthy" versus "n-1" total resource availability. Under normal operation, PVC will use all available resources and distribute VMs across all cluster nodes. However, during single-node failure or maintenance conditions, all VMs will be required to run on the remaining hypervisors. Thus, care should be taken during planning to ensure there is sufficient resources for the expected workload of the cluster.
 
-The general rule for available resource capacity planning can be though of as "1/3 of the total disks space, 2/3 of the total RAM, 2/3 of the total CPUs" for a 3-node cluster.
+The general values for default resource availability of a 3-node cluster for n-1 availability (1 node offline) are:
 
-For memory provisioning of VMs, PVC will warn the administrator, via a Degraded cluster state, if the "n-1" RAM quantity is exceeded by the total maximum allocation of all running VMs. This situation can be worked around with sufficient swap space on nodes to ensure there is overflow, however the warning cannot be overridden. If nodes are of mismatched sizes, the "n-1" RAM quantity is calculated by removing (one of) the largest node in the cluster and adding the remaining nodes' RAM counts together.
+  * 1/3 of the total data disk space (3 copies of all data, distributed across all 3 nodes)
+  * 2/3 of the total RAM
+  * 2/3 of the total CPU cores
 
-### System Memory Utilization
+For memory provisioning of VMs, PVC will warn the administrator, via a Degraded cluster state, if the "n-1" RAM quantity is exceeded by the total maximum allocation of all running VMs. If nodes are of mismatched sizes, the "n-1" RAM quantity is calculated by removing (one of) the largest node in the cluster and adding the remaining nodes' RAM counts together.
 
-By default, several components of PVC outside of VMs will have large memory allocations, most notably Ceph OSD processes and Zookeeper database processes. These processes should be considered when selecting the RAM allocation of nodes, and adjusted in the Ansible `group_vars` if lower defaults are required.
+### CPU
 
-#### Ceph OSD processes
+CPU resources are a very important part of the overall performance of a PVC cluster. Numerous aspects of the system require high-performance CPU cores, including the VM workloads themselves, the PVC databases, and, especially, the Ceph storage subsystem.
 
-By default, PVC will allow several GB (up to 4-6GB) of RAM allocation per OSD to maximize the available cache space and hence disk performance. This can be lowered as far as 939MB should the administrator require due to a low RAM configuration, but no further due to Ceph limitations; therefore at least 1GB of memory per storage OSD is required even in the most limited case.
+As a general rule, more cores, and faster cores, are always better, and real cores are preferable to SMT virtual cores in most cases.
 
-#### Zookeeper processes
+#### SMT
 
-By default, the Java heap and stack sizes are set to 256MB and 512MB respectively, yieliding a memory usage of 500+MB after serveral days or weeks of uptime. This can be lowered to 32M or less for lightly-used clusters should the administrator require due to a low RAM configuration.
+SMT in particular can be a contentious subject, and performance can vary wildly for different workloads; thus, while they are useful, in terms of performance calculations they should always be considered as an afterthought or "bonus" to assist with many VMs contending for resources, and base specifications should be done based on the number of real CPU cores instead.
 
-### Operating System and Architecture
+#### CPU core counts
 
-As an underlying OS, only Debian GNU/Linux 10.x "Buster" or 11.x "Bullseye" is supported by PVC. This is the operating system installed by the PVC [node installer](https://github.com/parallelvirtualcluster/pvc-installer) and expected by the PVC [Ansible configuration system](https://github.com/parallelvirtualcluster/pvc-ansible). Ubuntu or other Debian-derived distributions may work, but are not officially supported. PVC also makes use of a custom repository to provide the PVC software and (for Debian Buster) an updated version of Ceph beyond what is available in the base operating system, and this is only compatible officially with Debian 10 or 11. PVC will generally be upgraded regularly to support new Debian versions. As a rule, using the current versions of the official node installer and Ansible repository is the preferred and only supported method for deploying PVC.
+The following should be considered recommended minimums for CPU core allocations:
+
+  * PVC system daemons, including Zookeeper and PostgreSQL databases: 2 CPU cores
+  * Ceph Monitor and Manager processes: 1 CPU core
+  * Ceph OSD processes: 2 CPU cores *per OSD disk*
+  * Virtual Machines: 1 CPU core per vCPU in the largest spec'd VM (e.g. 12 vCPUs in a VM = 12 cores here)
+
+To provide an example, consider a cluster that would run 2 OSD disks per node, and want to run several VMs, the largest of which would require 12 vCPUs:
+
+  * PVC system: 2 cores
+  * Ceph Mon/Mgr: 1 core
+  * Ceph OSDs: 2 * 2 = 4 cores
+  * VMs: 12 cores
+
+This gives a total of 19 cores, and thus a 20+ core CPU would be recommended.
+
+Additional CPU cores, as previously mentioned, are always better. For instance, though 2 is the recommended minimum per OSD disk, better performance can be achieved if there are 4 cores available per OSD instead. This trade-off depends heavily on the required workload and VM specifications and should be carefully considered.
+
+#### CPU performance
+
+While CPU frequency is not a tell-all or even particularly useful metric across generations or manufacturers, within a specific generation and manufacturer, faster CPUs will almost always improve performance across the board, especially when considering the Ceph storage subsystem. If a 2.0GHz and a 2.6GHz CPU of the same core count are both available, the 2.6GHz one is almost always the better choice from a pure performance perspective. 
+
+### Memory
+
+Memory is extremely important to PVC clusters, and like CPU resources a not-insignificant amount of memory is required for the baseline cluster before VMs are considered.
+
+#### Memory allocations
+
+The following should be considered recommended minimums for memory allocations:
+
+  * PVC daemons: 1 GB
+  * Zookeeper database: 1 GB
+  * PostgreSQL database: 1 GB
+  * Ceph Monitor and Manager processes: 1 GB
+  * Ceph OSD processes: 1 GB *per OSD disk*
+
+All additional memory can be consumed by virtual machines.
+
+To provide an example, in the same cluster as mentioned in the CPU section:
+
+ * PVC system: 1 GB
+ * Zookeeper: 1 GB
+ * PostgreSQL: 1 GB
+ * Ceph Mon/Mgr: 1 GB
+ * Ceph OSDs: 2 * 1 GB = 2 GB
+
+This gives a total of 6 GB of memory for the base system, with VMs requiring additional memory.
+
+#### VM Memory Overprovisioning
+
+An important consideration is that the KVM hypervisor used by PVC will only allocate guest memory *as required by the guest*, but PVC tracks memory allocation based on the allocated maximum. Thus, for example, a VM may be allocated 8192 MB of memory in PVC, and thus the PVC system considers 8 GB "allocated" and "provisioned" to this VM, but if the actual guest is only using 500 MB of that memory, the actual memory usage on the hypervisor node will be 500 MB for that VM. Thus it is possible for "all" memory to be allocated on a node but there still be many GB of "free" memory. This is an intentional design decision to avoid excessive overprovisioning of memory and thus situations where non-VM processes become memory starved, as the PVC system itself does *not* track the usage by the aforementioned processes.
+
+#### Memory Performance
+
+Given the recommended CPU requirements, all PVC hypervisors should contain at least DDR3 memory, which is sufficiently performant for all tasks. Memory latency and performance, however, can become important especially in large NUMA systems, and especially with regards to the Ceph storage subsystem. Care should be taken to optimize the memory layout in nodes, for instance making use of all available memory channels in the CPU architecture and preferring 1 DIMM-per-channel (DPC) over 2 DPC.
+
+#### Ceph OSD memory utilization
+
+While the recommended *minimum* is 1 GB per OSD process, in reality, Ceph can allocate between 4 and 6 GB of memory per OSD process, especially for caching metadata and other frequently-used data. Thus, for maximum performance, 4 GB instead of 1 GB should be allocated per-OSD.
+
+#### Memory limit tuning
+
+The PVC Ansible deployment system allows the administrator to specify limits on some aspects of the aforementioned memory requirements, for instance limiting Zookeeper or Ceph OSD processes to lower amounts of memory. This is not recommended except in situations where memory is extremely constrained; in such situations adding additional memory to nodes is always preferable. For details and examples please see the Ansible variable files.
+
+### Disk
+
+#### System Disks
+
+The performance of system disks is of critical importance in the PVC cluster. At least 32GB of space are required, and at least 100GB is recommended to ensure optimal performance. The system disks should be fast SAS HDDs, SSDs, eMMC flash, class-10 SD, or other flash-based mediums, and RAID-1 is critical for reliability purposes, especially for more wear- or failure-sensitive media types.
+
+PVC will store the various databases on these disks, so overall performance can affect the responsiveness of the system. However note that no VM data is ever stored on system disks; this is provided exclusively by the Ceph data disks (OSDs).
+
+#### Ceph OSD disks
+
+All VM block devices are stored on Ceph OSD data disks. The default pool configuration of the Ceph storage subsystem uses a `copies=3` layout with a `host`-level failure domain; thus, in a 3-node cluster, each block of data is stored 3 times, once per node. This ensures that 2 copies of each piece of data are available even if a host is down, at the cost of 1/3 of the total overall storage space. Other configurations are possible, but this is the minimum recommended.
+
+The performance of VM disks will be dictated almost exclusively by the performance of these disks in combination with the CPU resources of the system as discussed previously. Very fast, robust, and resilient storage is highly recommended for OSD disks to maximize performance and longevity. High-performance SATA, SAS, or NVMe SSDs are recommended for this task, sized according to the expected workload. Spinning disks (HDDs) are *not* recommended for this purpose, and their very low random performance will significantly limit the overall storage performance of the cluster.
+
+Initially, it is optimal if all nodes contain the same number and same size of OSD disks, to ensure even distribution of the data across all disks and thus maximize performance. PVC supports adding additional OSDs at a later time, however the administrator should be cautious to always add new disks in parallel on all nodes at the same time, as otherwise the replication ratio will prevent the new space from being utilized. Thus, in a 3-node cluster, disks must be added 3-at-a-time to all 3 nodes, and these disks must be identically sized, in order to increase the total usable storage space by the value of one of these disks.
+
+In addition to the primary data disks, PVC also supports the offloading of the Ceph BlueStore OSD database and WAL functions of the OSDs onto a separate OSD database volume group on a dedicated storage device. In the normal use-case, this would be an extremely fast and endurant Intel Optane or similar extremely-performant NVMe SSD which is significantly faster than the primary data SSDs. This will help accelerate random write I/Os and metadata lookups, especially when using lower-performance SATA or SAS SSDs. Generally speaking this volume should be large enough to support 5% of the capacity of all OSDs on a node, with some room for future expansion. Only one such device and volume group is supported at this time.
+
+### Network
+
+Because PVC makes extensive use of cross-node communications, high-throughput and low-latency networking is critical. At a minimum, 10-gigabit networking is recommended to ensure suitable throughput for the storage subsystem as well as for VM traffic. Higher-speed networking can also improve performance, especially when using extremely fast Ceph OSD disks.
+
+A minimum of 2 network interfaces is recommended. These should then be combined into a logical aggregate (LAG) using 802.3ad (LACP) to provide redundant links and a boost in available bandwidth. Additional NICs can also be used to separate discrete parts of the networking stack, which will be discussed below.
+
+#### Remote IPMI-over-IP
+
+IPMI provides a method to manage the physical chassis' of nodes from outside of their operating system. Common implementations include Dell iDRAC, HP iLO, Cisco CIMC, and others.
+
+PVC nodes in production deployments should always feature an IPMI-over-IP interface of some kind, which is then reachable either in, or via, the Upstream system network (see [System Networks](#system-networks)). This requirement is discussed in more detail during the [Fencing and Recovery](#fencing-and-recovery) section below.
+
+## PVC Architecture
+
+### Operating System
+
+As an underlying OS, only Debian GNU/Linux 10.x "Buster" or 11.x "Bullseye" are supported by PVC. This is the operating system installed by the PVC [node installer](https://github.com/parallelvirtualcluster/pvc-installer) and expected by the PVC [Ansible configuration system](https://github.com/parallelvirtualcluster/pvc-ansible). Ubuntu or other Debian-derived distributions may work, but are not officially supported. PVC also makes use of a custom repository to provide the PVC software and (for Debian Buster) an updated version of Ceph beyond what is available in the base operating system, and this is only compatible officially with Debian 10 or 11. PVC will generally be upgraded regularly to support new Debian versions. As a rule, using the current versions of the official node installer and Ansible repository is the preferred and only supported method for deploying PVC.
 
 Currently, only the `amd64` (Intel 64 or AMD64) architecture is officially supported by PVC. Given the cross-platform nature of Python and the various software components in Debian, it may work on `armhf` or `arm64` systems as well, however this has not been tested by the author and is not officially supported at this time.
 
-## Storage Layout: Ceph and OSDs
+### Ceph Storage Layout
 
-PVC makes use of Ceph, a distributed, replicated, self-healing, and self-managing storage system to provide shared VM storage. While a PVC administrator is not required to understand Ceph for day-to-day administraton, and PVC provides interfaces to most of the common storage functions required to operate a cluster, at least some knowledge of Ceph is advisable.
+PVC makes use of Ceph, a distributed, replicated, self-healing, and self-managing storage system to provide shared VM storage. While a PVC administrator is not required to understand Ceph for day-to-day administration, and PVC provides interfaces to most of the common storage functions required to operate a cluster, at least some knowledge of Ceph is advisable.
 
 The Ceph subsystem of PVC creates a "hyperconverged" cluster whereby storage and VM hypervisor functions are collocated onto the same physical servers; PVC does not differentiate between "storage" and "compute" nodes, and while storage support can be disabled and an external Ceph cluster used, this is not recommended. The performance of the storage must be taken into account when sizing the nodes as mentioned above.
 
@@ -90,27 +184,25 @@ Disks must be balanced across all storage-containing nodes. For instance, adding
 
 PVC Ceph pools make use of the replication mechanism of Ceph to store multiple copies of each object, thus ensuring that data is always available even when a host is unavailable. Only "replica"-based Ceph redundancy is supported by PVC; erasure coded pools are not supported due to major performance impacts related to rewrites and random I/O as well as management overhead.
 
-The default replication level for a new pool is `copies=3, mincopies=2`. This will store 3 copies of each object, with a host-level failure domain, and will allow I/O as long as 2 copies are available. Thus, in a cluster of any size, all data is fully available even if a single host becomes unavailable. It will however use 3x the space for each piece of data stored, which must be considered when sizing the disk space for the cluster: a pool in this configuration, running on 3 nodes each with a single 400GB disk, will effectively have 400GB of total space available for use. As mentioned above, new disks must also be added in groups across nodes equal to the total number of `copies` to ensure new space is usable; for instance in a `copies=3` scheme, at least 3 disks must thus be added to different hosts at the same time for the avilable space to grow.
+The default replication level for a new pool is `copies=3, mincopies=2`. This will store 3 copies of each object, with a host-level failure domain, and will allow I/O as long as 2 copies are available. Thus, in a cluster of any size, all data is fully available even if a single host becomes unavailable. It will however use 3x the space for each piece of data stored, which must be considered when sizing the disk space for the cluster: a pool in this configuration, running on 3 nodes each with a single 400GB disk, will effectively have 400GB of total space available for use. As mentioned above, new disks must also be added in groups across nodes equal to the total number of `copies` to ensure new space is usable; for instance in a `copies=3` scheme, at least 3 disks must thus be added to different hosts at the same time for the available space to grow.
 
 Non-default values can also be set at pool creation time. For instance, one could create a `copies=3, mincopies=1` pool, which would allow I/O with two hosts down, but leaves the cluster susceptible to a write hole should a disk fail in this state; this configuration is not recommended in most situations. Alternatively, for additional resilience, one could create a `copies=4, mincopies=2` pool, which would also allow 2 hosts to fail, without a write hole, but would consume 4x the space for each piece of data stored and require new disks to be added in groups of 4 instead. Practically any combination of values is possible, however these 3 are the most relevant for most use-cases, and for most, especially small, clusters, the default is sufficient to provide solid redundancy and guard against host failures until the administrator can respond.
 
 Replication levels cannot be changed within PVC once a pool is created, however they can be changed via manual Ceph commands on a coordinator should the administrator require this, though discussion of this process is outside of the scope of this documentation. The administrator should carefully consider sizing, failure domains, and performance when first selecting storage devices and creating pools, to ensure the right level of resiliency versus data usage for their use-case and planned cluster size.
 
-## Physical network considerations
+### Networks
 
-At a minimum, a production PVC cluster should use at least two 1Gbps Ethernet interfaces, connected in an LACP or active-backup bond on one or more switches. On top of this bond, the various cluster networks should be configured as 802.3q vLANs. PVC is be able to support configurations without bonding or 802.1q vLAN support, using multiple physical interfaces and no bridged client networks, but this is strongly discouraged due to the added complexity this introduces; the switches chosen for the cluster should include these requirements as a minimum.
+At a minimum, a production PVC cluster should use at least two 10Gbps Ethernet interfaces, connected in an LACP or active-backup bond on one or more switches. On top of this bond, the various cluster networks should be configured as 802.3q vLANs. PVC is be able to support configurations without bonding or 802.1q vLAN support, using multiple physical interfaces and no bridged client networks, but this is strongly discouraged due to the added complexity this introduces; the switches chosen for the cluster should include these requirements as a minimum.
 
-More advanced physical network layouts are also possible. For instance, one could have two isolated networks. On the first network, each node has two 10Gbps Ethernet interfaces, which are combined in a bond across two redundant switch fabrics and that handle the upstream and cluster networks. On the second network, each node has an additional two 10Gbps, which are also combined in a bond across the redundant switch fabrics and handle the storage network. This configuration could support up to 10Gbps of aggregate client traffic while also supporting 10Gbps of aggregate storage traffic. Even more complex network configurations are possible if the cluster requires such performance. See the [Example Configurations](#example-configurations) section for some basic topology examples.
+More advanced physical network layouts are also possible. For instance, one could have two isolated networks. On the first network, each node has two 10Gbps Ethernet interfaces, which are combined in a bond across two redundant switch fabrics and that handle the upstream and cluster networks. On the second network, each node has an additional two 10Gbps, which are also combined in a bond across the redundant switch fabrics and handle the storage network. This configuration could support up to 10Gbps of aggregate client traffic while also supporting 10Gbps of aggregate storage traffic. Even more complex network configurations are possible if the cluster requires such performance. See the [Example System Diagrams](#example-system-diagrams) section for some basic topology examples.
 
 Only Ethernet networks are supported by PVC. More exotic interconnects such as Infiniband are not supported by default, and must be manually set up with Ethernet (e.g. EoIB) layers on top to be usable with PVC.
 
+Lower-speed networks (e.g. 1Gbps or 100Mbps) should not be used as these will severely bottleneck the performance of the storage subsystem. In an advanced split layout, it may be acceptable to use 1Gbps interfaces for VM guest networks, however the core system networks should always be a minimum of 10Gbps.
+
 PVC manages the IP addressing of all nodes itself and creates the required addresses during node daemon startup; thus, the on-boot network configuration of each interface should be set to "manual" with no IP addresses configured. This can be ignored safely, however, and the addresses specified manually in the networking configurations. PVC nodes use a split (`/etc/network/interfaces.d/<iface>`) network configuration model.
 
-## Network Layout: Considering the required networks
-
-A PVC cluster needs several different networks to operate properly; they are described in detail below and the administrator should ensure they account for all the required networks when planning the cluster.
-
-### PVC system networks
+### System Networks
 
 #### Upstream: Connecting the nodes to the wider world
 
@@ -166,7 +258,7 @@ Nodes in this network are generally assigned IPs automatically based on their no
 
 The administrator may choose to collocate the storage network on the same physical interface as the cluster network, or on a separate physical interface. This should be decided based on the size of the cluster and the perceived ratios of client network versus storage traffic. In large (>3 node) or storage-intensive clusters, this network should generally be a separate set of fast physical interfaces, separate from both the upstream and cluster networks, in order to maximize and isolate the storage bandwidth. If the administrator does choose to collocate these networks, they may also share the same IP address, thus eliminating any distinction between the Cluster and Storage networks. The PVC software handles this natively when the Cluster and Storage IPs of a node are identical.
 
-### PVC client networks
+### Client Networks
 
 #### Bridged (unmanaged) Client Networks
 
@@ -184,21 +276,75 @@ With this client network type, PVC is in full control of the network. No vLAN co
 
 NOTE: These networks may introduce a bottleneck and tromboning if there is a large amount of external and/or inter-network traffic on the cluster. The administrator should consider this carefully when deciding whether to use managed or bridged networks and properly evaluate the inter-network traffic requirements.
 
+#### SR-IOV Client Networks
+
+The third type of client network is the SR-IOV network. SR-IOV (Single-Root I/O Virtualization) is a technique and feature enabled on modern high-performance NICs (for instance, those from Intel or nVidia) which allows a single physical Ethernet port (a "PF" in SR-IOV terminology) to be split, at a hardware level, into multiple virtual Ethernet ports ("VF"s), which can then be managed separately. Starting with version 0.9.21, PVC support SR-IOV PF and VF configuration at the node level, and these VFs can be passed into VMs in two ways.
+
+SR-IOV's main benefit is to offload bridging and network functions from the hypervisor layer, and direct them onto the hardware itself. This can increase network throughput in some situations, as well as provide near-complete isolation of guest networks from the hypervisors (in contrast with bridges which *can* expose client traffic to the hypervisors, and VXLANs which *do* expose client traffic to the hypervisors). For instance, a VF can have a vLAN specified, and the tagging/untagging of packets is then carried out at the hardware layer.
+
+There are however caveats to working with SR-IOV. At the most basic level, the biggest difference with SR-IOV compared to the other two network types is that SR-IOV must be configured on a per-node basis. That is, each node must have SR-IOV explicitly enabled, it's specific PF devices defined, and a set of VFs created at PVC startup. Generally, with identical PVC nodes, this will not be a problem but is something to consider, especially if the servers are mismatched in any way. It is thus also possible to set some nodes with SR-IOV functionality, and others without, though care must be taken in this situation to set node limits in the VM metadata of any VMs which use SR-IOV VFs to prevent failed migrations.
+
+PFs are defined in the `pvcnoded.yml` configuration of each node, via the `sriov_device` list. Each PF can have an arbitrary number of VFs (`vfcount`) allocated, though each NIC vendor and model has specific limits. Once configured, specifically with Intel NICs, PFs (and specifically, the `vfcount` attribute in the driver) are immutable and cannot be changed easily without completely flushing the node and rebooting it, so care should be taken to select the desired settings as early in the cluster configuration as possible.
+
+Once created, VFs are also managed on a per-node basis. That is, each VF, on each host, even if they have the exact same device names, is managed separately. For instance, the PF `ens1f0` creating a VF `ens1f0v0` on "`hv1`", can have a different configuration from the identically-named VF `ens1f0v0` on "`hv2`". The administrator is responsible for ensuring consistency here, and for ensuring that devices do not overlap (e.g. assigning the same VF name to VMs on two separate nodes which might migrate to each other). PVC will however explicitly prevent two VMs from being assigned to the same VF on the same node, even if this may be technically possible in some cases.
+
+When attaching VFs to VMs, there are two supported modes: `macvtap`, and `hostdev`.
+
+`macvtap`, as the name suggests, uses the Linux `macvtap` driver to connect the VF to the VM. Once attached, the vNIC behaves just like a "bridged" network connection above, and like "bridged" connections, the "mode" of the NIC can be specified, defaulting to "virtio" but supporting various emulated devices instead. Note that in this mode, vLANs cannot be configured on the guest side; they must be specified in the VF configuration (`pvc network sriov vf set`) with one vLAN per VF. VMs with `macvtap` interfaces can be live migrated between nodes without issue, assuming there is a corresponding free VF on the destination node, and the SR-IOV functionality is transparent to the VM.
+
+`hostdev` is a direct PCIe pass-through method. With a VF attached to a VM in `hostdev` mode, the virtual PCIe NIC device itself becomes hidden from the node, and is visible only to the guest, where it appears as a discrete PCIe device. In this mode, vLANs and other attributes can be set on the guest side at will, though setting vLANs and other properties in the VF configuration is still supported. The main caveat to this mode is that VMs with connected `hostdev` SR-IOV VFs *cannot be live migrated between nodes*. Only a `shutdown` migration is supported, and, like `macvtap`, an identical PCIe device at the same bus address must be present on the target node. To prevent unexpected failures, PVC will explicitly set the VM metadata for the "migration method" to "shutdown" the first time that a `hostdev` VF is attached to it; if this changes later, the administrator must change this back explicitly.
+
+Generally speaking, SR-IOV connections are not recommended unless there is a good use-case for them. On modern hardware, software bridges are extremely performant, and are much simpler to manage. The functionality is provided for those rare use-cases where SR-IOV is absolutely required by the administrator, but care must be taken to understand all the requirements and caveats of SR-IOV before using it in production.
+
 #### Other Client Networks
 
 Future PVC versions may support other client network types, such as direct-routing between VMs.
 
-## Node Layout: Considering how nodes are laid out
+### Fencing and Recovery
 
-A production-grade PVC cluster requires at least 3 nodes running the PVC Daemon software. 1-node clusters are supported for very small clusters, home labs, and testing, but provide no redundancy; they should not be used in production situations.
+Self-management and self-healing are important components of PVC's design, and to accomplish this, PVC contains automated fencing and recovery functions to handle situations where nodes crash or become unreachable. PVC is then able, if properly configured, to directly power-cycle the failed node, and bring up any VMs that were running on it on the remaining hypervisors. This ensures that, while there might be a few minutes of downtime for VMs, they are recovered as quickly as possible without human intervention.
 
-### Node Functions: Coordinators versus Hypervisors
+To operate correctly, these functions require each node in the cluster to have a functional IPMI-over-IP setup with a configured user who is able to perform chassis power commands. This differs depending on the chassis manufacturer and model, and should be tested prior to deploying any production cluster. If IPMI is not configured correctly at node startup, the daemon will warn and disable automatic recovery of the node. The IPMI should be present in the Upstream system network (see [System Networks](#system-networks) above), or in another secured network which is reachable from the Upstream system network, whichever is more convenient for the layout of the networks.
 
-Within PVC, a given node can have one of two main functions: "Coordinator" or "Hypervisor".
+The general process is divided into 3 sections: detecting node failures, fencing nodes, and recovering from fenced nodes.
+
+#### Detecting Failed Nodes
+
+Within the PVC configuration, each node has 3 settings which determine the failure detection time. The first is the `keepalive_interval` setting. This is normally set to 5 seconds, and is the interval at which the node daemon of each node sends its keepalives (as well as gathers statistics about running VMs, Ceph components, etc.). This interval should never need to be changed, but is configurable for maximum flexibility in corner cases. During each keepalive, the node updates a specific key in the Zookeeper cluster with the current UNIX timestamp, which determines when the node was last alive. During their own keepalives, the other nodes check their peers' timestamps to confirm if they are updating normally. Note that, due to this happening during the peer keepalives, if all nodes lose contact with the Zookeeper database, they will *not* immediately begin fencing each other, since the keepalives will not complete; they will, however, upon recovery, jump immediately to the next section when they all realize that their last keepalives were over the threshold, and this situation is discussed there.
+
+The second option is the `fence_intervals` setting. This option determines how many keepalive intervals a node can miss before it is marked `dead` and a fencing sequence started. This is normally set to 6 intervals, which combined with the 5 second `keepalive_interval`, gives a total of 30 seconds (+/- up to another 5 second `keepalive_interval` for peers should they not line up) for the node to be without updates before fencing begins.
+
+The third setting is optional, and is best used in situations where the IPMI connectivity of a node is excessively flaky or can be impaired (e.g. georedundant clusters), or where VM uptime is more important than the burden of recovering from a split-brain situation, and is not as extensively tested. This option is `suicide_intervals`, and if set to a non-0 value, is the number of keepalive intervals before a node *itself* determines that it should forcibly power itself off, which should always be equal to or less than the normal `fence_intervals` setting. Naturally, the node must be somewhat functional to do this, and this can go very wrong, so using this option is not normally recommended.
+
+#### Fencing Nodes
+
+Once the cluster, and specifically one node in the cluster, has determined that a given node is `dead` due to a lack of keepalives, the fencing process starts. This spawns a dedicated child thread within the node daemon of the detecting node, which continually monitors the state of the `dead` node and then performs the fence.
+
+During the `dead` process, the failed node has 6 chances, called "saving throws", at `keepalive_interval` second windows, to send another keepalive before it is fenced. This additional, fixed, delay helps ensure that the cluster will gracefully recover from intermittent network failures or loss of Zookeeper contact, by providing nodes up to another 6 keepalive intervals to save themselves once the fence timer actually begins. This bring the total time, with default options, of a node stopping contact to a node being fenced, to between 60 and 65 seconds. This duration is considered by the author an acceptable compromise between speedy recovery and avoiding false positives (and hence larger outages).
+
+Once a node has been marked `dead` and has failed its 6 "saving throws", the fence process triggers an IPMI chassis reset sequence. First, the node is issued the standard IPMI `chassis power reset` command to trigger a cold system reset. Next, it waits a fixed 1 second and then issues a `chassis power on` signal to ensure the node is powered on (just in case it had already shut itself off). The node then waits a fixed 2 seconds, and then checks the current `chassis power status`. Using the results of these 3 commands, PVC is then able to determine with near certainty whether the node has truly been forced offline or not, and it can proceed to the next step.
+
+#### Recovery from Node Fences
+
+Once a node has been fenced, successfully or not, the system waits for one keepalive interval before proceeding.
+
+The cluster then determines what to do based both on the result of the fence (whether the node was determined to have been successfully cold-reset or not) and on two additional configuration values. The first, `successful_fence`, specifies what action to take when the fence was successful, and is either `migrate` (VMs to other nodes), the default, or `None` (no action). The second, `failed_fence`, is an identical choice for when the fence was unsuccessful, and defaults to `None`.
+
+If the fence was successful and `successful_fence` is set to `None`, then no migration takes place and the VMs on the fenced node will remain offline until the node recovers. If instead `successful_fence` is set to the default of `migrate`, the system will then begin migrating (and hence, starting) VMs that were active on the failed node to other nodes in the cluster. During this special `fence-flush` action, any stale RBD locks on the storage volumes are forcibly cleared, and this is considered safe since the fenced node is determined to have successfully been powered off and the VMs thus terminated. Once all VMs are migrated, the fenced node will then be set to a normal `flushed` state, as if it had been cleanly flushed before powering off. If and when the node returns to active, healthy service, either automatically (if the reset cleared the fault condition) or after human intervention, VMs can then migrate back and the cluster can resume normal operation; otherwise the cluster will remain in the degraded state until corrected.
+
+If the fence was unsuccessful and `failed_fence` is set to the default of `None`, no automatic recovery takes place, since the cluster cannot determine that it is safe to do so. This would most commonly occur during network partitions where the `dead` node potentially remains up with VMs running on it, and the cluster is now in a split-brain situation. The `suicide_interval` option mentioned above is provided for this specific situation, and would allow the administrator to set the `failed_fence` action to `migrate` as well, as they could be somewhat confident that the node will have forcibly terminated itself. However due to the inherent potential for danger in this scenario, it is recommended to leave these options at their defaults, and handle such situations manually instead, as well as ensuring proper network design to avoid the potential for such split-brain situations to occur.
+
+## Advanced Layouts
+
+### Coordinators versus Hypervisors
+
+While a normal basic PVC cluster would consist of 3, or perhaps 5, nodes, PVC is able to scale up much further by differentiating between "coordinator" and "hypervisor" nodes. Such a basic cluster would consist only of coordinator nodes. Scaling up however, it is prudent to add new nodes as hypervisor nodes instead to minimize database scaling problems.
 
 #### Coordinators
 
-Coordinators are a special set of 3 or 5 nodes with additional functionality. The coordinator nodes run, in addition to the PVC software itself, a number of databases and additional functions which are required by the whole cluster. An odd number of coordinators is *always* required to maintain quorum, though there are diminishing returns when creating more than 3. These additional functions are:
+Coordinators are a special set of 3 or 5 nodes with additional functionality. The coordinator nodes run, in addition to the PVC software itself, a number of databases and additional functions which are required by the whole cluster. An odd number of coordinators is *always* required to maintain quorum, though there are diminishing returns when creating more than 3. As mentioned above, generally for small clusters all nodes are coordinators.
+
+These additional functions are:
 
 0. The Zookeeper database cluster containing the cluster state and configuration
 0. The Patroni PostgreSQL database cluster containing DNS records for managed networks and provisioning configurations
@@ -221,9 +367,13 @@ PVC gracefully handles transitioning primary coordinator state, to minimize down
 
 #### Hypervisors
 
-Hypervisors consist of all other PVC nodes in the cluster. For small clusters (3 nodes), there will generally not be any non-coordinator nodes, though adding a 4th would require it to be a hypervisor to preserve quorum between the coordinators. Larger clusters should generally add new nodes as Hypervisors rather than coordinators to preserve the small set of coordinator nodes previously mentioned.
+Hypervisor nodes do not run any of the database or routing functionality of coordinator nodes, nor can they become the primary coordinator node (for obvious reasons). When scaling a cluster up beyond the initial 3, or perhaps 5, coordinator nodes, or when an even number of nodes (e.g. 4) may be desired, any nodes beyond the 3 coordinators should be added as hypervisors.
 
-### Geographic redundancy
+Hypervisor nodes are capable of running VMs and Ceph OSD disks, just like coordinator nodes, though the latter is optional.
+
+PVC has no limit to the number of hypervisor nodes that can connect to a set of coordinators, though beyond a dozen or so total nodes, a more scale-focused infrastructure solution may be warranted.
+
+### Georedundancy
 
 PVC supports geographic redundancy of nodes in order to facilitate disaster recovery scenarios when uptime is critical. Functionally, PVC behaves the same regardless of whether the 3 or more coordinators are in the same physical location, or remote physical locations.
 
@@ -231,33 +381,34 @@ When using geographic redundancy, there are several caveats to keep in mind:
 
 * The Ceph storage subsystem is latency-sensitive. With the default replication configuration, at least 2 writes must succeed for the write to return a success, so the total write latency of a write on any system will be equal to the maximum latency between any two nodes. It is recommended to keep all PVC nodes as "close" as possible latency-wise or storage performance may suffer.
 
-* The inter-node PVC networks must be layer-2 networks (broadcast domains). These networks must be spanned to all nodes in all locations.
+* The inter-node PVC networks (see [System Networks](#system-networks)) must be layer-2 networks (broadcast domains). These networks must be spanned to all nodes in all locations.
 
-* The number of sites and positioning of coordinators at those sites is important. A majority (at least 2 in a 3-coordinator cluster, or 3 in a 5-coordinator) of coordinators must be able to reach each other in a failure scenario for the cluster as a whole to remain functional. Thus, configurations such as 2 + 1 or 3 + 2 splits across 2 sites do *not* provide full redundancy, and the whole cluster will be down if the majority site is down. It is thus recommended to always have an odd number of sites to match the odd number of coordinators, for instance a 1 + 1 + 1 or 2 + 2 + 1 configuration. Also note that all hypervisors much be able to reach the majority coordinator group or their storage will be impacted as well.
+* The number of sites and positioning of coordinators at those sites is important. A majority (at least 2 in a 3-coordinator cluster, or 3 in a 5-coordinator cluster) of coordinators must be able to reach each other in a failure scenario for the cluster as a whole to remain functional. Thus, configurations such as 2 + 1 or 3 + 2 splits across 2 sites do *not* provide full redundancy, and the whole cluster will be down if the majority site is down. It is thus recommended to always have an odd number of sites to match the odd number of coordinators, for instance a 1 + 1 + 1 or 2 + 2 + 1 configuration. Also note that all hypervisors much be able to reach the majority coordinator group or their storage will be impacted as well.
 
-* Even if the PVC software itself is in an unmanageable state, VMs will continue to run if at all possible. However, since the storage subsystem makes use of the same quorum, losing more than half of the nodes will very likely result in storage interruption as well, which will affect running VMs.
+    This diagram outlines the supported and unsupported/unreliable georedundant configurations for 3 nodes. Care must always be taken to ensure that the cluster can operate with the loss of any given georeundant site.
+
+    ![georeundancy-caveats](/images/georedundancy-caveats.png)
+
+    *Above: Supported and unsupported/unreliable georedundant configurations* 
+
+* Even if the PVC software itself is in an unmanageable state, VMs will continue to run if at all possible. However, since the storage subsystem makes use of the same quorum, losing more than half of the coordinator nodes will very likely result in storage interruption as well, which will affect running VMs.
+
+* Nodes in remote geographic locations might not be able to be fenced by the remaining PVC nodes if the entire site is unreachable. The cluster will thus be unable to automatically recover VMs at the failed site should it go down. If at all possible, redundant links to georedundant sites are recommended to ensure there is always a network path. Note that the `suicide_interval` configuration option, while it might seem to help here, will not, because the remaining nodes will not be able to reliably confirm if the remote site actually *did* shut itself off. Thus automatic failover of georedundant sides is a potential deficiency that must be considered.
 
 If these requirements cannot be fulfilled, it may be best to have separate PVC clusters at each site and handle service redundancy at a higher layer to avoid a major disruption.
 
-## Example Configurations
+## Example System Diagrams
 
-This section provides diagrams of 3 possible node configurations. These diagrams can be extrapolated out to almost any possible configuration and number of nodes.
+This section provides diagrams of 2 best-practice cluster configurations. These diagrams can be extrapolated out to almost any possible configuration and number of nodes.
 
-#### Basic 3-node cluster
+#### Small 3-node cluster
 
-![3-node cluster](/images/3-node-cluster.png)
+![Small 3-node cluster](/images/pvc-3-node-cluster.png)
 
-*Above: A diagram of a simple 3-node cluster; all nodes are coordinators, single 1Gbps network interface per node, collapsed cluster and storage networks*
+*Above: A diagram of a simple 3-node cluster with all nodes as coordinators. Dual 10 Gbps network interface per node, unified physical networking with collapsed cluster and storage networks.*
 
-#### Mid-sized 8-node cluster with 3 coordinators
+#### Large 8-node cluster
 
-![8-node cluster](/images/8-node-cluster.png)
-
-*Above: A diagram of a mid-sized 8-node cluster with 3 coordinators, dual bonded 10Gbps network interfaces per node*
-
-#### Large 17-node cluster with 5 coordinators
-
-![17-node cluster](/images/17-node-cluster.png)
-
-*Above: A diagram of a large 17-node cluster with 5 coordinators, dual bonded 10Gbps network interfaces per node for both cluster/upstream and storage networks*
+![Large 8-node cluster](/images/pvc-8-node-cluster.png)
 
+*Above: A diagram of a large 8-node cluster with 3 coordinators and 5 hypervisors. Quad 10Gbps network interfaces per node, split physical networking into guest/cluster and storage networks.*

docs/index.md

@@ -1,176 +1,58 @@
-# PVC - The Parallel Virtual Cluster system
-
 <p align="center">
-<img alt="Logo banner" src="https://git.bonifacelabs.ca/uploads/-/system/project/avatar/135/pvc_logo.png"/>
+<img alt="Logo banner" src="images/pvc_logo_black.png"/>
 <br/><br/>
 <a href="https://github.com/parallelvirtualcluster/pvc"><img alt="License" src="https://img.shields.io/github/license/parallelvirtualcluster/pvc"/></a>
 <a href="https://github.com/parallelvirtualcluster/pvc/releases"><img alt="Release" src="https://img.shields.io/github/release-pre/parallelvirtualcluster/pvc"/></a>
 <a href="https://parallelvirtualcluster.readthedocs.io/en/latest/?badge=latest"><img alt="Documentation Status" src="https://readthedocs.org/projects/parallelvirtualcluster/badge/?version=latest"/></a>
 </p>
 
-PVC is a KVM+Ceph+Zookeeper-based, Free Software, scalable, redundant, self-healing, and self-managing private cloud solution designed with administrator simplicity in mind. It is built from the ground-up to be redundant at the host layer, allowing the cluster to gracefully handle the loss of nodes or their components, both due to hardware failure or due to maintenance. It is able to scale from a minimum of 3 nodes up to 12 or more nodes, while retaining performance and flexibility, allowing the administrator to build a small cluster today and grow it as needed.
+## What is PVC?
+
+PVC is a Linux KVM-based hyperconverged infrastructure (HCI) virtualization cluster solution that is fully Free Software, scalable, redundant, self-healing, self-managing, and designed for administrator simplicity. It is an alternative to other HCI solutions such as Harvester, Nutanix, and VMWare, as well as to other common virtualization stacks such as ProxMox and OpenStack.
+
+PVC is a complete HCI solution, built from well-known and well-trusted Free Software tools, to assist an administrator in creating and managing a cluster of servers to run virtual machines, as well as self-managing several important aspects including storage failover, node failure and recovery, virtual machine failure and recovery, and network plumbing. It is designed to act consistently, reliably, and unobtrusively, letting the administrator concentrate on more important things.
+
+PVC is highly scalable. From a minimum (production) node count of 3, up to 12 or more, and supporting many dozens of VMs, PVC scales along with your workload and requirements. Deploy a cluster once and grow it as your needs expand.
+
+As a consequence of its features, PVC makes administrating very high-uptime VMs extremely easy, featuring VM live migration, built-in always-enabled shared storage with transparent multi-node replication, and consistent network plumbing throughout the cluster. Nodes can also be seamlessly removed from or added to service, with zero VM downtime, to facilitate maintenance, upgrades, or other work.
+
+PVC also features an optional, fully customizable VM provisioning framework, designed to automate and simplify VM deployments using custom provisioning profiles, scripts, and CloudInit userdata API support.
+
+Installation of PVC is accomplished by two main components: a [Node installer ISO](https://github.com/parallelvirtualcluster/pvc-installer) which creates on-demand installer ISOs, and an [Ansible role framework](https://github.com/parallelvirtualcluster/pvc-ansible) to configure, bootstrap, and administrate the nodes. Once up, the cluster is managed via an HTTP REST API, accessible via a Python Click CLI client or WebUI.
+
+Just give it physical servers, and it will run your VMs without you having to think about it, all in just an hour or two of setup time.
+
+
+## What is it based on?
+
+The core node and API daemons, as well as the CLI API client, are written in Python 3 and are fully Free Software (GNU GPL v3). In addition to these, PVC makes use of the following software tools to provide a holistic hyperconverged infrastructure solution:
+
+  * Debian GNU/Linux as the base OS.
+  * Linux KVM, QEMU, and Libvirt for VM management.
+  * Linux `ip`, FRRouting, NFTables, DNSMasq, and PowerDNS for network management.
+  * Ceph for storage management.
+  * Apache Zookeeper for the primary cluster state database.
+  * Patroni PostgreSQL manager for the secondary relation databases (DNS aggregation, Provisioner configuration).
 
-The major goal of PVC is to be administrator friendly, providing the power of Enterprise-grade private clouds like OpenStack, Nutanix, and VMWare to homelabbers, SMBs, and small ISPs, without the cost or complexity. It believes in picking the best tool for a job and abstracting it behind the cluster as a whole, freeing the administrator from the boring and time-consuming task of selecting the best component, and letting them get on with the things that really matter. Administration can be done from a simple CLI or via a RESTful API capable of building full-featured web frontends or additional applications, taking a self-documenting approach to keep the administrator learning curvet as low as possible. Setup is easy and straightforward with an [ISO-based node installer](https://github.com/parallelvirtualcluster/pvc-installer) and [Ansible role framework](https://github.com/parallelvirtualcluster/pvc-ansible) designed to get a cluster up and running as quickly as possible. Build your cloud in an hour, grow it as you need, and never worry about it: just add physical servers.
 
 ## Getting Started
 
-To get started with PVC, please see the [About](https://parallelvirtualcluster.readthedocs.io/en/latest/about/) page for general information about the project, and the [Getting Started](https://parallelvirtualcluster.readthedocs.io/en/latest/getting-started/) page for details on configuring your cluster.
+To get started with PVC, please see the [About](https://parallelvirtualcluster.readthedocs.io/en/latest/about/) page for general information about the project, and the [Getting Started](https://parallelvirtualcluster.readthedocs.io/en/latest/getting-started/) page for details on configuring your first cluster.
+
 
 ## Changelog
 
-#### v0.9.20
+View the changelog in [CHANGELOG.md](https://github.com/parallelvirtualcluster/pvc/blob/master/CHANGELOG.md).
 
-  * [Daemons] Implemented a Zookeeper schema handler and version 0 schema
-  * [Daemons] Completes major refactoring of codebase to make use of the schema handler
-  * [Daemons] Adds support for dynamic chema changges and "hot reloading" of pvcnoded processes
-  * [Daemons] Adds a functional testing script for verifying operation against a test cluster
-  * [Daemons, CLI] Fixes several minor bugs found by the above script
-  * [Daemons, CLI] Add support for Debian 11 "Bullseye"
 
-#### v0.9.19
+## Screenshots
 
-  * [CLI] Corrects some flawed conditionals
-  * [API] Disables SQLAlchemy modification tracking functionality (not used by us)
-  * [Daemons] Implements new zkhandler module for improved reliability and reusability
-  * [Daemons] Refactors some code to use new zkhandler module
-  * [API, CLI] Adds support for "none" migration selector (uses cluster default instead)
-  * [Daemons] Moves some configuration keys to new /config tree
-  * [Node Daemon] Increases initial lock timeout for VM migrations to avoid out-of-sync potential
-  * [Provisioner] Support storing and using textual cluster network labels ("upstream", "storage", "cluster") in templates
-  * [API] Avoid duplicating existing node states
+While PVC's API and internals aren't very screenshot-worthy, here is some example output of the CLI tool.
 
-#### v0.9.18
+<p><img alt="Node listing" src="images/pvc-nodes.png"/><br/><i>Listing the nodes in a cluster</i></p>
 
-  * Adds VM rename functionality to API and CLI client
+<p><img alt="Network listing" src="images/pvc-networks.png"/><br/><i>Listing the networks in a cluster, showing 3 bridged and 1 IPv4-only managed networks</i></p>
 
-#### v0.9.17
-
-  * [CLI] Fixes bugs in log follow output
-
-#### v0.9.16
-
-  * Improves some CLI help messages
-  * Skips empty local cluster in CLI
-  * Adjusts how confirmations happen during VM modify restarts
-  * Fixes bug around corrupted VM log files
-  * Fixes bug around subprocess pipe exceptions
-
-#### v0.9.15
-
-  * [CLI] Adds additional verification (--yes) to several VM management commands
-  * [CLI] Adds a method to override --yes/confirmation requirements via envvar (PVC_UNSAFE)
-  * [CLI] Adds description fields to PVC clusters in CLI
-
-#### v0.9.14
-
-  * Fixes bugs around cloned volume provisioning
-  * Fixes some minor visual bugs
-  * Minor license update (from GPL3+ to GPL3)
-  * Adds qemu-guest-agent support to provisioner-created VMs by default
-
-#### v0.9.13
-
-  * Adds nicer startup messages for daemons
-  * Adds additional API field for stored_bytes to pool stats
-  * Fixes sorting issues with snapshot lists
-  * Fixes missing increment/decrement of snapshot_count on volumes
-  * Fixes bad calls in pool element API endpoints
-  * Fixes inconsistent bytes_tohuman behaviour in daemons
-  * Adds validation and maximum volume size on creation (must be smaller than the pool free space)
-
-#### v0.9.12
-
-  * Fixes a bug in the pvcnoded service unit file causing a Zookeeper startup race condition
-
-#### v0.9.11
-
-  * Documentation updates
-  * Adds VNC information to VM info
-  * Goes back to external Ceph commands for disk usage
-
-#### v0.9.10
-
-  * Moves OSD stats uploading to primary, eliminating reporting failures while hosts are down
-  * Documentation updates
-  * Significantly improves RBD locking behaviour in several situations, eliminating cold-cluster start issues and failed VM boot-ups after crashes
-  * Fixes some timeout delays with fencing
-  * Fixes bug in validating YAML provisioner userdata
-
-#### v0.9.9
-
-  * Adds documentation updates
-  * Removes single-element list stripping and fixes surrounding bugs
-  * Adds additional fields to some API endpoints for ease of parsing by clients
-  * Fixes bugs with network configuration
-
-#### v0.9.8
-
-  * Adds support for cluster backup/restore
-  * Moves location of `init` command in CLI to make room for the above
-  * Cleans up some invalid help messages from the API
-
-#### v0.9.7
-
-  * Fixes bug with provisioner system template modifications
-
-#### v0.9.6
-
-  * Fixes bug with migrations
-
-#### v0.9.5
-
-  * Fixes bug with line count in log follow
-  * Fixes bug with disk stat output being None
-  * Adds short pretty health output
-  * Documentation updates
-
-#### v0.9.4
-
-  * Fixes major bug in OVA parser
-
-#### v0.9.3
-
-  * Fixes bugs with image & OVA upload parsing
-
-#### v0.9.2
-
-  * Major linting of the codebase with flake8; adds linting tools
-  * Implements CLI-based modification of VM vCPUs, memory, networks, and disks without directly editing XML
-  * Fixes bug where `pvc vm log -f` would show all 1000 lines before starting
-  * Fixes bug in default provisioner libvirt schema (`drive` -> `driver` typo)
-
-#### v0.9.1
-
-  * Added per-VM migration method feature
-  * Fixed bug with provisioner system template listing
-
-#### v0.9.0
-
-Numerous small improvements and bugfixes. This release is suitable for general use and is pre-release-quality software.
-
-This release introduces an updated version scheme; all future stable releases until 1.0.0 is ready will be made under this 0.9.z naming. This does not represent semantic versioning and all changes (feature, improvement, or bugfix) will be considered for inclusion in this release train.
-
-#### v0.8
-
-Numerous improvements and bugfixes. This release is suitable for general use and is pre-release-quality software.
-
-#### v0.7
-
-Numerous improvements and bugfixes, revamped documentation. This release is suitable for general use and is beta-quality software.
-
-#### v0.6
-
-Numerous improvements and bugfixes, full implementation of the provisioner, full implementation of the API CLI client (versus direct CLI client). This release is suitable for general use and is beta-quality software.
-
-#### v0.5
-
-First public release; fully implements the VM, network, and storage managers, the HTTP API, and the pvc-ansible framework for deploying and bootstrapping a cluster. This release is suitable for general use, though it is still alpha-quality software and should be expected to change significantly until 1.0 is released.
-
-#### v0.4
-
-Full implementation of virtual management and virtual networking functionality. Partial implementation of storage functionality.
-
-#### v0.3
-
-Basic implementation of virtual management functionality.
+<p><img alt="VM listing and migration" src="images/pvc-migration.png"/><br/><i>Listing a limited set of VMs and migrating one with status updates</i></p>
 
+<p><img alt="Node logs" src="images/pvc-nodelog.png"/><br/><i>Viewing the logs of a node (keepalives and VM [un]migration)</i></p>

docs/manuals/ansible.md

@@ -450,6 +450,13 @@ pvc_nodes:
     ipmi_password: "{{ passwd_ipmi_host }}"
 
 pvc_bridge_device: bondU
+pvc_bridge_mtu: 1500
+
+pvc_sriov_enable: True
+pvc_sriov_device:
+  - phy: ens1f0
+    mtu: 9000
+    vfcount: 6
 
 pvc_upstream_device: "{{ networks['upstream']['device'] }}"
 pvc_upstream_mtu: "{{ networks['upstream']['mtu'] }}"
@@ -901,6 +908,24 @@ The IPMI password for the node management controller. Unless a per-host override
 
 The device name of the underlying network interface to be used for "bridged"-type client networks. For each "bridged"-type network, an IEEE 802.3q vLAN and bridge will be created on top of this device to pass these networks. In most cases, using the reflexive `networks['cluster']['raw_device']` or `networks['upstream']['raw_device']` from the Base role is sufficient.
 
+#### `pvc_bridge_mtu`
+
+* *required*
+
+The MTU of the underlying network interface to be used for "bridged"-type client networks. This is the maximum MTU such networks can use.
+
+#### `pvc_sriov_enable`
+
+* *optional*
+
+Whether to enable or disable SR-IOV functionality.
+
+#### `pvc_sriov_device`
+
+* *optional*
+
+A list of SR-IOV devices. See the Daemon manual for details.
+
 #### `pvc_<network>_*`
 
 The next set of entries is hard-coded to use the values from the global `networks` list. It should not need to be changed under most circumstances. Refer to the previous sections for specific notes about each entry.

docs/manuals/daemon.md

@@ -146,6 +146,12 @@ pvc:
         console_log_lines: 1000
       networking:
         bridge_device: ens4
+        bridge_mtu: 1500
+        sriov_enable: True
+        sriov_device:
+          - phy: ens1f0
+            mtu: 9000
+            vfcount: 7
         upstream:
           device: ens4
           mtu: 1500
@@ -422,6 +428,41 @@ How many lines of VM console logs to keep in the Zookeeper database for each VM.
 
 The network interface device used to create Bridged client network vLANs on. For most clusters, should match the underlying device of the various static networks (e.g. `ens4` or `bond0`), though may also use a separate network interface.
 
+#### `system` → `configuration` → `networking` → `bridge_mtu`
+
+* *optional*
+* *requires* `functions` → `enable_networking`
+
+The network interface MTU for the Bridged client network device. This is the maximum MTU a bridged client network can use.
+
+#### `system` → `configuration` → `networking` → `sriov_enable`
+
+* *optional*, defaults to `False`
+* *requires* `functions` → `enable_networking`
+
+Enables (or disables) SR-IOV functionality in PVC. If enabled, at least one `sriov_device` entry should be specified.
+
+#### `system` → `configuration` → `networking` → `sriov_device`
+
+* *optional*
+* *requires* `functions` → `enable_networking`
+
+Contains a list of SR-IOV PF (physical function) devices and their basic configuration. Each element contains the following entries:
+
+##### `phy`:
+
+* *required*
+
+The raw Linux network device with SR-IOV PF functionality.
+
+##### `mtu`
+
+The MTU of the PF device, set on daemon startup.
+
+##### `vfcount`
+
+The number of VF devices to create on this PF. VF devices are then managed via PVC on a per-node basis.
+
 #### `system` → `configuration` → `networking`
 
 * *optional*

docs/manuals/swagger.json

@@ -144,6 +144,19 @@
             },
             "type": "object"
         },
+        "NodeLog": {
+            "properties": {
+                "data": {
+                    "description": "The recent log text",
+                    "type": "string"
+                },
+                "name": {
+                    "description": "The name of the Node",
+                    "type": "string"
+                }
+            },
+            "type": "object"
+        },
         "VMLog": {
             "properties": {
                 "data": {
@@ -215,6 +228,23 @@
             },
             "type": "object"
         },
+        "VMTags": {
+            "properties": {
+                "name": {
+                    "description": "The name of the VM",
+                    "type": "string"
+                },
+                "tags": {
+                    "description": "The tag(s) of the VM",
+                    "items": {
+                        "id": "VMTag",
+                        "type": "object"
+                    },
+                    "type": "array"
+                }
+            },
+            "type": "object"
+        },
         "acl": {
             "properties": {
                 "description": {
@@ -334,6 +364,10 @@
                     },
                     "type": "object"
                 },
+                "mtu": {
+                    "description": "The MTU of the network, if set; empty otherwise",
+                    "type": "integer"
+                },
                 "name_servers": {
                     "description": "The configured DNS nameservers of the network for NS records (\"managed\" networks only)",
                     "items": {
@@ -464,6 +498,10 @@
                     "description": "The current operating system type",
                     "type": "string"
                 },
+                "pvc_version": {
+                    "description": "The current running PVC node daemon version",
+                    "type": "string"
+                },
                 "running_domains": {
                     "description": "The list of running domains (VMs) by UUID",
                     "type": "string"
@@ -486,6 +524,14 @@
         },
         "osd": {
             "properties": {
+                "db_device": {
+                    "description": "The OSD database/WAL block device (logical volume); empty if not applicable",
+                    "type": "string"
+                },
+                "device": {
+                    "description": "The OSD data block device",
+                    "type": "string"
+                },
                 "id": {
                     "description": "The Ceph ID of the OSD",
                     "type": "string (containing integer)"
@@ -764,6 +810,99 @@
             },
             "type": "object"
         },
+        "sriov_pf": {
+            "properties": {
+                "mtu": {
+                    "description": "The MTU of the SR-IOV PF device",
+                    "type": "string"
+                },
+                "phy": {
+                    "description": "The name of the SR-IOV PF device",
+                    "type": "string"
+                },
+                "vfs": {
+                    "items": {
+                        "description": "The PHY name of a VF of this PF",
+                        "type": "string"
+                    },
+                    "type": "list"
+                }
+            },
+            "type": "object"
+        },
+        "sriov_vf": {
+            "properties": {
+                "config": {
+                    "id": "sriov_vf_config",
+                    "properties": {
+                        "link_state": {
+                            "description": "The current SR-IOV VF link state (either enabled, disabled, or auto)",
+                            "type": "string"
+                        },
+                        "query_rss": {
+                            "description": "Whether VF RSS querying is enabled or disabled",
+                            "type": "boolean"
+                        },
+                        "spoof_check": {
+                            "description": "Whether device spoof checking is enabled or disabled",
+                            "type": "boolean"
+                        },
+                        "trust": {
+                            "description": "Whether guest device trust is enabled or disabled",
+                            "type": "boolean"
+                        },
+                        "tx_rate_max": {
+                            "description": "The maximum TX rate of the SR-IOV VF device",
+                            "type": "string"
+                        },
+                        "tx_rate_min": {
+                            "description": "The minimum TX rate of the SR-IOV VF device",
+                            "type": "string"
+                        },
+                        "vlan_id": {
+                            "description": "The tagged vLAN ID of the SR-IOV VF device",
+                            "type": "string"
+                        },
+                        "vlan_qos": {
+                            "description": "The QOS group of the tagged vLAN",
+                            "type": "string"
+                        }
+                    },
+                    "type": "object"
+                },
+                "mac": {
+                    "description": "The current MAC address of the VF device",
+                    "type": "string"
+                },
+                "mtu": {
+                    "description": "The current MTU of the VF device",
+                    "type": "integer"
+                },
+                "pf": {
+                    "description": "The name of the SR-IOV PF parent of this VF device",
+                    "type": "string"
+                },
+                "phy": {
+                    "description": "The name of the SR-IOV VF device",
+                    "type": "string"
+                },
+                "usage": {
+                    "id": "sriov_vf_usage",
+                    "properties": {
+                        "domain": {
+                            "description": "The UUID of the domain the SR-IOV VF is currently used by",
+                            "type": "boolean"
+                        },
+                        "used": {
+                            "description": "Whether the SR-IOV VF is currently used by a VM or not",
+                            "type": "boolean"
+                        }
+                    },
+                    "type": "object"
+                }
+            },
+            "type": "object"
+        },
         "storage-template": {
             "properties": {
                 "disks": {
@@ -826,6 +965,7 @@
         "storagebenchmark": {
             "properties": {
                 "benchmark_result": {
+                    "description": "A format 0 test result",
                     "properties": {
                         "test_name": {
                             "properties": {
@@ -959,6 +1099,10 @@
                 "job": {
                     "description": "The job name (an ISO date) of the test result",
                     "type": "string"
+                },
+                "test_format": {
+                    "description": "The PVC benchmark format of the results",
+                    "type": "integer"
                 }
             },
             "type": "object"
@@ -1273,6 +1417,28 @@
                     "description": "The current state of the VM",
                     "type": "string"
                 },
+                "tags": {
+                    "description": "The tag(s) of the VM",
+                    "items": {
+                        "id": "VMTag",
+                        "properties": {
+                            "name": {
+                                "description": "The name of the tag",
+                                "type": "string"
+                            },
+                            "protected": {
+                                "description": "Whether the tag is protected or not",
+                                "type": "boolean"
+                            },
+                            "type": {
+                                "description": "The type of the tag (user, system)",
+                                "type": "string"
+                            }
+                        },
+                        "type": "object"
+                    },
+                    "type": "array"
+                },
                 "type": {
                     "description": "The type of the VM",
                     "type": "string"
@@ -1459,8 +1625,15 @@
         },
         "/api/v1/initialize": {
             "post": {
-                "description": "Note: Normally used only once during cluster bootstrap; checks for the existence of the \"/primary_node\" key before proceeding and returns 400 if found",
+                "description": "<br/>If the 'overwrite' option is not True, the cluster will return 400 if the `/config/primary_node` key is found. If 'overwrite' is True, the existing cluster<br/>data will be erased and new, empty data written in its place.<br/><br/>All node daemons should be stopped before running this command, and the API daemon started manually to avoid undefined behavior.",
                 "parameters": [
+                    {
+                        "description": "A flag to enable or disable (default) overwriting existing data",
+                        "in": "query",
+                        "name": "overwrite",
+                        "required": false,
+                        "type": "bool"
+                    },
                     {
                         "description": "A confirmation string to ensure that the API consumer really means it",
                         "in": "query",
@@ -1596,6 +1769,12 @@
                         "required": true,
                         "type": "string"
                     },
+                    {
+                        "description": "The MTU of the network; defaults to the underlying interface MTU if not set",
+                        "in": "query",
+                        "name": "mtu",
+                        "type": "integer"
+                    },
                     {
                         "description": "The DNS domain of the network (\"managed\" networks only)",
                         "in": "query",
@@ -1741,6 +1920,12 @@
                         "required": true,
                         "type": "string"
                     },
+                    {
+                        "description": "The MTU of the network; defaults to the underlying interface MTU if not set",
+                        "in": "query",
+                        "name": "mtu",
+                        "type": "integer"
+                    },
                     {
                         "description": "The DNS domain of the network (\"managed\" networks only)",
                         "in": "query",
@@ -1824,6 +2009,12 @@
                         "name": "description",
                         "type": "string"
                     },
+                    {
+                        "description": "The MTU of the network",
+                        "in": "query",
+                        "name": "mtu",
+                        "type": "integer"
+                    },
                     {
                         "description": "The DNS domain of the network (\"managed\" networks only)",
                         "in": "query",
@@ -2311,7 +2502,7 @@
                 "description": "",
                 "parameters": [
                     {
-                        "description": "A search limit; fuzzy by default, use ^/$ to force exact matches",
+                        "description": "A search limit in the name, tags, or an exact UUID; fuzzy by default, use ^/$ to force exact matches",
                         "in": "query",
                         "name": "limit",
                         "required": false,
@@ -2522,6 +2713,38 @@
                 ]
             }
         },
+        "/api/v1/node/{node}/log": {
+            "get": {
+                "description": "",
+                "parameters": [
+                    {
+                        "description": "The number of lines to retrieve",
+                        "in": "query",
+                        "name": "lines",
+                        "required": false,
+                        "type": "integer"
+                    }
+                ],
+                "responses": {
+                    "200": {
+                        "description": "OK",
+                        "schema": {
+                            "$ref": "#/definitions/NodeLog"
+                        }
+                    },
+                    "404": {
+                        "description": "Node not found",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    }
+                },
+                "summary": "Return the recent logs of {node}",
+                "tags": [
+                    "node"
+                ]
+            }
+        },
         "/api/v1/provisioner/create": {
             "post": {
                 "description": "Note: Starts a background job in the pvc-provisioner-worker Celery worker while returning a task ID; the task ID can be used to query the \"GET /provisioner/status/<task_id>\" endpoint for the job status",
@@ -4453,6 +4676,181 @@
                 ]
             }
         },
+        "/api/v1/sriov/pf": {
+            "get": {
+                "description": "",
+                "responses": {
+                    "200": {
+                        "description": "OK",
+                        "schema": {
+                            "$ref": "#/definitions/sriov_pf"
+                        }
+                    }
+                },
+                "summary": "Return a list of SR-IOV PFs on a given node",
+                "tags": [
+                    "network / sriov"
+                ]
+            }
+        },
+        "/api/v1/sriov/pf/{node}": {
+            "get": {
+                "description": "",
+                "responses": {
+                    "200": {
+                        "description": "OK",
+                        "schema": {
+                            "$ref": "#/definitions/sriov_pf"
+                        }
+                    }
+                },
+                "summary": "Return a list of SR-IOV PFs on node {node}",
+                "tags": [
+                    "network / sriov"
+                ]
+            }
+        },
+        "/api/v1/sriov/vf": {
+            "get": {
+                "description": "",
+                "responses": {
+                    "200": {
+                        "description": "OK",
+                        "schema": {
+                            "$ref": "#/definitions/sriov_vf"
+                        }
+                    }
+                },
+                "summary": "Return a list of SR-IOV VFs on a given node, optionally limited to those in the specified PF",
+                "tags": [
+                    "network / sriov"
+                ]
+            }
+        },
+        "/api/v1/sriov/vf/{node}": {
+            "get": {
+                "description": "",
+                "responses": {
+                    "200": {
+                        "description": "OK",
+                        "schema": {
+                            "$ref": "#/definitions/sriov_vf"
+                        }
+                    }
+                },
+                "summary": "Return a list of SR-IOV VFs on node {node}, optionally limited to those in the specified PF",
+                "tags": [
+                    "network / sriov"
+                ]
+            }
+        },
+        "/api/v1/sriov/vf/{node}/{vf}": {
+            "get": {
+                "description": "",
+                "responses": {
+                    "200": {
+                        "description": "OK",
+                        "schema": {
+                            "$ref": "#/definitions/sriov_vf"
+                        }
+                    },
+                    "404": {
+                        "description": "Not found",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    }
+                },
+                "summary": "Return information about {vf} on {node}",
+                "tags": [
+                    "network / sriov"
+                ]
+            },
+            "put": {
+                "description": "",
+                "parameters": [
+                    {
+                        "description": "The vLAN ID for vLAN tagging (0 is disabled)",
+                        "in": "query",
+                        "name": "vlan_id",
+                        "required": false,
+                        "type": "integer"
+                    },
+                    {
+                        "description": "The vLAN QOS priority (0 is disabled)",
+                        "in": "query",
+                        "name": "vlan_qos",
+                        "required": false,
+                        "type": "integer"
+                    },
+                    {
+                        "description": "The minimum TX rate (0 is disabled)",
+                        "in": "query",
+                        "name": "tx_rate_min",
+                        "required": false,
+                        "type": "integer"
+                    },
+                    {
+                        "description": "The maximum TX rate (0 is disabled)",
+                        "in": "query",
+                        "name": "tx_rate_max",
+                        "required": false,
+                        "type": "integer"
+                    },
+                    {
+                        "description": "The administrative link state",
+                        "enum": [
+                            "auto",
+                            "enable",
+                            "disable"
+                        ],
+                        "in": "query",
+                        "name": "link_state",
+                        "required": false,
+                        "type": "string"
+                    },
+                    {
+                        "description": "Enable or disable spoof checking",
+                        "in": "query",
+                        "name": "spoof_check",
+                        "required": false,
+                        "type": "boolean"
+                    },
+                    {
+                        "description": "Enable or disable VF user trust",
+                        "in": "query",
+                        "name": "trust",
+                        "required": false,
+                        "type": "boolean"
+                    },
+                    {
+                        "description": "Enable or disable query RSS support",
+                        "in": "query",
+                        "name": "query_rss",
+                        "required": false,
+                        "type": "boolean"
+                    }
+                ],
+                "responses": {
+                    "200": {
+                        "description": "OK",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    },
+                    "400": {
+                        "description": "Bad request",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    }
+                },
+                "summary": "Set the configuration of {vf} on {node}",
+                "tags": [
+                    "network / sriov"
+                ]
+            }
+        },
         "/api/v1/status": {
             "get": {
                 "description": "",
@@ -4648,6 +5046,20 @@
                         "name": "weight",
                         "required": true,
                         "type": "number"
+                    },
+                    {
+                        "description": "Whether to use an external OSD DB LV device",
+                        "in": "query",
+                        "name": "ext_db",
+                        "required": false,
+                        "type": "boolean"
+                    },
+                    {
+                        "description": "Decimal ratio of total OSD size for the external OSD DB LV device, default 0.05 (5%)",
+                        "in": "query",
+                        "name": "ext_db_ratio",
+                        "required": false,
+                        "type": "float"
                     }
                 ],
                 "responses": {
@@ -4770,6 +5182,45 @@
                 ]
             }
         },
+        "/api/v1/storage/ceph/osddb": {
+            "post": {
+                "description": "Note: This task may take up to 30s to complete and return",
+                "parameters": [
+                    {
+                        "description": "The PVC node to create the OSD DB volume group on",
+                        "in": "query",
+                        "name": "node",
+                        "required": true,
+                        "type": "string"
+                    },
+                    {
+                        "description": "The block device (e.g. \"/dev/sdb\", \"/dev/disk/by-path/...\", etc.) to create the OSD DB volume group on",
+                        "in": "query",
+                        "name": "device",
+                        "required": true,
+                        "type": "string"
+                    }
+                ],
+                "responses": {
+                    "200": {
+                        "description": "OK",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    },
+                    "400": {
+                        "description": "Bad request",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    }
+                },
+                "summary": "Add a Ceph OSD database volume group to the cluster",
+                "tags": [
+                    "storage / ceph"
+                ]
+            }
+        },
         "/api/v1/storage/ceph/pool": {
             "get": {
                 "description": "",
@@ -5516,7 +5967,7 @@
                 "description": "",
                 "parameters": [
                     {
-                        "description": "A name search limit; fuzzy by default, use ^/$ to force exact matches",
+                        "description": "A search limit in the name, tags, or an exact UUID; fuzzy by default, use ^/$ to force exact matches",
                         "in": "query",
                         "name": "limit",
                         "required": false,
@@ -5535,6 +5986,20 @@
                         "name": "state",
                         "required": false,
                         "type": "string"
+                    },
+                    {
+                        "description": "Limit list to VMs with this tag",
+                        "in": "query",
+                        "name": "tag",
+                        "required": false,
+                        "type": "string"
+                    },
+                    {
+                        "description": "Negate the specified node, state, or tag limit(s)",
+                        "in": "query",
+                        "name": "negate",
+                        "required": false,
+                        "type": "boolean"
                     }
                 ],
                 "responses": {
@@ -5610,6 +6075,26 @@
                         "name": "migration_method",
                         "required": false,
                         "type": "string"
+                    },
+                    {
+                        "description": "The user tag(s) of the VM",
+                        "in": "query",
+                        "items": {
+                            "type": "string"
+                        },
+                        "name": "user_tags",
+                        "required": false,
+                        "type": "array"
+                    },
+                    {
+                        "description": "The protected user tag(s) of the VM",
+                        "in": "query",
+                        "items": {
+                            "type": "string"
+                        },
+                        "name": "protected_tags",
+                        "required": false,
+                        "type": "array"
                     }
                 ],
                 "responses": {
@@ -5721,7 +6206,8 @@
                             "mem",
                             "vcpus",
                             "load",
-                            "vms"
+                            "vms",
+                            "none (cluster default)"
                         ],
                         "in": "query",
                         "name": "selector",
@@ -5747,6 +6233,26 @@
                         "name": "migration_method",
                         "required": false,
                         "type": "string"
+                    },
+                    {
+                        "description": "The user tag(s) of the VM",
+                        "in": "query",
+                        "items": {
+                            "type": "string"
+                        },
+                        "name": "user_tags",
+                        "required": false,
+                        "type": "array"
+                    },
+                    {
+                        "description": "The protected user tag(s) of the VM",
+                        "in": "query",
+                        "items": {
+                            "type": "string"
+                        },
+                        "name": "protected_tags",
+                        "required": false,
+                        "type": "array"
                     }
                 ],
                 "responses": {
@@ -5837,6 +6343,68 @@
                 ]
             }
         },
+        "/api/v1/vm/{vm}/device": {
+            "delete": {
+                "description": "",
+                "parameters": [
+                    {
+                        "description": "The raw Libvirt XML definition of the device to detach",
+                        "in": "query",
+                        "name": "xml",
+                        "required": true,
+                        "type": "string"
+                    }
+                ],
+                "responses": {
+                    "200": {
+                        "description": "OK",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    },
+                    "400": {
+                        "description": "Bad request",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    }
+                },
+                "summary": "Hot-detach device XML to {vm}",
+                "tags": [
+                    "vm"
+                ]
+            },
+            "post": {
+                "description": "",
+                "parameters": [
+                    {
+                        "description": "The raw Libvirt XML definition of the device to attach",
+                        "in": "query",
+                        "name": "xml",
+                        "required": true,
+                        "type": "string"
+                    }
+                ],
+                "responses": {
+                    "200": {
+                        "description": "OK",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    },
+                    "400": {
+                        "description": "Bad request",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    }
+                },
+                "summary": "Hot-attach device XML to {vm}",
+                "tags": [
+                    "vm"
+                ]
+            }
+        },
         "/api/v1/vm/{vm}/locks": {
             "post": {
                 "description": "",
@@ -5871,7 +6439,7 @@
                         }
                     },
                     "404": {
-                        "description": "Not found",
+                        "description": "VM not found",
                         "schema": {
                             "$ref": "#/definitions/Message"
                         }
@@ -5945,6 +6513,12 @@
                         "schema": {
                             "$ref": "#/definitions/Message"
                         }
+                    },
+                    "404": {
+                        "description": "VM not found",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
                     }
                 },
                 "summary": "Set the metadata of {vm}",
@@ -6132,6 +6706,84 @@
                     "vm"
                 ]
             }
+        },
+        "/api/v1/vm/{vm}/tags": {
+            "get": {
+                "description": "",
+                "responses": {
+                    "200": {
+                        "description": "OK",
+                        "schema": {
+                            "$ref": "#/definitions/VMTags"
+                        }
+                    },
+                    "404": {
+                        "description": "VM not found",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    }
+                },
+                "summary": "Return the tags of {vm}",
+                "tags": [
+                    "vm"
+                ]
+            },
+            "post": {
+                "description": "",
+                "parameters": [
+                    {
+                        "description": "The action to perform with the tag",
+                        "enum": [
+                            "add",
+                            "remove"
+                        ],
+                        "in": "query",
+                        "name": "action",
+                        "required": true,
+                        "type": "string"
+                    },
+                    {
+                        "description": "The text value of the tag",
+                        "in": "query",
+                        "name": "tag",
+                        "required": true,
+                        "type": "string"
+                    },
+                    {
+                        "default": false,
+                        "description": "Set the protected state of the tag",
+                        "in": "query",
+                        "name": "protected",
+                        "required": false,
+                        "type": "boolean"
+                    }
+                ],
+                "responses": {
+                    "200": {
+                        "description": "OK",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    },
+                    "400": {
+                        "description": "Bad request",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    },
+                    "404": {
+                        "description": "VM not found",
+                        "schema": {
+                            "$ref": "#/definitions/Message"
+                        }
+                    }
+                },
+                "summary": "Set the tags of {vm}",
+                "tags": [
+                    "vm"
+                ]
+            }
         }
     },
     "swagger": "2.0"

lint

@@ -6,8 +6,8 @@ if ! which flake8 &>/dev/null; then
 fi
 
 flake8 \
-    --ignore=E501 \
-    --exclude=api-daemon/migrations/versions,api-daemon/provisioner/examples
+    --ignore=E201,E202,E222,E501,E241,F522,F523,F541 \
+    --exclude=debian,api-daemon/migrations/versions,api-daemon/provisioner/examples
 ret=$?
 if [[ $ret -eq 0 ]]; then
     echo "No linting issues found!"

mkdocs.yml

@@ -3,3 +3,4 @@ theme: readthedocs
 markdown_extensions:
   - toc:
       permalink: yes
+      toc_depth: '1-4'

node-daemon/pvc-flush.service

@@ -2,7 +2,7 @@
 
 [Unit]
 Description = Parallel Virtual Cluster autoflush daemon
-After = pvcnoded.service pvcapid.service zookeeper.service libvirtd.service ssh.service ceph.target
+After = pvcnoded.service pvcapid.service zookeeper.service libvirtd.service ssh.service ceph.target network-online.target
 Wants = pvcnoded.service
 PartOf = pvc.target
 

node-daemon/pvcnoded.py

@@ -20,3 +20,5 @@
 ###############################################################################
 
 import pvcnoded.Daemon  # noqa: F401
+
+pvcnoded.Daemon.entrypoint()

node-daemon/pvcnoded.sample.yaml

@@ -140,6 +140,8 @@ pvc:
         file_logging: True
         # stdout_logging: Enable or disable logging to stdout (i.e. journald)
         stdout_logging: True
+        # zookeeper_logging: Enable ot disable logging to Zookeeper (for `pvc node log` functionality)
+        zookeeper_logging: True
         # log_colours: Enable or disable ANSI colours in log output
         log_colours: True
         # log_dates: Enable or disable date strings in log output
@@ -152,26 +154,46 @@ pvc:
         log_keepalive_storage_details: True
         # console_log_lines: Number of console log lines to store in Zookeeper per VM
         console_log_lines: 1000
+        # node_log_lines: Number of node log lines to store in Zookeeper per node
+        node_log_lines: 2000
       # networking: PVC networking configuration
       # OPTIONAL if enable_networking: False
       networking:
-        # bridge_device: Underlying device to use for bridged vLAN networks; usually the device underlying <cluster>
+        # bridge_device: Underlying device to use for bridged vLAN networks; usually the device of <cluster>
         bridge_device: ens4
+        # bridge_mtu: The MTU of the underlying device used for bridged vLAN networks, and thus the maximum
+        #             MTU of the overlying bridge devices.
+        bridge_mtu: 1500
+        # sriov_enable: Enable or disable (default if absent) SR-IOV network support
+        sriov_enable: False
+        # sriov_device: Underlying device(s) to use for SR-IOV networks; can be bridge_device or other NIC(s)
+        sriov_device:
+            # The physical device name
+          - phy: ens1f1 
+            # The preferred MTU of the physical device; OPTIONAL - defaults to the interface default if unset
+            mtu: 9000
+            # The number of VFs to enable on this device
+            # NOTE: This defines the maximum number of VMs which can be provisioned on this physical device; VMs
+            #       are allocated to these VFs manually by the administrator and thus all nodes should have the
+            #       same number
+            # NOTE: This value cannot be changed at runtime on Intel(R) NICs; the node will need to be restarted
+            #       if this value changes
+            vfcount: 8
         # upstream: Upstream physical interface device
         upstream:
           # device: Upstream interface device name
           device: ens4
           # mtu: Upstream interface MTU; use 9000 for jumbo frames (requires switch support)
           mtu: 1500
-          # address: Upstream interface IP address, options: None, by-id, <static>/<mask>
-          address: None
+          # address: Upstream interface IP address, options: by-id, <static>/<mask>
+          address: by-id
         # cluster: Cluster (VNIC) physical interface device
         cluster:
           # device: Cluster (VNIC) interface device name
           device: ens4
           # mtu: Cluster (VNIC) interface MTU; use 9000 for jumbo frames (requires switch support)
           mtu: 1500
-          # address: Cluster (VNIC) interface IP address, options: None, by-id, <static>/<mask>
+          # address: Cluster (VNIC) interface IP address, options: by-id, <static>/<mask>
           address: by-id
         # storage: Storage (Ceph OSD) physical interface device
         storage:
@@ -179,7 +201,7 @@ pvc:
           device: ens4
           # mtu: Storage (Ceph OSD) interface MTU; use 9000 for jumbo frames (requires switch support)
           mtu: 1500
-          # address: Storage (Ceph OSD) interface IP address, options: None, by-id, <static>/<mask>
+          # address: Storage (Ceph OSD) interface IP address, options: by-id, <static>/<mask>
           address: by-id
       # storage; PVC storage configuration
       # OPTIONAL if enable_storage: False

node-daemon/pvcnoded/CephInstance.py

@@ -1,428 +0,0 @@
-#!/usr/bin/env python3
-
-# CephInstance.py - Class implementing a PVC node Ceph instance
-# Part of the Parallel Virtual Cluster (PVC) system
-#
-#    Copyright (C) 2018-2021 Joshua M. Boniface <joshua@boniface.me>
-#
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License as published by
-#    the Free Software Foundation, version 3.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
-#
-###############################################################################
-
-import time
-import json
-import psutil
-
-import daemon_lib.common as common
-
-
-class CephOSDInstance(object):
-    def __init__(self, zkhandler, this_node, osd_id):
-        self.zkhandler = zkhandler
-        self.this_node = this_node
-        self.osd_id = osd_id
-        self.node = None
-        self.size = None
-        self.stats = dict()
-
-        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('osd.node', self.osd_id))
-        def watch_osd_node(data, stat, event=''):
-            if event and event.type == 'DELETED':
-                # The key has been deleted after existing before; terminate this watcher
-                # because this class instance is about to be reaped in Daemon.py
-                return False
-
-            try:
-                data = data.decode('ascii')
-            except AttributeError:
-                data = ''
-
-            if data and data != self.node:
-                self.node = data
-
-        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('osd.stats', self.osd_id))
-        def watch_osd_stats(data, stat, event=''):
-            if event and event.type == 'DELETED':
-                # The key has been deleted after existing before; terminate this watcher
-                # because this class instance is about to be reaped in Daemon.py
-                return False
-
-            try:
-                data = data.decode('ascii')
-            except AttributeError:
-                data = ''
-
-            if data and data != self.stats:
-                self.stats = json.loads(data)
-
-
-def add_osd(zkhandler, logger, node, device, weight):
-    # We are ready to create a new OSD on this node
-    logger.out('Creating new OSD disk on block device {}'.format(device), state='i')
-    try:
-        # 1. Create an OSD; we do this so we know what ID will be gen'd
-        retcode, stdout, stderr = common.run_os_command('ceph osd create')
-        if retcode:
-            print('ceph osd create')
-            print(stdout)
-            print(stderr)
-            raise
-        osd_id = stdout.rstrip()
-
-        # 2. Remove that newly-created OSD
-        retcode, stdout, stderr = common.run_os_command('ceph osd rm {}'.format(osd_id))
-        if retcode:
-            print('ceph osd rm')
-            print(stdout)
-            print(stderr)
-            raise
-
-        # 3a. Zap the disk to ensure it is ready to go
-        logger.out('Zapping disk {}'.format(device), state='i')
-        retcode, stdout, stderr = common.run_os_command('ceph-volume lvm zap --destroy {}'.format(device))
-        if retcode:
-            print('ceph-volume lvm zap')
-            print(stdout)
-            print(stderr)
-            raise
-
-        # 3b. Create the OSD for real
-        logger.out('Preparing LVM for new OSD disk with ID {} on {}'.format(osd_id, device), state='i')
-        retcode, stdout, stderr = common.run_os_command(
-            'ceph-volume lvm prepare --bluestore --data {device}'.format(
-                osdid=osd_id,
-                device=device
-            )
-        )
-        if retcode:
-            print('ceph-volume lvm prepare')
-            print(stdout)
-            print(stderr)
-            raise
-
-        # 4a. Get OSD FSID
-        logger.out('Getting OSD FSID for ID {} on {}'.format(osd_id, device), state='i')
-        retcode, stdout, stderr = common.run_os_command(
-            'ceph-volume lvm list {device}'.format(
-                osdid=osd_id,
-                device=device
-            )
-        )
-        for line in stdout.split('\n'):
-            if 'osd fsid' in line:
-                osd_fsid = line.split()[-1]
-
-        if not osd_fsid:
-            print('ceph-volume lvm list')
-            print('Could not find OSD fsid in data:')
-            print(stdout)
-            print(stderr)
-            raise
-
-        # 4b. Activate the OSD
-        logger.out('Activating new OSD disk with ID {}'.format(osd_id, device), state='i')
-        retcode, stdout, stderr = common.run_os_command(
-            'ceph-volume lvm activate --bluestore {osdid} {osdfsid}'.format(
-                osdid=osd_id,
-                osdfsid=osd_fsid
-            )
-        )
-        if retcode:
-            print('ceph-volume lvm activate')
-            print(stdout)
-            print(stderr)
-            raise
-
-        # 5. Add it to the crush map
-        logger.out('Adding new OSD disk with ID {} to CRUSH map'.format(osd_id), state='i')
-        retcode, stdout, stderr = common.run_os_command(
-            'ceph osd crush add osd.{osdid} {weight} root=default host={node}'.format(
-                osdid=osd_id,
-                weight=weight,
-                node=node
-            )
-        )
-        if retcode:
-            print('ceph osd crush add')
-            print(stdout)
-            print(stderr)
-            raise
-        time.sleep(0.5)
-
-        # 6. Verify it started
-        retcode, stdout, stderr = common.run_os_command(
-            'systemctl status ceph-osd@{osdid}'.format(
-                osdid=osd_id
-            )
-        )
-        if retcode:
-            print('systemctl status')
-            print(stdout)
-            print(stderr)
-            raise
-
-        # 7. Add the new OSD to the list
-        logger.out('Adding new OSD disk with ID {} to Zookeeper'.format(osd_id), state='i')
-        zkhandler.write([
-            (('osd', osd_id), ''),
-            (('osd.node', osd_id), node),
-            (('osd.device', osd_id), device),
-            (('osd.stats', osd_id), '{}'),
-        ])
-
-        # Log it
-        logger.out('Created new OSD disk with ID {}'.format(osd_id), state='o')
-        return True
-    except Exception as e:
-        # Log it
-        logger.out('Failed to create new OSD disk: {}'.format(e), state='e')
-        return False
-
-
-def remove_osd(zkhandler, logger, osd_id, osd_obj):
-    logger.out('Removing OSD disk {}'.format(osd_id), state='i')
-    try:
-        # 1. Verify the OSD is present
-        retcode, stdout, stderr = common.run_os_command('ceph osd ls')
-        osd_list = stdout.split('\n')
-        if osd_id not in osd_list:
-            logger.out('Could not find OSD {} in the cluster'.format(osd_id), state='e')
-            return True
-
-        # 1. Set the OSD out so it will flush
-        logger.out('Setting out OSD disk with ID {}'.format(osd_id), state='i')
-        retcode, stdout, stderr = common.run_os_command('ceph osd out {}'.format(osd_id))
-        if retcode:
-            print('ceph osd out')
-            print(stdout)
-            print(stderr)
-            raise
-
-        # 2. Wait for the OSD to flush
-        logger.out('Flushing OSD disk with ID {}'.format(osd_id), state='i')
-        osd_string = str()
-        while True:
-            try:
-                retcode, stdout, stderr = common.run_os_command('ceph pg dump osds --format json')
-                dump_string = json.loads(stdout)
-                for osd in dump_string:
-                    if str(osd['osd']) == osd_id:
-                        osd_string = osd
-                num_pgs = osd_string['num_pgs']
-                if num_pgs > 0:
-                    time.sleep(5)
-                else:
-                    raise
-            except Exception:
-                break
-
-        # 3. Stop the OSD process and wait for it to be terminated
-        logger.out('Stopping OSD disk with ID {}'.format(osd_id), state='i')
-        retcode, stdout, stderr = common.run_os_command('systemctl stop ceph-osd@{}'.format(osd_id))
-        if retcode:
-            print('systemctl stop')
-            print(stdout)
-            print(stderr)
-            raise
-
-        # FIXME: There has to be a better way to do this /shrug
-        while True:
-            is_osd_up = False
-            # Find if there is a process named ceph-osd with arg '--id {id}'
-            for p in psutil.process_iter(attrs=['name', 'cmdline']):
-                if 'ceph-osd' == p.info['name'] and '--id {}'.format(osd_id) in ' '.join(p.info['cmdline']):
-                    is_osd_up = True
-            # If there isn't, continue
-            if not is_osd_up:
-                break
-
-        # 4. Determine the block devices
-        retcode, stdout, stderr = common.run_os_command('readlink /var/lib/ceph/osd/ceph-{}/block'.format(osd_id))
-        vg_name = stdout.split('/')[-2]  # e.g. /dev/ceph-<uuid>/osd-block-<uuid>
-        retcode, stdout, stderr = common.run_os_command('vgs --separator , --noheadings -o pv_name {}'.format(vg_name))
-        pv_block = stdout.strip()
-
-        # 5. Zap the volumes
-        logger.out('Zapping OSD disk with ID {} on {}'.format(osd_id, pv_block), state='i')
-        retcode, stdout, stderr = common.run_os_command('ceph-volume lvm zap --destroy {}'.format(pv_block))
-        if retcode:
-            print('ceph-volume lvm zap')
-            print(stdout)
-            print(stderr)
-            raise
-
-        # 6. Purge the OSD from Ceph
-        logger.out('Purging OSD disk with ID {}'.format(osd_id), state='i')
-        retcode, stdout, stderr = common.run_os_command('ceph osd purge {} --yes-i-really-mean-it'.format(osd_id))
-        if retcode:
-            print('ceph osd purge')
-            print(stdout)
-            print(stderr)
-            raise
-
-        # 7. Delete OSD from ZK
-        logger.out('Deleting OSD disk with ID {} from Zookeeper'.format(osd_id), state='i')
-        zkhandler.delete(('osd', osd_id), recursive=True)
-
-        # Log it
-        logger.out('Removed OSD disk with ID {}'.format(osd_id), state='o')
-        return True
-    except Exception as e:
-        # Log it
-        logger.out('Failed to purge OSD disk with ID {}: {}'.format(osd_id, e), state='e')
-        return False
-
-
-class CephPoolInstance(object):
-    def __init__(self, zkhandler, this_node, name):
-        self.zkhandler = zkhandler
-        self.this_node = this_node
-        self.name = name
-        self.pgs = ''
-        self.stats = dict()
-
-        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('pool.pgs', self.name))
-        def watch_pool_node(data, stat, event=''):
-            if event and event.type == 'DELETED':
-                # The key has been deleted after existing before; terminate this watcher
-                # because this class instance is about to be reaped in Daemon.py
-                return False
-
-            try:
-                data = data.decode('ascii')
-            except AttributeError:
-                data = ''
-
-            if data and data != self.pgs:
-                self.pgs = data
-
-        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('pool.stats', self.name))
-        def watch_pool_stats(data, stat, event=''):
-            if event and event.type == 'DELETED':
-                # The key has been deleted after existing before; terminate this watcher
-                # because this class instance is about to be reaped in Daemon.py
-                return False
-
-            try:
-                data = data.decode('ascii')
-            except AttributeError:
-                data = ''
-
-            if data and data != self.stats:
-                self.stats = json.loads(data)
-
-
-class CephVolumeInstance(object):
-    def __init__(self, zkhandler, this_node, pool, name):
-        self.zkhandler = zkhandler
-        self.this_node = this_node
-        self.pool = pool
-        self.name = name
-        self.stats = dict()
-
-        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('volume.stats', f'{self.pool}/{self.name}'))
-        def watch_volume_stats(data, stat, event=''):
-            if event and event.type == 'DELETED':
-                # The key has been deleted after existing before; terminate this watcher
-                # because this class instance is about to be reaped in Daemon.py
-                return False
-
-            try:
-                data = data.decode('ascii')
-            except AttributeError:
-                data = ''
-
-            if data and data != self.stats:
-                self.stats = json.loads(data)
-
-
-class CephSnapshotInstance(object):
-    def __init__(self, zkhandler, this_node, pool, volume, name):
-        self.zkhandler = zkhandler
-        self.this_node = this_node
-        self.pool = pool
-        self.volume = volume
-        self.name = name
-        self.stats = dict()
-
-        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('snapshot.stats', f'{self.pool}/{self.volume}/{self.name}'))
-        def watch_snapshot_stats(data, stat, event=''):
-            if event and event.type == 'DELETED':
-                # The key has been deleted after existing before; terminate this watcher
-                # because this class instance is about to be reaped in Daemon.py
-                return False
-
-            try:
-                data = data.decode('ascii')
-            except AttributeError:
-                data = ''
-
-            if data and data != self.stats:
-                self.stats = json.loads(data)
-
-
-# Primary command function
-# This command pipe is only used for OSD adds and removes
-def run_command(zkhandler, logger, this_node, data, d_osd):
-    # Get the command and args
-    command, args = data.split()
-
-    # Adding a new OSD
-    if command == 'osd_add':
-        node, device, weight = args.split(',')
-        if node == this_node.name:
-            # Lock the command queue
-            zk_lock = zkhandler.writelock('base.cmd.ceph')
-            with zk_lock:
-                # Add the OSD
-                result = add_osd(zkhandler, logger, node, device, weight)
-                # Command succeeded
-                if result:
-                    # Update the command queue
-                    zkhandler.write([
-                        ('base.cmd.ceph', 'success-{}'.format(data))
-                    ])
-                # Command failed
-                else:
-                    # Update the command queue
-                    zkhandler.write([
-                        ('base.cmd.ceph', 'failure-{}'.format(data))
-                    ])
-                # Wait 1 seconds before we free the lock, to ensure the client hits the lock
-                time.sleep(1)
-
-    # Removing an OSD
-    elif command == 'osd_remove':
-        osd_id = args
-
-        # Verify osd_id is in the list
-        if d_osd[osd_id] and d_osd[osd_id].node == this_node.name:
-            # Lock the command queue
-            zk_lock = zkhandler.writelock('base.cmd.ceph')
-            with zk_lock:
-                # Remove the OSD
-                result = remove_osd(zkhandler, logger, osd_id, d_osd[osd_id])
-                # Command succeeded
-                if result:
-                    # Update the command queue
-                    zkhandler.write([
-                        ('base.cmd.ceph', 'success-{}'.format(data))
-                    ])
-                # Command failed
-                else:
-                    # Update the command queue
-                    zkhandler.write([
-                        ('base.cmd.ceph', 'failure-{}'.format(data))
-                    ])
-                # Wait 1 seconds before we free the lock, to ensure the client hits the lock
-                time.sleep(1)

node-daemon/pvcnoded/dnsmasq-zookeeper-leases.py

@@ -1,4 +1,4 @@
-#!/usr/bin/python3
+#!/usr/bin/env python3
 
 # dnsmasq-zookeeper-leases.py - DNSMASQ leases script for Zookeeper
 # Part of the Parallel Virtual Cluster (PVC) system

node-daemon/pvcnoded/objects/CephInstance.py

@@ -0,0 +1,586 @@
+#!/usr/bin/env python3
+
+# CephInstance.py - Class implementing a PVC node Ceph instance
+# Part of the Parallel Virtual Cluster (PVC) system
+#
+#    Copyright (C) 2018-2021 Joshua M. Boniface <joshua@boniface.me>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, version 3.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+#
+###############################################################################
+
+import time
+import json
+import psutil
+
+import daemon_lib.common as common
+
+from distutils.util import strtobool
+from re import search
+
+
+class CephOSDInstance(object):
+    def __init__(self, zkhandler, this_node, osd_id):
+        self.zkhandler = zkhandler
+        self.this_node = this_node
+        self.osd_id = osd_id
+        self.node = None
+        self.size = None
+        self.stats = dict()
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('osd.node', self.osd_id))
+        def watch_osd_node(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = ''
+
+            if data and data != self.node:
+                self.node = data
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('osd.stats', self.osd_id))
+        def watch_osd_stats(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = ''
+
+            if data and data != self.stats:
+                self.stats = json.loads(data)
+
+    @staticmethod
+    def add_osd(zkhandler, logger, node, device, weight, ext_db_flag=False, ext_db_ratio=0.05):
+        # We are ready to create a new OSD on this node
+        logger.out('Creating new OSD disk on block device {}'.format(device), state='i')
+        try:
+            # 1. Create an OSD; we do this so we know what ID will be gen'd
+            retcode, stdout, stderr = common.run_os_command('ceph osd create')
+            if retcode:
+                print('ceph osd create')
+                print(stdout)
+                print(stderr)
+                raise Exception
+            osd_id = stdout.rstrip()
+
+            # 2. Remove that newly-created OSD
+            retcode, stdout, stderr = common.run_os_command('ceph osd rm {}'.format(osd_id))
+            if retcode:
+                print('ceph osd rm')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            # 3a. Zap the disk to ensure it is ready to go
+            logger.out('Zapping disk {}'.format(device), state='i')
+            retcode, stdout, stderr = common.run_os_command('ceph-volume lvm zap --destroy {}'.format(device))
+            if retcode:
+                print('ceph-volume lvm zap')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            dev_flags = "--data {}".format(device)
+
+            # 3b. Prepare the logical volume if ext_db_flag
+            if ext_db_flag:
+                _, osd_size_bytes, _ = common.run_os_command('blockdev --getsize64 {}'.format(device))
+                osd_size_bytes = int(osd_size_bytes)
+                result = CephOSDInstance.create_osd_db_lv(zkhandler, logger, osd_id, ext_db_ratio, osd_size_bytes)
+                if not result:
+                    raise Exception
+                db_device = "osd-db/osd-{}".format(osd_id)
+                dev_flags += " --block.db {}".format(db_device)
+            else:
+                db_device = ""
+
+            # 3c. Create the OSD for real
+            logger.out('Preparing LVM for new OSD disk with ID {} on {}'.format(osd_id, device), state='i')
+            retcode, stdout, stderr = common.run_os_command(
+                'ceph-volume lvm prepare --bluestore {devices}'.format(
+                    osdid=osd_id,
+                    devices=dev_flags
+                )
+            )
+            if retcode:
+                print('ceph-volume lvm prepare')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            # 4a. Get OSD FSID
+            logger.out('Getting OSD FSID for ID {} on {}'.format(osd_id, device), state='i')
+            retcode, stdout, stderr = common.run_os_command(
+                'ceph-volume lvm list {device}'.format(
+                    osdid=osd_id,
+                    device=device
+                )
+            )
+            for line in stdout.split('\n'):
+                if 'osd fsid' in line:
+                    osd_fsid = line.split()[-1]
+
+            if not osd_fsid:
+                print('ceph-volume lvm list')
+                print('Could not find OSD fsid in data:')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            # 4b. Activate the OSD
+            logger.out('Activating new OSD disk with ID {}'.format(osd_id, device), state='i')
+            retcode, stdout, stderr = common.run_os_command(
+                'ceph-volume lvm activate --bluestore {osdid} {osdfsid}'.format(
+                    osdid=osd_id,
+                    osdfsid=osd_fsid
+                )
+            )
+            if retcode:
+                print('ceph-volume lvm activate')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            # 5. Add it to the crush map
+            logger.out('Adding new OSD disk with ID {} to CRUSH map'.format(osd_id), state='i')
+            retcode, stdout, stderr = common.run_os_command(
+                'ceph osd crush add osd.{osdid} {weight} root=default host={node}'.format(
+                    osdid=osd_id,
+                    weight=weight,
+                    node=node
+                )
+            )
+            if retcode:
+                print('ceph osd crush add')
+                print(stdout)
+                print(stderr)
+                raise Exception
+            time.sleep(0.5)
+
+            # 6. Verify it started
+            retcode, stdout, stderr = common.run_os_command(
+                'systemctl status ceph-osd@{osdid}'.format(
+                    osdid=osd_id
+                )
+            )
+            if retcode:
+                print('systemctl status')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            # 7. Add the new OSD to the list
+            logger.out('Adding new OSD disk with ID {} to Zookeeper'.format(osd_id), state='i')
+            zkhandler.write([
+                (('osd', osd_id), ''),
+                (('osd.node', osd_id), node),
+                (('osd.device', osd_id), device),
+                (('osd.db_device', osd_id), db_device),
+                (('osd.stats', osd_id), '{}'),
+            ])
+
+            # Log it
+            logger.out('Created new OSD disk with ID {}'.format(osd_id), state='o')
+            return True
+        except Exception as e:
+            # Log it
+            logger.out('Failed to create new OSD disk: {}'.format(e), state='e')
+            return False
+
+    @staticmethod
+    def remove_osd(zkhandler, logger, osd_id, osd_obj):
+        logger.out('Removing OSD disk {}'.format(osd_id), state='i')
+        try:
+            # 1. Verify the OSD is present
+            retcode, stdout, stderr = common.run_os_command('ceph osd ls')
+            osd_list = stdout.split('\n')
+            if osd_id not in osd_list:
+                logger.out('Could not find OSD {} in the cluster'.format(osd_id), state='e')
+                return True
+
+            # 1. Set the OSD out so it will flush
+            logger.out('Setting out OSD disk with ID {}'.format(osd_id), state='i')
+            retcode, stdout, stderr = common.run_os_command('ceph osd out {}'.format(osd_id))
+            if retcode:
+                print('ceph osd out')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            # 2. Wait for the OSD to flush
+            logger.out('Flushing OSD disk with ID {}'.format(osd_id), state='i')
+            osd_string = str()
+            while True:
+                try:
+                    retcode, stdout, stderr = common.run_os_command('ceph pg dump osds --format json')
+                    dump_string = json.loads(stdout)
+                    for osd in dump_string:
+                        if str(osd['osd']) == osd_id:
+                            osd_string = osd
+                    num_pgs = osd_string['num_pgs']
+                    if num_pgs > 0:
+                        time.sleep(5)
+                    else:
+                        raise Exception
+                except Exception:
+                    break
+
+            # 3. Stop the OSD process and wait for it to be terminated
+            logger.out('Stopping OSD disk with ID {}'.format(osd_id), state='i')
+            retcode, stdout, stderr = common.run_os_command('systemctl stop ceph-osd@{}'.format(osd_id))
+            if retcode:
+                print('systemctl stop')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            # FIXME: There has to be a better way to do this /shrug
+            while True:
+                is_osd_up = False
+                # Find if there is a process named ceph-osd with arg '--id {id}'
+                for p in psutil.process_iter(attrs=['name', 'cmdline']):
+                    if 'ceph-osd' == p.info['name'] and '--id {}'.format(osd_id) in ' '.join(p.info['cmdline']):
+                        is_osd_up = True
+                # If there isn't, continue
+                if not is_osd_up:
+                    break
+
+            # 4. Determine the block devices
+            retcode, stdout, stderr = common.run_os_command('readlink /var/lib/ceph/osd/ceph-{}/block'.format(osd_id))
+            vg_name = stdout.split('/')[-2]  # e.g. /dev/ceph-<uuid>/osd-block-<uuid>
+            retcode, stdout, stderr = common.run_os_command('vgs --separator , --noheadings -o pv_name {}'.format(vg_name))
+            pv_block = stdout.strip()
+
+            # 5. Zap the volumes
+            logger.out('Zapping OSD disk with ID {} on {}'.format(osd_id, pv_block), state='i')
+            retcode, stdout, stderr = common.run_os_command('ceph-volume lvm zap --destroy {}'.format(pv_block))
+            if retcode:
+                print('ceph-volume lvm zap')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            # 6. Purge the OSD from Ceph
+            logger.out('Purging OSD disk with ID {}'.format(osd_id), state='i')
+            retcode, stdout, stderr = common.run_os_command('ceph osd purge {} --yes-i-really-mean-it'.format(osd_id))
+            if retcode:
+                print('ceph osd purge')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            # 7. Remove the DB device
+            if zkhandler.exists(('osd.db_device', osd_id)):
+                db_device = zkhandler.read(('osd.db_device', osd_id))
+                logger.out('Removing OSD DB logical volume "{}"'.format(db_device), state='i')
+                retcode, stdout, stderr = common.run_os_command('lvremove --yes --force {}'.format(db_device))
+
+            # 8. Delete OSD from ZK
+            logger.out('Deleting OSD disk with ID {} from Zookeeper'.format(osd_id), state='i')
+            zkhandler.delete(('osd', osd_id), recursive=True)
+
+            # Log it
+            logger.out('Removed OSD disk with ID {}'.format(osd_id), state='o')
+            return True
+        except Exception as e:
+            # Log it
+            logger.out('Failed to purge OSD disk with ID {}: {}'.format(osd_id, e), state='e')
+            return False
+
+    @staticmethod
+    def add_db_vg(zkhandler, logger, device):
+        logger.out('Creating new OSD database volume group on block device {}'.format(device), state='i')
+        try:
+            # 0. Check if an existsing volume group exists
+            retcode, stdout, stderr = common.run_os_command(
+                'vgdisplay osd-db'
+            )
+            if retcode != 5:
+                logger.out('Ceph OSD database VG "osd-db" already exists', state='e')
+                return False
+
+            # 1. Create an empty partition table
+            logger.out('Creating partitions on block device {}'.format(device), state='i')
+            retcode, stdout, stderr = common.run_os_command(
+                'sgdisk --clear {}'.format(device)
+            )
+            if retcode:
+                print('sgdisk create partition table')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            retcode, stdout, stderr = common.run_os_command(
+                'sgdisk --new 1:: --typecode 1:8e00 {}'.format(device)
+            )
+            if retcode:
+                print('sgdisk create pv partition')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            # Handle the partition ID portion
+            if search(r'by-path', device) or search(r'by-id', device):
+                # /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0 -> pci-0000:03:00.0-scsi-0:1:0:0-part1
+                partition = '{}-part1'.format(device)
+            elif search(r'nvme', device):
+                # /dev/nvme0n1 -> nvme0n1p1
+                partition = '{}p1'.format(device)
+            else:
+                # /dev/sda -> sda1
+                # No other '/dev/disk/by-*' types are valid for raw block devices anyways
+                partition = '{}1'.format(device)
+
+            # 2. Create the PV
+            logger.out('Creating PV on block device {}'.format(partition), state='i')
+            retcode, stdout, stderr = common.run_os_command(
+                'pvcreate --force {}'.format(partition)
+            )
+            if retcode:
+                print('pv creation')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            # 2. Create the VG (named 'osd-db')
+            logger.out('Creating VG "osd-db" on block device {}'.format(partition), state='i')
+            retcode, stdout, stderr = common.run_os_command(
+                'vgcreate --force osd-db {}'.format(partition)
+            )
+            if retcode:
+                print('vg creation')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            # Log it
+            logger.out('Created new OSD database volume group on block device {}'.format(device), state='o')
+            return True
+        except Exception as e:
+            # Log it
+            logger.out('Failed to create OSD database volume group: {}'.format(e), state='e')
+            return False
+
+    @staticmethod
+    def create_osd_db_lv(zkhandler, logger, osd_id, ext_db_ratio, osd_size_bytes):
+        logger.out('Creating new OSD database logical volume for OSD ID {}'.format(osd_id), state='i')
+        try:
+            # 0. Check if an existsing logical volume exists
+            retcode, stdout, stderr = common.run_os_command(
+                'lvdisplay osd-db/osd{}'.format(osd_id)
+            )
+            if retcode != 5:
+                logger.out('Ceph OSD database LV "osd-db/osd{}" already exists'.format(osd_id), state='e')
+                return False
+
+            # 1. Determine LV sizing
+            osd_db_size = int(osd_size_bytes * ext_db_ratio / 1024 / 1024)
+
+            # 2. Create the LV
+            logger.out('Creating DB LV "osd-db/osd-{}" of {}M ({} * {})'.format(osd_id, osd_db_size, osd_size_bytes, ext_db_ratio), state='i')
+            retcode, stdout, stderr = common.run_os_command(
+                'lvcreate --yes --name osd-{} --size {} osd-db'.format(osd_id, osd_db_size)
+            )
+            if retcode:
+                print('db lv creation')
+                print(stdout)
+                print(stderr)
+                raise Exception
+
+            # Log it
+            logger.out('Created new OSD database logical volume "osd-db/osd-{}"'.format(osd_id), state='o')
+            return True
+        except Exception as e:
+            # Log it
+            logger.out('Failed to create OSD database logical volume: {}'.format(e), state='e')
+            return False
+
+
+class CephPoolInstance(object):
+    def __init__(self, zkhandler, this_node, name):
+        self.zkhandler = zkhandler
+        self.this_node = this_node
+        self.name = name
+        self.pgs = ''
+        self.stats = dict()
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('pool.pgs', self.name))
+        def watch_pool_node(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = ''
+
+            if data and data != self.pgs:
+                self.pgs = data
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('pool.stats', self.name))
+        def watch_pool_stats(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = ''
+
+            if data and data != self.stats:
+                self.stats = json.loads(data)
+
+
+class CephVolumeInstance(object):
+    def __init__(self, zkhandler, this_node, pool, name):
+        self.zkhandler = zkhandler
+        self.this_node = this_node
+        self.pool = pool
+        self.name = name
+        self.stats = dict()
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('volume.stats', f'{self.pool}/{self.name}'))
+        def watch_volume_stats(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = ''
+
+            if data and data != self.stats:
+                self.stats = json.loads(data)
+
+
+class CephSnapshotInstance(object):
+    def __init__(self, zkhandler, this_node, pool, volume, name):
+        self.zkhandler = zkhandler
+        self.this_node = this_node
+        self.pool = pool
+        self.volume = volume
+        self.name = name
+        self.stats = dict()
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('snapshot.stats', f'{self.pool}/{self.volume}/{self.name}'))
+        def watch_snapshot_stats(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = ''
+
+            if data and data != self.stats:
+                self.stats = json.loads(data)
+
+
+# Primary command function
+# This command pipe is only used for OSD adds and removes
+def ceph_command(zkhandler, logger, this_node, data, d_osd):
+    # Get the command and args
+    command, args = data.split()
+
+    # Adding a new OSD
+    if command == 'osd_add':
+        node, device, weight, ext_db_flag, ext_db_ratio = args.split(',')
+        ext_db_flag = bool(strtobool(ext_db_flag))
+        ext_db_ratio = float(ext_db_ratio)
+        if node == this_node.name:
+            # Lock the command queue
+            zk_lock = zkhandler.writelock('base.cmd.ceph')
+            with zk_lock:
+                # Add the OSD
+                result = CephOSDInstance.add_osd(zkhandler, logger, node, device, weight, ext_db_flag, ext_db_ratio)
+                # Command succeeded
+                if result:
+                    # Update the command queue
+                    zkhandler.write([
+                        ('base.cmd.ceph', 'success-{}'.format(data))
+                    ])
+                # Command failed
+                else:
+                    # Update the command queue
+                    zkhandler.write([
+                        ('base.cmd.ceph', 'failure-{}'.format(data))
+                    ])
+                # Wait 1 seconds before we free the lock, to ensure the client hits the lock
+                time.sleep(1)
+
+    # Removing an OSD
+    elif command == 'osd_remove':
+        osd_id = args
+
+        # Verify osd_id is in the list
+        if d_osd[osd_id] and d_osd[osd_id].node == this_node.name:
+            # Lock the command queue
+            zk_lock = zkhandler.writelock('base.cmd.ceph')
+            with zk_lock:
+                # Remove the OSD
+                result = CephOSDInstance.remove_osd(zkhandler, logger, osd_id, d_osd[osd_id])
+                # Command succeeded
+                if result:
+                    # Update the command queue
+                    zkhandler.write([
+                        ('base.cmd.ceph', 'success-{}'.format(data))
+                    ])
+                # Command failed
+                else:
+                    # Update the command queue
+                    zkhandler.write([
+                        ('base.cmd.ceph', 'failure-{}'.format(data))
+                    ])
+                # Wait 1 seconds before we free the lock, to ensure the client hits the lock
+                time.sleep(1)
+
+    # Adding a new DB VG
+    elif command == 'db_vg_add':
+        node, device = args.split(',')
+        if node == this_node.name:
+            # Lock the command queue
+            zk_lock = zkhandler.writelock('base.cmd.ceph')
+            with zk_lock:
+                # Add the VG
+                result = CephOSDInstance.add_db_vg(zkhandler, logger, device)
+                # Command succeeded
+                if result:
+                    # Update the command queue
+                    zkhandler.write([
+                        ('base.cmd.ceph', 'success-{}'.format(data))
+                    ])
+                # Command failed
+                else:
+                    # Update the command queue
+                    zkhandler.write([
+                        ('base.cmd.ceph', 'failure={}'.format(data))
+                    ])
+                # Wait 1 seconds before we free the lock, to ensure the client hits the lock
+                time.sleep(1)

node-daemon/pvcnoded/objects/DNSAggregatorInstance.py

@@ -74,7 +74,7 @@ class PowerDNSInstance(object):
         self.dns_server_daemon = None
 
         # Floating upstreams
-        self.vni_floatingipaddr, self.vni_cidrnetmask = self.config['vni_floating_ip'].split('/')
+        self.cluster_floatingipaddr, self.cluster_cidrnetmask = self.config['cluster_floating_ip'].split('/')
         self.upstream_floatingipaddr, self.upstream_cidrnetmask = self.config['upstream_floating_ip'].split('/')
 
     def start(self):
@@ -91,7 +91,7 @@ class PowerDNSInstance(object):
             '--disable-syslog=yes',              # Log only to stdout (which is then captured)
             '--disable-axfr=no',                 # Allow AXFRs
             '--allow-axfr-ips=0.0.0.0/0',        # Allow AXFRs to anywhere
-            '--local-address={},{}'.format(self.vni_floatingipaddr, self.upstream_floatingipaddr),  # Listen on floating IPs
+            '--local-address={},{}'.format(self.cluster_floatingipaddr, self.upstream_floatingipaddr),  # Listen on floating IPs
             '--local-port=53',                   # On port 53
             '--log-dns-details=on',              # Log details
             '--loglevel=3',                      # Log info

node-daemon/pvcnoded/objects/MetadataAPIInstance.py

@@ -180,7 +180,7 @@ class MetadataAPIInstance(object):
         client_macaddr = host_information.get('mac_address', None)
 
         # Find the VM with that MAC address - we can't assume that the hostname is actually right
-        _discard, vm_list = pvc_vm.get_list(self.zkhandler, None, None, None)
+        _discard, vm_list = pvc_vm.get_list(self.zkhandler, None, None, None, None)
         vm_details = dict()
         for vm in vm_list:
             try:

node-daemon/pvcnoded/objects/NodeInstance.py

@@ -21,7 +21,7 @@
 
 import time
 
-from threading import Thread
+from threading import Thread, Event
 
 import daemon_lib.common as common
 
@@ -38,10 +38,10 @@ class NodeInstance(object):
         # Which node is primary
         self.primary_node = None
         # States
-        self.daemon_mode = self.zkhandler.read(('node.mode.daemon', self.name))
+        self.daemon_mode = self.zkhandler.read(('node.mode', self.name))
         self.daemon_state = 'stop'
         self.router_state = 'client'
-        self.domain_state = 'ready'
+        self.domain_state = 'flushed'
         # Object lists
         self.d_node = d_node
         self.d_network = d_network
@@ -65,9 +65,9 @@ class NodeInstance(object):
             self.upstream_dev = self.config['upstream_dev']
             self.upstream_floatingipaddr = self.config['upstream_floating_ip'].split('/')[0]
             self.upstream_ipaddr, self.upstream_cidrnetmask = self.config['upstream_dev_ip'].split('/')
-            self.vni_dev = self.config['vni_dev']
-            self.vni_floatingipaddr = self.config['vni_floating_ip'].split('/')[0]
-            self.vni_ipaddr, self.vni_cidrnetmask = self.config['vni_dev_ip'].split('/')
+            self.cluster_dev = self.config['cluster_dev']
+            self.cluster_floatingipaddr = self.config['cluster_floating_ip'].split('/')[0]
+            self.cluster_ipaddr, self.cluster_cidrnetmask = self.config['cluster_dev_ip'].split('/')
             self.storage_dev = self.config['storage_dev']
             self.storage_floatingipaddr = self.config['storage_floating_ip'].split('/')[0]
             self.storage_ipaddr, self.storage_cidrnetmask = self.config['storage_dev_ip'].split('/')
@@ -76,16 +76,17 @@ class NodeInstance(object):
             self.upstream_floatingipaddr = None
             self.upstream_ipaddr = None
             self.upstream_cidrnetmask = None
-            self.vni_dev = None
-            self.vni_floatingipaddr = None
-            self.vni_ipaddr = None
-            self.vni_cidrnetmask = None
+            self.cluster_dev = None
+            self.cluster_floatingipaddr = None
+            self.cluster_ipaddr = None
+            self.cluster_cidrnetmask = None
             self.storage_dev = None
             self.storage_floatingipaddr = None
             self.storage_ipaddr = None
             self.storage_cidrnetmask = None
         # Threads
         self.flush_thread = None
+        self.flush_event = Event()
         # Flags
         self.flush_stopper = False
 
@@ -159,8 +160,8 @@ class NodeInstance(object):
                     if self.flush_thread is not None:
                         self.logger.out('Waiting for previous migration to complete'.format(self.name), state='i')
                         self.flush_stopper = True
-                        while self.flush_stopper:
-                            time.sleep(0.1)
+                        self.flush_event.wait()
+                        self.flush_event.clear()
 
                     # Do flushing in a thread so it doesn't block the migrates out
                     if self.domain_state == 'flush':
@@ -246,7 +247,7 @@ class NodeInstance(object):
             if data != self.domain_list:
                 self.domain_list = data
 
-        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('node.count.provisioned_domainss', self.name))
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('node.count.provisioned_domains', self.name))
         def watch_node_domainscount(data, stat, event=''):
             if event and event.type == 'DELETED':
                 # The key has been deleted after existing before; terminate this watcher
@@ -387,13 +388,13 @@ class NodeInstance(object):
         # 2. Add Cluster & Storage floating IP
         self.logger.out(
             'Creating floating management IP {}/{} on interface {}'.format(
-                self.vni_floatingipaddr,
-                self.vni_cidrnetmask,
+                self.cluster_floatingipaddr,
+                self.cluster_cidrnetmask,
                 'brcluster'
             ),
             state='o'
         )
-        common.createIPAddress(self.vni_floatingipaddr, self.vni_cidrnetmask, 'brcluster')
+        common.createIPAddress(self.cluster_floatingipaddr, self.cluster_cidrnetmask, 'brcluster')
         self.logger.out(
             'Creating floating storage IP {}/{} on interface {}'.format(
                 self.storage_floatingipaddr,
@@ -466,7 +467,6 @@ class NodeInstance(object):
                 """
                 patronictl
                     -c /etc/patroni/config.yml
-                    -d zookeeper://localhost:2181
                     switchover
                     --candidate {}
                     --force
@@ -600,13 +600,13 @@ class NodeInstance(object):
         # 6. Remove Cluster & Storage floating IP
         self.logger.out(
             'Removing floating management IP {}/{} from interface {}'.format(
-                self.vni_floatingipaddr,
-                self.vni_cidrnetmask,
+                self.cluster_floatingipaddr,
+                self.cluster_cidrnetmask,
                 'brcluster'
             ),
             state='o'
         )
-        common.removeIPAddress(self.vni_floatingipaddr, self.vni_cidrnetmask, 'brcluster')
+        common.removeIPAddress(self.cluster_floatingipaddr, self.cluster_cidrnetmask, 'brcluster')
         self.logger.out(
             'Removing floating storage IP {}/{} from interface {}'.format(
                 self.storage_floatingipaddr,
@@ -680,6 +680,7 @@ class NodeInstance(object):
             # Allow us to cancel the operation
             if self.flush_stopper:
                 self.logger.out('Aborting node flush'.format(self.name), state='i')
+                self.flush_event.set()
                 self.flush_thread = None
                 self.flush_stopper = False
                 return
@@ -712,6 +713,7 @@ class NodeInstance(object):
 
             # Wait for the VM to migrate so the next VM's free RAM count is accurate (they migrate in serial anyways)
             ticks = 0
+            self.logger.out('Waiting for migration of VM "{}"'.format(dom_uuid), state='i')
             while self.zkhandler.read(('domain.state', dom_uuid)) in ['migrate', 'unmigrate', 'shutdown']:
                 ticks += 1
                 if ticks > 600:
@@ -734,6 +736,7 @@ class NodeInstance(object):
             # Allow us to cancel the operation
             if self.flush_stopper:
                 self.logger.out('Aborting node unflush'.format(self.name), state='i')
+                self.flush_event.set()
                 self.flush_thread = None
                 self.flush_stopper = False
                 return
@@ -767,8 +770,14 @@ class NodeInstance(object):
             ])
 
             # Wait for the VM to migrate back
+            ticks = 0
+            self.logger.out('Waiting for migration of VM "{}"'.format(dom_uuid), state='i')
             while self.zkhandler.read(('domain.state', dom_uuid)) in ['migrate', 'unmigrate', 'shutdown']:
-                time.sleep(0.1)
+                ticks += 1
+                if ticks > 600:
+                    # Abort if we've waited for 120 seconds, the VM is messed and just continue
+                    break
+                time.sleep(0.2)
 
         self.zkhandler.write([
             (('node.state.domain', self.name), 'ready')

node-daemon/pvcnoded/objects/SRIOVVFInstance.py

@@ -0,0 +1,210 @@
+#!/usr/bin/env python3
+
+# SRIOVVFInstance.py - Class implementing a PVC SR-IOV VF and run by pvcnoded
+# Part of the Parallel Virtual Cluster (PVC) system
+#
+#    Copyright (C) 2018-2021 Joshua M. Boniface <joshua@boniface.me>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, version 3.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+#
+###############################################################################
+
+import daemon_lib.common as common
+
+
+def boolToOnOff(state):
+    if state and str(state) == 'True':
+        return 'on'
+    else:
+        return 'off'
+
+
+class SRIOVVFInstance(object):
+    # Initialization function
+    def __init__(self, vf, zkhandler, config, logger, this_node):
+        self.vf = vf
+        self.zkhandler = zkhandler
+        self.config = config
+        self.logger = logger
+        self.this_node = this_node
+        self.myhostname = self.this_node.name
+
+        self.pf = self.zkhandler.read(('node.sriov.vf', self.myhostname, 'sriov_vf.pf', self.vf))
+        self.mtu = self.zkhandler.read(('node.sriov.vf', self.myhostname, 'sriov_vf.mtu', self.vf))
+        self.vfid = self.vf.replace('{}v'.format(self.pf), '')
+
+        self.logger.out('Setting MTU to {}'.format(self.mtu), state='i', prefix='SR-IOV VF {}'.format(self.vf))
+        common.run_os_command('ip link set {} mtu {}'.format(self.vf, self.mtu))
+
+        # These properties are set via the DataWatch functions, to ensure they are configured on the system
+        self.mac = None
+        self.vlan_id = None
+        self.vlan_qos = None
+        self.tx_rate_min = None
+        self.tx_rate_max = None
+        self.spoof_check = None
+        self.link_state = None
+        self.trust = None
+        self.query_rss = None
+
+        # Zookeeper handlers for changed configs
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('node.sriov.vf', self.myhostname) + self.zkhandler.schema.path('sriov_vf.mac', self.vf))
+        def watch_vf_mac(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = '00:00:00:00:00:00'
+
+            if data != self.mac:
+                self.mac = data
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('node.sriov.vf', self.myhostname) + self.zkhandler.schema.path('sriov_vf.config.vlan_id', self.vf))
+        def watch_vf_vlan_id(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = '0'
+
+            if data != self.vlan_id:
+                self.vlan_id = data
+                self.logger.out('Setting vLAN ID to {}'.format(self.vlan_id), state='i', prefix='SR-IOV VF {}'.format(self.vf))
+                common.run_os_command('ip link set {} vf {} vlan {} qos {}'.format(self.pf, self.vfid, self.vlan_id, self.vlan_qos))
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('node.sriov.vf', self.myhostname) + self.zkhandler.schema.path('sriov_vf.config.vlan_qos', self.vf))
+        def watch_vf_vlan_qos(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = '0'
+
+            if data != self.vlan_qos:
+                self.vlan_qos = data
+                self.logger.out('Setting vLAN QOS to {}'.format(self.vlan_qos), state='i', prefix='SR-IOV VF {}'.format(self.vf))
+                common.run_os_command('ip link set {} vf {} vlan {} qos {}'.format(self.pf, self.vfid, self.vlan_id, self.vlan_qos))
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('node.sriov.vf', self.myhostname) + self.zkhandler.schema.path('sriov_vf.config.tx_rate_min', self.vf))
+        def watch_vf_tx_rate_min(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = '0'
+
+            if data != self.tx_rate_min:
+                self.tx_rate_min = data
+                self.logger.out('Setting minimum TX rate to {}'.format(self.tx_rate_min), state='i', prefix='SR-IOV VF {}'.format(self.vf))
+                common.run_os_command('ip link set {} vf {} min_tx_rate {}'.format(self.pf, self.vfid, self.tx_rate_min))
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('node.sriov.vf', self.myhostname) + self.zkhandler.schema.path('sriov_vf.config.tx_rate_max', self.vf))
+        def watch_vf_tx_rate_max(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; termaxate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = '0'
+
+            if data != self.tx_rate_max:
+                self.tx_rate_max = data
+                self.logger.out('Setting maximum TX rate to {}'.format(self.tx_rate_max), state='i', prefix='SR-IOV VF {}'.format(self.vf))
+                common.run_os_command('ip link set {} vf {} max_tx_rate {}'.format(self.pf, self.vfid, self.tx_rate_max))
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('node.sriov.vf', self.myhostname) + self.zkhandler.schema.path('sriov_vf.config.spoof_check', self.vf))
+        def watch_vf_spoof_check(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = '0'
+
+            if data != self.spoof_check:
+                self.spoof_check = data
+                self.logger.out('Setting spoof checking {}'.format(boolToOnOff(self.spoof_check)), state='i', prefix='SR-IOV VF {}'.format(self.vf))
+                common.run_os_command('ip link set {} vf {} spoofchk {}'.format(self.pf, self.vfid, boolToOnOff(self.spoof_check)))
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('node.sriov.vf', self.myhostname) + self.zkhandler.schema.path('sriov_vf.config.link_state', self.vf))
+        def watch_vf_link_state(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = 'on'
+
+            if data != self.link_state:
+                self.link_state = data
+                self.logger.out('Setting link state to {}'.format(boolToOnOff(self.link_state)), state='i', prefix='SR-IOV VF {}'.format(self.vf))
+                common.run_os_command('ip link set {} vf {} state {}'.format(self.pf, self.vfid, self.link_state))
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('node.sriov.vf', self.myhostname) + self.zkhandler.schema.path('sriov_vf.config.trust', self.vf))
+        def watch_vf_trust(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = 'off'
+
+            if data != self.trust:
+                self.trust = data
+                self.logger.out('Setting trust mode {}'.format(boolToOnOff(self.trust)), state='i', prefix='SR-IOV VF {}'.format(self.vf))
+                common.run_os_command('ip link set {} vf {} trust {}'.format(self.pf, self.vfid, boolToOnOff(self.trust)))
+
+        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('node.sriov.vf', self.myhostname) + self.zkhandler.schema.path('sriov_vf.config.query_rss', self.vf))
+        def watch_vf_query_rss(data, stat, event=''):
+            if event and event.type == 'DELETED':
+                # The key has been deleted after existing before; terminate this watcher
+                # because this class instance is about to be reaped in Daemon.py
+                return False
+
+            try:
+                data = data.decode('ascii')
+            except AttributeError:
+                data = 'off'
+
+            if data != self.query_rss:
+                self.query_rss = data
+                self.logger.out('Setting RSS query ability {}'.format(boolToOnOff(self.query_rss)), state='i', prefix='SR-IOV VF {}'.format(self.vf))
+                common.run_os_command('ip link set {} vf {} query_rss {}'.format(self.pf, self.vfid, boolToOnOff(self.query_rss)))

node-daemon/pvcnoded/objects/VMInstance.py

@@ -28,88 +28,15 @@ from threading import Thread
 
 from xml.etree import ElementTree
 
+from re import match
+
 import daemon_lib.common as common
 
-import pvcnoded.VMConsoleWatcherInstance as VMConsoleWatcherInstance
+import pvcnoded.objects.VMConsoleWatcherInstance as VMConsoleWatcherInstance
 
 import daemon_lib.common as daemon_common
 
 
-def flush_locks(zkhandler, logger, dom_uuid, this_node=None):
-    logger.out('Flushing RBD locks for VM "{}"'.format(dom_uuid), state='i')
-    # Get the list of RBD images
-    rbd_list = zkhandler.read(('domain.storage.volumes', dom_uuid)).split(',')
-
-    for rbd in rbd_list:
-        # Check if a lock exists
-        lock_list_retcode, lock_list_stdout, lock_list_stderr = common.run_os_command('rbd lock list --format json {}'.format(rbd))
-        if lock_list_retcode != 0:
-            logger.out('Failed to obtain lock list for volume "{}"'.format(rbd), state='e')
-            continue
-
-        try:
-            lock_list = json.loads(lock_list_stdout)
-        except Exception as e:
-            logger.out('Failed to parse lock list for volume "{}": {}'.format(rbd, e), state='e')
-            continue
-
-        # If there's at least one lock
-        if lock_list:
-            # Loop through the locks
-            for lock in lock_list:
-                if this_node is not None and zkhandler.read(('domain.state', dom_uuid)) != 'stop' and lock['address'].split(':')[0] != this_node.storage_ipaddr:
-                    logger.out('RBD lock does not belong to this host (lock owner: {}): freeing this lock would be unsafe, aborting'.format(lock['address'].split(':')[0], state='e'))
-                    zkhandler.write([
-                        (('domain.state', dom_uuid), 'fail'),
-                        (('domain.failed_reason', dom_uuid), 'Could not safely free RBD lock {} ({}) on volume {}; stop VM and flush locks manually'.format(lock['id'], lock['address'], rbd)),
-                    ])
-                    break
-                # Free the lock
-                lock_remove_retcode, lock_remove_stdout, lock_remove_stderr = common.run_os_command('rbd lock remove {} "{}" "{}"'.format(rbd, lock['id'], lock['locker']))
-                if lock_remove_retcode != 0:
-                    logger.out('Failed to free RBD lock "{}" on volume "{}": {}'.format(lock['id'], rbd, lock_remove_stderr), state='e')
-                    zkhandler.write([
-                        (('domain.state', dom_uuid), 'fail'),
-                        (('domain.failed_reason', dom_uuid), 'Could not free RBD lock {} ({}) on volume {}: {}'.format(lock['id'], lock['address'], rbd, lock_remove_stderr)),
-                    ])
-                    break
-                logger.out('Freed RBD lock "{}" on volume "{}"'.format(lock['id'], rbd), state='o')
-
-    return True
-
-
-# Primary command function
-def run_command(zkhandler, logger, this_node, data):
-    # Get the command and args
-    command, args = data.split()
-
-    # Flushing VM RBD locks
-    if command == 'flush_locks':
-        dom_uuid = args
-
-        # Verify that the VM is set to run on this node
-        if this_node.d_domain[dom_uuid].getnode() == this_node.name:
-            # Lock the command queue
-            zk_lock = zkhandler.writelock('base.cmd.domain')
-            with zk_lock:
-                # Flush the lock
-                result = flush_locks(zkhandler, logger, dom_uuid, this_node)
-                # Command succeeded
-                if result:
-                    # Update the command queue
-                    zkhandler.write([
-                        ('base.cmd.domain', 'success-{}'.format(data))
-                    ])
-                # Command failed
-                else:
-                    # Update the command queue
-                    zkhandler.write([
-                        ('base.cmd.domain', 'failure-{}'.format(data))
-                    ])
-                # Wait 1 seconds before we free the lock, to ensure the client hits the lock
-                time.sleep(1)
-
-
 class VMInstance(object):
     # Initialization function
     def __init__(self, domuuid, zkhandler, config, logger, this_node):
@@ -140,6 +67,9 @@ class VMInstance(object):
         self.inshutdown = False
         self.instop = False
 
+        # State thread
+        self.state_thread = None
+
         # Libvirt domuuid
         self.dom = self.lookupByUUID(self.domuuid)
 
@@ -156,8 +86,8 @@ class VMInstance(object):
 
             # Perform a management command
             self.logger.out('Updating state of VM {}'.format(self.domuuid), state='i')
-            state_thread = Thread(target=self.manage_vm_state, args=(), kwargs={})
-            state_thread.start()
+            self.state_thread = Thread(target=self.manage_vm_state, args=(), kwargs={})
+            self.state_thread.start()
 
     # Get data functions
     def getstate(self):
@@ -238,6 +168,34 @@ class VMInstance(object):
                 (('domain.console.vnc', self.domuuid), '')
             ])
 
+    # Attach a device to the running domain
+    def attach_device(self, xml_spec):
+        if not self.dom:
+            self.logger.out('Cannot attach device to non-running domain', state='w', prefix='Domain {}'.format(self.domuuid))
+            return False
+
+        try:
+            self.logger.out('Attaching new device to VM', state='i', prefix='Domain {}'.format(self.domuuid))
+            self.dom.attachDevice(xml_spec)
+            return True
+        except Exception as e:
+            self.logger.out('Failed to attach device: {}'.format(e), state='e', prefix='Domain {}'.format(self.domuuid))
+            return False
+
+    # Detach a device from the running domain
+    def detach_device(self, xml_spec):
+        if not self.dom:
+            self.logger.out('Cannot detach device from non-running domain', state='w', prefix='Domain {}'.format(self.domuuid))
+            return False
+
+        try:
+            self.logger.out('Detaching device from VM', state='i', prefix='Domain {}'.format(self.domuuid))
+            self.dom.detachDevice(xml_spec)
+            return True
+        except Exception as e:
+            self.logger.out('Failed to detach device: {}'.format(e), state='e', prefix='Domain {}'.format(self.domuuid))
+            return False
+
     # Start up the VM
     def start_vm(self):
         # Start the log watcher
@@ -265,7 +223,7 @@ class VMInstance(object):
         if self.getdom() is None or self.getdom().state()[0] != libvirt.VIR_DOMAIN_RUNNING:
             # Flush locks
             self.logger.out('Flushing RBD locks', state='i', prefix='Domain {}'.format(self.domuuid))
-            flush_locks(self.zkhandler, self.logger, self.domuuid, self.this_node)
+            VMInstance.flush_locks(self.zkhandler, self.logger, self.domuuid, self.this_node)
             if self.zkhandler.read(('domain.state', self.domuuid)) == 'fail':
                 lv_conn.close()
                 self.dom = None
@@ -335,6 +293,13 @@ class VMInstance(object):
         self.instop = True
         try:
             self.dom.destroy()
+            time.sleep(0.2)
+            try:
+                if self.getdom().state()[0] == libvirt.VIR_DOMAIN_RUNNING:
+                    # It didn't terminate, try again
+                    self.dom.destroy()
+            except libvirt.libvirtError:
+                pass
         except AttributeError:
             self.logger.out('Failed to terminate VM', state='e', prefix='Domain {}'.format(self.domuuid))
         self.removeDomainFromList()
@@ -351,6 +316,13 @@ class VMInstance(object):
         self.instop = True
         try:
             self.dom.destroy()
+            time.sleep(0.2)
+            try:
+                if self.getdom().state()[0] == libvirt.VIR_DOMAIN_RUNNING:
+                    # It didn't terminate, try again
+                    self.dom.destroy()
+            except libvirt.libvirtError:
+                pass
         except AttributeError:
             self.logger.out('Failed to stop VM', state='e', prefix='Domain {}'.format(self.domuuid))
         self.removeDomainFromList()
@@ -380,7 +352,7 @@ class VMInstance(object):
 
             # Abort shutdown if the state changes to start
             current_state = self.zkhandler.read(('domain.state', self.domuuid))
-            if current_state not in ['shutdown', 'restart']:
+            if current_state not in ['shutdown', 'restart', 'migrate']:
                 self.logger.out('Aborting VM shutdown due to state change', state='i', prefix='Domain {}'.format(self.domuuid))
                 is_aborted = True
                 break
@@ -456,42 +428,12 @@ class VMInstance(object):
         migrate_lock_node.acquire()
         migrate_lock_state.acquire()
 
-        time.sleep(0.2)  # Initial delay for the first writer to grab the lock
-
         # Don't try to migrate a node to itself, set back to start
         if self.node == self.lastnode or self.node == self.this_node.name:
             abort_migrate('Target node matches the current active node during initial check')
             return
 
-        # Synchronize nodes A (I am reader)
-        lock = self.zkhandler.readlock(('domain.migrate.sync_lock', self.domuuid))
-        self.logger.out('Acquiring read lock for synchronization phase A', state='i', prefix='Domain {}'.format(self.domuuid))
-        lock.acquire()
-        self.logger.out('Acquired read lock for synchronization phase A', state='o', prefix='Domain {}'.format(self.domuuid))
-        if self.zkhandler.read(('domain.migrate.sync_lock', self.domuuid)) == '':
-            self.logger.out('Waiting for peer', state='i', prefix='Domain {}'.format(self.domuuid))
-            ticks = 0
-            while self.zkhandler.read(('domain.migrate.sync_lock', self.domuuid)) == '':
-                time.sleep(0.1)
-                ticks += 1
-                if ticks > 300:
-                    self.logger.out('Timed out waiting 30s for peer', state='e', prefix='Domain {}'.format(self.domuuid))
-                    aborted = True
-                    break
-        self.logger.out('Releasing read lock for synchronization phase A', state='i', prefix='Domain {}'.format(self.domuuid))
-        lock.release()
-        self.logger.out('Released read lock for synchronization phase A', state='o', prefix='Domain {}'.format(self.domuuid))
-
-        if aborted:
-            abort_migrate('Timed out waiting for peer')
-            return
-
-        # Synchronize nodes B (I am writer)
-        lock = self.zkhandler.writelock(('domain.migrate.sync_lock', self.domuuid))
-        self.logger.out('Acquiring write lock for synchronization phase B', state='i', prefix='Domain {}'.format(self.domuuid))
-        lock.acquire()
-        self.logger.out('Acquired write lock for synchronization phase B', state='o', prefix='Domain {}'.format(self.domuuid))
-        time.sleep(0.5)  # Time for reader to acquire the lock
+        time.sleep(0.5)  # Initial delay for the first writer to grab the lock
 
         def migrate_live():
             self.logger.out('Setting up live migration', state='i', prefix='Domain {}'.format(self.domuuid))
@@ -523,21 +465,22 @@ class VMInstance(object):
             dest_lv_conn.close()
             self.console_log_instance.stop()
             self.removeDomainFromList()
-
             return True
 
         def migrate_shutdown():
             self.logger.out('Shutting down VM for offline migration', state='i', prefix='Domain {}'.format(self.domuuid))
-            self.zkhandler.write([
-                (('domain.state', self.domuuid), 'shutdown')
-            ])
-            while self.zkhandler.read(('domain.state', self.domuuid)) != 'stop':
-                time.sleep(0.5)
+            self.shutdown_vm()
             return True
 
-        do_migrate_shutdown = False
+        self.logger.out('Acquiring lock for migration phase B', state='i', prefix='Domain {}'.format(self.domuuid))
+        lock = self.zkhandler.exclusivelock(('domain.migrate.sync_lock', self.domuuid))
+        try:
+            lock.acquire(timeout=30.0)
+        except Exception:
+            abort_migrate('Timed out waiting for peer')
+            return
+        self.logger.out('Acquired lock for migration phase B', state='o', prefix='Domain {}'.format(self.domuuid))
         migrate_live_result = False
-
         # Do a final verification
         if self.node == self.lastnode or self.node == self.this_node.name:
             abort_migrate('Target node matches the current active node during final check')
@@ -545,7 +488,6 @@ class VMInstance(object):
         if self.node != target_node:
             abort_migrate('Target node changed during preparation')
             return
-
         if not force_shutdown:
             # A live migrate is attemped 3 times in succession
             ticks = 0
@@ -560,59 +502,20 @@ class VMInstance(object):
                     break
         else:
             migrate_live_result = False
-
         if not migrate_live_result:
             if force_live:
                 self.logger.out('Could not live migrate VM while live migration enforced', state='e', prefix='Domain {}'.format(self.domuuid))
                 aborted = True
             else:
-                do_migrate_shutdown = True
-
-        self.logger.out('Releasing write lock for synchronization phase B', state='i', prefix='Domain {}'.format(self.domuuid))
-        lock.release()
-        self.logger.out('Released write lock for synchronization phase B', state='o', prefix='Domain {}'.format(self.domuuid))
-
+                migrate_shutdown()
         if aborted:
             abort_migrate('Live migration failed and is required')
             return
-
-        # Synchronize nodes C (I am writer)
-        lock = self.zkhandler.writelock(('domain.migrate.sync_lock', self.domuuid))
-        self.logger.out('Acquiring write lock for synchronization phase C', state='i', prefix='Domain {}'.format(self.domuuid))
-        lock.acquire()
-        self.logger.out('Acquired write lock for synchronization phase C', state='o', prefix='Domain {}'.format(self.domuuid))
-        time.sleep(0.5)  # Time for reader to acquire the lock
-
-        if do_migrate_shutdown:
-            migrate_shutdown()
-
-        self.logger.out('Releasing write lock for synchronization phase C', state='i', prefix='Domain {}'.format(self.domuuid))
-        lock.release()
-        self.logger.out('Released write lock for synchronization phase C', state='o', prefix='Domain {}'.format(self.domuuid))
-
-        # Synchronize nodes D (I am reader)
-        lock = self.zkhandler.readlock(('domain.migrate.sync_lock', self.domuuid))
-        self.logger.out('Acquiring read lock for synchronization phase D', state='i', prefix='Domain {}'.format(self.domuuid))
-        lock.acquire()
-        self.logger.out('Acquired read lock for synchronization phase D', state='o', prefix='Domain {}'.format(self.domuuid))
-
         self.last_currentnode = self.zkhandler.read(('domain.node', self.domuuid))
         self.last_lastnode = self.zkhandler.read(('domain.last_node', self.domuuid))
-
-        self.logger.out('Releasing read lock for synchronization phase D', state='i', prefix='Domain {}'.format(self.domuuid))
-        lock.release()
-        self.logger.out('Released read lock for synchronization phase D', state='o', prefix='Domain {}'.format(self.domuuid))
-
-        # Wait for the receive side to complete before we declare all-done and release locks
-        ticks = 0
-        while self.zkhandler.read(('domain.migrate.sync_lock', self.domuuid)) != '':
-            time.sleep(0.1)
-            ticks += 1
-            if ticks > 100:
-                self.logger.out('Sync lock clear exceeded 10s timeout, continuing', state='w', prefix='Domain {}'.format(self.domuuid))
-                break
         migrate_lock_node.release()
         migrate_lock_state.release()
+        lock.release()
 
         self.inmigrate = False
         return
@@ -625,57 +528,31 @@ class VMInstance(object):
 
         self.inreceive = True
 
-        self.logger.out('Receiving VM migration from node "{}"'.format(self.node), state='i', prefix='Domain {}'.format(self.domuuid))
-
-        # Short delay to ensure sender is in sync
-        time.sleep(0.5)
+        self.logger.out('Receiving VM migration from node "{}"'.format(self.last_currentnode), state='i', prefix='Domain {}'.format(self.domuuid))
 
         # Ensure our lock key is populated
         self.zkhandler.write([
             (('domain.migrate.sync_lock', self.domuuid), self.domuuid)
         ])
 
-        # Synchronize nodes A (I am writer)
-        lock = self.zkhandler.writelock(('domain.migrate.sync_lock', self.domuuid))
-        self.logger.out('Acquiring write lock for synchronization phase A', state='i', prefix='Domain {}'.format(self.domuuid))
-        lock.acquire()
-        self.logger.out('Acquired write lock for synchronization phase A', state='o', prefix='Domain {}'.format(self.domuuid))
-        time.sleep(1)  # Time for reader to acquire the lock
-        self.logger.out('Releasing write lock for synchronization phase A', state='i', prefix='Domain {}'.format(self.domuuid))
+        self.logger.out('Acquiring lock for migration phase A', state='i', prefix='Domain {}'.format(self.domuuid))
+        lock = self.zkhandler.exclusivelock(('domain.migrate.sync_lock', self.domuuid))
+        try:
+            lock.acquire(timeout=30.0)
+        except Exception:
+            self.logger.out('Failed to acquire exclusive lock for VM', state='w')
+            return
+        self.logger.out('Acquired lock for migration phase A', state='o', prefix='Domain {}'.format(self.domuuid))
+        # Exactly twice the amount of time that the other side is waiting
+        time.sleep(1)
         lock.release()
-        self.logger.out('Released write lock for synchronization phase A', state='o', prefix='Domain {}'.format(self.domuuid))
-        time.sleep(0.1)  # Time for new writer to acquire the lock
 
-        # Synchronize nodes B (I am reader)
-        lock = self.zkhandler.readlock(('domain.migrate.sync_lock', self.domuuid))
-        self.logger.out('Acquiring read lock for synchronization phase B', state='i', prefix='Domain {}'.format(self.domuuid))
+        self.logger.out('Acquiring lock for phase C', state='i', prefix='Domain {}'.format(self.domuuid))
         lock.acquire()
-        self.logger.out('Acquired read lock for synchronization phase B', state='o', prefix='Domain {}'.format(self.domuuid))
-        self.logger.out('Releasing read lock for synchronization phase B', state='i', prefix='Domain {}'.format(self.domuuid))
-        lock.release()
-        self.logger.out('Released read lock for synchronization phase B', state='o', prefix='Domain {}'.format(self.domuuid))
-
-        # Synchronize nodes C (I am reader)
-        lock = self.zkhandler.readlock(('domain.migrate.sync_lock', self.domuuid))
-        self.logger.out('Acquiring read lock for synchronization phase C', state='i', prefix='Domain {}'.format(self.domuuid))
-        lock.acquire()
-        self.logger.out('Acquired read lock for synchronization phase C', state='o', prefix='Domain {}'.format(self.domuuid))
-
+        self.logger.out('Acquired lock for migration phase C', state='o', prefix='Domain {}'.format(self.domuuid))
         # Set the updated data
         self.last_currentnode = self.zkhandler.read(('domain.node', self.domuuid))
         self.last_lastnode = self.zkhandler.read(('domain.last_node', self.domuuid))
-
-        self.logger.out('Releasing read lock for synchronization phase C', state='i', prefix='Domain {}'.format(self.domuuid))
-        lock.release()
-        self.logger.out('Released read lock for synchronization phase C', state='o', prefix='Domain {}'.format(self.domuuid))
-
-        # Synchronize nodes D (I am writer)
-        lock = self.zkhandler.writelock(('domain.migrate.sync_lock', self.domuuid))
-        self.logger.out('Acquiring write lock for synchronization phase D', state='i', prefix='Domain {}'.format(self.domuuid))
-        lock.acquire()
-        self.logger.out('Acquired write lock for synchronization phase D', state='o', prefix='Domain {}'.format(self.domuuid))
-        time.sleep(0.5)  # Time for reader to acquire the lock
-
         self.state = self.zkhandler.read(('domain.state', self.domuuid))
         self.dom = self.lookupByUUID(self.domuuid)
         if self.dom:
@@ -707,10 +584,7 @@ class VMInstance(object):
                 else:
                     # The send failed or was aborted
                     self.logger.out('Migrate aborted or failed; VM in state {}'.format(self.state), state='w', prefix='Domain {}'.format(self.domuuid))
-
-        self.logger.out('Releasing write lock for synchronization phase D', state='i', prefix='Domain {}'.format(self.domuuid))
         lock.release()
-        self.logger.out('Released write lock for synchronization phase D', state='o', prefix='Domain {}'.format(self.domuuid))
 
         self.zkhandler.write([
             (('domain.migrate.sync_lock', self.domuuid), '')
@@ -726,6 +600,7 @@ class VMInstance(object):
         self.state = self.zkhandler.read(('domain.state', self.domuuid))
         self.node = self.zkhandler.read(('domain.node', self.domuuid))
         self.lastnode = self.zkhandler.read(('domain.last_node', self.domuuid))
+        self.migration_method = self.zkhandler.read(('domain.meta.migrate_method', self.domuuid))
 
         # Check the current state of the VM
         try:
@@ -829,6 +704,9 @@ class VMInstance(object):
                     else:
                         self.terminate_vm()
 
+        self.state_thread = None
+        return
+
     # This function is a wrapper for libvirt.lookupByUUID which fixes some problems
     # 1. Takes a text UUID and handles converting it to bytes
     # 2. Try's it and returns a sensible value if not
@@ -866,3 +744,100 @@ class VMInstance(object):
 
         # Return the dom object (or None)
         return dom
+
+    # Flush the locks of a VM based on UUID
+    @staticmethod
+    def flush_locks(zkhandler, logger, dom_uuid, this_node=None):
+        logger.out('Flushing RBD locks for VM "{}"'.format(dom_uuid), state='i')
+        # Get the list of RBD images
+        rbd_list = zkhandler.read(('domain.storage.volumes', dom_uuid)).split(',')
+
+        for rbd in rbd_list:
+            # Check if a lock exists
+            lock_list_retcode, lock_list_stdout, lock_list_stderr = common.run_os_command('rbd lock list --format json {}'.format(rbd))
+            if lock_list_retcode != 0:
+                logger.out('Failed to obtain lock list for volume "{}"'.format(rbd), state='e')
+                continue
+
+            try:
+                lock_list = json.loads(lock_list_stdout)
+            except Exception as e:
+                logger.out('Failed to parse lock list for volume "{}": {}'.format(rbd, e), state='e')
+                continue
+
+            # If there's at least one lock
+            if lock_list:
+                # Loop through the locks
+                for lock in lock_list:
+                    if this_node is not None and zkhandler.read(('domain.state', dom_uuid)) != 'stop' and lock['address'].split(':')[0] != this_node.storage_ipaddr:
+                        logger.out('RBD lock does not belong to this host (lock owner: {}): freeing this lock would be unsafe, aborting'.format(lock['address'].split(':')[0], state='e'))
+                        zkhandler.write([
+                            (('domain.state', dom_uuid), 'fail'),
+                            (('domain.failed_reason', dom_uuid), 'Could not safely free RBD lock {} ({}) on volume {}; stop VM and flush locks manually'.format(lock['id'], lock['address'], rbd)),
+                        ])
+                        break
+                    # Free the lock
+                    lock_remove_retcode, lock_remove_stdout, lock_remove_stderr = common.run_os_command('rbd lock remove {} "{}" "{}"'.format(rbd, lock['id'], lock['locker']))
+                    if lock_remove_retcode != 0:
+                        logger.out('Failed to free RBD lock "{}" on volume "{}": {}'.format(lock['id'], rbd, lock_remove_stderr), state='e')
+                        zkhandler.write([
+                            (('domain.state', dom_uuid), 'fail'),
+                            (('domain.failed_reason', dom_uuid), 'Could not free RBD lock {} ({}) on volume {}: {}'.format(lock['id'], lock['address'], rbd, lock_remove_stderr)),
+                        ])
+                        break
+                    logger.out('Freed RBD lock "{}" on volume "{}"'.format(lock['id'], rbd), state='o')
+
+        return True
+
+
+# Primary command function
+def vm_command(zkhandler, logger, this_node, data):
+    # Get the command and args
+    command, dom_uuid, *args = data.split()
+
+    if match('success-.*', command) or match('failure-.*', command):
+        return
+
+    logger.out('Getting command "{}" for domain "{}"'.format(command, dom_uuid), state='i')
+
+    # Verify that the VM is set to run on this node
+    domain = this_node.d_domain.get(dom_uuid, None)
+    if domain is None:
+        return False
+
+    if domain.getnode() != this_node.name:
+        return
+
+    # Lock the command queue
+    zk_lock = zkhandler.writelock('base.cmd.domain')
+    with zk_lock:
+        # Flushing VM RBD locks
+        if command == 'flush_locks':
+            result = VMInstance.flush_locks(zkhandler, logger, dom_uuid, this_node)
+        # Attaching a device
+        elif command == 'attach_device':
+            xml_spec = ' '.join(args)
+            result = domain.attach_device(xml_spec)
+        # Detaching a device
+        elif command == 'detach_device':
+            xml_spec = ' '.join(args)
+            result = domain.detach_device(xml_spec)
+        # Command not defined
+        else:
+            result = False
+
+        # Command succeeded
+        if result:
+            # Update the command queue
+            zkhandler.write([
+                ('base.cmd.domain', 'success-{}'.format(data))
+            ])
+        # Command failed
+        else:
+            # Update the command queue
+            zkhandler.write([
+                ('base.cmd.domain', 'failure-{}'.format(data))
+            ])
+
+        # Wait 1 seconds before we free the lock, to ensure the client hits the lock
+        time.sleep(1)

node-daemon/pvcnoded/objects/VXNetworkInstance.py

@@ -36,30 +36,40 @@ class VXNetworkInstance(object):
         self.logger = logger
         self.this_node = this_node
         self.dns_aggregator = dns_aggregator
-        self.vni_dev = config['vni_dev']
-        self.vni_mtu = config['vni_mtu']
+        self.cluster_dev = config['cluster_dev']
+        self.cluster_mtu = config['cluster_mtu']
         self.bridge_dev = config['bridge_dev']
+        self.bridge_mtu = config['bridge_mtu']
 
         self.nettype = self.zkhandler.read(('network.type', self.vni))
         if self.nettype == 'bridged':
+            self.base_nic = 'vlan{}'.format(self.vni)
+            self.bridge_nic = 'vmbr{}'.format(self.vni)
+            self.max_mtu = self.bridge_mtu
             self.logger.out(
                 'Creating new bridged network',
                 prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
             )
             self.init_bridged()
         elif self.nettype == 'managed':
+            self.base_nic = 'vxlan{}'.format(self.vni)
+            self.bridge_nic = 'vmbr{}'.format(self.vni)
+            self.max_mtu = self.cluster_mtu - 50
             self.logger.out(
                 'Creating new managed network',
                 prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
             )
             self.init_managed()
         else:
+            self.base_nic = None
+            self.bridge_nic = None
+            self.max_mtu = 0
             self.logger.out(
                 'Invalid network type {}'.format(self.nettype),
                 prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
             )
             pass
 
@@ -68,8 +78,10 @@ class VXNetworkInstance(object):
         self.old_description = None
         self.description = None
 
-        self.vlan_nic = 'vlan{}'.format(self.vni)
-        self.bridge_nic = 'vmbr{}'.format(self.vni)
+        try:
+            self.vx_mtu = self.zkhandler.read(('network.mtu', self.vni))
+        except Exception:
+            self.vx_mtu = None
 
         # Zookeper handlers for changed states
         @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('network', self.vni))
@@ -83,6 +95,23 @@ class VXNetworkInstance(object):
                 self.old_description = self.description
                 self.description = data.decode('ascii')
 
+        # Try block for migration purposes
+        try:
+            @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('network.mtu', self.vni))
+            def watch_network_mtu(data, stat, event=''):
+                if event and event.type == 'DELETED':
+                    # The key has been deleted after existing before; terminate this watcher
+                    # because this class instance is about to be reaped in Daemon.py
+                    return False
+
+                if data and str(self.vx_mtu) != data.decode('ascii'):
+                    self.vx_mtu = data.decode('ascii')
+                    self.validateNetworkMTU()
+                    self.updateNetworkMTU()
+        except Exception:
+            self.validateNetworkMTU()
+            self.updateNetworkMTU()
+
         self.createNetworkBridged()
 
     # Initialize a managed network
@@ -102,8 +131,10 @@ class VXNetworkInstance(object):
         self.dhcp4_start = self.zkhandler.read(('network.ip4.dhcp_start', self.vni))
         self.dhcp4_end = self.zkhandler.read(('network.ip4.dhcp_end', self.vni))
 
-        self.vxlan_nic = 'vxlan{}'.format(self.vni)
-        self.bridge_nic = 'vmbr{}'.format(self.vni)
+        try:
+            self.vx_mtu = self.zkhandler.read(('network.mtu', self.vni))
+        except Exception:
+            self.vx_mtu = None
 
         self.nftables_netconf_filename = '{}/networks/{}.nft'.format(self.config['nft_dynamic_directory'], self.vni)
         self.firewall_rules = []
@@ -138,7 +169,7 @@ add rule inet filter input tcp dport 80 meta iifname {bridgenic} counter accept
 # Block traffic into the router from network
 add rule inet filter input meta iifname {bridgenic} counter drop
 """.format(
-            vxlannic=self.vxlan_nic,
+            vxlannic=self.base_nic,
             bridgenic=self.bridge_nic
         )
 
@@ -147,14 +178,14 @@ add rule inet filter forward ip daddr {netaddr4} counter jump {vxlannic}-in
 add rule inet filter forward ip saddr {netaddr4} counter jump {vxlannic}-out
 """.format(
             netaddr4=self.ip4_network,
-            vxlannic=self.vxlan_nic,
+            vxlannic=self.base_nic,
         )
         self.firewall_rules_v6 = """# Jump from forward chain to this chain when matching net (IPv4)
 add rule inet filter forward ip6 daddr {netaddr6} counter jump {vxlannic}-in
 add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
 """.format(
             netaddr6=self.ip6_network,
-            vxlannic=self.vxlan_nic,
+            vxlannic=self.base_nic,
         )
 
         self.firewall_rules_in = self.zkhandler.children(('network.rule.in', self.vni))
@@ -209,6 +240,23 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                     self.stopDHCPServer()
                     self.startDHCPServer()
 
+        # Try block for migration purposes
+        try:
+            @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('network.mtu', self.vni))
+            def watch_network_mtu(data, stat, event=''):
+                if event and event.type == 'DELETED':
+                    # The key has been deleted after existing before; terminate this watcher
+                    # because this class instance is about to be reaped in Daemon.py
+                    return False
+
+                if data and str(self.vx_mtu) != data.decode('ascii'):
+                    self.vx_mtu = data.decode('ascii')
+                    self.validateNetworkMTU()
+                    self.updateNetworkMTU()
+        except Exception:
+            self.validateNetworkMTU()
+            self.updateNetworkMTU()
+
         @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('network.ip6.network', self.vni))
         def watch_network_ip6_network(data, stat, event=''):
             if event and event.type == 'DELETED':
@@ -383,6 +431,66 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
     def getvni(self):
         return self.vni
 
+    def validateNetworkMTU(self):
+        update_mtu = False
+
+        # Explicitly set the MTU to max_mtu if unset (in Zookeeper too assuming the key exists)
+        if self.vx_mtu == '' or self.vx_mtu is None:
+            self.logger.out(
+                'MTU not specified; setting to maximum MTU {} instead'.format(self.max_mtu),
+                prefix='VNI {}'.format(self.vni),
+                state='w'
+            )
+            self.vx_mtu = self.max_mtu
+            update_mtu = True
+
+        # Set MTU to an integer (if it's not)
+        if not isinstance(self.vx_mtu, int):
+            self.vx_mtu = int(self.vx_mtu)
+
+        # Ensure the MTU is valid
+        if self.vx_mtu > self.max_mtu:
+            self.logger.out(
+                'MTU {} is larger than maximum MTU {}; setting to maximum MTU instead'.format(self.vx_mtu, self.max_mtu),
+                prefix='VNI {}'.format(self.vni),
+                state='w'
+            )
+            self.vx_mtu = self.max_mtu
+            update_mtu = True
+
+        if update_mtu:
+            # Try block for migration purposes
+            try:
+                self.zkhandler.write([
+                    (('network.mtu', self.vni), self.vx_mtu)
+                ])
+            except Exception as e:
+                self.logger.out(
+                    'Could not update MTU in Zookeeper: {}'.format(e),
+                    prefix='VNI {}'.format(self.vni),
+                    state='w'
+                )
+
+    def updateNetworkMTU(self):
+        self.logger.out(
+            'Setting network MTU to {}'.format(self.vx_mtu),
+            prefix='VNI {}'.format(self.vni),
+            state='i'
+        )
+        # Set MTU of base and bridge NICs
+        common.run_os_command(
+            'ip link set {} mtu {} up'.format(
+                self.base_nic,
+                self.vx_mtu
+            )
+        )
+        common.run_os_command(
+            'ip link set {} mtu {} up'.format(
+                self.bridge_nic,
+                self.vx_mtu
+            )
+        )
+
     def updateDHCPReservations(self, old_reservations_list, new_reservations_list):
         for reservation in new_reservations_list:
             if reservation not in old_reservations_list:
@@ -411,7 +519,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
         self.logger.out(
             'Updating firewall rules',
             prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
         )
         ordered_acls_in = {}
         ordered_acls_out = {}
@@ -458,18 +566,18 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
     def createNetworkBridged(self):
         self.logger.out(
             'Creating bridged vLAN device {} on interface {}'.format(
-                self.vlan_nic,
+                self.base_nic,
                 self.bridge_dev
             ),
             prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
         )
 
         # Create vLAN interface
         common.run_os_command(
             'ip link add link {} name {} type vlan id {}'.format(
                 self.bridge_dev,
-                self.vlan_nic,
+                self.base_nic,
                 self.vni
             )
         )
@@ -480,20 +588,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
             )
         )
 
-        # Set MTU of vLAN and bridge NICs
-        vx_mtu = self.vni_mtu
-        common.run_os_command(
-            'ip link set {} mtu {} up'.format(
-                self.vlan_nic,
-                vx_mtu
-            )
-        )
-        common.run_os_command(
-            'ip link set {} mtu {} up'.format(
-                self.bridge_nic,
-                vx_mtu
-            )
-        )
+        self.updateNetworkMTU()
 
         # Disable tx checksum offload on bridge interface (breaks DHCP on Debian < 9)
         common.run_os_command(
@@ -513,7 +608,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
         common.run_os_command(
             'brctl addif {} {}'.format(
                 self.bridge_nic,
-                self.vlan_nic
+                self.base_nic
             )
         )
 
@@ -521,18 +616,18 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
     def createNetworkManaged(self):
         self.logger.out(
             'Creating VXLAN device on interface {}'.format(
-                self.vni_dev
+                self.cluster_dev
             ),
             prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
         )
 
         # Create VXLAN interface
         common.run_os_command(
             'ip link add {} type vxlan id {} dstport 4789 dev {}'.format(
-                self.vxlan_nic,
+                self.base_nic,
                 self.vni,
-                self.vni_dev
+                self.cluster_dev
             )
         )
         # Create bridge interface
@@ -542,20 +637,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
             )
         )
 
-        # Set MTU of VXLAN and bridge NICs
-        vx_mtu = self.vni_mtu - 50
-        common.run_os_command(
-            'ip link set {} mtu {} up'.format(
-                self.vxlan_nic,
-                vx_mtu
-            )
-        )
-        common.run_os_command(
-            'ip link set {} mtu {} up'.format(
-                self.bridge_nic,
-                vx_mtu
-            )
-        )
+        self.updateNetworkMTU()
 
         # Disable tx checksum offload on bridge interface (breaks DHCP on Debian < 9)
         common.run_os_command(
@@ -575,7 +657,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
         common.run_os_command(
             'brctl addif {} {}'.format(
                 self.bridge_nic,
-                self.vxlan_nic
+                self.base_nic
             )
         )
 
@@ -600,7 +682,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                     self.bridge_nic
                 ),
                 prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
             )
             common.createIPAddress(self.ip6_gateway, self.ip6_cidrnetmask, self.bridge_nic)
 
@@ -613,7 +695,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                     self.bridge_nic
                 ),
                 prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
             )
             common.createIPAddress(self.ip4_gateway, self.ip4_cidrnetmask, self.bridge_nic)
 
@@ -624,7 +706,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                     self.bridge_nic
                 ),
                 prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
             )
 
             # Recreate the environment we need for dnsmasq
@@ -716,10 +798,10 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
     def removeNetworkBridged(self):
         self.logger.out(
             'Removing VNI device on interface {}'.format(
-                self.vni_dev
+                self.cluster_dev
             ),
             prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
         )
         common.run_os_command(
             'ip link set {} down'.format(
@@ -728,13 +810,13 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
         )
         common.run_os_command(
             'ip link set {} down'.format(
-                self.vlan_nic
+                self.base_nic
             )
         )
         common.run_os_command(
             'brctl delif {} {}'.format(
                 self.bridge_nic,
-                self.vlan_nic
+                self.base_nic
             )
         )
         common.run_os_command(
@@ -744,7 +826,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
         )
         common.run_os_command(
             'ip link delete {}'.format(
-                self.vlan_nic
+                self.base_nic
             )
         )
 
@@ -752,10 +834,10 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
     def removeNetworkManaged(self):
         self.logger.out(
             'Removing VNI device on interface {}'.format(
-                self.vni_dev
+                self.cluster_dev
             ),
             prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
         )
         common.run_os_command(
             'ip link set {} down'.format(
@@ -764,13 +846,13 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
         )
         common.run_os_command(
             'ip link set {} down'.format(
-                self.vxlan_nic
+                self.base_nic
             )
         )
         common.run_os_command(
             'brctl delif {} {}'.format(
                 self.bridge_nic,
-                self.vxlan_nic
+                self.base_nic
             )
         )
         common.run_os_command(
@@ -780,7 +862,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
         )
         common.run_os_command(
             'ip link delete {}'.format(
-                self.vxlan_nic
+                self.base_nic
             )
         )
 
@@ -788,7 +870,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
         self.logger.out(
             'Removing firewall rules',
             prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
         )
 
         try:
@@ -815,7 +897,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                 self.bridge_nic
             ),
             prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
         )
         common.removeIPAddress(self.ip6_gateway, self.ip6_cidrnetmask, self.bridge_nic)
 
@@ -827,7 +909,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                 self.bridge_nic
             ),
             prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
         )
         common.removeIPAddress(self.ip4_gateway, self.ip4_cidrnetmask, self.bridge_nic)
 
@@ -838,7 +920,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                     self.bridge_nic
                 ),
                 prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
             )
             # Terminate, then kill
             self.dhcp_server_daemon.signal('term')

node-daemon/pvcnoded/util/config.py

@@ -0,0 +1,385 @@
+#!/usr/bin/env python3
+
+# config.py - Utility functions for pvcnoded configuration parsing
+# Part of the Parallel Virtual Cluster (PVC) system
+#
+#    Copyright (C) 2018-2021 Joshua M. Boniface <joshua@boniface.me>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, version 3.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+#
+###############################################################################
+
+import os
+import subprocess
+import yaml
+from socket import gethostname
+from re import findall
+from psutil import cpu_count
+from ipaddress import ip_address, ip_network
+
+
+class MalformedConfigurationError(Exception):
+    """
+    An except when parsing the PVC Node daemon configuration file
+    """
+    def __init__(self, error=None):
+        self.msg = f'ERROR: Configuration file is malformed: {error}'
+
+    def __str__(self):
+        return str(self.msg)
+
+
+def get_static_data():
+    """
+    Data that is obtained once at node startup for use later
+    """
+    staticdata = list()
+    staticdata.append(str(cpu_count()))  # CPU count
+    staticdata.append(
+        subprocess.run(
+            ['uname', '-r'], stdout=subprocess.PIPE
+        ).stdout.decode('ascii').strip()
+    )
+    staticdata.append(
+        subprocess.run(
+            ['uname', '-o'], stdout=subprocess.PIPE
+        ).stdout.decode('ascii').strip()
+    )
+    staticdata.append(
+        subprocess.run(
+            ['uname', '-m'], stdout=subprocess.PIPE
+        ).stdout.decode('ascii').strip()
+    )
+
+    return staticdata
+
+
+def get_configuration_path():
+    try:
+        return os.environ['PVCD_CONFIG_FILE']
+    except KeyError:
+        print('ERROR: The "PVCD_CONFIG_FILE" environment variable must be set.')
+        os._exit(1)
+
+
+def get_hostname():
+    node_fqdn = gethostname()
+    node_hostname = node_fqdn.split('.', 1)[0]
+    node_domain = ''.join(node_fqdn.split('.', 1)[1:])
+    try:
+        node_id = findall(r'\d+', node_hostname)[-1]
+    except IndexError:
+        node_id = 0
+
+    return node_fqdn, node_hostname, node_domain, node_id
+
+
+def validate_floating_ip(config, network):
+    if network not in ['cluster', 'storage', 'upstream']:
+        return False, f'Specified network type "{network}" is not valid'
+
+    floating_key = f'{network}_floating_ip'
+    network_key = f'{network}_network'
+
+    # Verify the network provided is valid
+    try:
+        network = ip_network(config[network_key])
+    except Exception:
+        return False, f'Network address {config[network_key]} for {network_key} is not valid'
+
+    # Verify that the floating IP is valid (and in the network)
+    try:
+        floating_address = ip_address(config[floating_key].split('/')[0])
+        if floating_address not in list(network.hosts()):
+            raise
+    except Exception:
+        return False, f'Floating address {config[floating_key]} for {floating_key} is not valid'
+
+    return True, ''
+
+
+def get_configuration():
+    """
+    Parse the configuration of the node daemon.
+    """
+    pvcnoded_config_file = get_configuration_path()
+
+    print('Loading configuration from file "{}"'.format(pvcnoded_config_file))
+
+    with open(pvcnoded_config_file, 'r') as cfgfile:
+        try:
+            o_config = yaml.load(cfgfile, Loader=yaml.SafeLoader)
+        except Exception as e:
+            print('ERROR: Failed to parse configuration file: {}'.format(e))
+            os._exit(1)
+
+    node_fqdn, node_hostname, node_domain, node_id = get_hostname()
+
+    # Create the configuration dictionary
+    config = dict()
+
+    # Get the initial base configuration
+    try:
+        o_base = o_config['pvc']
+        o_cluster = o_config['pvc']['cluster']
+    except Exception as e:
+        raise MalformedConfigurationError(e)
+
+    config_general = {
+        'node':          o_base.get('node', node_hostname),
+        'node_hostname': node_hostname,
+        'node_fqdn':     node_fqdn,
+        'node_domain':   node_domain,
+        'node_id':       node_id,
+        'coordinators':  o_cluster.get('coordinators', list()),
+        'debug':         o_base.get('debug', False),
+    }
+
+    config = {**config, **config_general}
+
+    # Get the functions configuration
+    try:
+        o_functions = o_config['pvc']['functions']
+    except Exception as e:
+        raise MalformedConfigurationError(e)
+
+    config_functions = {
+        'enable_hypervisor': o_functions.get('enable_hypervisor', False),
+        'enable_networking': o_functions.get('enable_networking', False),
+        'enable_storage':    o_functions.get('enable_storage', False),
+        'enable_api':        o_functions.get('enable_api', False),
+    }
+
+    config = {**config, **config_functions}
+
+    # Get the directory configuration
+    try:
+        o_directories = o_config['pvc']['system']['configuration']['directories']
+    except Exception as e:
+        raise MalformedConfigurationError(e)
+
+    config_directories = {
+        'dynamic_directory':     o_directories.get('dynamic_directory', None),
+        'log_directory':         o_directories.get('log_directory', None),
+        'console_log_directory': o_directories.get('console_log_directory', None),
+    }
+
+    # Define our dynamic directory schema
+    config_directories['dnsmasq_dynamic_directory'] = config_directories['dynamic_directory'] + '/dnsmasq'
+    config_directories['pdns_dynamic_directory'] = config_directories['dynamic_directory'] + '/pdns'
+    config_directories['nft_dynamic_directory'] = config_directories['dynamic_directory'] + '/nft'
+
+    # Define our log directory schema
+    config_directories['dnsmasq_log_directory'] = config_directories['log_directory'] + '/dnsmasq'
+    config_directories['pdns_log_directory'] = config_directories['log_directory'] + '/pdns'
+    config_directories['nft_log_directory'] = config_directories['log_directory'] + '/nft'
+
+    config = {**config, **config_directories}
+
+    # Get the logging configuration
+    try:
+        o_logging = o_config['pvc']['system']['configuration']['logging']
+    except Exception as e:
+        raise MalformedConfigurationError(e)
+
+    config_logging = {
+        'file_logging':                  o_logging.get('file_logging', False),
+        'stdout_logging':                o_logging.get('stdout_logging', False),
+        'zookeeper_logging':             o_logging.get('zookeeper_logging', False),
+        'log_colours':                   o_logging.get('log_colours', False),
+        'log_dates':                     o_logging.get('log_dates', False),
+        'log_keepalives':                o_logging.get('log_keepalives', False),
+        'log_keepalive_cluster_details': o_logging.get('log_keepalive_cluster_details', False),
+        'log_keepalive_storage_details': o_logging.get('log_keepalive_storage_details', False),
+        'console_log_lines':             o_logging.get('console_log_lines', False),
+        'node_log_lines':                o_logging.get('node_log_lines', False),
+    }
+
+    config = {**config, **config_logging}
+
+    # Get the interval configuration
+    try:
+        o_intervals = o_config['pvc']['system']['intervals']
+    except Exception as e:
+        raise MalformedConfigurationError(e)
+
+    config_intervals = {
+        'vm_shutdown_timeout':  int(o_intervals.get('vm_shutdown_timeout', 60)),
+        'keepalive_interval':   int(o_intervals.get('keepalive_interval', 5)),
+        'fence_intervals':      int(o_intervals.get('fence_intervals', 6)),
+        'suicide_intervals':    int(o_intervals.get('suicide_interval', 0)),
+    }
+
+    config = {**config, **config_intervals}
+
+    # Get the fencing configuration
+    try:
+        o_fencing = o_config['pvc']['system']['fencing']
+        o_fencing_actions = o_fencing['actions']
+        o_fencing_ipmi = o_fencing['ipmi']
+    except Exception as e:
+        raise MalformedConfigurationError(e)
+
+    config_fencing = {
+        'successful_fence': o_fencing_actions.get('successful_fence', None),
+        'failed_fence':     o_fencing_actions.get('failed_fence', None),
+        'ipmi_hostname':    o_fencing_ipmi.get('host', f'{node_hostname}-lom.{node_domain}'),
+        'ipmi_username':    o_fencing_ipmi.get('user', 'null'),
+        'ipmi_password':    o_fencing_ipmi.get('pass', 'null'),
+    }
+
+    config = {**config, **config_fencing}
+
+    # Get the migration configuration
+    try:
+        o_migration = o_config['pvc']['system']['migration']
+    except Exception as e:
+        raise MalformedConfigurationError(e)
+
+    config_migration = {
+        'migration_target_selector': o_migration.get('target_selector', 'mem'),
+    }
+
+    config = {**config, **config_migration}
+
+    if config['enable_networking']:
+        # Get the node networks configuration
+        try:
+            o_networks = o_config['pvc']['cluster']['networks']
+            o_network_cluster = o_networks['cluster']
+            o_network_storage = o_networks['storage']
+            o_network_upstream = o_networks['upstream']
+            o_sysnetworks = o_config['pvc']['system']['configuration']['networking']
+            o_sysnetwork_cluster = o_sysnetworks['cluster']
+            o_sysnetwork_storage = o_sysnetworks['storage']
+            o_sysnetwork_upstream = o_sysnetworks['upstream']
+        except Exception as e:
+            raise MalformedConfigurationError(e)
+
+        config_networks = {
+            'cluster_domain':       o_network_cluster.get('domain', None),
+            'cluster_network':      o_network_cluster.get('network', None),
+            'cluster_floating_ip':  o_network_cluster.get('floating_ip', None),
+            'cluster_dev':          o_sysnetwork_cluster.get('device', None),
+            'cluster_mtu':          o_sysnetwork_cluster.get('mtu', None),
+            'cluster_dev_ip':       o_sysnetwork_cluster.get('address', None),
+            'storage_domain':       o_network_storage.get('domain', None),
+            'storage_network':      o_network_storage.get('network', None),
+            'storage_floating_ip':  o_network_storage.get('floating_ip', None),
+            'storage_dev':          o_sysnetwork_storage.get('device', None),
+            'storage_mtu':          o_sysnetwork_storage.get('mtu', None),
+            'storage_dev_ip':       o_sysnetwork_storage.get('address', None),
+            'upstream_domain':      o_network_upstream.get('domain', None),
+            'upstream_network':     o_network_upstream.get('network', None),
+            'upstream_floating_ip': o_network_upstream.get('floating_ip', None),
+            'upstream_gateway':     o_network_upstream.get('gateway', None),
+            'upstream_dev':         o_sysnetwork_upstream.get('device', None),
+            'upstream_mtu':         o_sysnetwork_upstream.get('mtu', None),
+            'upstream_dev_ip':      o_sysnetwork_upstream.get('address', None),
+            'bridge_dev':           o_sysnetworks.get('bridge_device', None),
+            'bridge_mtu':           o_sysnetworks.get('bridge_mtu', 1500),
+            'enable_sriov':         o_sysnetworks.get('sriov_enable', False),
+            'sriov_device':         o_sysnetworks.get('sriov_device', list())
+        }
+
+        config = {**config, **config_networks}
+
+        for network_type in ['cluster', 'storage', 'upstream']:
+            result, msg = validate_floating_ip(config, network_type)
+            if not result:
+                raise MalformedConfigurationError(msg)
+
+            address_key = '{}_dev_ip'.format(network_type)
+            network_key = f'{network_type}_network'
+            network = ip_network(config[network_key])
+            # With autoselection of addresses, construct an IP from the relevant network
+            if config[address_key] == 'by-id':
+                # The NodeID starts at 1, but indexes start at 0
+                address_id = int(config['node_id']) - 1
+                # Grab the nth address from the network
+                config[address_key] = '{}/{}'.format(list(network.hosts())[address_id], network.prefixlen)
+            # Validate the provided IP instead
+            else:
+                try:
+                    address = ip_address(config[address_key].split('/')[0])
+                    if address not in list(network.hosts()):
+                        raise
+                except Exception:
+                    raise MalformedConfigurationError(
+                        f'IP address {config[address_key]} for {address_key} is not valid'
+                    )
+
+        # Get the PowerDNS aggregator database configuration
+        try:
+            o_pdnsdb = o_config['pvc']['coordinator']['dns']['database']
+        except Exception as e:
+            raise MalformedConfigurationError(e)
+
+        config_pdnsdb = {
+            'pdns_postgresql_host':     o_pdnsdb.get('host', None),
+            'pdns_postgresql_port':     o_pdnsdb.get('port', None),
+            'pdns_postgresql_dbname':   o_pdnsdb.get('name', None),
+            'pdns_postgresql_user':     o_pdnsdb.get('user', None),
+            'pdns_postgresql_password': o_pdnsdb.get('pass', None),
+        }
+
+        config = {**config, **config_pdnsdb}
+
+        # Get the Cloud-Init Metadata database configuration
+        try:
+            o_metadatadb = o_config['pvc']['coordinator']['metadata']['database']
+        except Exception as e:
+            raise MalformedConfigurationError(e)
+
+        config_metadatadb = {
+            'metadata_postgresql_host':     o_metadatadb.get('host', None),
+            'metadata_postgresql_port':     o_metadatadb.get('port', None),
+            'metadata_postgresql_dbname':   o_metadatadb.get('name', None),
+            'metadata_postgresql_user':     o_metadatadb.get('user', None),
+            'metadata_postgresql_password': o_metadatadb.get('pass', None),
+        }
+
+        config = {**config, **config_metadatadb}
+
+    if config['enable_storage']:
+        # Get the storage configuration
+        try:
+            o_storage = o_config['pvc']['system']['configuration']['storage']
+        except Exception as e:
+            raise MalformedConfigurationError(e)
+
+        config_storage = {
+            'ceph_config_file':   o_storage.get('ceph_config_file', None),
+            'ceph_admin_keyring': o_storage.get('ceph_admin_keyring', None),
+        }
+
+        config = {**config, **config_storage}
+
+        # Add our node static data to the config
+        config['static_data'] = get_static_data()
+
+    return config
+
+
+def validate_directories(config):
+    if not os.path.exists(config['dynamic_directory']):
+        os.makedirs(config['dynamic_directory'])
+        os.makedirs(config['dnsmasq_dynamic_directory'])
+        os.makedirs(config['pdns_dynamic_directory'])
+        os.makedirs(config['nft_dynamic_directory'])
+
+    if not os.path.exists(config['log_directory']):
+        os.makedirs(config['log_directory'])
+        os.makedirs(config['dnsmasq_log_directory'])
+        os.makedirs(config['pdns_log_directory'])
+        os.makedirs(config['nft_log_directory'])

node-daemon/pvcnoded/util/fencing.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 
-# fencing.py - PVC daemon function library, node fencing functions
+# fencing.py - Utility functions for pvcnoded fencing
 # Part of the Parallel Virtual Cluster (PVC) system
 #
 #    Copyright (C) 2018-2021 Joshua M. Boniface <joshua@boniface.me>
@@ -22,13 +22,14 @@
 import time
 
 import daemon_lib.common as common
-import pvcnoded.VMInstance as VMInstance
+
+from pvcnoded.objects.VMInstance import VMInstance
 
 
 #
 # Fence thread entry function
 #
-def fenceNode(node_name, zkhandler, config, logger):
+def fence_node(node_name, zkhandler, config, logger):
     # We allow exactly 6 saving throws (30 seconds) for the host to come back online or we kill it
     failcount_limit = 6
     failcount = 0
@@ -40,13 +41,13 @@ def fenceNode(node_name, zkhandler, config, logger):
         # Is it still 'dead'
         if node_daemon_state == 'dead':
             failcount += 1
-            logger.out('Node "{}" failed {}/{} saving throws'.format(node_name, failcount, failcount_limit), state='w')
+            logger.out('Node "{}" failed {}/{} saving throws'.format(node_name, failcount, failcount_limit), state='s')
         # It changed back to something else so it must be alive
         else:
             logger.out('Node "{}" passed a saving throw; canceling fence'.format(node_name), state='o')
             return
 
-    logger.out('Fencing node "{}" via IPMI reboot signal'.format(node_name), state='w')
+    logger.out('Fencing node "{}" via IPMI reboot signal'.format(node_name), state='s')
 
     # Get IPMI information
     ipmi_hostname = zkhandler.read(('node.ipmi.hostname', node_name))
@@ -54,9 +55,16 @@ def fenceNode(node_name, zkhandler, config, logger):
     ipmi_password = zkhandler.read(('node.ipmi.password', node_name))
 
     # Shoot it in the head
-    fence_status = rebootViaIPMI(ipmi_hostname, ipmi_username, ipmi_password, logger)
+    fence_status = reboot_via_ipmi(ipmi_hostname, ipmi_username, ipmi_password, logger)
+
     # Hold to ensure the fence takes effect and system stabilizes
-    time.sleep(config['keepalive_interval'] * 2)
+    logger.out('Waiting {}s for fence of node "{}" to take effect'.format(config['keepalive_interval'], node_name), state='i')
+    time.sleep(config['keepalive_interval'])
+    if fence_status:
+        logger.out('Marking node "{}" as fenced'.format(node_name), state='i')
+        zkhandler.write([
+            (('node.state.daemon', node_name), 'fenced')
+        ])
 
     # Force into secondary network state if needed
     if node_name in config['coordinators']:
@@ -118,12 +126,13 @@ def migrateFromFencedNode(zkhandler, node_name, config, logger):
     zkhandler.write([
         (('node.state.domain', node_name), 'flushed')
     ])
+    logger.out('All VMs flushed from dead node "{}" to new hosts'.format(node_name), state='i')
 
 
 #
 # Perform an IPMI fence
 #
-def rebootViaIPMI(ipmi_hostname, ipmi_user, ipmi_password, logger):
+def reboot_via_ipmi(ipmi_hostname, ipmi_user, ipmi_password, logger):
     # Forcibly reboot the node
     ipmi_command_reset = '/usr/bin/ipmitool -I lanplus -H {} -U {} -P {} chassis power reset'.format(
         ipmi_hostname, ipmi_user, ipmi_password
@@ -131,44 +140,56 @@ def rebootViaIPMI(ipmi_hostname, ipmi_user, ipmi_password, logger):
     ipmi_reset_retcode, ipmi_reset_stdout, ipmi_reset_stderr = common.run_os_command(ipmi_command_reset)
 
     if ipmi_reset_retcode != 0:
-        logger.out('Failed to reboot dead node', state='e')
-        print(ipmi_reset_stderr)
-        return False
+        logger.out(f'Failed to reboot dead node: {ipmi_reset_stderr}', state='e')
+
+    time.sleep(1)
+
+    # Power on the node (just in case it is offline)
+    ipmi_command_start = '/usr/bin/ipmitool -I lanplus -H {} -U {} -P {} chassis power on'.format(
+        ipmi_hostname, ipmi_user, ipmi_password
+    )
+    ipmi_start_retcode, ipmi_start_stdout, ipmi_start_stderr = common.run_os_command(ipmi_command_start)
 
     time.sleep(2)
 
-    # Ensure the node is powered on
+    # Check the chassis power state
+    logger.out('Checking power state of dead node', state='i')
     ipmi_command_status = '/usr/bin/ipmitool -I lanplus -H {} -U {} -P {} chassis power status'.format(
         ipmi_hostname, ipmi_user, ipmi_password
     )
     ipmi_status_retcode, ipmi_status_stdout, ipmi_status_stderr = common.run_os_command(ipmi_command_status)
 
-    # Trigger a power start if needed
-    if ipmi_status_stdout != "Chassis Power is on":
-        ipmi_command_start = '/usr/bin/ipmitool -I lanplus -H {} -U {} -P {} chassis power on'.format(
-            ipmi_hostname, ipmi_user, ipmi_password
-        )
-        ipmi_start_retcode, ipmi_start_stdout, ipmi_start_stderr = common.run_os_command(ipmi_command_start)
-
-        if ipmi_start_retcode != 0:
-            logger.out('Failed to start powered-off dead node', state='e')
-            print(ipmi_reset_stderr)
+    if ipmi_reset_retcode == 0:
+        if ipmi_status_stdout.strip() == "Chassis Power is on":
+            # We successfully rebooted the node and it is powered on; this is a succeessful fence
+            logger.out('Successfully rebooted dead node', state='o')
+            return True
+        elif ipmi_status_stdout.strip() == "Chassis Power is off":
+            # We successfully rebooted the node but it is powered off; this might be expected or not, but the node is confirmed off so we can call it a successful fence
+            logger.out('Chassis power is in confirmed off state after successfuly IPMI reboot; proceeding with fence-flush', state='o')
+            return True
+        else:
+            # We successfully rebooted the node but it is in some unknown power state; since this might indicate a silent failure, we must call it a failed fence
+            logger.out('Chassis power is in an unknown state ({}) after successful IPMI reboot; not performing fence-flush'.format(ipmi_status_stdout.strip()), state='e')
+            return False
+    else:
+        if ipmi_status_stdout.strip() == "Chassis Power is off":
+            # We failed to reboot the node but it is powered off; it has probably suffered a serious hardware failure, but the node is confirmed off so we can call it a successful fence
+            logger.out('Chassis power is in confirmed off state after failed IPMI reboot; proceeding with fence-flush', state='o')
+            return True
+        else:
+            # We failed to reboot the node but it is in some unknown power state (including "on"); since this might indicate a silent failure, we must call it a failed fence
+            logger.out('Chassis power is not in confirmed off state after failed IPMI reboot; not performing fence-flush', state='e')
             return False
-
-    # Declare success
-    logger.out('Successfully rebooted dead node', state='o')
-    return True
 
 
 #
 # Verify that IPMI connectivity to this host exists (used during node init)
 #
-def verifyIPMI(ipmi_hostname, ipmi_user, ipmi_password):
-    ipmi_command_status = '/usr/bin/ipmitool -I lanplus -H {} -U {} -P {} chassis power status'.format(
-        ipmi_hostname, ipmi_user, ipmi_password
-    )
-    ipmi_status_retcode, ipmi_status_stdout, ipmi_status_stderr = common.run_os_command(ipmi_command_status, timeout=2)
-    if ipmi_status_retcode == 0 and ipmi_status_stdout != "Chassis Power is on":
+def verify_ipmi(ipmi_hostname, ipmi_user, ipmi_password):
+    ipmi_command = f'/usr/bin/ipmitool -I lanplus -H {ipmi_hostname} -U {ipmi_user} -P {ipmi_password} chassis power status'
+    retcode, stdout, stderr = common.run_os_command(ipmi_command, timeout=2)
+    if retcode == 0 and stdout.strip() == "Chassis Power is on":
         return True
     else:
         return False

node-daemon/pvcnoded/util/keepalive.py

@@ -0,0 +1,739 @@
+#!/usr/bin/env python3
+
+# keepalive.py - Utility functions for pvcnoded Keepalives
+# Part of the Parallel Virtual Cluster (PVC) system
+#
+#    Copyright (C) 2018-2021 Joshua M. Boniface <joshua@boniface.me>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, version 3.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+#
+###############################################################################
+
+import pvcnoded.util.fencing
+
+import daemon_lib.common as common
+
+from apscheduler.schedulers.background import BackgroundScheduler
+from rados import Rados
+from xml.etree import ElementTree
+from queue import Queue
+from threading import Thread
+from datetime import datetime
+
+import json
+import re
+import libvirt
+import psutil
+import os
+import time
+
+
+# State table for pretty stats
+libvirt_vm_states = {
+    0: "NOSTATE",
+    1: "RUNNING",
+    2: "BLOCKED",
+    3: "PAUSED",
+    4: "SHUTDOWN",
+    5: "SHUTOFF",
+    6: "CRASHED",
+    7: "PMSUSPENDED"
+}
+
+
+def start_keepalive_timer(logger, config, zkhandler, this_node):
+    keepalive_interval = config['keepalive_interval']
+    logger.out(f'Starting keepalive timer ({keepalive_interval} second interval)', state='s')
+    keepalive_timer = BackgroundScheduler()
+    keepalive_timer.add_job(
+        node_keepalive,
+        args=(logger, config, zkhandler, this_node),
+        trigger='interval',
+        seconds=keepalive_interval)
+    keepalive_timer.start()
+    return keepalive_timer
+
+
+def stop_keepalive_timer(logger, keepalive_timer):
+    try:
+        keepalive_timer.shutdown()
+        logger.out('Stopping keepalive timer', state='s')
+    except Exception:
+        logger.out('Failed to stop keepalive timer', state='w')
+
+
+# Ceph stats update function
+def collect_ceph_stats(logger, config, zkhandler, this_node, queue):
+    pool_list = zkhandler.children('base.pool')
+    osd_list = zkhandler.children('base.osd')
+
+    debug = config['debug']
+    if debug:
+        logger.out("Thread starting", state='d', prefix='ceph-thread')
+
+    # Connect to the Ceph cluster
+    try:
+        ceph_conn = Rados(conffile=config['ceph_config_file'], conf=dict(keyring=config['ceph_admin_keyring']))
+        if debug:
+            logger.out("Connecting to cluster", state='d', prefix='ceph-thread')
+        ceph_conn.connect(timeout=1)
+    except Exception as e:
+        logger.out('Failed to open connection to Ceph cluster: {}'.format(e), state='e')
+        return
+
+    if debug:
+        logger.out("Getting health stats from monitor", state='d', prefix='ceph-thread')
+
+    # Get Ceph cluster health for local status output
+    command = {"prefix": "health", "format": "json"}
+    try:
+        health_status = json.loads(ceph_conn.mon_command(json.dumps(command), b'', timeout=1)[1])
+        ceph_health = health_status['status']
+    except Exception as e:
+        logger.out('Failed to obtain Ceph health data: {}'.format(e), state='e')
+        ceph_health = 'HEALTH_UNKN'
+
+    if ceph_health in ['HEALTH_OK']:
+        ceph_health_colour = logger.fmt_green
+    elif ceph_health in ['HEALTH_UNKN']:
+        ceph_health_colour = logger.fmt_cyan
+    elif ceph_health in ['HEALTH_WARN']:
+        ceph_health_colour = logger.fmt_yellow
+    else:
+        ceph_health_colour = logger.fmt_red
+
+    # Primary-only functions
+    if this_node.router_state == 'primary':
+        if debug:
+            logger.out("Set ceph health information in zookeeper (primary only)", state='d', prefix='ceph-thread')
+
+        command = {"prefix": "status", "format": "pretty"}
+        ceph_status = ceph_conn.mon_command(json.dumps(command), b'', timeout=1)[1].decode('ascii')
+        try:
+            zkhandler.write([
+                ('base.storage', str(ceph_status))
+            ])
+        except Exception as e:
+            logger.out('Failed to set Ceph status data: {}'.format(e), state='e')
+
+        if debug:
+            logger.out("Set ceph rados df information in zookeeper (primary only)", state='d', prefix='ceph-thread')
+
+        # Get rados df info
+        command = {"prefix": "df", "format": "pretty"}
+        ceph_df = ceph_conn.mon_command(json.dumps(command), b'', timeout=1)[1].decode('ascii')
+        try:
+            zkhandler.write([
+                ('base.storage.util', str(ceph_df))
+            ])
+        except Exception as e:
+            logger.out('Failed to set Ceph utilization data: {}'.format(e), state='e')
+
+        if debug:
+            logger.out("Set pool information in zookeeper (primary only)", state='d', prefix='ceph-thread')
+
+        # Get pool info
+        command = {"prefix": "df", "format": "json"}
+        ceph_df_output = ceph_conn.mon_command(json.dumps(command), b'', timeout=1)[1].decode('ascii')
+        try:
+            ceph_pool_df_raw = json.loads(ceph_df_output)['pools']
+        except Exception as e:
+            logger.out('Failed to obtain Pool data (ceph df): {}'.format(e), state='w')
+            ceph_pool_df_raw = []
+
+        retcode, stdout, stderr = common.run_os_command('rados df --format json', timeout=1)
+        try:
+            rados_pool_df_raw = json.loads(stdout)['pools']
+        except Exception as e:
+            logger.out('Failed to obtain Pool data (rados df): {}'.format(e), state='w')
+            rados_pool_df_raw = []
+
+        pool_count = len(ceph_pool_df_raw)
+        if debug:
+            logger.out("Getting info for {} pools".format(pool_count), state='d', prefix='ceph-thread')
+        for pool_idx in range(0, pool_count):
+            try:
+                # Combine all the data for this pool
+                ceph_pool_df = ceph_pool_df_raw[pool_idx]
+                rados_pool_df = rados_pool_df_raw[pool_idx]
+                pool = ceph_pool_df
+                pool.update(rados_pool_df)
+
+                # Ignore any pools that aren't in our pool list
+                if pool['name'] not in pool_list:
+                    if debug:
+                        logger.out("Pool {} not in pool list {}".format(pool['name'], pool_list), state='d', prefix='ceph-thread')
+                    continue
+                else:
+                    if debug:
+                        logger.out("Parsing data for pool {}".format(pool['name']), state='d', prefix='ceph-thread')
+
+                # Assemble a useful data structure
+                pool_df = {
+                    'id': pool['id'],
+                    'stored_bytes': pool['stats']['stored'],
+                    'free_bytes': pool['stats']['max_avail'],
+                    'used_bytes': pool['stats']['bytes_used'],
+                    'used_percent': pool['stats']['percent_used'],
+                    'num_objects': pool['stats']['objects'],
+                    'num_object_clones': pool['num_object_clones'],
+                    'num_object_copies': pool['num_object_copies'],
+                    'num_objects_missing_on_primary': pool['num_objects_missing_on_primary'],
+                    'num_objects_unfound': pool['num_objects_unfound'],
+                    'num_objects_degraded': pool['num_objects_degraded'],
+                    'read_ops': pool['read_ops'],
+                    'read_bytes': pool['read_bytes'],
+                    'write_ops': pool['write_ops'],
+                    'write_bytes': pool['write_bytes']
+                }
+
+                # Write the pool data to Zookeeper
+                zkhandler.write([
+                    (('pool.stats', pool['name']), str(json.dumps(pool_df)))
+                ])
+            except Exception as e:
+                # One or more of the status commands timed out, just continue
+                logger.out('Failed to format and send pool data: {}'.format(e), state='w')
+                pass
+
+    # Only grab OSD stats if there are OSDs to grab (otherwise `ceph osd df` hangs)
+    osds_this_node = 0
+    if len(osd_list) > 0:
+        # Get data from Ceph OSDs
+        if debug:
+            logger.out("Get data from Ceph OSDs", state='d', prefix='ceph-thread')
+
+        # Parse the dump data
+        osd_dump = dict()
+
+        command = {"prefix": "osd dump", "format": "json"}
+        osd_dump_output = ceph_conn.mon_command(json.dumps(command), b'', timeout=1)[1].decode('ascii')
+        try:
+            osd_dump_raw = json.loads(osd_dump_output)['osds']
+        except Exception as e:
+            logger.out('Failed to obtain OSD data: {}'.format(e), state='w')
+            osd_dump_raw = []
+
+        if debug:
+            logger.out("Loop through OSD dump", state='d', prefix='ceph-thread')
+        for osd in osd_dump_raw:
+            osd_dump.update({
+                str(osd['osd']): {
+                    'uuid': osd['uuid'],
+                    'up': osd['up'],
+                    'in': osd['in'],
+                    'primary_affinity': osd['primary_affinity']
+                }
+            })
+
+        # Parse the df data
+        if debug:
+            logger.out("Parse the OSD df data", state='d', prefix='ceph-thread')
+
+        osd_df = dict()
+
+        command = {"prefix": "osd df", "format": "json"}
+        try:
+            osd_df_raw = json.loads(ceph_conn.mon_command(json.dumps(command), b'', timeout=1)[1])['nodes']
+        except Exception as e:
+            logger.out('Failed to obtain OSD data: {}'.format(e), state='w')
+            osd_df_raw = []
+
+        if debug:
+            logger.out("Loop through OSD df", state='d', prefix='ceph-thread')
+        for osd in osd_df_raw:
+            osd_df.update({
+                str(osd['id']): {
+                    'utilization': osd['utilization'],
+                    'var': osd['var'],
+                    'pgs': osd['pgs'],
+                    'kb': osd['kb'],
+                    'weight': osd['crush_weight'],
+                    'reweight': osd['reweight'],
+                }
+            })
+
+        # Parse the status data
+        if debug:
+            logger.out("Parse the OSD status data", state='d', prefix='ceph-thread')
+
+        osd_status = dict()
+
+        command = {"prefix": "osd status", "format": "pretty"}
+        try:
+            osd_status_raw = ceph_conn.mon_command(json.dumps(command), b'', timeout=1)[1].decode('ascii')
+        except Exception as e:
+            logger.out('Failed to obtain OSD status data: {}'.format(e), state='w')
+            osd_status_raw = []
+
+        if debug:
+            logger.out("Loop through OSD status data", state='d', prefix='ceph-thread')
+
+        for line in osd_status_raw.split('\n'):
+            # Strip off colour
+            line = re.sub(r'\x1b(\[.*?[@-~]|\].*?(\x07|\x1b\\))', '', line)
+            # Split it for parsing
+            line = line.split()
+            if len(line) > 1 and line[1].isdigit():
+                # This is an OSD line so parse it
+                osd_id = line[1]
+                node = line[3].split('.')[0]
+                used = line[5]
+                avail = line[7]
+                wr_ops = line[9]
+                wr_data = line[11]
+                rd_ops = line[13]
+                rd_data = line[15]
+                state = line[17]
+                osd_status.update({
+                    str(osd_id): {
+                        'node': node,
+                        'used': used,
+                        'avail': avail,
+                        'wr_ops': wr_ops,
+                        'wr_data': wr_data,
+                        'rd_ops': rd_ops,
+                        'rd_data': rd_data,
+                        'state': state
+                    }
+                })
+
+        # Merge them together into a single meaningful dict
+        if debug:
+            logger.out("Merge OSD data together", state='d', prefix='ceph-thread')
+
+        osd_stats = dict()
+
+        for osd in osd_list:
+            if zkhandler.read(('osd.node', osd)) == config['node_hostname']:
+                osds_this_node += 1
+            try:
+                this_dump = osd_dump[osd]
+                this_dump.update(osd_df[osd])
+                this_dump.update(osd_status[osd])
+                osd_stats[osd] = this_dump
+            except KeyError as e:
+                # One or more of the status commands timed out, just continue
+                logger.out('Failed to parse OSD stats into dictionary: {}'.format(e), state='w')
+
+        # Upload OSD data for the cluster (primary-only)
+        if this_node.router_state == 'primary':
+            if debug:
+                logger.out("Trigger updates for each OSD", state='d', prefix='ceph-thread')
+
+            for osd in osd_list:
+                try:
+                    stats = json.dumps(osd_stats[osd])
+                    zkhandler.write([
+                        (('osd.stats', osd), str(stats))
+                    ])
+                except KeyError as e:
+                    # One or more of the status commands timed out, just continue
+                    logger.out('Failed to upload OSD stats from dictionary: {}'.format(e), state='w')
+
+    ceph_conn.shutdown()
+
+    queue.put(ceph_health_colour)
+    queue.put(ceph_health)
+    queue.put(osds_this_node)
+
+    if debug:
+        logger.out("Thread finished", state='d', prefix='ceph-thread')
+
+
+# VM stats update function
+def collect_vm_stats(logger, config, zkhandler, this_node, queue):
+    debug = config['debug']
+    if debug:
+        logger.out("Thread starting", state='d', prefix='vm-thread')
+
+    # Connect to libvirt
+    libvirt_name = "qemu:///system"
+    if debug:
+        logger.out("Connecting to libvirt", state='d', prefix='vm-thread')
+    try:
+        lv_conn = libvirt.open(libvirt_name)
+        if lv_conn is None:
+            raise Exception
+    except Exception:
+        logger.out('Failed to open connection to "{}"'.format(libvirt_name), state='e')
+        return
+
+    memalloc = 0
+    memprov = 0
+    vcpualloc = 0
+    # Toggle state management of dead VMs to restart them
+    if debug:
+        logger.out("Toggle state management of dead VMs to restart them", state='d', prefix='vm-thread')
+    # Make a copy of the d_domain; if not, and it changes in flight, this can fail
+    fixed_d_domain = this_node.d_domain.copy()
+    for domain, instance in fixed_d_domain.items():
+        if domain in this_node.domain_list:
+            # Add the allocated memory to our memalloc value
+            memalloc += instance.getmemory()
+            memprov += instance.getmemory()
+            vcpualloc += instance.getvcpus()
+            if instance.getstate() == 'start' and instance.getnode() == this_node.name:
+                if instance.getdom() is not None:
+                    try:
+                        if instance.getdom().state()[0] != libvirt.VIR_DOMAIN_RUNNING:
+                            logger.out("VM {} has failed".format(instance.domname), state='w', prefix='vm-thread')
+                            raise
+                    except Exception:
+                        # Toggle a state "change"
+                        logger.out("Resetting state to {} for VM {}".format(instance.getstate(), instance.domname), state='i', prefix='vm-thread')
+                        zkhandler.write([
+                            (('domain.state', domain), instance.getstate())
+                        ])
+        elif instance.getnode() == this_node.name:
+            memprov += instance.getmemory()
+
+    # Get list of running domains from Libvirt
+    running_domains = lv_conn.listAllDomains(libvirt.VIR_CONNECT_LIST_DOMAINS_ACTIVE)
+
+    # Get statistics from any running VMs
+    for domain in running_domains:
+        try:
+            # Get basic information about the VM
+            tree = ElementTree.fromstring(domain.XMLDesc())
+            domain_uuid = domain.UUIDString()
+            domain_name = domain.name()
+
+            # Get all the raw information about the VM
+            if debug:
+                logger.out("Getting general statistics for VM {}".format(domain_name), state='d', prefix='vm-thread')
+            domain_state, domain_maxmem, domain_mem, domain_vcpus, domain_cputime = domain.info()
+            # We can't properly gather stats from a non-running VMs so continue
+            if domain_state != libvirt.VIR_DOMAIN_RUNNING:
+                continue
+            domain_memory_stats = domain.memoryStats()
+            domain_cpu_stats = domain.getCPUStats(True)[0]
+        except Exception as e:
+            if debug:
+                try:
+                    logger.out("Failed getting VM information for {}: {}".format(domain.name(), e), state='d', prefix='vm-thread')
+                except Exception:
+                    pass
+            continue
+
+        # Ensure VM is present in the domain_list
+        if domain_uuid not in this_node.domain_list:
+            this_node.domain_list.append(domain_uuid)
+
+        if debug:
+            logger.out("Getting disk statistics for VM {}".format(domain_name), state='d', prefix='vm-thread')
+        domain_disk_stats = []
+        try:
+            for disk in tree.findall('devices/disk'):
+                disk_name = disk.find('source').get('name')
+                if not disk_name:
+                    disk_name = disk.find('source').get('file')
+                disk_stats = domain.blockStats(disk.find('target').get('dev'))
+                domain_disk_stats.append({
+                    "name": disk_name,
+                    "rd_req": disk_stats[0],
+                    "rd_bytes": disk_stats[1],
+                    "wr_req": disk_stats[2],
+                    "wr_bytes": disk_stats[3],
+                    "err": disk_stats[4]
+                })
+        except Exception as e:
+            if debug:
+                try:
+                    logger.out("Failed getting disk stats for {}: {}".format(domain.name(), e), state='d', prefix='vm-thread')
+                except Exception:
+                    pass
+            continue
+
+        if debug:
+            logger.out("Getting network statistics for VM {}".format(domain_name), state='d', prefix='vm-thread')
+        domain_network_stats = []
+        try:
+            for interface in tree.findall('devices/interface'):
+                interface_type = interface.get('type')
+                if interface_type not in ['bridge']:
+                    continue
+                interface_name = interface.find('target').get('dev')
+                interface_bridge = interface.find('source').get('bridge')
+                interface_stats = domain.interfaceStats(interface_name)
+                domain_network_stats.append({
+                    "name": interface_name,
+                    "bridge": interface_bridge,
+                    "rd_bytes": interface_stats[0],
+                    "rd_packets": interface_stats[1],
+                    "rd_errors": interface_stats[2],
+                    "rd_drops": interface_stats[3],
+                    "wr_bytes": interface_stats[4],
+                    "wr_packets": interface_stats[5],
+                    "wr_errors": interface_stats[6],
+                    "wr_drops": interface_stats[7]
+                })
+        except Exception as e:
+            if debug:
+                try:
+                    logger.out("Failed getting network stats for {}: {}".format(domain.name(), e), state='d', prefix='vm-thread')
+                except Exception:
+                    pass
+            continue
+
+        # Create the final dictionary
+        domain_stats = {
+            "state": libvirt_vm_states[domain_state],
+            "maxmem": domain_maxmem,
+            "livemem": domain_mem,
+            "cpus": domain_vcpus,
+            "cputime": domain_cputime,
+            "mem_stats": domain_memory_stats,
+            "cpu_stats": domain_cpu_stats,
+            "disk_stats": domain_disk_stats,
+            "net_stats": domain_network_stats
+        }
+
+        if debug:
+            logger.out("Writing statistics for VM {} to Zookeeper".format(domain_name), state='d', prefix='vm-thread')
+
+        try:
+            zkhandler.write([
+                (('domain.stats', domain_uuid), str(json.dumps(domain_stats)))
+            ])
+        except Exception as e:
+            if debug:
+                logger.out("Failed to write domain statistics: {}".format(e), state='d', prefix='vm-thread')
+
+    # Close the Libvirt connection
+    lv_conn.close()
+
+    queue.put(len(running_domains))
+    queue.put(memalloc)
+    queue.put(memprov)
+    queue.put(vcpualloc)
+
+    if debug:
+        logger.out("Thread finished", state='d', prefix='vm-thread')
+
+
+# Keepalive update function
+def node_keepalive(logger, config, zkhandler, this_node):
+    debug = config['debug']
+    if debug:
+        logger.out("Keepalive starting", state='d', prefix='main-thread')
+
+    # Set the migration selector in Zookeeper for clients to read
+    if config['enable_hypervisor']:
+        if this_node.router_state == 'primary':
+            try:
+                if zkhandler.read('base.config.migration_target_selector') != config['migration_target_selector']:
+                    raise
+            except Exception:
+                zkhandler.write([
+                    ('base.config.migration_target_selector', config['migration_target_selector'])
+                ])
+
+    # Set the upstream IP in Zookeeper for clients to read
+    if config['enable_networking']:
+        if this_node.router_state == 'primary':
+            try:
+                if zkhandler.read('base.config.upstream_ip') != config['upstream_floating_ip']:
+                    raise
+            except Exception:
+                zkhandler.write([
+                    ('base.config.upstream_ip', config['upstream_floating_ip'])
+                ])
+
+    # Get past state and update if needed
+    if debug:
+        logger.out("Get past state and update if needed", state='d', prefix='main-thread')
+
+    past_state = zkhandler.read(('node.state.daemon', this_node.name))
+    if past_state != 'run' and past_state != 'shutdown':
+        this_node.daemon_state = 'run'
+        zkhandler.write([
+            (('node.state.daemon', this_node.name), 'run')
+        ])
+    else:
+        this_node.daemon_state = 'run'
+
+    # Ensure the primary key is properly set
+    if debug:
+        logger.out("Ensure the primary key is properly set", state='d', prefix='main-thread')
+    if this_node.router_state == 'primary':
+        if zkhandler.read('base.config.primary_node') != this_node.name:
+            zkhandler.write([
+                ('base.config.primary_node', this_node.name)
+            ])
+
+    # Run VM statistics collection in separate thread for parallelization
+    if config['enable_hypervisor']:
+        vm_thread_queue = Queue()
+        vm_stats_thread = Thread(target=collect_vm_stats, args=(logger, config, zkhandler, this_node, vm_thread_queue), kwargs={})
+        vm_stats_thread.start()
+
+    # Run Ceph status collection in separate thread for parallelization
+    if config['enable_storage']:
+        ceph_thread_queue = Queue()
+        ceph_stats_thread = Thread(target=collect_ceph_stats, args=(logger, config, zkhandler, this_node, ceph_thread_queue), kwargs={})
+        ceph_stats_thread.start()
+
+    # Get node performance statistics
+    this_node.memtotal = int(psutil.virtual_memory().total / 1024 / 1024)
+    this_node.memused = int(psutil.virtual_memory().used / 1024 / 1024)
+    this_node.memfree = int(psutil.virtual_memory().free / 1024 / 1024)
+    this_node.cpuload = os.getloadavg()[0]
+
+    # Join against running threads
+    if config['enable_hypervisor']:
+        vm_stats_thread.join(timeout=config['keepalive_interval'])
+        if vm_stats_thread.is_alive():
+            logger.out('VM stats gathering exceeded timeout, continuing', state='w')
+    if config['enable_storage']:
+        ceph_stats_thread.join(timeout=config['keepalive_interval'])
+        if ceph_stats_thread.is_alive():
+            logger.out('Ceph stats gathering exceeded timeout, continuing', state='w')
+
+    # Get information from thread queues
+    if config['enable_hypervisor']:
+        try:
+            this_node.domains_count = vm_thread_queue.get(timeout=config['keepalive_interval'])
+            this_node.memalloc = vm_thread_queue.get(timeout=config['keepalive_interval'])
+            this_node.memprov = vm_thread_queue.get(timeout=config['keepalive_interval'])
+            this_node.vcpualloc = vm_thread_queue.get(timeout=config['keepalive_interval'])
+        except Exception:
+            logger.out('VM stats queue get exceeded timeout, continuing', state='w')
+    else:
+        this_node.domains_count = 0
+        this_node.memalloc = 0
+        this_node.memprov = 0
+        this_node.vcpualloc = 0
+
+    if config['enable_storage']:
+        try:
+            ceph_health_colour = ceph_thread_queue.get(timeout=config['keepalive_interval'])
+            ceph_health = ceph_thread_queue.get(timeout=config['keepalive_interval'])
+            osds_this_node = ceph_thread_queue.get(timeout=config['keepalive_interval'])
+        except Exception:
+            logger.out('Ceph stats queue get exceeded timeout, continuing', state='w')
+            ceph_health_colour = logger.fmt_cyan
+            ceph_health = 'UNKNOWN'
+            osds_this_node = '?'
+
+    # Set our information in zookeeper
+    keepalive_time = int(time.time())
+    if debug:
+        logger.out("Set our information in zookeeper", state='d', prefix='main-thread')
+    try:
+        zkhandler.write([
+            (('node.memory.total', this_node.name), str(this_node.memtotal)),
+            (('node.memory.used', this_node.name), str(this_node.memused)),
+            (('node.memory.free', this_node.name), str(this_node.memfree)),
+            (('node.memory.allocated', this_node.name), str(this_node.memalloc)),
+            (('node.memory.provisioned', this_node.name), str(this_node.memprov)),
+            (('node.vcpu.allocated', this_node.name), str(this_node.vcpualloc)),
+            (('node.cpu.load', this_node.name), str(this_node.cpuload)),
+            (('node.count.provisioned_domains', this_node.name), str(this_node.domains_count)),
+            (('node.running_domains', this_node.name), ' '.join(this_node.domain_list)),
+            (('node.keepalive', this_node.name), str(keepalive_time)),
+        ])
+    except Exception:
+        logger.out('Failed to set keepalive data', state='e')
+
+    # Display node information to the terminal
+    if config['log_keepalives']:
+        if this_node.router_state == 'primary':
+            cst_colour = logger.fmt_green
+        elif this_node.router_state == 'secondary':
+            cst_colour = logger.fmt_blue
+        else:
+            cst_colour = logger.fmt_cyan
+        logger.out(
+            '{}{} keepalive @ {}{} [{}{}{}]'.format(
+                logger.fmt_purple,
+                config['node_hostname'],
+                datetime.now(),
+                logger.fmt_end,
+                logger.fmt_bold + cst_colour,
+                this_node.router_state,
+                logger.fmt_end
+            ),
+            state='t'
+        )
+        if config['log_keepalive_cluster_details']:
+            logger.out(
+                '{bold}Maintenance:{nofmt} {maint}  '
+                '{bold}Active VMs:{nofmt} {domcount}  '
+                '{bold}Networks:{nofmt} {netcount}  '
+                '{bold}Load:{nofmt} {load}  '
+                '{bold}Memory [MiB]: VMs:{nofmt} {allocmem}  '
+                '{bold}Used:{nofmt} {usedmem}  '
+                '{bold}Free:{nofmt} {freemem}'.format(
+                    bold=logger.fmt_bold,
+                    nofmt=logger.fmt_end,
+                    maint=this_node.maintenance,
+                    domcount=this_node.domains_count,
+                    netcount=len(zkhandler.children('base.network')),
+                    load=this_node.cpuload,
+                    freemem=this_node.memfree,
+                    usedmem=this_node.memused,
+                    allocmem=this_node.memalloc,
+                ),
+                state='t'
+            )
+        if config['enable_storage'] and config['log_keepalive_storage_details']:
+            logger.out(
+                '{bold}Ceph cluster status:{nofmt} {health_colour}{health}{nofmt}  '
+                '{bold}Total OSDs:{nofmt} {total_osds}  '
+                '{bold}Node OSDs:{nofmt} {node_osds}  '
+                '{bold}Pools:{nofmt} {total_pools}  '.format(
+                    bold=logger.fmt_bold,
+                    health_colour=ceph_health_colour,
+                    nofmt=logger.fmt_end,
+                    health=ceph_health,
+                    total_osds=len(zkhandler.children('base.osd')),
+                    node_osds=osds_this_node,
+                    total_pools=len(zkhandler.children('base.pool'))
+                ),
+                state='t'
+            )
+
+    # Look for dead nodes and fence them
+    if not this_node.maintenance:
+        if debug:
+            logger.out("Look for dead nodes and fence them", state='d', prefix='main-thread')
+        if config['daemon_mode'] == 'coordinator':
+            for node_name in zkhandler.children('base.node'):
+                try:
+                    node_daemon_state = zkhandler.read(('node.state.daemon', node_name))
+                    node_keepalive = int(zkhandler.read(('node.keepalive', node_name)))
+                except Exception:
+                    node_daemon_state = 'unknown'
+                    node_keepalive = 0
+
+                # Handle deadtime and fencng if needed
+                # (A node is considered dead when its keepalive timer is >6*keepalive_interval seconds
+                # out-of-date while in 'start' state)
+                node_deadtime = int(time.time()) - (int(config['keepalive_interval']) * int(config['fence_intervals']))
+                if node_keepalive < node_deadtime and node_daemon_state == 'run':
+                    logger.out('Node {} seems dead - starting monitor for fencing'.format(node_name), state='w')
+                    zk_lock = zkhandler.writelock(('node.state.daemon', node_name))
+                    with zk_lock:
+                        # Ensures that, if we lost the lock race and come out of waiting,
+                        # we won't try to trigger our own fence thread.
+                        if zkhandler.read(('node.state.daemon', node_name)) != 'dead':
+                            fence_thread = Thread(target=pvcnoded.util.fencing.fence_node, args=(node_name, zkhandler, config, logger), kwargs={})
+                            fence_thread.start()
+                            # Write the updated data after we start the fence thread
+                            zkhandler.write([
+                                (('node.state.daemon', node_name), 'dead')
+                            ])
+
+    if debug:
+        logger.out("Keepalive finished", state='d', prefix='main-thread')

node-daemon/pvcnoded/util/libvirt.py

@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+
+# libvirt.py - Utility functions for pvcnoded libvirt
+# Part of the Parallel Virtual Cluster (PVC) system
+#
+#    Copyright (C) 2018-2021 Joshua M. Boniface <joshua@boniface.me>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, version 3.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+#
+###############################################################################
+
+import libvirt
+
+
+def validate_libvirtd(logger, config):
+    if config['enable_hypervisor']:
+        libvirt_check_name = f'qemu+tcp://{config["node_hostname"]}/system'
+        logger.out(f'Connecting to Libvirt daemon at {libvirt_check_name}', state='i')
+        try:
+            lv_conn = libvirt.open(libvirt_check_name)
+            lv_conn.close()
+        except Exception as e:
+            logger.out(f'Failed to connect to Libvirt daemon: {e}', state='e')
+            return False
+
+    return True