Bump version to 0.9.41

Add bridge_mtu config to docs
Adjust log type of object setup message
2021-10-09 19:39:21 -04:00 · 2021-10-09 19:28:50 -04:00 · 2021-10-09 19:23:12 -04:00 · 2021-10-09 19:21:47 -04:00 · 2021-10-09 19:11:42 -04:00 · 2021-10-09 19:09:38 -04:00
24 changed files with 666 additions and 711 deletions
--- a/.version
+++ b/.version
@ -1 +1 @@
-0.9.39
+0.9.41
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -0,0 +1,302 @@
+## PVC Changelog
+
+###### [v0.9.41](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.41)
+
+  * Fixes a bad conditional check in IPMI verification
+  * Implements per-network MTU configuration; NOTE: Requires new keys in pvcnoded.yaml (`bridge_mtu`) and Ansible group_vars (`pvc_bridge_mtu`)
+
+###### [v0.9.40](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.40)
+
+  * [Docs] Documentation updates for new Changelog file
+  * [Node Daemon] Fixes bug with schema updates
+
+###### [v0.9.39](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.39)
+
+  * [Documentation] Update several documentation sections
+  * [API Daemon/CLI Client] Add negate flag for VM option limits (node, tag, state)
+  * [Build] Add linting check to build-and-deploy.sh
+
+###### [v0.9.38](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.38)
+
+  * [All] Significantly improve storage benchmark format and reporting
+
+###### [v0.9.37](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.37)
+
+  * [All] Adds support for configurable OSD DB size ratios
+  * [Node Daemon] Fixes bugs with OSD creation
+  * [Node Daemon] Fixes exception bugs in CephInstance
+  * [CLI Client] Adjusts descriptions around Ceph OSDs
+  * [Node Daemon] Fixes ordering of pvc-flush unit
+  * [Node Daemon] Fixes bugs in fence handling and libvirt keepalive
+  * [Node Daemon] Simplifies locking for and speeds up VM migrations
+  * [Node Daemon] Fixes bugs in queue get timeouts
+  * [API Daemon] Adjusts benchmark test jobs configuration and naming
+
+###### [v0.9.36](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.36)
+
+  * [Node Daemon] Fixes a bug during early cleanup
+  * [All] Adds support for OSD database/WAL block devices to improve Ceph performance; NOTE: Applies only to new OSDs
+
+###### [v0.9.35](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.35)
+
+  * [Node Daemon] Fixes several bugs and crashes in node daemon
+  * [General] Updates linting rules for newer Flake8 linter
+  * [Daemons/CLI client] Adds VM network and disk hot attach/detach support; NOTE: Changes the default behaviour of `pvc vm network add`/`remove` and `pvc vm volume add`/`remove`
+  * [API Daemon] Adds checks for pool size when resizing volumes
+  * [API Daemon] Adds checks for RAM and vCPU sizes when defining or modifying VMs
+
+###### [v0.9.34](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.34)
+
+  * [Provisioner] Adds support for filesystem arguments containing =
+  * [CLI Client] Fixes bug with pvc provisioner status output formatting
+  * [Node Daemon] Fixes minor typo in startup message
+
+###### [v0.9.33](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.33)
+
+  * [Node Daemon] A major refactoring of the node daemon
+  * [CLI Client] Fixes output errors if a node has no provisioner data
+  * [Packages] Fixes issues with including __pycache__ directories in .deb files
+
+###### [v0.9.32](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.32)
+
+  * [CLI Client] Fixes some incorrect colours in network lists
+  * [Documentation] Adds documentation screenshots of CLI client
+  * [Node Daemon] Fixes a bug if VM stats gathering fails
+
+###### [v0.9.31](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.31)
+
+  * [Packages] Cleans up obsolete Suggests lines
+  * [Node Daemon] Adjusts log text of VM migrations to show the correct source node
+  * [API Daemon] Adjusts the OVA importer to support floppy RASD types for compatability
+  * [API Daemon] Ensures that volume resize commands without a suffix get B appended
+  * [API Daemon] Removes the explicit setting of image-features in PVC; defaulting to the limited set has been moved to the ceph.conf configuration on nodes via PVC Ansible
+
+###### [v0.9.30](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.30)
+
+  * [Node Daemon] Fixes bug with schema validation
+
+###### [v0.9.29](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.29)
+
+  * [Node Daemon] Corrects numerous bugs with node logging framework
+
+###### [v0.9.28](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.28)
+
+  * [CLI Client] Revamp confirmation options for "vm modify" command
+
+###### [v0.9.27](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.27)
+
+  * [CLI Client] Fixes a bug with vm modify command when passed a file
+
+###### [v0.9.26](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.26)
+
+  * [Node Daemon] Corrects some bad assumptions about fencing results during hardware failures
+  * [All] Implements VM tagging functionality
+  * [All] Implements Node log access via PVC functionality
+
+###### [v0.9.25](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.25)
+
+  * [Node Daemon] Returns to Rados library calls for Ceph due to performance problems
+  * [Node Daemon] Adds a date output to keepalive messages
+  * [Daemons] Configures ZK connection logging only for persistent connections
+  * [API Provisioner] Add context manager-based chroot to Debootstrap example script
+  * [Node Daemon] Fixes a bug where shutdown daemon state was overwritten
+
+###### [v0.9.24](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.24)
+
+  * [Node Daemon] Removes Rados module polling of Ceph cluster and returns to command-based polling for timeout purposes, and removes some flaky return statements
+  * [Node Daemon] Removes flaky Zookeeper connection renewals that caused problems
+  * [CLI Client] Allow raw lists of clusters from `pvc cluster list`
+  * [API Daemon] Fixes several issues when getting VM data without stats
+  * [API Daemon] Fixes issues with removing VMs while disks are still in use (failed provisioning, etc.)
+
+###### [v0.9.23](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.23)
+
+  * [Daemons] Fixes a critical overwriting bug in zkhandler when schema paths are not yet valid
+  * [Node Daemon] Ensures the daemon mode is updated on every startup (fixes the side effect of the above bug in 0.9.22)
+
+###### [v0.9.22](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.22)
+
+  * [API Daemon] Drastically improves performance when getting large lists (e.g. VMs)
+  * [Daemons] Adds profiler functions for use in debug mode
+  * [Daemons] Improves reliability of ZK locking
+  * [Daemons] Adds the new logo in ASCII form to the Daemon startup message
+  * [Node Daemon] Fixes bug where VMs would sometimes not stop
+  * [Node Daemon] Code cleanups in various classes
+  * [Node Daemon] Fixes a bug when reading node schema data
+  * [All] Adds node PVC version information to the list output
+  * [CLI Client] Improves the style and formatting of list output including a new header line
+  * [API Worker] Fixes a bug that prevented the storage benchmark job from running
+
+###### [v0.9.21](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.21)
+
+  * [API Daemon] Ensures VMs stop before removing them
+  * [Node Daemon] Fixes a bug with VM shutdowns not timing out
+  * [Documentation] Adds information about georedundancy caveats
+  * [All] Adds support for SR-IOV NICs (hostdev and macvtap) and surrounding documentation
+  * [Node Daemon] Fixes a bug where shutdown aborted migrations unexpectedly
+  * [Node Daemon] Fixes a bug where the migration method was not updated realtime
+  * [Node Daemon] Adjusts the Patroni commands to remove reference to Zookeeper path
+  * [CLI Client] Adjusts several help messages and fixes some typos
+  * [CLI Client] Converts the CLI client to a proper Python module
+  * [API Daemon] Improves VM list performance
+  * [API Daemon] Adjusts VM list matching critera (only matches against the UUID if it's a full UUID)
+  * [API Worker] Fixes incompatibility between Deb 10 and 11 in launching Celery worker
+  * [API Daemon] Corrects several bugs with initialization command
+  * [Documentation] Adds a shiny new logo and revamps introduction text
+
+###### [v0.9.20](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.20)
+
+  * [Daemons] Implemented a Zookeeper schema handler and version 0 schema
+  * [Daemons] Completes major refactoring of codebase to make use of the schema handler
+  * [Daemons] Adds support for dynamic chema changges and "hot reloading" of pvcnoded processes
+  * [Daemons] Adds a functional testing script for verifying operation against a test cluster
+  * [Daemons, CLI] Fixes several minor bugs found by the above script
+  * [Daemons, CLI] Add support for Debian 11 "Bullseye"
+
+###### [v0.9.19](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.19)
+
+  * [CLI] Corrects some flawed conditionals
+  * [API] Disables SQLAlchemy modification tracking functionality (not used by us)
+  * [Daemons] Implements new zkhandler module for improved reliability and reusability
+  * [Daemons] Refactors some code to use new zkhandler module
+  * [API, CLI] Adds support for "none" migration selector (uses cluster default instead)
+  * [Daemons] Moves some configuration keys to new /config tree
+  * [Node Daemon] Increases initial lock timeout for VM migrations to avoid out-of-sync potential
+  * [Provisioner] Support storing and using textual cluster network labels ("upstream", "storage", "cluster") in templates
+  * [API] Avoid duplicating existing node states
+
+###### [v0.9.18](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.18)
+
+  * Adds VM rename functionality to API and CLI client
+
+###### [v0.9.17](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.17)
+
+  * [CLI] Fixes bugs in log follow output
+
+###### [v0.9.16](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.16)
+
+  * Improves some CLI help messages
+  * Skips empty local cluster in CLI
+  * Adjusts how confirmations happen during VM modify restarts
+  * Fixes bug around corrupted VM log files
+  * Fixes bug around subprocess pipe exceptions
+
+###### [v0.9.15](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.15)
+
+  * [CLI] Adds additional verification (--yes) to several VM management commands
+  * [CLI] Adds a method to override --yes/confirmation requirements via envvar (PVC_UNSAFE)
+  * [CLI] Adds description fields to PVC clusters in CLI
+
+###### [v0.9.14](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.14)
+
+  * Fixes bugs around cloned volume provisioning
+  * Fixes some minor visual bugs
+  * Minor license update (from GPL3+ to GPL3)
+  * Adds qemu-guest-agent support to provisioner-created VMs by default
+
+###### [v0.9.13](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.13)
+
+  * Adds nicer startup messages for daemons
+  * Adds additional API field for stored_bytes to pool stats
+  * Fixes sorting issues with snapshot lists
+  * Fixes missing increment/decrement of snapshot_count on volumes
+  * Fixes bad calls in pool element API endpoints
+  * Fixes inconsistent bytes_tohuman behaviour in daemons
+  * Adds validation and maximum volume size on creation (must be smaller than the pool free space)
+
+###### [v0.9.12](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.12)
+
+  * Fixes a bug in the pvcnoded service unit file causing a Zookeeper startup race condition
+
+###### [v0.9.11](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.11)
+
+  * Documentation updates
+  * Adds VNC information to VM info
+  * Goes back to external Ceph commands for disk usage
+
+###### [v0.9.10](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.10)
+
+  * Moves OSD stats uploading to primary, eliminating reporting failures while hosts are down
+  * Documentation updates
+  * Significantly improves RBD locking behaviour in several situations, eliminating cold-cluster start issues and failed VM boot-ups after crashes
+  * Fixes some timeout delays with fencing
+  * Fixes bug in validating YAML provisioner userdata
+
+###### [v0.9.9](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.9)
+
+  * Adds documentation updates
+  * Removes single-element list stripping and fixes surrounding bugs
+  * Adds additional fields to some API endpoints for ease of parsing by clients
+  * Fixes bugs with network configuration
+
+###### [v0.9.8](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.8)
+
+  * Adds support for cluster backup/restore
+  * Moves location of `init` command in CLI to make room for the above
+  * Cleans up some invalid help messages from the API
+
+###### [v0.9.7](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.7)
+
+  * Fixes bug with provisioner system template modifications
+
+###### [v0.9.6](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.6)
+
+  * Fixes bug with migrations
+
+###### [v0.9.5](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.5)
+
+  * Fixes bug with line count in log follow
+  * Fixes bug with disk stat output being None
+  * Adds short pretty health output
+  * Documentation updates
+
+###### [v0.9.4](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.4)
+
+  * Fixes major bug in OVA parser
+
+###### [v0.9.3](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.3)
+
+  * Fixes bugs with image & OVA upload parsing
+
+###### [v0.9.2](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.2)
+
+  * Major linting of the codebase with flake8; adds linting tools
+  * Implements CLI-based modification of VM vCPUs, memory, networks, and disks without directly editing XML
+  * Fixes bug where `pvc vm log -f` would show all 1000 lines before starting
+  * Fixes bug in default provisioner libvirt schema (`drive` -> `driver` typo)
+
+###### [v0.9.1](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.1)
+
+  * Added per-VM migration method feature
+  * Fixed bug with provisioner system template listing
+
+###### [v0.9.0](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.0)
+
+Numerous small improvements and bugfixes. This release is suitable for general use and is pre-release-quality software.
+
+This release introduces an updated version scheme; all future stable releases until 1.0.0 is ready will be made under this 0.9.z naming. This does not represent semantic versioning and all changes (feature, improvement, or bugfix) will be considered for inclusion in this release train.
+
+###### [v0.8](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.8)
+
+Numerous improvements and bugfixes. This release is suitable for general use and is pre-release-quality software.
+
+###### [v0.7](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.7)
+
+Numerous improvements and bugfixes, revamped documentation. This release is suitable for general use and is beta-quality software.
+
+###### [v0.6](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.6)
+
+Numerous improvements and bugfixes, full implementation of the provisioner, full implementation of the API CLI client (versus direct CLI client). This release is suitable for general use and is beta-quality software.
+
+###### [v0.5](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.5)
+
+First public release; fully implements the VM, network, and storage managers, the HTTP API, and the pvc-ansible framework for deploying and bootstrapping a cluster. This release is suitable for general use, though it is still alpha-quality software and should be expected to change significantly until 1.0 is released.
+
+###### [v0.4](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.4)
+
+Full implementation of virtual management and virtual networking functionality. Partial implementation of storage functionality.
+
+###### [v0.3](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.3)
+
+Basic implementation of virtual management functionality.
+
--- a/README.md
+++ b/README.md
@ -8,7 +8,7 @@

 ## What is PVC?

-PVC is a virtual machine-based hyperconverged infrastructure (HCI) virtualization cluster solution that is fully Free Software, scalable, redundant, self-healing, self-managing, and designed for administrator simplicity. It is an alternative to other HCI solutions such as Harvester, Nutanix, and VMWare, as well as to other common virtualization stacks such as ProxMox and OpenStack.
+PVC is a Linux KVM-based hyperconverged infrastructure (HCI) virtualization cluster solution that is fully Free Software, scalable, redundant, self-healing, self-managing, and designed for administrator simplicity. It is an alternative to other HCI solutions such as Harvester, Nutanix, and VMWare, as well as to other common virtualization stacks such as ProxMox and OpenStack.

 PVC is a complete HCI solution, built from well-known and well-trusted Free Software tools, to assist an administrator in creating and managing a cluster of servers to run virtual machines, as well as self-managing several important aspects including storage failover, node failure and recovery, virtual machine failure and recovery, and network plumbing. It is designed to act consistently, reliably, and unobtrusively, letting the administrator concentrate on more important things.

@ -40,6 +40,11 @@ The core node and API daemons, as well as the CLI API client, are written in Pyt
 To get started with PVC, please see the [About](https://parallelvirtualcluster.readthedocs.io/en/latest/about/) page for general information about the project, and the [Getting Started](https://parallelvirtualcluster.readthedocs.io/en/latest/getting-started/) page for details on configuring your first cluster.


+## Changelog
+
+View the changelog in [CHANGELOG.md](https://github.com/parallelvirtualcluster/pvc/blob/master/CHANGELOG.md).
+
+
 ## Screenshots

 While PVC's API and internals aren't very screenshot-worthy, here is some example output of the CLI tool.
@ -51,297 +56,3 @@ While PVC's API and internals aren't very screenshot-worthy, here is some exampl
 <p><img alt="VM listing and migration" src="docs/images/pvc-migration.png"/><br/><i>Listing a limited set of VMs and migrating one with status updates</i></p>

 <p><img alt="Node logs" src="docs/images/pvc-nodelog.png"/><br/><i>Viewing the logs of a node (keepalives and VM [un]migration)</i></p>
-
-
-## Changelog
-
-###### v0.9.39
-
-  * [Documentation] Update several documentation sections
-  * [API Daemon/CLI Client] Add negate flag for VM option limits (node, tag, state)
-  * [Build] Add linting check to build-and-deploy.sh
-
-###### v0.9.38
-
-  * [All] Significantly improve storage benchmark format and reporting
-
-###### v0.9.37
-
-  * [All] Adds support for configurable OSD DB size ratios
-  * [Node Daemon] Fixes bugs with OSD creation
-  * [Node Daemon] Fixes exception bugs in CephInstance
-  * [CLI Client] Adjusts descriptions around Ceph OSDs
-  * [Node Daemon] Fixes ordering of pvc-flush unit
-  * [Node Daemon] Fixes bugs in fence handling and libvirt keepalive
-  * [Node Daemon] Simplifies locking for and speeds up VM migrations
-  * [Node Daemon] Fixes bugs in queue get timeouts
-  * [API Daemon] Adjusts benchmark test jobs configuration and naming
-
-###### v0.9.36
-
-  * [Node Daemon] Fixes a bug during early cleanup
-  * [All] Adds support for OSD database/WAL block devices to improve Ceph performance; NOTE: Applies only to new OSDs
-
-###### v0.9.35
-
-  * [Node Daemon] Fixes several bugs and crashes in node daemon
-  * [General] Updates linting rules for newer Flake8 linter
-  * [Daemons/CLI client] Adds VM network and disk hot attach/detach support; NOTE: Changes the default behaviour of `pvc vm network add`/`remove` and `pvc vm volume add`/`remove`
-  * [API Daemon] Adds checks for pool size when resizing volumes
-  * [API Daemon] Adds checks for RAM and vCPU sizes when defining or modifying VMs
-
-###### v0.9.34
-
-  * [Provisioner] Adds support for filesystem arguments containing =
-  * [CLI Client] Fixes bug with pvc provisioner status output formatting
-  * [Node Daemon] Fixes minor typo in startup message
-
-###### v0.9.33
-
-  * [Node Daemon] A major refactoring of the node daemon
-  * [CLI Client] Fixes output errors if a node has no provisioner data
-  * [Packages] Fixes issues with including __pycache__ directories in .deb files
-
-###### v0.9.32
-
-  * [CLI Client] Fixes some incorrect colours in network lists
-  * [Documentation] Adds documentation screenshots of CLI client
-  * [Node Daemon] Fixes a bug if VM stats gathering fails
-
-###### v0.9.31
-
-  * [Packages] Cleans up obsolete Suggests lines
-  * [Node Daemon] Adjusts log text of VM migrations to show the correct source node
-  * [API Daemon] Adjusts the OVA importer to support floppy RASD types for compatability
-  * [API Daemon] Ensures that volume resize commands without a suffix get B appended
-  * [API Daemon] Removes the explicit setting of image-features in PVC; defaulting to the limited set has been moved to the ceph.conf configuration on nodes via PVC Ansible
-
-###### v0.9.30
-
-  * [Node Daemon] Fixes bug with schema validation
-
-###### v0.9.29
-
-  * [Node Daemon] Corrects numerous bugs with node logging framework
-
-###### v0.9.28
-
-  * [CLI Client] Revamp confirmation options for "vm modify" command
-
-###### v0.9.27
-
-  * [CLI Client] Fixes a bug with vm modify command when passed a file
-
-###### v0.9.26
-
-  * [Node Daemon] Corrects some bad assumptions about fencing results during hardware failures
-  * [All] Implements VM tagging functionality
-  * [All] Implements Node log access via PVC functionality
-
-###### v0.9.25
-
-  * [Node Daemon] Returns to Rados library calls for Ceph due to performance problems
-  * [Node Daemon] Adds a date output to keepalive messages
-  * [Daemons] Configures ZK connection logging only for persistent connections
-  * [API Provisioner] Add context manager-based chroot to Debootstrap example script
-  * [Node Daemon] Fixes a bug where shutdown daemon state was overwritten
-
-###### v0.9.24
-
-  * [Node Daemon] Removes Rados module polling of Ceph cluster and returns to command-based polling for timeout purposes, and removes some flaky return statements
-  * [Node Daemon] Removes flaky Zookeeper connection renewals that caused problems
-  * [CLI Client] Allow raw lists of clusters from `pvc cluster list`
-  * [API Daemon] Fixes several issues when getting VM data without stats
-  * [API Daemon] Fixes issues with removing VMs while disks are still in use (failed provisioning, etc.)
-
-###### v0.9.23
-
-  * [Daemons] Fixes a critical overwriting bug in zkhandler when schema paths are not yet valid
-  * [Node Daemon] Ensures the daemon mode is updated on every startup (fixes the side effect of the above bug in 0.9.22)
-
-###### v0.9.22
-
-  * [API Daemon] Drastically improves performance when getting large lists (e.g. VMs)
-  * [Daemons] Adds profiler functions for use in debug mode
-  * [Daemons] Improves reliability of ZK locking
-  * [Daemons] Adds the new logo in ASCII form to the Daemon startup message
-  * [Node Daemon] Fixes bug where VMs would sometimes not stop
-  * [Node Daemon] Code cleanups in various classes
-  * [Node Daemon] Fixes a bug when reading node schema data
-  * [All] Adds node PVC version information to the list output
-  * [CLI Client] Improves the style and formatting of list output including a new header line
-  * [API Worker] Fixes a bug that prevented the storage benchmark job from running
-
-###### v0.9.21
-
-  * [API Daemon] Ensures VMs stop before removing them
-  * [Node Daemon] Fixes a bug with VM shutdowns not timing out
-  * [Documentation] Adds information about georedundancy caveats
-  * [All] Adds support for SR-IOV NICs (hostdev and macvtap) and surrounding documentation
-  * [Node Daemon] Fixes a bug where shutdown aborted migrations unexpectedly
-  * [Node Daemon] Fixes a bug where the migration method was not updated realtime
-  * [Node Daemon] Adjusts the Patroni commands to remove reference to Zookeeper path
-  * [CLI Client] Adjusts several help messages and fixes some typos
-  * [CLI Client] Converts the CLI client to a proper Python module
-  * [API Daemon] Improves VM list performance
-  * [API Daemon] Adjusts VM list matching critera (only matches against the UUID if it's a full UUID)
-  * [API Worker] Fixes incompatibility between Deb 10 and 11 in launching Celery worker
-  * [API Daemon] Corrects several bugs with initialization command
-  * [Documentation] Adds a shiny new logo and revamps introduction text
-
-###### v0.9.20
-
-  * [Daemons] Implemented a Zookeeper schema handler and version 0 schema
-  * [Daemons] Completes major refactoring of codebase to make use of the schema handler
-  * [Daemons] Adds support for dynamic chema changges and "hot reloading" of pvcnoded processes
-  * [Daemons] Adds a functional testing script for verifying operation against a test cluster
-  * [Daemons, CLI] Fixes several minor bugs found by the above script
-  * [Daemons, CLI] Add support for Debian 11 "Bullseye"
-
-###### v0.9.19
-
-  * [CLI] Corrects some flawed conditionals
-  * [API] Disables SQLAlchemy modification tracking functionality (not used by us)
-  * [Daemons] Implements new zkhandler module for improved reliability and reusability
-  * [Daemons] Refactors some code to use new zkhandler module
-  * [API, CLI] Adds support for "none" migration selector (uses cluster default instead)
-  * [Daemons] Moves some configuration keys to new /config tree
-  * [Node Daemon] Increases initial lock timeout for VM migrations to avoid out-of-sync potential
-  * [Provisioner] Support storing and using textual cluster network labels ("upstream", "storage", "cluster") in templates
-  * [API] Avoid duplicating existing node states
-
-###### v0.9.18
-
-  * Adds VM rename functionality to API and CLI client
-
-###### v0.9.17
-
-  * [CLI] Fixes bugs in log follow output
-
-###### v0.9.16
-
-  * Improves some CLI help messages
-  * Skips empty local cluster in CLI
-  * Adjusts how confirmations happen during VM modify restarts
-  * Fixes bug around corrupted VM log files
-  * Fixes bug around subprocess pipe exceptions
-
-###### v0.9.15
-
-  * [CLI] Adds additional verification (--yes) to several VM management commands
-  * [CLI] Adds a method to override --yes/confirmation requirements via envvar (PVC_UNSAFE)
-  * [CLI] Adds description fields to PVC clusters in CLI
-
-###### v0.9.14
-
-  * Fixes bugs around cloned volume provisioning
-  * Fixes some minor visual bugs
-  * Minor license update (from GPL3+ to GPL3)
-  * Adds qemu-guest-agent support to provisioner-created VMs by default
-
-###### v0.9.13
-
-  * Adds nicer startup messages for daemons
-  * Adds additional API field for stored_bytes to pool stats
-  * Fixes sorting issues with snapshot lists
-  * Fixes missing increment/decrement of snapshot_count on volumes
-  * Fixes bad calls in pool element API endpoints
-  * Fixes inconsistent bytes_tohuman behaviour in daemons
-  * Adds validation and maximum volume size on creation (must be smaller than the pool free space)
-
-###### v0.9.12
-
-  * Fixes a bug in the pvcnoded service unit file causing a Zookeeper startup race condition
-
-###### v0.9.11
-
-  * Documentation updates
-  * Adds VNC information to VM info
-  * Goes back to external Ceph commands for disk usage
-
-###### v0.9.10
-
-  * Moves OSD stats uploading to primary, eliminating reporting failures while hosts are down
-  * Documentation updates
-  * Significantly improves RBD locking behaviour in several situations, eliminating cold-cluster start issues and failed VM boot-ups after crashes
-  * Fixes some timeout delays with fencing
-  * Fixes bug in validating YAML provisioner userdata
-
-###### v0.9.9
-
-  * Adds documentation updates
-  * Removes single-element list stripping and fixes surrounding bugs
-  * Adds additional fields to some API endpoints for ease of parsing by clients
-  * Fixes bugs with network configuration
-
-###### v0.9.8
-
-  * Adds support for cluster backup/restore
-  * Moves location of `init` command in CLI to make room for the above
-  * Cleans up some invalid help messages from the API
-
-###### v0.9.7
-
-  * Fixes bug with provisioner system template modifications
-
-###### v0.9.6
-
-  * Fixes bug with migrations
-
-###### v0.9.5
-
-  * Fixes bug with line count in log follow
-  * Fixes bug with disk stat output being None
-  * Adds short pretty health output
-  * Documentation updates
-
-###### v0.9.4
-
-  * Fixes major bug in OVA parser
-
-###### v0.9.3
-
-  * Fixes bugs with image & OVA upload parsing
-
-###### v0.9.2
-
-  * Major linting of the codebase with flake8; adds linting tools
-  * Implements CLI-based modification of VM vCPUs, memory, networks, and disks without directly editing XML
-  * Fixes bug where `pvc vm log -f` would show all 1000 lines before starting
-  * Fixes bug in default provisioner libvirt schema (`drive` -> `driver` typo)
-
-###### v0.9.1
-
-  * Added per-VM migration method feature
-  * Fixed bug with provisioner system template listing
-
-###### v0.9.0
-
-Numerous small improvements and bugfixes. This release is suitable for general use and is pre-release-quality software.
-
-This release introduces an updated version scheme; all future stable releases until 1.0.0 is ready will be made under this 0.9.z naming. This does not represent semantic versioning and all changes (feature, improvement, or bugfix) will be considered for inclusion in this release train.
-
-###### v0.8
-
-Numerous improvements and bugfixes. This release is suitable for general use and is pre-release-quality software.
-
-###### v0.7
-
-Numerous improvements and bugfixes, revamped documentation. This release is suitable for general use and is beta-quality software.
-
-###### v0.6
-
-Numerous improvements and bugfixes, full implementation of the provisioner, full implementation of the API CLI client (versus direct CLI client). This release is suitable for general use and is beta-quality software.
-
-###### v0.5
-
-First public release; fully implements the VM, network, and storage managers, the HTTP API, and the pvc-ansible framework for deploying and bootstrapping a cluster. This release is suitable for general use, though it is still alpha-quality software and should be expected to change significantly until 1.0 is released.
-
-###### v0.4
-
-Full implementation of virtual management and virtual networking functionality. Partial implementation of storage functionality.
-
-###### v0.3
-
-Basic implementation of virtual management functionality.
-
--- a/api-daemon/pvcapid/Daemon.py
+++ b/api-daemon/pvcapid/Daemon.py
@ -25,7 +25,7 @@ import yaml
 from distutils.util import strtobool as dustrtobool

 # Daemon version
-version = '0.9.39'
+version = '0.9.41'

 # API version
 API_VERSION = 1.0
--- a/api-daemon/pvcapid/flaskapi.py
+++ b/api-daemon/pvcapid/flaskapi.py
@ -2109,6 +2109,9 @@ class API_Network_Root(Resource):
                  enum:
                    - managed
                    - bridged
+                mtu:
+                  type: integer
+                  description: The MTU of the network, if set; empty otherwise
                domain:
                  type: string
                  description: The DNS domain of the network ("managed" networks only)
@ -2169,6 +2172,7 @@ class API_Network_Root(Resource):
        {'name': 'vni', 'required': True},
        {'name': 'description', 'required': True},
        {'name': 'nettype', 'choices': ('managed', 'bridged'), 'helptext': 'A valid nettype must be specified', 'required': True},
+        {'name': 'mtu'},
        {'name': 'domain'},
        {'name': 'name_servers'},
        {'name': 'ip4_network'},
@ -2205,6 +2209,10 @@ class API_Network_Root(Resource):
            enum:
              - managed
              - bridged
+          - in: query
+            name: mtu
+            type: integer
+            description: The MTU of the network; defaults to the underlying interface MTU if not set
          - in: query
            name: domain
            type: string
@ -2261,6 +2269,7 @@ class API_Network_Root(Resource):
            reqargs.get('vni', None),
            reqargs.get('description', None),
            reqargs.get('nettype', None),
+            reqargs.get('mtu', ''),
            reqargs.get('domain', None),
            name_servers,
            reqargs.get('ip4_network', None),
@ -2301,6 +2310,7 @@ class API_Network_Element(Resource):
    @RequestParser([
        {'name': 'description', 'required': True},
        {'name': 'nettype', 'choices': ('managed', 'bridged'), 'helptext': 'A valid nettype must be specified', 'required': True},
+        {'name': 'mtu'},
        {'name': 'domain'},
        {'name': 'name_servers'},
        {'name': 'ip4_network'},
@ -2332,6 +2342,10 @@ class API_Network_Element(Resource):
            enum:
              - managed
              - bridged
+          - in: query
+            name: mtu
+            type: integer
+            description: The MTU of the network; defaults to the underlying interface MTU if not set
          - in: query
            name: domain
            type: string
@ -2388,6 +2402,7 @@ class API_Network_Element(Resource):
            reqargs.get('vni', None),
            reqargs.get('description', None),
            reqargs.get('nettype', None),
+            reqargs.get('mtu', ''),
            reqargs.get('domain', None),
            name_servers,
            reqargs.get('ip4_network', None),
@ -2401,6 +2416,7 @@ class API_Network_Element(Resource):

    @RequestParser([
        {'name': 'description'},
+        {'name': 'mtu'},
        {'name': 'domain'},
        {'name': 'name_servers'},
        {'name': 'ip4_network'},
@ -2424,6 +2440,10 @@ class API_Network_Element(Resource):
            name: description
            type: string
            description: The description of the network
+          - in: query
+            name: mtu
+            type: integer
+            description: The MTU of the network
          - in: query
            name: domain
            type: string
@ -2484,6 +2504,7 @@ class API_Network_Element(Resource):
        return api_helper.net_modify(
            vni,
            reqargs.get('description', None),
+            reqargs.get('mtu', None),
            reqargs.get('domain', None),
            name_servers,
            reqargs.get('ip4_network', None),
--- a/api-daemon/pvcapid/helper.py
+++ b/api-daemon/pvcapid/helper.py
@ -927,7 +927,7 @@ def net_list(zkhandler, limit=None, is_fuzzy=True):


@ZKConnection(config)
-def net_add(zkhandler, vni, description, nettype, domain, name_servers,
+def net_add(zkhandler, vni, description, nettype, mtu, domain, name_servers,
            ip4_network, ip4_gateway, ip6_network, ip6_gateway,
            dhcp4_flag, dhcp4_start, dhcp4_end):
    """
@ -935,7 +935,7 @@ def net_add(zkhandler, vni, description, nettype, domain, name_servers,
    """
    if dhcp4_flag:
        dhcp4_flag = bool(strtobool(dhcp4_flag))
-    retflag, retdata = pvc_network.add_network(zkhandler, vni, description, nettype, domain, name_servers,
+    retflag, retdata = pvc_network.add_network(zkhandler, vni, description, nettype, mtu, domain, name_servers,
                                               ip4_network, ip4_gateway, ip6_network, ip6_gateway,
                                               dhcp4_flag, dhcp4_start, dhcp4_end)

@ -951,7 +951,7 @@ def net_add(zkhandler, vni, description, nettype, domain, name_servers,


@ZKConnection(config)
-def net_modify(zkhandler, vni, description, domain, name_servers,
+def net_modify(zkhandler, vni, description, mtu, domain, name_servers,
               ip4_network, ip4_gateway,
               ip6_network, ip6_gateway,
               dhcp4_flag, dhcp4_start, dhcp4_end):
@ -960,7 +960,7 @@ def net_modify(zkhandler, vni, description, domain, name_servers,
    """
    if dhcp4_flag is not None:
        dhcp4_flag = bool(strtobool(dhcp4_flag))
-    retflag, retdata = pvc_network.modify_network(zkhandler, vni, description, domain, name_servers,
+    retflag, retdata = pvc_network.modify_network(zkhandler, vni, description, mtu, domain, name_servers,
                                                  ip4_network, ip4_gateway, ip6_network, ip6_gateway,
                                                  dhcp4_flag, dhcp4_start, dhcp4_end)

--- a/25
+++ b/25
@ -15,31 +15,26 @@ echo "# Write the changelog below; comments will be ignored" >> ${changelog_file
 $EDITOR ${changelog_file}

 changelog="$( cat ${changelog_file} | grep -v '^#' | sed 's/^*/  */' )"
+rm ${changelog_file}

 sed -i "s,version = '${current_version}',version = '${new_version}'," node-daemon/pvcnoded/Daemon.py
 sed -i "s,version = '${current_version}',version = '${new_version}'," api-daemon/pvcapid/Daemon.py
 sed -i "s,version='${current_version}',version='${new_version}'," client-cli/setup.py
 echo ${new_version} > .version

-readme_tmpdir=$( mktemp -d )
-cp README.md ${readme_tmpdir}/
-cp docs/index.md ${readme_tmpdir}/
-pushd ${readme_tmpdir} &>/dev/null
+changelog_tmpdir=$( mktemp -d )
+cp CHANGELOG.md ${changelog_tmpdir}/
+pushd ${changelog_tmpdir} &>/dev/null

-echo -e "\n###### v${new_version}\n\n${changelog}" >> middle
+echo -e "\n###### [v${new_version}](https://github.com/parallelvirtualcluster/pvc/releases/tag/v${new_version})\n\n${changelog}" >> middle

-csplit README.md "/## Changelog/1" &>/dev/null
-cat xx00 middle xx01 > README.md
-rm xx00 xx01
-
-csplit index.md "/## Changelog/1" &>/dev/null
-cat xx00 middle xx01 > index.md
+csplit CHANGELOG.md "/## PVC Changelog/1" &>/dev/null
+cat xx00 middle xx01 > CHANGELOG.md
 rm xx00 xx01

 popd &>/dev/null
-mv ${readme_tmpdir}/README.md README.md
-mv ${readme_tmpdir}/index.md docs/index.md
-rm -r ${readme_tmpdir}
+mv ${changelog_tmpdir}/CHANGELOG.md CHANGELOG.md
+rm -r ${changelog_tmpdir}

 deb_changelog_orig="$( cat debian/changelog )"
 deb_changelog_new="pvc (${new_version}-0) unstable; urgency=high\n\n${changelog}\n\n -- $( git config --get user.name ) <$( git config --get user.email )>  $( date --rfc-email )\n"
@ -49,7 +44,7 @@ echo -e "${deb_changelog_new}" >> ${deb_changelog_file}
 echo -e "${deb_changelog_orig}" >> ${deb_changelog_file}
 mv ${deb_changelog_file} debian/changelog

-git add node-daemon/pvcnoded/Daemon.py api-daemon/pvcapid/Daemon.py client-cli/setup.py README.md docs/index.md debian/changelog .version
+git add node-daemon/pvcnoded/Daemon.py api-daemon/pvcapid/Daemon.py client-cli/setup.py debian/changelog CHANGELOG.md .version
 git commit -v

 echo
--- a/client-cli/pvc/cli_lib/network.py
+++ b/client-cli/pvc/cli_lib/network.py
@ -100,7 +100,7 @@ def net_list(config, limit):
        return False, response.json().get('message', '')


-def net_add(config, vni, description, nettype, domain, name_servers, ip4_network, ip4_gateway, ip6_network, ip6_gateway, dhcp4_flag, dhcp4_start, dhcp4_end):
+def net_add(config, vni, description, nettype, mtu, domain, name_servers, ip4_network, ip4_gateway, ip6_network, ip6_gateway, dhcp4_flag, dhcp4_start, dhcp4_end):
    """
    Add new network

@ -112,6 +112,7 @@ def net_add(config, vni, description, nettype, domain, name_servers, ip4_network
        'vni': vni,
        'description': description,
        'nettype': nettype,
+        'mtu': mtu,
        'domain': domain,
        'name_servers': name_servers,
        'ip4_network': ip4_network,
@ -132,7 +133,7 @@ def net_add(config, vni, description, nettype, domain, name_servers, ip4_network
    return retstatus, response.json().get('message', '')


-def net_modify(config, net, description, domain, name_servers, ip4_network, ip4_gateway, ip6_network, ip6_gateway, dhcp4_flag, dhcp4_start, dhcp4_end):
+def net_modify(config, net, description, mtu, domain, name_servers, ip4_network, ip4_gateway, ip6_network, ip6_gateway, dhcp4_flag, dhcp4_start, dhcp4_end):
    """
    Modify a network

@ -143,6 +144,8 @@ def net_modify(config, net, description, domain, name_servers, ip4_network, ip4_
    params = dict()
    if description is not None:
        params['description'] = description
+    if mtu is not None:
+        params['mtu'] = mtu
    if domain is not None:
        params['domain'] = domain
    if name_servers is not None:
@ -519,6 +522,7 @@ def format_info(config, network_information, long_output):
    # Basic information
    ainformation.append('{}VNI:{}            {}'.format(ansiprint.purple(), ansiprint.end(), network_information['vni']))
    ainformation.append('{}Type:{}           {}'.format(ansiprint.purple(), ansiprint.end(), network_information['type']))
+    ainformation.append('{}MTU:{}            {}'.format(ansiprint.purple(), ansiprint.end(), network_information['mtu']))
    ainformation.append('{}Description:{}    {}'.format(ansiprint.purple(), ansiprint.end(), network_information['description']))
    if network_information['type'] == 'managed':
        ainformation.append('{}Domain:{}         {}'.format(ansiprint.purple(), ansiprint.end(), network_information['domain']))
@ -575,6 +579,7 @@ def format_list(config, network_list):
    net_vni_length = 5
    net_description_length = 12
    net_nettype_length = 8
+    net_mtu_length = 4
    net_domain_length = 6
    net_v6_flag_length = 6
    net_dhcp6_flag_length = 7
@ -589,6 +594,10 @@ def format_list(config, network_list):
        _net_description_length = len(network_information['description']) + 1
        if _net_description_length > net_description_length:
            net_description_length = _net_description_length
+        # mtu column
+        _net_mtu_length = len(str(network_information['mtu'])) + 1
+        if _net_mtu_length > net_mtu_length:
+            net_mtu_length = _net_mtu_length
        # domain column
        _net_domain_length = len(network_information['domain']) + 1
        if _net_domain_length > net_domain_length:
@ -599,14 +608,15 @@ def format_list(config, network_list):
        bold=ansiprint.bold(),
        end_bold=ansiprint.end(),
        networks_header_length=net_vni_length + net_description_length + 1,
-        config_header_length=net_nettype_length + net_domain_length + net_v6_flag_length + net_dhcp6_flag_length + net_v4_flag_length + net_dhcp4_flag_length + 6,
+        config_header_length=net_nettype_length + net_mtu_length + net_domain_length + net_v6_flag_length + net_dhcp6_flag_length + net_v4_flag_length + net_dhcp4_flag_length + 7,
        networks_header='Networks ' + ''.join(['-' for _ in range(9, net_vni_length + net_description_length)]),
-        config_header='Config ' + ''.join(['-' for _ in range(7, net_nettype_length + net_domain_length + net_v6_flag_length + net_dhcp6_flag_length + net_v4_flag_length + net_dhcp4_flag_length + 5)]))
+        config_header='Config ' + ''.join(['-' for _ in range(7, net_nettype_length + net_mtu_length + net_domain_length + net_v6_flag_length + net_dhcp6_flag_length + net_v4_flag_length + net_dhcp4_flag_length + 6)]))
    )
    network_list_output.append('{bold}\
 {net_vni: <{net_vni_length}} \
 {net_description: <{net_description_length}} \
 {net_nettype: <{net_nettype_length}} \
+{net_mtu: <{net_mtu_length}} \
 {net_domain: <{net_domain_length}}  \
 {net_v6_flag: <{net_v6_flag_length}} \
 {net_dhcp6_flag: <{net_dhcp6_flag_length}} \
@ -618,6 +628,7 @@ def format_list(config, network_list):
        net_vni_length=net_vni_length,
        net_description_length=net_description_length,
        net_nettype_length=net_nettype_length,
+        net_mtu_length=net_mtu_length,
        net_domain_length=net_domain_length,
        net_v6_flag_length=net_v6_flag_length,
        net_dhcp6_flag_length=net_dhcp6_flag_length,
@ -626,6 +637,7 @@ def format_list(config, network_list):
        net_vni='VNI',
        net_description='Description',
        net_nettype='Type',
+        net_mtu='MTU',
        net_domain='Domain',
        net_v6_flag='IPv6',
        net_dhcp6_flag='DHCPv6',
@ -649,6 +661,7 @@ def format_list(config, network_list):
 {net_vni: <{net_vni_length}} \
 {net_description: <{net_description_length}} \
 {net_nettype: <{net_nettype_length}} \
+{net_mtu: <{net_mtu_length}} \
 {net_domain: <{net_domain_length}}  \
 {v6_flag_colour}{net_v6_flag: <{net_v6_flag_length}}{colour_off} \
 {dhcp6_flag_colour}{net_dhcp6_flag: <{net_dhcp6_flag_length}}{colour_off} \
@ -660,6 +673,7 @@ def format_list(config, network_list):
            net_vni_length=net_vni_length,
            net_description_length=net_description_length,
            net_nettype_length=net_nettype_length,
+            net_mtu_length=net_mtu_length,
            net_domain_length=net_domain_length,
            net_v6_flag_length=net_v6_flag_length,
            net_dhcp6_flag_length=net_dhcp6_flag_length,
@ -668,6 +682,7 @@ def format_list(config, network_list):
            net_vni=network_information['vni'],
            net_description=network_information['description'],
            net_nettype=network_information['type'],
+            net_mtu=network_information['mtu'],
            net_domain=network_information['domain'],
            net_v6_flag=v6_flag,
            v6_flag_colour=v6_flag_colour,
--- a/client-cli/pvc/pvc.py
+++ b/client-cli/pvc/pvc.py
@ -1874,6 +1874,11 @@ def cli_network():
    type=click.Choice(['managed', 'bridged']),
    help='Network type; managed networks control IP addressing; bridged networks are simple vLAN bridges. All subsequent options are unused for bridged networks.'
 )
+@click.option(
+    '-m', '--mtu', 'mtu',
+    default='',
+    help='MTU of the network interfaces.'
+)
@click.option(
    '-n', '--domain', 'domain',
    default=None,
@ -1924,10 +1929,12 @@ def cli_network():
    'vni'
 )
@cluster_req
-def net_add(vni, description, nettype, domain, ip_network, ip_gateway, ip6_network, ip6_gateway, dhcp_flag, dhcp_start, dhcp_end, name_servers):
+def net_add(vni, description, nettype, mtu, domain, ip_network, ip_gateway, ip6_network, ip6_gateway, dhcp_flag, dhcp_start, dhcp_end, name_servers):
    """
    Add a new virtual network with VXLAN identifier VNI.

+    NOTE: The MTU must be equal to, or less than, the underlying device MTU (either the node 'bridge_mtu' for bridged networks, or the node 'cluster_mtu' minus 50 for managed networks). Is only required if the device MTU should be lower than the underlying physical device MTU for compatibility. If unset, defaults to the underlying device MTU which will be set explcitly when the network is added to the nodes.
+
    Examples:

    pvc network add 101 --description my-bridged-net --type bridged
@ -1941,7 +1948,7 @@ def net_add(vni, description, nettype, domain, ip_network, ip_gateway, ip6_netwo
    IPv6 is fully supported with --ipnet6 and --gateway6 in addition to or instead of IPv4. PVC will configure DHCPv6 in a semi-managed configuration for the network if set.
    """

-    retcode, retmsg = pvc_network.net_add(config, vni, description, nettype, domain, name_servers, ip_network, ip_gateway, ip6_network, ip6_gateway, dhcp_flag, dhcp_start, dhcp_end)
+    retcode, retmsg = pvc_network.net_add(config, vni, description, nettype, mtu, domain, name_servers, ip_network, ip_gateway, ip6_network, ip6_gateway, dhcp_flag, dhcp_start, dhcp_end)
    cleanup(retcode, retmsg)


@ -1954,6 +1961,11 @@ def net_add(vni, description, nettype, domain, ip_network, ip_gateway, ip6_netwo
    default=None,
    help='Description of the network; must be unique and not contain whitespace.'
 )
+@click.option(
+    '-m', '--mtu', 'mtu',
+    default=None,
+    help='MTU of the network interfaces.'
+)
@click.option(
    '-n', '--domain', 'domain',
    default=None,
@ -2004,16 +2016,18 @@ def net_add(vni, description, nettype, domain, ip_network, ip_gateway, ip6_netwo
    'vni'
 )
@cluster_req
-def net_modify(vni, description, domain, name_servers, ip6_network, ip6_gateway, ip4_network, ip4_gateway, dhcp_flag, dhcp_start, dhcp_end):
+def net_modify(vni, description, mtu, domain, name_servers, ip6_network, ip6_gateway, ip4_network, ip4_gateway, dhcp_flag, dhcp_start, dhcp_end):
    """
    Modify details of virtual network VNI. All fields optional; only specified fields will be updated.

+    NOTE: The MTU must be equal to, or less than, the underlying device MTU (either the node 'bridge_mtu' for bridged networks, or the node 'cluster_mtu' minus 50 for managed networks). Is only required if the device MTU should be lower than the underlying physical device MTU for compatibility. To reset an explicit MTU to the default underlying device MTU, specify '--mtu' with a quoted empty string argument.
+
    Example:

    pvc network modify 1001 --gateway 10.1.1.1 --dhcp
    """

-    retcode, retmsg = pvc_network.net_modify(config, vni, description, domain, name_servers, ip4_network, ip4_gateway, ip6_network, ip6_gateway, dhcp_flag, dhcp_start, dhcp_end)
+    retcode, retmsg = pvc_network.net_modify(config, vni, description, mtu, domain, name_servers, ip4_network, ip4_gateway, ip6_network, ip6_gateway, dhcp_flag, dhcp_start, dhcp_end)
    cleanup(retcode, retmsg)


--- a/client-cli/setup.py
+++ b/client-cli/setup.py
@ -2,7 +2,7 @@ from setuptools import setup

 setup(
    name='pvc',
-    version='0.9.39',
+    version='0.9.41',
    packages=['pvc', 'pvc.cli_lib'],
    install_requires=[
        'Click',
--- a/daemon-common/migrations/versions/6.json
+++ b/daemon-common/migrations/versions/6.json
@ -0,0 +1 @@
+{"version": "6", "root": "", "base": {"root": "", "schema": "/schema", "schema.version": "/schema/version", "config": "/config", "config.maintenance": "/config/maintenance", "config.primary_node": "/config/primary_node", "config.primary_node.sync_lock": "/config/primary_node/sync_lock", "config.upstream_ip": "/config/upstream_ip", "config.migration_target_selector": "/config/migration_target_selector", "cmd": "/cmd", "cmd.node": "/cmd/nodes", "cmd.domain": "/cmd/domains", "cmd.ceph": "/cmd/ceph", "logs": "/logs", "node": "/nodes", "domain": "/domains", "network": "/networks", "storage": "/ceph", "storage.util": "/ceph/util", "osd": "/ceph/osds", "pool": "/ceph/pools", "volume": "/ceph/volumes", "snapshot": "/ceph/snapshots"}, "logs": {"node": "", "messages": "/messages"}, "node": {"name": "", "keepalive": "/keepalive", "mode": "/daemonmode", "data.active_schema": "/activeschema", "data.latest_schema": "/latestschema", "data.static": "/staticdata", "data.pvc_version": "/pvcversion", "running_domains": "/runningdomains", "count.provisioned_domains": "/domainscount", "count.networks": "/networkscount", "state.daemon": "/daemonstate", "state.router": "/routerstate", "state.domain": "/domainstate", "cpu.load": "/cpuload", "vcpu.allocated": "/vcpualloc", "memory.total": "/memtotal", "memory.used": "/memused", "memory.free": "/memfree", "memory.allocated": "/memalloc", "memory.provisioned": "/memprov", "ipmi.hostname": "/ipmihostname", "ipmi.username": "/ipmiusername", "ipmi.password": "/ipmipassword", "sriov": "/sriov", "sriov.pf": "/sriov/pf", "sriov.vf": "/sriov/vf"}, "sriov_pf": {"phy": "", "mtu": "/mtu", "vfcount": "/vfcount"}, "sriov_vf": {"phy": "", "pf": "/pf", "mtu": "/mtu", "mac": "/mac", "phy_mac": "/phy_mac", "config": "/config", "config.vlan_id": "/config/vlan_id", "config.vlan_qos": "/config/vlan_qos", "config.tx_rate_min": "/config/tx_rate_min", "config.tx_rate_max": "/config/tx_rate_max", "config.spoof_check": "/config/spoof_check", "config.link_state": "/config/link_state", "config.trust": "/config/trust", "config.query_rss": "/config/query_rss", "pci": "/pci", "pci.domain": "/pci/domain", "pci.bus": "/pci/bus", "pci.slot": "/pci/slot", "pci.function": "/pci/function", "used": "/used", "used_by": "/used_by"}, "domain": {"name": "", "xml": "/xml", "state": "/state", "profile": "/profile", "stats": "/stats", "node": "/node", "last_node": "/lastnode", "failed_reason": "/failedreason", "storage.volumes": "/rbdlist", "console.log": "/consolelog", "console.vnc": "/vnc", "meta.autostart": "/node_autostart", "meta.migrate_method": "/migration_method", "meta.node_selector": "/node_selector", "meta.node_limit": "/node_limit", "meta.tags": "/tags", "migrate.sync_lock": "/migrate_sync_lock"}, "tag": {"name": "", "type": "/type", "protected": "/protected"}, "network": {"vni": "", "type": "/nettype", "mtu": "/mtu", "rule": "/firewall_rules", "rule.in": "/firewall_rules/in", "rule.out": "/firewall_rules/out", "nameservers": "/name_servers", "domain": "/domain", "reservation": "/dhcp4_reservations", "lease": "/dhcp4_leases", "ip4.gateway": "/ip4_gateway", "ip4.network": "/ip4_network", "ip4.dhcp": "/dhcp4_flag", "ip4.dhcp_start": "/dhcp4_start", "ip4.dhcp_end": "/dhcp4_end", "ip6.gateway": "/ip6_gateway", "ip6.network": "/ip6_network", "ip6.dhcp": "/dhcp6_flag"}, "reservation": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname"}, "lease": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname", "expiry": "/expiry", "client_id": "/clientid"}, "rule": {"description": "", "rule": "/rule", "order": "/order"}, "osd": {"id": "", "node": "/node", "device": "/device", "db_device": "/db_device", "stats": "/stats"}, "pool": {"name": "", "pgs": "/pgs", "stats": "/stats"}, "volume": {"name": "", "stats": "/stats"}, "snapshot": {"name": "", "stats": "/stats"}}
--- a/daemon-common/network.py
+++ b/daemon-common/network.py
@ -135,6 +135,7 @@ def getNetworkACLs(zkhandler, vni, _direction):
 def getNetworkInformation(zkhandler, vni):
    description = zkhandler.read(('network', vni))
    nettype = zkhandler.read(('network.type', vni))
+    mtu = zkhandler.read(('network.mtu', vni))
    domain = zkhandler.read(('network.domain', vni))
    name_servers = zkhandler.read(('network.nameservers', vni))
    ip6_network = zkhandler.read(('network.ip6.network', vni))
@ -151,6 +152,7 @@ def getNetworkInformation(zkhandler, vni):
        'vni': int(vni),
        'description': description,
        'type': nettype,
+        'mtu': mtu,
        'domain': domain,
        'name_servers': name_servers.split(','),
        'ip6': {
@ -235,7 +237,7 @@ def isValidIP(ipaddr):
 #
 # Direct functions
 #
-def add_network(zkhandler, vni, description, nettype,
+def add_network(zkhandler, vni, description, nettype, mtu,
                domain, name_servers, ip4_network, ip4_gateway, ip6_network, ip6_gateway,
                dhcp4_flag, dhcp4_start, dhcp4_end):
    # Ensure start and end DHCP ranges are set if the flag is set
@ -267,6 +269,7 @@ def add_network(zkhandler, vni, description, nettype,
    result = zkhandler.write([
        (('network', vni), description),
        (('network.type', vni), nettype),
+        (('network.mtu', vni), mtu),
        (('network.domain', vni), domain),
        (('network.nameservers', vni), name_servers),
        (('network.ip6.network', vni), ip6_network),
@ -290,13 +293,15 @@ def add_network(zkhandler, vni, description, nettype,
        return False, 'ERROR: Failed to add network.'


-def modify_network(zkhandler, vni, description=None, domain=None, name_servers=None,
+def modify_network(zkhandler, vni, description=None, mtu=None, domain=None, name_servers=None,
                   ip4_network=None, ip4_gateway=None, ip6_network=None, ip6_gateway=None,
                   dhcp4_flag=None, dhcp4_start=None, dhcp4_end=None):
    # Add the modified parameters to Zookeeper
    update_data = list()
    if description is not None:
        update_data.append((('network', vni), description))
+    if mtu is not None:
+        update_data.append((('network.mtu', vni), mtu))
    if domain is not None:
        update_data.append((('network.domain', vni), domain))
    if name_servers is not None:
--- a/daemon-common/zkhandler.py
+++ b/daemon-common/zkhandler.py
@ -466,7 +466,7 @@ class ZKHandler(object):
 #
 class ZKSchema(object):
    # Current version
-    _version = 5
+    _version = 6

    # Root for doing nested keys
    _schema_root = ''
@ -595,6 +595,7 @@ class ZKSchema(object):
        'network': {
            'vni': '',  # The root key
            'type': '/nettype',
+            'mtu': '/mtu',
            'rule': '/firewall_rules',
            'rule.in': '/firewall_rules/in',
            'rule.out': '/firewall_rules/out',
--- a/debian/changelog
+++ b/debian/changelog
@ -1,3 +1,17 @@
+pvc (0.9.41-0) unstable; urgency=high
+
+  * Fixes a bad conditional check in IPMI verification
+  * Implements per-network MTU configuration; NOTE: Requires new keys in pvcnoded.yaml (`bridge_mtu`) and Ansible group_vars (`pvc_bridge_mtu`)
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Sat, 09 Oct 2021 19:39:21 -0400
+
+pvc (0.9.40-0) unstable; urgency=high
+
+  * [Docs] Documentation updates for new Changelog file
+  * [Node Daemon] Fixes bug with schema updates
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Thu, 07 Oct 2021 14:42:04 -0400
+
 pvc (0.9.39-0) unstable; urgency=high

  * [Documentation] Update several documentation sections
--- a/docs/cluster-architecture.md
+++ b/docs/cluster-architecture.md
@ -2,23 +2,24 @@

 - [PVC Cluster Architecture considerations](#pvc-cluster-architecture-considerations)
  * [Node Specification](#node-specification)
-    - [n-1 Redundancy](#n-1-redundancy)
-    - [CPU](#cpu)
-    - [Memory](#memory)
-    - [Disk](#disk)
-    - [Network](#network)
-  * [PVC architecture](#pvc-architecture)
-    - [Operating System](#operating-system)
-    - [Ceph Storage Layout](#ceph-storage-layout)
-    - [Networks](#networks)
-      + [System Networks](#system-networks)
-      + [Client Networks](#client-networks)
-  * [Advanced Layouts](#advanced-layouts)
-    - [Coordinators versus Hypervisors](#coordinators-versus-hypervisors)
-    - [Georedundancy](#georedundancy)
-  * [Example System Diagrams](#example-system-diagrams)
-    - [Small 3-node cluster](#small-3-node-cluster)
-    - [Large 8-node cluster](#large-8-node-cluster)
+    + [n-1 Redundancy](#n-1-redundancy)
+    + [CPU](#cpu)
+    + [Memory](#memory)
+    + [Disk](#disk)
+    + [Network](#network)
+  * [PVC architecture](#pvc+architecture)
+    + [Operating System](#operating-system)
+    + [Ceph Storage Layout](#ceph-storage-layout)
+    + [Networks](#networks)
+      - [System Networks](#system+networks)
+      - [Client Networks](#client+networks)
+    + [Fencing and Recovery](#fencing-and-recovery)
+  * [Advanced Layouts](#advanced+layouts)
+    + [Coordinators versus Hypervisors](#coordinators-versus-hypervisors)
+    + [Georedundancy](#georedundancy)
+  * [Example System Diagrams](#example+system-diagrams)
+    + [Small 3-node cluster](#small-3-node-cluster)
+    + [Large 8-node cluster](#large-8-node-cluster)

 This document contains considerations the administrator should make when preparing for and building a PVC cluster. It is important that prospective PVC administrators read this document *thoroughly* before deploying a cluster to ensure they understand the requirements, caveats, and important details about how PVC operates.

@ -36,11 +37,12 @@ The following table provides recommended minimum specifications for each compone
 | System disk (SSD/HDD/USB/SD/eMMC) | 2x 100GB RAID-1 |
 | Data disk (SSD only) | 1x 400GB |
 | Network interfaces | 2x 10Gbps (LACP LAG) |
-| Total CPU cores (healthy) | 24 |
-| Total CPU cores (n-1) | 16 |
-| Total RAM (healthy) | 96GB |
-| Total RAM (n-1) | 64GB |
-| Total disk space | 400GB |
+| Remote IPMI-over-IP | Available and connected |
+| Total CPU cores (3 nodes healthy) | 24 |
+| Total CPU cores (3 nodes n-1) | 16 |
+| Total RAM (3 nodes healthy) | 96GB |
+| Total RAM (3 nodes n-1) | 64GB |
+| Total disk space (3 nodes) | 400GB |

 For testing, or low-budget homelab applications, some aspects can be further tuned down, however consider the following sections carefully.

@ -156,6 +158,12 @@ Because PVC makes extensive use of cross-node communications, high-throughput an

 A minimum of 2 network interfaces is recommended. These should then be combined into a logical aggregate (LAG) using 802.3ad (LACP) to provide redundant links and a boost in available bandwidth. Additional NICs can also be used to separate discrete parts of the networking stack, which will be discussed below.

+#### Remote IPMI-over-IP
+
+IPMI provides a method to manage the physical chassis' of nodes from outside of their operating system. Common implementations include Dell iDRAC, HP iLO, Cisco CIMC, and others.
+
+PVC nodes in production deployments should always feature an IPMI-over-IP interface of some kind, which is then reachable either in, or via, the Upstream system network (see [System Networks](#system-networks)). This requirement is discussed in more detail during the [Fencing and Recovery](#fencing-and-recovery) section below.
+
 ## PVC Architecture

 ### Operating System
@ -182,7 +190,7 @@ Non-default values can also be set at pool creation time. For instance, one coul

 Replication levels cannot be changed within PVC once a pool is created, however they can be changed via manual Ceph commands on a coordinator should the administrator require this, though discussion of this process is outside of the scope of this documentation. The administrator should carefully consider sizing, failure domains, and performance when first selecting storage devices and creating pools, to ensure the right level of resiliency versus data usage for their use-case and planned cluster size.

-## Networks
+### Networks

 At a minimum, a production PVC cluster should use at least two 10Gbps Ethernet interfaces, connected in an LACP or active-backup bond on one or more switches. On top of this bond, the various cluster networks should be configured as 802.3q vLANs. PVC is be able to support configurations without bonding or 802.1q vLAN support, using multiple physical interfaces and no bridged client networks, but this is strongly discouraged due to the added complexity this introduces; the switches chosen for the cluster should include these requirements as a minimum.

@ -292,6 +300,40 @@ Generally speaking, SR-IOV connections are not recommended unless there is a goo

 Future PVC versions may support other client network types, such as direct-routing between VMs.

+### Fencing and Recovery
+
+Self-management and self-healing are important components of PVC's design, and to accomplish this, PVC contains automated fencing and recovery functions to handle situations where nodes crash or become unreachable. PVC is then able, if properly configured, to directly power-cycle the failed node, and bring up any VMs that were running on it on the remaining hypervisors. This ensures that, while there might be a few minutes of downtime for VMs, they are recovered as quickly as possible without human intervention.
+
+To operate correctly, these functions require each node in the cluster to have a functional IPMI-over-IP setup with a configured user who is able to perform chassis power commands. This differs depending on the chassis manufacturer and model, and should be tested prior to deploying any production cluster. If IPMI is not configured correctly at node startup, the daemon will warn and disable automatic recovery of the node. The IPMI should be present in the Upstream system network (see [System Networks](#system-networks) above), or in another secured network which is reachable from the Upstream system network, whichever is more convenient for the layout of the networks.
+
+The general process is divided into 3 sections: detecting node failures, fencing nodes, and recovering from fenced nodes.
+
+#### Detecting Failed Nodes
+
+Within the PVC configuration, each node has 3 settings which determine the failure detection time. The first is the `keepalive_interval` setting. This is normally set to 5 seconds, and is the interval at which the node daemon of each node sends its keepalives (as well as gathers statistics about running VMs, Ceph components, etc.). This interval should never need to be changed, but is configurable for maximum flexibility in corner cases. During each keepalive, the node updates a specific key in the Zookeeper cluster with the current UNIX timestamp, which determines when the node was last alive. During their own keepalives, the other nodes check their peers' timestamps to confirm if they are updating normally. Note that, due to this happening during the peer keepalives, if all nodes lose contact with the Zookeeper database, they will *not* immediately begin fencing each other, since the keepalives will not complete; they will, however, upon recovery, jump immediately to the next section when they all realize that their last keepalives were over the threshold, and this situation is discussed there.
+
+The second option is the `fence_intervals` setting. This option determines how many keepalive intervals a node can miss before it is marked `dead` and a fencing sequence started. This is normally set to 6 intervals, which combined with the 5 second `keepalive_interval`, gives a total of 30 seconds (+/- up to another 5 second `keepalive_interval` for peers should they not line up) for the node to be without updates before fencing begins.
+
+The third setting is optional, and is best used in situations where the IPMI connectivity of a node is excessively flaky or can be impaired (e.g. georedundant clusters), or where VM uptime is more important than the burden of recovering from a split-brain situation, and is not as extensively tested. This option is `suicide_intervals`, and if set to a non-0 value, is the number of keepalive intervals before a node *itself* determines that it should forcibly power itself off, which should always be equal to or less than the normal `fence_intervals` setting. Naturally, the node must be somewhat functional to do this, and this can go very wrong, so using this option is not normally recommended.
+
+#### Fencing Nodes
+
+Once the cluster, and specifically one node in the cluster, has determined that a given node is `dead` due to a lack of keepalives, the fencing process starts. This spawns a dedicated child thread within the node daemon of the detecting node, which continually monitors the state of the `dead` node and then performs the fence.
+
+During the `dead` process, the failed node has 6 chances, called "saving throws", at `keepalive_interval` second windows, to send another keepalive before it is fenced. This additional, fixed, delay helps ensure that the cluster will gracefully recover from intermittent network failures or loss of Zookeeper contact, by providing nodes up to another 6 keepalive intervals to save themselves once the fence timer actually begins. This bring the total time, with default options, of a node stopping contact to a node being fenced, to between 60 and 65 seconds. This duration is considered by the author an acceptable compromise between speedy recovery and avoiding false positives (and hence larger outages).
+
+Once a node has been marked `dead` and has failed its 6 "saving throws", the fence process triggers an IPMI chassis reset sequence. First, the node is issued the standard IPMI `chassis power reset` command to trigger a cold system reset. Next, it waits a fixed 1 second and then issues a `chassis power on` signal to ensure the node is powered on (just in case it had already shut itself off). The node then waits a fixed 2 seconds, and then checks the current `chassis power status`. Using the results of these 3 commands, PVC is then able to determine with near certainty whether the node has truly been forced offline or not, and it can proceed to the next step.
+
+#### Recovery from Node Fences
+
+Once a node has been fenced, successfully or not, the system waits for one keepalive interval before proceeding.
+
+The cluster then determines what to do based both on the result of the fence (whether the node was determined to have been successfully cold-reset or not) and on two additional configuration values. The first, `successful_fence`, specifies what action to take when the fence was successful, and is either `migrate` (VMs to other nodes), the default, or `None` (no action). The second, `failed_fence`, is an identical choice for when the fence was unsuccessful, and defaults to `None`.
+
+If the fence was successful and `successful_fence` is set to `None`, then no migration takes place and the VMs on the fenced node will remain offline until the node recovers. If instead `successful_fence` is set to the default of `migrate`, the system will then begin migrating (and hence, starting) VMs that were active on the failed node to other nodes in the cluster. During this special `fence-flush` action, any stale RBD locks on the storage volumes are forcibly cleared, and this is considered safe since the fenced node is determined to have successfully been powered off and the VMs thus terminated. Once all VMs are migrated, the fenced node will then be set to a normal `flushed` state, as if it had been cleanly flushed before powering off. If and when the node returns to active, healthy service, either automatically (if the reset cleared the fault condition) or after human intervention, VMs can then migrate back and the cluster can resume normal operation; otherwise the cluster will remain in the degraded state until corrected.
+
+If the fence was unsuccessful and `failed_fence` is set to the default of `None`, no automatic recovery takes place, since the cluster cannot determine that it is safe to do so. This would most commonly occur during network partitions where the `dead` node potentially remains up with VMs running on it, and the cluster is now in a split-brain situation. The `suicide_interval` option mentioned above is provided for this specific situation, and would allow the administrator to set the `failed_fence` action to `migrate` as well, as they could be somewhat confident that the node will have forcibly terminated itself. However due to the inherent potential for danger in this scenario, it is recommended to leave these options at their defaults, and handle such situations manually instead, as well as ensuring proper network design to avoid the potential for such split-brain situations to occur.
+
 ## Advanced Layouts

 ### Coordinators versus Hypervisors
--- a/docs/index.md
+++ b/docs/index.md
@ -8,7 +8,7 @@

 ## What is PVC?

-PVC is a virtual machine-based hyperconverged infrastructure (HCI) virtualization cluster solution that is fully Free Software, scalable, redundant, self-healing, self-managing, and designed for administrator simplicity. It is an alternative to other HCI solutions such as Harvester, Nutanix, and VMWare, as well as to other common virtualization stacks such as ProxMox and OpenStack.
+PVC is a Linux KVM-based hyperconverged infrastructure (HCI) virtualization cluster solution that is fully Free Software, scalable, redundant, self-healing, self-managing, and designed for administrator simplicity. It is an alternative to other HCI solutions such as Harvester, Nutanix, and VMWare, as well as to other common virtualization stacks such as ProxMox and OpenStack.

 PVC is a complete HCI solution, built from well-known and well-trusted Free Software tools, to assist an administrator in creating and managing a cluster of servers to run virtual machines, as well as self-managing several important aspects including storage failover, node failure and recovery, virtual machine failure and recovery, and network plumbing. It is designed to act consistently, reliably, and unobtrusively, letting the administrator concentrate on more important things.

@ -40,6 +40,11 @@ The core node and API daemons, as well as the CLI API client, are written in Pyt
 To get started with PVC, please see the [About](https://parallelvirtualcluster.readthedocs.io/en/latest/about/) page for general information about the project, and the [Getting Started](https://parallelvirtualcluster.readthedocs.io/en/latest/getting-started/) page for details on configuring your first cluster.


+## Changelog
+
+View the changelog in [CHANGELOG.md](https://github.com/parallelvirtualcluster/pvc/blob/master/CHANGELOG.md).
+
+
 ## Screenshots

 While PVC's API and internals aren't very screenshot-worthy, here is some example output of the CLI tool.
@ -51,297 +56,3 @@ While PVC's API and internals aren't very screenshot-worthy, here is some exampl
 <p><img alt="VM listing and migration" src="images/pvc-migration.png"/><br/><i>Listing a limited set of VMs and migrating one with status updates</i></p>

 <p><img alt="Node logs" src="images/pvc-nodelog.png"/><br/><i>Viewing the logs of a node (keepalives and VM [un]migration)</i></p>
-
-
-## Changelog
-
-###### v0.9.39
-
-  * [Documentation] Update several documentation sections
-  * [API Daemon/CLI Client] Add negate flag for VM option limits (node, tag, state)
-  * [Build] Add linting check to build-and-deploy.sh
-
-###### v0.9.38
-
-  * [All] Significantly improve storage benchmark format and reporting
-
-###### v0.9.37
-
-  * [All] Adds support for configurable OSD DB size ratios
-  * [Node Daemon] Fixes bugs with OSD creation
-  * [Node Daemon] Fixes exception bugs in CephInstance
-  * [CLI Client] Adjusts descriptions around Ceph OSDs
-  * [Node Daemon] Fixes ordering of pvc-flush unit
-  * [Node Daemon] Fixes bugs in fence handling and libvirt keepalive
-  * [Node Daemon] Simplifies locking for and speeds up VM migrations
-  * [Node Daemon] Fixes bugs in queue get timeouts
-  * [API Daemon] Adjusts benchmark test jobs configuration and naming
-
-###### v0.9.36
-
-  * [Node Daemon] Fixes a bug during early cleanup
-  * [All] Adds support for OSD database/WAL block devices to improve Ceph performance; NOTE: Applies only to new OSDs
-
-###### v0.9.35
-
-  * [Node Daemon] Fixes several bugs and crashes in node daemon
-  * [General] Updates linting rules for newer Flake8 linter
-  * [Daemons/CLI client] Adds VM network and disk hot attach/detach support; NOTE: Changes the default behaviour of `pvc vm network add`/`remove` and `pvc vm volume add`/`remove`
-  * [API Daemon] Adds checks for pool size when resizing volumes
-  * [API Daemon] Adds checks for RAM and vCPU sizes when defining or modifying VMs
-
-###### v0.9.34
-
-  * [Provisioner] Adds support for filesystem arguments containing =
-  * [CLI Client] Fixes bug with pvc provisioner status output formatting
-  * [Node Daemon] Fixes minor typo in startup message
-
-###### v0.9.33
-
-  * [Node Daemon] A major refactoring of the node daemon
-  * [CLI Client] Fixes output errors if a node has no provisioner data
-  * [Packages] Fixes issues with including __pycache__ directories in .deb files
-
-###### v0.9.32
-
-  * [CLI Client] Fixes some incorrect colours in network lists
-  * [Documentation] Adds documentation screenshots of CLI client
-  * [Node Daemon] Fixes a bug if VM stats gathering fails
-
-###### v0.9.31
-
-  * [Packages] Cleans up obsolete Suggests lines
-  * [Node Daemon] Adjusts log text of VM migrations to show the correct source node
-  * [API Daemon] Adjusts the OVA importer to support floppy RASD types for compatability
-  * [API Daemon] Ensures that volume resize commands without a suffix get B appended
-  * [API Daemon] Removes the explicit setting of image-features in PVC; defaulting to the limited set has been moved to the ceph.conf configuration on nodes via PVC Ansible
-
-###### v0.9.30
-
-  * [Node Daemon] Fixes bug with schema validation
-
-###### v0.9.29
-
-  * [Node Daemon] Corrects numerous bugs with node logging framework
-
-###### v0.9.28
-
-  * [CLI Client] Revamp confirmation options for "vm modify" command
-
-###### v0.9.27
-
-  * [CLI Client] Fixes a bug with vm modify command when passed a file
-
-###### v0.9.26
-
-  * [Node Daemon] Corrects some bad assumptions about fencing results during hardware failures
-  * [All] Implements VM tagging functionality
-  * [All] Implements Node log access via PVC functionality
-
-###### v0.9.25
-
-  * [Node Daemon] Returns to Rados library calls for Ceph due to performance problems
-  * [Node Daemon] Adds a date output to keepalive messages
-  * [Daemons] Configures ZK connection logging only for persistent connections
-  * [API Provisioner] Add context manager-based chroot to Debootstrap example script
-  * [Node Daemon] Fixes a bug where shutdown daemon state was overwritten
-
-###### v0.9.24
-
-  * [Node Daemon] Removes Rados module polling of Ceph cluster and returns to command-based polling for timeout purposes, and removes some flaky return statements
-  * [Node Daemon] Removes flaky Zookeeper connection renewals that caused problems
-  * [CLI Client] Allow raw lists of clusters from `pvc cluster list`
-  * [API Daemon] Fixes several issues when getting VM data without stats
-  * [API Daemon] Fixes issues with removing VMs while disks are still in use (failed provisioning, etc.)
-
-###### v0.9.23
-
-  * [Daemons] Fixes a critical overwriting bug in zkhandler when schema paths are not yet valid
-  * [Node Daemon] Ensures the daemon mode is updated on every startup (fixes the side effect of the above bug in 0.9.22)
-
-###### v0.9.22
-
-  * [API Daemon] Drastically improves performance when getting large lists (e.g. VMs)
-  * [Daemons] Adds profiler functions for use in debug mode
-  * [Daemons] Improves reliability of ZK locking
-  * [Daemons] Adds the new logo in ASCII form to the Daemon startup message
-  * [Node Daemon] Fixes bug where VMs would sometimes not stop
-  * [Node Daemon] Code cleanups in various classes
-  * [Node Daemon] Fixes a bug when reading node schema data
-  * [All] Adds node PVC version information to the list output
-  * [CLI Client] Improves the style and formatting of list output including a new header line
-  * [API Worker] Fixes a bug that prevented the storage benchmark job from running
-
-###### v0.9.21
-
-  * [API Daemon] Ensures VMs stop before removing them
-  * [Node Daemon] Fixes a bug with VM shutdowns not timing out
-  * [Documentation] Adds information about georedundancy caveats
-  * [All] Adds support for SR-IOV NICs (hostdev and macvtap) and surrounding documentation
-  * [Node Daemon] Fixes a bug where shutdown aborted migrations unexpectedly
-  * [Node Daemon] Fixes a bug where the migration method was not updated realtime
-  * [Node Daemon] Adjusts the Patroni commands to remove reference to Zookeeper path
-  * [CLI Client] Adjusts several help messages and fixes some typos
-  * [CLI Client] Converts the CLI client to a proper Python module
-  * [API Daemon] Improves VM list performance
-  * [API Daemon] Adjusts VM list matching critera (only matches against the UUID if it's a full UUID)
-  * [API Worker] Fixes incompatibility between Deb 10 and 11 in launching Celery worker
-  * [API Daemon] Corrects several bugs with initialization command
-  * [Documentation] Adds a shiny new logo and revamps introduction text
-
-###### v0.9.20
-
-  * [Daemons] Implemented a Zookeeper schema handler and version 0 schema
-  * [Daemons] Completes major refactoring of codebase to make use of the schema handler
-  * [Daemons] Adds support for dynamic chema changges and "hot reloading" of pvcnoded processes
-  * [Daemons] Adds a functional testing script for verifying operation against a test cluster
-  * [Daemons, CLI] Fixes several minor bugs found by the above script
-  * [Daemons, CLI] Add support for Debian 11 "Bullseye"
-
-###### v0.9.19
-
-  * [CLI] Corrects some flawed conditionals
-  * [API] Disables SQLAlchemy modification tracking functionality (not used by us)
-  * [Daemons] Implements new zkhandler module for improved reliability and reusability
-  * [Daemons] Refactors some code to use new zkhandler module
-  * [API, CLI] Adds support for "none" migration selector (uses cluster default instead)
-  * [Daemons] Moves some configuration keys to new /config tree
-  * [Node Daemon] Increases initial lock timeout for VM migrations to avoid out-of-sync potential
-  * [Provisioner] Support storing and using textual cluster network labels ("upstream", "storage", "cluster") in templates
-  * [API] Avoid duplicating existing node states
-
-###### v0.9.18
-
-  * Adds VM rename functionality to API and CLI client
-
-###### v0.9.17
-
-  * [CLI] Fixes bugs in log follow output
-
-###### v0.9.16
-
-  * Improves some CLI help messages
-  * Skips empty local cluster in CLI
-  * Adjusts how confirmations happen during VM modify restarts
-  * Fixes bug around corrupted VM log files
-  * Fixes bug around subprocess pipe exceptions
-
-###### v0.9.15
-
-  * [CLI] Adds additional verification (--yes) to several VM management commands
-  * [CLI] Adds a method to override --yes/confirmation requirements via envvar (PVC_UNSAFE)
-  * [CLI] Adds description fields to PVC clusters in CLI
-
-###### v0.9.14
-
-  * Fixes bugs around cloned volume provisioning
-  * Fixes some minor visual bugs
-  * Minor license update (from GPL3+ to GPL3)
-  * Adds qemu-guest-agent support to provisioner-created VMs by default
-
-###### v0.9.13
-
-  * Adds nicer startup messages for daemons
-  * Adds additional API field for stored_bytes to pool stats
-  * Fixes sorting issues with snapshot lists
-  * Fixes missing increment/decrement of snapshot_count on volumes
-  * Fixes bad calls in pool element API endpoints
-  * Fixes inconsistent bytes_tohuman behaviour in daemons
-  * Adds validation and maximum volume size on creation (must be smaller than the pool free space)
-
-###### v0.9.12
-
-  * Fixes a bug in the pvcnoded service unit file causing a Zookeeper startup race condition
-
-###### v0.9.11
-
-  * Documentation updates
-  * Adds VNC information to VM info
-  * Goes back to external Ceph commands for disk usage
-
-###### v0.9.10
-
-  * Moves OSD stats uploading to primary, eliminating reporting failures while hosts are down
-  * Documentation updates
-  * Significantly improves RBD locking behaviour in several situations, eliminating cold-cluster start issues and failed VM boot-ups after crashes
-  * Fixes some timeout delays with fencing
-  * Fixes bug in validating YAML provisioner userdata
-
-###### v0.9.9
-
-  * Adds documentation updates
-  * Removes single-element list stripping and fixes surrounding bugs
-  * Adds additional fields to some API endpoints for ease of parsing by clients
-  * Fixes bugs with network configuration
-
-###### v0.9.8
-
-  * Adds support for cluster backup/restore
-  * Moves location of `init` command in CLI to make room for the above
-  * Cleans up some invalid help messages from the API
-
-###### v0.9.7
-
-  * Fixes bug with provisioner system template modifications
-
-###### v0.9.6
-
-  * Fixes bug with migrations
-
-###### v0.9.5
-
-  * Fixes bug with line count in log follow
-  * Fixes bug with disk stat output being None
-  * Adds short pretty health output
-  * Documentation updates
-
-###### v0.9.4
-
-  * Fixes major bug in OVA parser
-
-###### v0.9.3
-
-  * Fixes bugs with image & OVA upload parsing
-
-###### v0.9.2
-
-  * Major linting of the codebase with flake8; adds linting tools
-  * Implements CLI-based modification of VM vCPUs, memory, networks, and disks without directly editing XML
-  * Fixes bug where `pvc vm log -f` would show all 1000 lines before starting
-  * Fixes bug in default provisioner libvirt schema (`drive` -> `driver` typo)
-
-###### v0.9.1
-
-  * Added per-VM migration method feature
-  * Fixed bug with provisioner system template listing
-
-###### v0.9.0
-
-Numerous small improvements and bugfixes. This release is suitable for general use and is pre-release-quality software.
-
-This release introduces an updated version scheme; all future stable releases until 1.0.0 is ready will be made under this 0.9.z naming. This does not represent semantic versioning and all changes (feature, improvement, or bugfix) will be considered for inclusion in this release train.
-
-###### v0.8
-
-Numerous improvements and bugfixes. This release is suitable for general use and is pre-release-quality software.
-
-###### v0.7
-
-Numerous improvements and bugfixes, revamped documentation. This release is suitable for general use and is beta-quality software.
-
-###### v0.6
-
-Numerous improvements and bugfixes, full implementation of the provisioner, full implementation of the API CLI client (versus direct CLI client). This release is suitable for general use and is beta-quality software.
-
-###### v0.5
-
-First public release; fully implements the VM, network, and storage managers, the HTTP API, and the pvc-ansible framework for deploying and bootstrapping a cluster. This release is suitable for general use, though it is still alpha-quality software and should be expected to change significantly until 1.0 is released.
-
-###### v0.4
-
-Full implementation of virtual management and virtual networking functionality. Partial implementation of storage functionality.
-
-###### v0.3
-
-Basic implementation of virtual management functionality.
-
--- a/docs/manuals/ansible.md
+++ b/docs/manuals/ansible.md
@ -450,6 +450,7 @@ pvc_nodes:
    ipmi_password: "{{ passwd_ipmi_host }}"

 pvc_bridge_device: bondU
+pvc_bridge_mtu: 1500

 pvc_sriov_enable: True
 pvc_sriov_device:
@ -907,6 +908,12 @@ The IPMI password for the node management controller. Unless a per-host override

 The device name of the underlying network interface to be used for "bridged"-type client networks. For each "bridged"-type network, an IEEE 802.3q vLAN and bridge will be created on top of this device to pass these networks. In most cases, using the reflexive `networks['cluster']['raw_device']` or `networks['upstream']['raw_device']` from the Base role is sufficient.

+#### `pvc_bridge_mtu`
+
+* *required*
+
+The MTU of the underlying network interface to be used for "bridged"-type client networks. This is the maximum MTU such networks can use.
+
 #### `pvc_sriov_enable`

 * *optional*
--- a/docs/manuals/daemon.md
+++ b/docs/manuals/daemon.md
@ -146,6 +146,7 @@ pvc:
        console_log_lines: 1000
      networking:
        bridge_device: ens4
+        bridge_mtu: 1500
        sriov_enable: True
        sriov_device:
          - phy: ens1f0
@ -427,6 +428,13 @@ How many lines of VM console logs to keep in the Zookeeper database for each VM.

 The network interface device used to create Bridged client network vLANs on. For most clusters, should match the underlying device of the various static networks (e.g. `ens4` or `bond0`), though may also use a separate network interface.

+#### `system` → `configuration` → `networking` → `bridge_mtu`
+
+* *optional*
+* *requires* `functions` → `enable_networking`
+
+The network interface MTU for the Bridged client network device. This is the maximum MTU a bridged client network can use.
+
 #### `system` → `configuration` → `networking` → `sriov_enable`

 * *optional*, defaults to `False`
--- a/docs/manuals/swagger.json
+++ b/docs/manuals/swagger.json
@ -364,6 +364,10 @@
                    },
                    "type": "object"
                },
+                "mtu": {
+                    "description": "The MTU of the network, if set; empty otherwise",
+                    "type": "integer"
+                },
                "name_servers": {
                    "description": "The configured DNS nameservers of the network for NS records (\"managed\" networks only)",
                    "items": {
@ -1765,6 +1769,12 @@
                        "required": true,
                        "type": "string"
                    },
+                    {
+                        "description": "The MTU of the network; defaults to the underlying interface MTU if not set",
+                        "in": "query",
+                        "name": "mtu",
+                        "type": "integer"
+                    },
                    {
                        "description": "The DNS domain of the network (\"managed\" networks only)",
                        "in": "query",
@ -1910,6 +1920,12 @@
                        "required": true,
                        "type": "string"
                    },
+                    {
+                        "description": "The MTU of the network; defaults to the underlying interface MTU if not set",
+                        "in": "query",
+                        "name": "mtu",
+                        "type": "integer"
+                    },
                    {
                        "description": "The DNS domain of the network (\"managed\" networks only)",
                        "in": "query",
@ -1993,6 +2009,12 @@
                        "name": "description",
                        "type": "string"
                    },
+                    {
+                        "description": "The MTU of the network",
+                        "in": "query",
+                        "name": "mtu",
+                        "type": "integer"
+                    },
                    {
                        "description": "The DNS domain of the network (\"managed\" networks only)",
                        "in": "query",
--- a/node-daemon/pvcnoded.sample.yaml
+++ b/node-daemon/pvcnoded.sample.yaml
@ -161,6 +161,9 @@ pvc:
      networking:
        # bridge_device: Underlying device to use for bridged vLAN networks; usually the device of <cluster>
        bridge_device: ens4
+        # bridge_mtu: The MTU of the underlying device used for bridged vLAN networks, and thus the maximum
+        #             MTU of the overlying bridge devices.
+        bridge_mtu: 1500
        # sriov_enable: Enable or disable (default if absent) SR-IOV network support
        sriov_enable: False
        # sriov_device: Underlying device(s) to use for SR-IOV networks; can be bridge_device or other NIC(s)
--- a/node-daemon/pvcnoded/Daemon.py
+++ b/node-daemon/pvcnoded/Daemon.py
@ -48,7 +48,7 @@ import re
 import json

 # Daemon version
-version = '0.9.39'
+version = '0.9.41'


 ##########################################################
@ -142,7 +142,7 @@ def entrypoint():

        # Prevent any keepalive updates while this happens
        if keepalive_timer is not None:
-            pvcnoded.util.keepalive.stop_keepalive_timer()
+            pvcnoded.util.keepalive.stop_keepalive_timer(logger, keepalive_timer)
            sleep(1)

        # Perform the migration (primary only)
@ -316,7 +316,7 @@ def entrypoint():
    pvcnoded.util.networking.create_nft_configuration(logger, config)

    # Create our object dictionaries
-    logger.out('Setting up objects', state='i')
+    logger.out('Setting up objects', state='s')

    d_node = dict()
    node_list = list()
--- a/node-daemon/pvcnoded/objects/VXNetworkInstance.py
+++ b/node-daemon/pvcnoded/objects/VXNetworkInstance.py
@ -39,27 +39,37 @@ class VXNetworkInstance(object):
        self.cluster_dev = config['cluster_dev']
        self.cluster_mtu = config['cluster_mtu']
        self.bridge_dev = config['bridge_dev']
+        self.bridge_mtu = config['bridge_mtu']

        self.nettype = self.zkhandler.read(('network.type', self.vni))
        if self.nettype == 'bridged':
+            self.base_nic = 'vlan{}'.format(self.vni)
+            self.bridge_nic = 'vmbr{}'.format(self.vni)
+            self.max_mtu = self.bridge_mtu
            self.logger.out(
                'Creating new bridged network',
                prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
            )
            self.init_bridged()
        elif self.nettype == 'managed':
+            self.base_nic = 'vxlan{}'.format(self.vni)
+            self.bridge_nic = 'vmbr{}'.format(self.vni)
+            self.max_mtu = self.cluster_mtu - 50
            self.logger.out(
                'Creating new managed network',
                prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
            )
            self.init_managed()
        else:
+            self.base_nic = None
+            self.bridge_nic = None
+            self.max_mtu = 0
            self.logger.out(
                'Invalid network type {}'.format(self.nettype),
                prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
            )
            pass

@ -68,8 +78,10 @@ class VXNetworkInstance(object):
        self.old_description = None
        self.description = None

-        self.vlan_nic = 'vlan{}'.format(self.vni)
-        self.bridge_nic = 'vmbr{}'.format(self.vni)
+        try:
+            self.vx_mtu = self.zkhandler.read(('network.mtu', self.vni))
+        except Exception:
+            self.vx_mtu = None

        # Zookeper handlers for changed states
        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('network', self.vni))
@ -83,6 +95,23 @@ class VXNetworkInstance(object):
                self.old_description = self.description
                self.description = data.decode('ascii')

+        # Try block for migration purposes
+        try:
+            @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('network.mtu', self.vni))
+            def watch_network_mtu(data, stat, event=''):
+                if event and event.type == 'DELETED':
+                    # The key has been deleted after existing before; terminate this watcher
+                    # because this class instance is about to be reaped in Daemon.py
+                    return False
+
+                if data and str(self.vx_mtu) != data.decode('ascii'):
+                    self.vx_mtu = data.decode('ascii')
+                    self.validateNetworkMTU()
+                    self.updateNetworkMTU()
+        except Exception:
+            self.validateNetworkMTU()
+            self.updateNetworkMTU()
+
        self.createNetworkBridged()

    # Initialize a managed network
@ -102,8 +131,10 @@ class VXNetworkInstance(object):
        self.dhcp4_start = self.zkhandler.read(('network.ip4.dhcp_start', self.vni))
        self.dhcp4_end = self.zkhandler.read(('network.ip4.dhcp_end', self.vni))

-        self.vxlan_nic = 'vxlan{}'.format(self.vni)
-        self.bridge_nic = 'vmbr{}'.format(self.vni)
+        try:
+            self.vx_mtu = self.zkhandler.read(('network.mtu', self.vni))
+        except Exception:
+            self.vx_mtu = None

        self.nftables_netconf_filename = '{}/networks/{}.nft'.format(self.config['nft_dynamic_directory'], self.vni)
        self.firewall_rules = []
@ -138,7 +169,7 @@ add rule inet filter input tcp dport 80 meta iifname {bridgenic} counter accept
 # Block traffic into the router from network
 add rule inet filter input meta iifname {bridgenic} counter drop
 """.format(
-            vxlannic=self.vxlan_nic,
+            vxlannic=self.base_nic,
            bridgenic=self.bridge_nic
        )

@ -147,14 +178,14 @@ add rule inet filter forward ip daddr {netaddr4} counter jump {vxlannic}-in
 add rule inet filter forward ip saddr {netaddr4} counter jump {vxlannic}-out
 """.format(
            netaddr4=self.ip4_network,
-            vxlannic=self.vxlan_nic,
+            vxlannic=self.base_nic,
        )
        self.firewall_rules_v6 = """# Jump from forward chain to this chain when matching net (IPv4)
 add rule inet filter forward ip6 daddr {netaddr6} counter jump {vxlannic}-in
 add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
 """.format(
            netaddr6=self.ip6_network,
-            vxlannic=self.vxlan_nic,
+            vxlannic=self.base_nic,
        )

        self.firewall_rules_in = self.zkhandler.children(('network.rule.in', self.vni))
@ -209,6 +240,23 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                    self.stopDHCPServer()
                    self.startDHCPServer()

+        # Try block for migration purposes
+        try:
+            @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('network.mtu', self.vni))
+            def watch_network_mtu(data, stat, event=''):
+                if event and event.type == 'DELETED':
+                    # The key has been deleted after existing before; terminate this watcher
+                    # because this class instance is about to be reaped in Daemon.py
+                    return False
+
+                if data and str(self.vx_mtu) != data.decode('ascii'):
+                    self.vx_mtu = data.decode('ascii')
+                    self.validateNetworkMTU()
+                    self.updateNetworkMTU()
+        except Exception:
+            self.validateNetworkMTU()
+            self.updateNetworkMTU()
+
        @self.zkhandler.zk_conn.DataWatch(self.zkhandler.schema.path('network.ip6.network', self.vni))
        def watch_network_ip6_network(data, stat, event=''):
            if event and event.type == 'DELETED':
@ -383,6 +431,66 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
    def getvni(self):
        return self.vni

+    def validateNetworkMTU(self):
+        update_mtu = False
+
+        # Explicitly set the MTU to max_mtu if unset (in Zookeeper too assuming the key exists)
+        if self.vx_mtu == '' or self.vx_mtu is None:
+            self.logger.out(
+                'MTU not specified; setting to maximum MTU {} instead'.format(self.max_mtu),
+                prefix='VNI {}'.format(self.vni),
+                state='w'
+            )
+            self.vx_mtu = self.max_mtu
+            update_mtu = True
+
+        # Set MTU to an integer (if it's not)
+        if not isinstance(self.vx_mtu, int):
+            self.vx_mtu = int(self.vx_mtu)
+
+        # Ensure the MTU is valid
+        if self.vx_mtu > self.max_mtu:
+            self.logger.out(
+                'MTU {} is larger than maximum MTU {}; setting to maximum MTU instead'.format(self.vx_mtu, self.max_mtu),
+                prefix='VNI {}'.format(self.vni),
+                state='w'
+            )
+            self.vx_mtu = self.max_mtu
+            update_mtu = True
+
+        if update_mtu:
+            # Try block for migration purposes
+            try:
+                self.zkhandler.write([
+                    (('network.mtu', self.vni), self.vx_mtu)
+                ])
+            except Exception as e:
+                self.logger.out(
+                    'Could not update MTU in Zookeeper: {}'.format(e),
+                    prefix='VNI {}'.format(self.vni),
+                    state='w'
+                )
+
+    def updateNetworkMTU(self):
+        self.logger.out(
+            'Setting network MTU to {}'.format(self.vx_mtu),
+            prefix='VNI {}'.format(self.vni),
+            state='i'
+        )
+        # Set MTU of base and bridge NICs
+        common.run_os_command(
+            'ip link set {} mtu {} up'.format(
+                self.base_nic,
+                self.vx_mtu
+            )
+        )
+        common.run_os_command(
+            'ip link set {} mtu {} up'.format(
+                self.bridge_nic,
+                self.vx_mtu
+            )
+        )
+
    def updateDHCPReservations(self, old_reservations_list, new_reservations_list):
        for reservation in new_reservations_list:
            if reservation not in old_reservations_list:
@ -411,7 +519,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
        self.logger.out(
            'Updating firewall rules',
            prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
        )
        ordered_acls_in = {}
        ordered_acls_out = {}
@ -458,18 +566,18 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
    def createNetworkBridged(self):
        self.logger.out(
            'Creating bridged vLAN device {} on interface {}'.format(
-                self.vlan_nic,
+                self.base_nic,
                self.bridge_dev
            ),
            prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
        )

        # Create vLAN interface
        common.run_os_command(
            'ip link add link {} name {} type vlan id {}'.format(
                self.bridge_dev,
-                self.vlan_nic,
+                self.base_nic,
                self.vni
            )
        )
@ -480,20 +588,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
            )
        )

-        # Set MTU of vLAN and bridge NICs
-        vx_mtu = self.cluster_mtu
-        common.run_os_command(
-            'ip link set {} mtu {} up'.format(
-                self.vlan_nic,
-                vx_mtu
-            )
-        )
-        common.run_os_command(
-            'ip link set {} mtu {} up'.format(
-                self.bridge_nic,
-                vx_mtu
-            )
-        )
+        self.updateNetworkMTU()

        # Disable tx checksum offload on bridge interface (breaks DHCP on Debian < 9)
        common.run_os_command(
@ -513,7 +608,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
        common.run_os_command(
            'brctl addif {} {}'.format(
                self.bridge_nic,
-                self.vlan_nic
+                self.base_nic
            )
        )

@ -524,13 +619,13 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                self.cluster_dev
            ),
            prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
        )

        # Create VXLAN interface
        common.run_os_command(
            'ip link add {} type vxlan id {} dstport 4789 dev {}'.format(
-                self.vxlan_nic,
+                self.base_nic,
                self.vni,
                self.cluster_dev
            )
@ -542,20 +637,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
            )
        )

-        # Set MTU of VXLAN and bridge NICs
-        vx_mtu = self.cluster_mtu - 50
-        common.run_os_command(
-            'ip link set {} mtu {} up'.format(
-                self.vxlan_nic,
-                vx_mtu
-            )
-        )
-        common.run_os_command(
-            'ip link set {} mtu {} up'.format(
-                self.bridge_nic,
-                vx_mtu
-            )
-        )
+        self.updateNetworkMTU()

        # Disable tx checksum offload on bridge interface (breaks DHCP on Debian < 9)
        common.run_os_command(
@ -575,7 +657,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
        common.run_os_command(
            'brctl addif {} {}'.format(
                self.bridge_nic,
-                self.vxlan_nic
+                self.base_nic
            )
        )

@ -600,7 +682,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                    self.bridge_nic
                ),
                prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
            )
            common.createIPAddress(self.ip6_gateway, self.ip6_cidrnetmask, self.bridge_nic)

@ -613,7 +695,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                    self.bridge_nic
                ),
                prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
            )
            common.createIPAddress(self.ip4_gateway, self.ip4_cidrnetmask, self.bridge_nic)

@ -624,7 +706,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                    self.bridge_nic
                ),
                prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
            )

            # Recreate the environment we need for dnsmasq
@ -719,7 +801,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                self.cluster_dev
            ),
            prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
        )
        common.run_os_command(
            'ip link set {} down'.format(
@ -728,13 +810,13 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
        )
        common.run_os_command(
            'ip link set {} down'.format(
-                self.vlan_nic
+                self.base_nic
            )
        )
        common.run_os_command(
            'brctl delif {} {}'.format(
                self.bridge_nic,
-                self.vlan_nic
+                self.base_nic
            )
        )
        common.run_os_command(
@ -744,7 +826,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
        )
        common.run_os_command(
            'ip link delete {}'.format(
-                self.vlan_nic
+                self.base_nic
            )
        )

@ -755,7 +837,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                self.cluster_dev
            ),
            prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
        )
        common.run_os_command(
            'ip link set {} down'.format(
@ -764,13 +846,13 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
        )
        common.run_os_command(
            'ip link set {} down'.format(
-                self.vxlan_nic
+                self.base_nic
            )
        )
        common.run_os_command(
            'brctl delif {} {}'.format(
                self.bridge_nic,
-                self.vxlan_nic
+                self.base_nic
            )
        )
        common.run_os_command(
@ -780,7 +862,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
        )
        common.run_os_command(
            'ip link delete {}'.format(
-                self.vxlan_nic
+                self.base_nic
            )
        )

@ -788,7 +870,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
        self.logger.out(
            'Removing firewall rules',
            prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
        )

        try:
@ -815,7 +897,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                self.bridge_nic
            ),
            prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
        )
        common.removeIPAddress(self.ip6_gateway, self.ip6_cidrnetmask, self.bridge_nic)

@ -827,7 +909,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                self.bridge_nic
            ),
            prefix='VNI {}'.format(self.vni),
-            state='o'
+            state='i'
        )
        common.removeIPAddress(self.ip4_gateway, self.ip4_cidrnetmask, self.bridge_nic)

@ -838,7 +920,7 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
                    self.bridge_nic
                ),
                prefix='VNI {}'.format(self.vni),
-                state='o'
+                state='i'
            )
            # Terminate, then kill
            self.dhcp_server_daemon.signal('term')
--- a/node-daemon/pvcnoded/util/config.py
+++ b/node-daemon/pvcnoded/util/config.py
@ -287,6 +287,7 @@ def get_configuration():
            'upstream_mtu':         o_sysnetwork_upstream.get('mtu', None),
            'upstream_dev_ip':      o_sysnetwork_upstream.get('address', None),
            'bridge_dev':           o_sysnetworks.get('bridge_device', None),
+            'bridge_mtu':           o_sysnetworks.get('bridge_mtu', 1500),
            'enable_sriov':         o_sysnetworks.get('sriov_enable', False),
            'sriov_device':         o_sysnetworks.get('sriov_device', list())
        }
--- a/node-daemon/pvcnoded/util/fencing.py
+++ b/node-daemon/pvcnoded/util/fencing.py
@ -189,7 +189,7 @@ def reboot_via_ipmi(ipmi_hostname, ipmi_user, ipmi_password, logger):
 def verify_ipmi(ipmi_hostname, ipmi_user, ipmi_password):
    ipmi_command = f'/usr/bin/ipmitool -I lanplus -H {ipmi_hostname} -U {ipmi_user} -P {ipmi_password} chassis power status'
    retcode, stdout, stderr = common.run_os_command(ipmi_command, timeout=2)
-    if retcode == 0 and stdout.strip() != "Chassis Power is on":
+    if retcode == 0 and stdout.strip() == "Chassis Power is on":
        return True
    else:
        return False
Author	SHA1	Message	Date
Joshua M. Boniface	f13cc04b89	Bump version to 0.9.41	2021-10-09 19:39:21 -04:00
Joshua M. Boniface	4ed537ee3b	Add bridge_mtu config to docs	2021-10-09 19:28:50 -04:00
Joshua M. Boniface	95e01f38d5	Adjust log type of object setup message	2021-10-09 19:23:12 -04:00
Joshua M. Boniface	3122d73bf5	Avoid duplicate runs of MTU set It wasn't the validator duplicating, but the update duplicating, so avoid that happening properly this time.	2021-10-09 19:21:47 -04:00
Joshua M. Boniface	7ed8ef179c	Revert "Avoid duplicate runs of MTU validator" This reverts commit `56021c443a`.	2021-10-09 19:11:42 -04:00
Joshua M. Boniface	caead02b2a	Set all log messages to information state None of these were "success" messages and thus shouldn't have been ok state.	2021-10-09 19:09:38 -04:00
Joshua M. Boniface	87bc5f93e6	Avoid duplicate runs of MTU validator	2021-10-09 19:07:41 -04:00
Joshua M. Boniface	203893559e	Use correct isinstance instead of type	2021-10-09 19:03:31 -04:00
Joshua M. Boniface	2c51bb0705	Move MTU validation to function Prevents code duplication and ensures validation runs when an MTU is updated, not just on network creation.	2021-10-09 19:01:45 -04:00
Joshua M. Boniface	46d3daf686	Add logger message when setting MTU	2021-10-09 18:56:18 -04:00
Joshua M. Boniface	e9d05aa24e	Ensure vx_mtu is always an int()	2021-10-09 18:52:50 -04:00
Joshua M. Boniface	d2c18d7b46	Fix bad header length in network list	2021-10-09 18:50:32 -04:00
Joshua M. Boniface	6ce28c43af	Add MTU value checking and log messages Ensures that if a specified MTU is more than the maximum it is set to the maximum instead, and adds warning messages for both situations.	2021-10-09 18:48:56 -04:00
Joshua M. Boniface	87cda72ca9	Fix invalid schema key Addresses #144	2021-10-09 18:42:33 -04:00
Joshua M. Boniface	8f71a6d2f6	Add MTU support to network add/modify commands Addresses #144	2021-10-09 18:06:21 -04:00
Joshua M. Boniface	c45f8f5bd5	Have VXNetworkInstance set MTU if unset Makes this explicit in Zookeeper if a network is unset, post-migration (schema version 6). Addresses #144	2021-10-09 17:52:57 -04:00
Joshua M. Boniface	24de0f4189	Add MTU to network creation/modification Addresses #144	2021-10-09 17:51:32 -04:00
Joshua M. Boniface	3690a2c1e0	Fix migration bugs and invalid vx_mtu Addresses #144	2021-10-09 17:35:10 -04:00
Joshua M. Boniface	50d8aa0586	Add handlers for client network MTUs Refactors some of the code in VXNetworkInterface to handle MTUs in a more streamlined fashion. Also fixes a bug whereby bridge client networks were being explicitly given the cluster dev MTU which might not be correct. Now adds support for this option explicitly in the configs, and defaults to 1500 for safety (the standard Ethernet MTU). Addresses #144	2021-10-09 17:02:27 -04:00
Joshua M. Boniface	db6e65712d	Make n-1 values clearer	2021-10-07 18:11:15 -04:00
Joshua M. Boniface	cf8e16543c	Correct levels in TOC	2021-10-07 18:08:28 -04:00
Joshua M. Boniface	1a4fcdcc2d	Correct spelling errors	2021-10-07 18:07:06 -04:00
Joshua M. Boniface	9a71db0800	Add documentation sections on IPMI and fencing	2021-10-07 18:05:47 -04:00
Joshua M. Boniface	6ee4c55071	Correct flawed conditional in verify_ipmi	2021-10-07 15:11:19 -04:00
Joshua M. Boniface	c27359c4bf	Bump version to 0.9.40	2021-10-07 14:42:04 -04:00
Joshua M. Boniface	46078932c3	Correct bad stop_keepalive_timer call	2021-10-07 14:41:12 -04:00
Joshua M. Boniface	c89699bc6f	Remove redundant wording from header	2021-10-07 12:20:04 -04:00
Joshua M. Boniface	1b9507e4f5	Replace headers with links in CHANGELOG.md	2021-10-07 12:17:44 -04:00
Joshua M. Boniface	3db7ac48f4	Add missing period to changelog sentence	2021-10-07 12:10:35 -04:00
Joshua M. Boniface	1830ec6465	Move changelog into dedicated file The changelog was getting far too long for the README/docs index to support, so move it into CHANGELOG.md and link to it instead.	2021-10-07 12:09:26 -04:00
 @ -1 +1 @@
 .9.39
 .9.41
				`@ -0,0 +1 @@`
				{"version": "6", "root": "", "base": {"root": "", "schema": "/schema", "schema.version": "/schema/version", "config": "/config", "config.maintenance": "/config/maintenance", "config.primary_node": "/config/primary_node", "config.primary_node.sync_lock": "/config/primary_node/sync_lock", "config.upstream_ip": "/config/upstream_ip", "config.migration_target_selector": "/config/migration_target_selector", "cmd": "/cmd", "cmd.node": "/cmd/nodes", "cmd.domain": "/cmd/domains", "cmd.ceph": "/cmd/ceph", "logs": "/logs", "node": "/nodes", "domain": "/domains", "network": "/networks", "storage": "/ceph", "storage.util": "/ceph/util", "osd": "/ceph/osds", "pool": "/ceph/pools", "volume": "/ceph/volumes", "snapshot": "/ceph/snapshots"}, "logs": {"node": "", "messages": "/messages"}, "node": {"name": "", "keepalive": "/keepalive", "mode": "/daemonmode", "data.active_schema": "/activeschema", "data.latest_schema": "/latestschema", "data.static": "/staticdata", "data.pvc_version": "/pvcversion", "running_domains": "/runningdomains", "count.provisioned_domains": "/domainscount", "count.networks": "/networkscount", "state.daemon": "/daemonstate", "state.router": "/routerstate", "state.domain": "/domainstate", "cpu.load": "/cpuload", "vcpu.allocated": "/vcpualloc", "memory.total": "/memtotal", "memory.used": "/memused", "memory.free": "/memfree", "memory.allocated": "/memalloc", "memory.provisioned": "/memprov", "ipmi.hostname": "/ipmihostname", "ipmi.username": "/ipmiusername", "ipmi.password": "/ipmipassword", "sriov": "/sriov", "sriov.pf": "/sriov/pf", "sriov.vf": "/sriov/vf"}, "sriov_pf": {"phy": "", "mtu": "/mtu", "vfcount": "/vfcount"}, "sriov_vf": {"phy": "", "pf": "/pf", "mtu": "/mtu", "mac": "/mac", "phy_mac": "/phy_mac", "config": "/config", "config.vlan_id": "/config/vlan_id", "config.vlan_qos": "/config/vlan_qos", "config.tx_rate_min": "/config/tx_rate_min", "config.tx_rate_max": "/config/tx_rate_max", "config.spoof_check": "/config/spoof_check", "config.link_state": "/config/link_state", "config.trust": "/config/trust", "config.query_rss": "/config/query_rss", "pci": "/pci", "pci.domain": "/pci/domain", "pci.bus": "/pci/bus", "pci.slot": "/pci/slot", "pci.function": "/pci/function", "used": "/used", "used_by": "/used_by"}, "domain": {"name": "", "xml": "/xml", "state": "/state", "profile": "/profile", "stats": "/stats", "node": "/node", "last_node": "/lastnode", "failed_reason": "/failedreason", "storage.volumes": "/rbdlist", "console.log": "/consolelog", "console.vnc": "/vnc", "meta.autostart": "/node_autostart", "meta.migrate_method": "/migration_method", "meta.node_selector": "/node_selector", "meta.node_limit": "/node_limit", "meta.tags": "/tags", "migrate.sync_lock": "/migrate_sync_lock"}, "tag": {"name": "", "type": "/type", "protected": "/protected"}, "network": {"vni": "", "type": "/nettype", "mtu": "/mtu", "rule": "/firewall_rules", "rule.in": "/firewall_rules/in", "rule.out": "/firewall_rules/out", "nameservers": "/name_servers", "domain": "/domain", "reservation": "/dhcp4_reservations", "lease": "/dhcp4_leases", "ip4.gateway": "/ip4_gateway", "ip4.network": "/ip4_network", "ip4.dhcp": "/dhcp4_flag", "ip4.dhcp_start": "/dhcp4_start", "ip4.dhcp_end": "/dhcp4_end", "ip6.gateway": "/ip6_gateway", "ip6.network": "/ip6_network", "ip6.dhcp": "/dhcp6_flag"}, "reservation": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname"}, "lease": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname", "expiry": "/expiry", "client_id": "/clientid"}, "rule": {"description": "", "rule": "/rule", "order": "/order"}, "osd": {"id": "", "node": "/node", "device": "/device", "db_device": "/db_device", "stats": "/stats"}, "pool": {"name": "", "pgs": "/pgs", "stats": "/stats"}, "volume": {"name": "", "stats": "/stats"}, "snapshot": {"name": "", "stats": "/stats"}}