Move drafts and fix links
This commit is contained in:
@ -0,0 +1,121 @@
|
||||
---
|
||||
title: "Burlington's Half-Assed Bike Infrastructure"
|
||||
description: "Venting my frustrations about this city's poor bike infrastructure"
|
||||
date: 2024-04-23
|
||||
tags:
|
||||
- Hometown
|
||||
- Politics
|
||||
- Civics
|
||||
---
|
||||
|
||||
This post is a little different than my usual post here, touching more on local politics, infrastructure, and issues in my home city of Burlington, Ontario, and written with some fairly poor organization of thoughts. I, however, couldn't think of another place to put this sort of long-form content (read: long rant), so it's going here for now.
|
||||
|
||||
## What Burlington Has
|
||||
|
||||
Burlington's topography and layout is important for the criticisms below, so it's worth spending a moment describing it.
|
||||
|
||||
Burlington is a large commuter suburb at the southern end of Lake Ontario, with Toronto about 50km to the northeast and Hamilton directly to the south and west. The city follows a broadly east-west grid structure, with 4 major north-south roads (Brant Stree, Guelph Line, Walkers Line, and Appleby Line) spaced at 2km increments. There are further north-south roads (Burloak to the far east bordering Oakville and Bronte Creek Provincial Park, and King Road and Waterdown Road further to the west) but the main portion of the city is contained by by those first four. Downtown is at the south end of Brant St. by Lake Ontario.
|
||||
|
||||
Through the center of the city, east-west, run both the Queen Elizabeth Way highway and a set of GO Transit tracks. Between and just north of them is a major industrial corridor that forms the boundary between the residential "North" and "South" Burlington. There are a number of other major east-west roads that will be mentioned; from south to north, they are Lakeshore Drive, New Street, Fairview Road, Harvester Road, Mainway, Upper Middle Road, and finally Dundas Street.
|
||||
|
||||
At the top boundary of the developed area are Dundas Street and (west of roughly Walkers Line) Highway 407ETR; all land allocation north of these is Greenbelt land and cannot be developed.
|
||||
|
||||
The topography is broadly sloping down from the Niagara Escarpment in the north to Lake Ontario in the south, with the highest peak in the (developed) city in the north-west at the top of Brant Street, and the lowest points along the lakeshore.
|
||||
|
||||
Lastly, there are two major northeast-southwest power lines that, due to the exact positioning, criss-cross the city on a diagonal; one in North Burlington, and one in South Burlington. A multi-use path follows both.
|
||||
|
||||
Here is the map:
|
||||
|
||||
|
||||
*Map courtesy of the City of Burlington*
|
||||
|
||||
Burlington seems like a great city to feature comprehensive bike infrastructure, but unfortunately it is thoroughly half-assed. Through nearly a century of car-centric planning and development, what should be a perfectly bikeable city is instead a congested wasteland of asphalt, concrete, and single-family sprawl.
|
||||
|
||||
## Burlington's Bike Infrastructure: The Good
|
||||
|
||||
There are a few good parts, so I'd like to list those first.
|
||||
|
||||
1. Multi-use paths along hydro corridors. These help provide both excellent recreational trails for casual or fitness cyclists, as well as transportation for some trips, for example North Burlington -> Downtown via the Crosstown Trail (north) or South Burlington -> Downtown via the Centennial Bikeway.
|
||||
|
||||
2. Multi-use paths north-south. A few of these are scattered throughout various neighbourhoods, for example the Palmer Trail in Palmer (my home neighbourhood), an unnamed extension to the Centennial Bikeway in Dynes, and another unnamed trail in Downtown. These do help cyclists cut through a number of residential areas, but as we'll see later, are not very long or effective.
|
||||
|
||||
3. Multi-use paths along major roads. A few of these exists: Appleby Line, Walkers Line and Brant Street feature multi-use paths beside sidewalks for a signfiicant part of their northern sections, and Upper Middle features a multi-use path along its length. Lakeshore Drive features a multi-use path instead of a sidewalk on its south side, as do Harvester Street and Dundas Street.
|
||||
|
||||
4. Bike Lanes and sharrow indications are present everywhere. However, as I'll outline below, these aren't *really* bike infrastructure, and are at the root of what I see as the major problems with cycling in Burlington, but we'll get there.
|
||||
|
||||
5. Neighbourhoods are being calmed. Most neighbourhoods in the city now post a 40km/h (instead of 50km/h) speed limit, and many roads have had speed control devices added to slow drivers down, making these roads much safer for cycling.
|
||||
|
||||
|
||||
## Burlington's Bike Infrastructure: The Bad & The Ugly
|
||||
|
||||
But, there are a lot of problems with bike infrastructure in the city. I'll outline what first, before getting into the why.
|
||||
|
||||
1. Multi-use paths are in disarray. I've been a resident of Burlington for my entire life - 36 years - and I do not remember most of the road-paralleled multi-use paths ever being repaved. Some sections are in such poor condition that they are lost to grass/weeds, are so bumpy as to put mountain bikes under strain, let alone road bikes, and others are simply haphazardly cut as an afterthought, with sharp drops at driveways or dangerous swerves needed to descend onto cross streets.
|
||||
|
||||
2. Multi-use paths are very limited. Except for those north-south hydro corridor and short neighbourhood paths, the remaining bike infrastructure is limited only to major roads, and not all of them.
|
||||
|
||||
3. Major roads are dangerous. 60+km/h speeds on wide [stroads](https://en.wikipedia.org/wiki/Stroad) with frequent, poorly-timed traffic lights is a deathtrap for bicyclists.
|
||||
|
||||
4. Bike lanes are, at best, bad. Painted bicycle gutters and (*shudder*) sharrows are not bike infrastructure. A painted line does not stop a two-tonne SUV driven by a distracted and impatient motorist from running you down. Worse still, in this city as in many others, bicycle lanes start and end abruptly with no rhyme or reason, forcing cyclings into sharing lanes with speeding vehicles frequently. Combined with the previous point, this makes for several extremely dangerous sections of road.
|
||||
|
||||
5. Highway onramps on the major roads are a deathtrap. With no approach lanes and drivers who routinely fail to signal these, it becomes a dangerous guessing game of who is going where while waiting to cross them, or braving 70+km/h speeding vehicles cutting you off.
|
||||
|
||||
6. There are far too many stop signs. Stop signs are the devil. They're inefficient for cyclists (unless, of course, cyclist priority and an [Idaho Stop](https://en.wikipedia.org/wiki/Idaho_stop) system is added) and actually add danger due to a cyclist having to start from zero afterwards (see below).
|
||||
|
||||
## What our Brilliant Government has tried to do
|
||||
|
||||
Really, outlining the problems above really just serves to put into context the assinine and, as stated before, half-assed measures this city and its governmnent has taken to "improve cycling", almost none of which do anything at all. I will outline 3 key case studies here.
|
||||
|
||||
### Bicycle Lanes No One Needed
|
||||
|
||||
In 2020, the city launched a project to replace 1 lane of car traffic in each direction along New Street with a bike lane, reducing the car lanes on this very busy stretch of road from 2 to 1. These were, of course, implemented in the most half-assed way possible, using paint and nothing more. Sure, a buffered bike lane is better than no buffer, but when a 1-km backlog of cars has built up, nothing is stopping them from cutting ahead through this purely-paint "infrastructure", putting everyone else at risk. And I saw this happen.
|
||||
|
||||
Predictably, in a car-centric sprawl city, the backlash from carbrain drivers was swift. However, it may surprise you, I agree with them. It was a poor place to put a bike lane, because the route was already serviced by a well-used multi-use path, the Centennial Bikeway, one of the few in the city! It was a pointless waste of millions of dollars that was ripped back out (read: painted over) 2 years later.
|
||||
|
||||
The reasons for the failure here are clear. In a bid to make a major street "bicycle-friendly", they ignored already-existing bicycle infrastructure in favour of painted bicycle gutters, inconveniencing motorists on a busy road with a solution that did nothing at all to actually protect bicyclists from the (now increased) vehicular traffic. It is a case study of a solution in need of a problem: someone wanted to get paid to paint some bike lanes, and by golly they did - twice even - when a much better solution already existed.
|
||||
|
||||
|
||||
|
||||
### Another Bike Lane to Nowhere
|
||||
|
||||
Bike lanes on Walkers Line, Guelph Line, and other places start and end abruptly. But, the fact that these lanes exist atall is a problem, because a bicycle gutter to nowhere is almost worse than nothing.
|
||||
|
||||
### The Most Dangerous and Pointless "Solution" to the Highway Onramps
|
||||
|
||||
On, the highway onramps. As I pointed out above, the design of most of them makes them incredibly dangerous to cross, not just for cyclists, but for pedestrians as well. What was the brilliant solution? Implement the most bizarre "stop here" on a bike possible. Beyold this magnificence:
|
||||
|
||||
|
||||
|
||||
Now, for someone used to a car, it may not be too obvious why this is an utterly terrible design. Let me list the reasons:
|
||||
|
||||
1. Bikes are slow to get moving after a stop, which will always happen here due to the poor road design in general.
|
||||
2. The more time a bike is in the path of a car (say, due to #1), the more danger they are in.
|
||||
3. A crossing is more dangerous (especially with #2 and #1) than a simple merge at speed.
|
||||
4. These are on hills (due to the overpass), making #1 even worse.
|
||||
5. They are abrupt and small, leaving little room to maneuver into a good position to both view oncoming traffic and proceed.
|
||||
6. They are placed so close to the main lanes as to provide very little gap to observe cars coming onto the onramp.
|
||||
|
||||
It's hard to know even where to begin with improving these. A few thoughts include...
|
||||
|
||||
* Setting them at least 50m back from the road, so it's much clearer when cars are coming up the lane.
|
||||
* Widening them into a proper path colocated with the sidewalk, to provide more room to slow down, stop, and turn.
|
||||
|
||||
But really, neither of these address the *actual* problem here. The problem is of speeding cars approaching a highway onramp at 60+km/h, without an approach lane, not signalling, and thus forcing a wait for a significant gap. Like most problems related to bikes-on-roads, the problem *should* be solved by modifications to driver behaviour and road design, not adding more inconveniences for cyclists. Here's how I'd design these:
|
||||
|
||||
|
||||
|
||||
If you don't see why this is lightyears better, I don't know what to say.
|
||||
|
||||
## Actually Improving Bike Infrastructure
|
||||
|
||||
OK, enough whinging. What should be done to actually *improve* the bike infrastructure in this city? It's really simple.
|
||||
|
||||
* Fix the shitty bike paths! Huge numbers of the bike paths are in disarray as outlined above. Repave them, now. Not "soon", because "soon" was literally 20 years ago. There is already a significant network of paths in the city, so improve them to help encourage cycling trips where possible.
|
||||
|
||||
* Add more bike paths. Instead of futzing around with bike lanes, widen the sidewalks and convert them (partially or fully) to multi-use paths. Do this on every major road; most have plenty of space.
|
||||
|
||||
* Build better crossings. Slow down traffic at dangerous crossings, add approach lanes, and give bicyclists more room to maneuver.
|
||||
|
||||
* Add more connecting paths through residential areas, and between the areas of the city. Specifically, I mean tunnels under the railway line(s) and across the highway, to avoid cyclists (and pedestrians!) from having to take the obnoxious concrete jungles that are the current highway overpasses. Just two would be a huge boon to the connectivity of the city.
|
||||
|
||||
And that's the end of my rant. Hopefully I've educated someone on the pitfalls of the current designs and "ideas", and can offer some more obvious solutions.
|
||||
153
content/en/posts/.drafts/custom-pdu-3/index.md
Normal file
153
content/en/posts/.drafts/custom-pdu-3/index.md
Normal file
@ -0,0 +1,153 @@
|
||||
+++
|
||||
date = "2021-04-03T00:00:00-04:00"
|
||||
tags = ["diy","homelab","buildlog"]
|
||||
title = "A Custom Monitored PDU"
|
||||
description = "Building a custom power monitoring PDU for fun and profit"
|
||||
type = "post"
|
||||
weight = 1
|
||||
draft = true
|
||||
+++
|
||||
|
||||
As a veteran homelabber, one thing that always comes up is power usage. Electricity is, unfortunately, not free, even if I do get half of it from "too cheap to meter" nuclear power here in Ontario. And servers can use a lot of it. Some systems provide on-demand power monitoring via their IPMI/BMC interfaces, but not all do. And figuring out how much power the other, non-intelligent, systems use can be a hassle.
|
||||
|
||||
The most obvious solution is what is normally called a "per-port monitored PDU". Typically used by colocation providers and large enterprises, these power distribution units (PDUs) allow the administrator to see the actual power usage out of each individual port at a given time, both for billing and monitoring purposes. They're the perfect solution to the problem of not knowing how much power you are using.
|
||||
|
||||
But these PDUs are not cheap. New, the cheapest ones I've been able to find run over $1400 USD, and they're gigantic 5-foot monsters designed for full-sized 42U+ racks. Add in dual circuits/UPSes, and the cost doubles. There has to be a better way.
|
||||
|
||||
Well, there is. With a decent amount of electrical know-how, some programming, 3D printing, and a lot of patience, I've been able to build myself several custom PDUs. Read on to know how!
|
||||
|
||||
**DISCLAIMER/WARNING:** While I am not a professional licensed electrician, I've spent a large portion of my life working with A/C electricity, pretty much from the time I could walk and hold a screwdriver, including a year as an Electrical associate at The Home Depot, as well as numerous home projects and two previous PDUs. I know what I'm doing. **Working with mains electricity is very dangerous, especially if you do not know what you're doing.** This post is provided as a curiosity for most, and a build log/guide only for those who are well-versed in working with this sort of thing. **Do not try this at home. I will not provide advice or guidance on any aspect of any similar project(s) outside of the scope of this document. Contact an electrician if in doubt.**
|
||||
|
||||
## PDU 1.0 and 2.0 - Hall Effect sensors
|
||||
|
||||
My first two forrays into the custom PDU project were simple devices that used the ACS714 [Hall effect](https://en.wikipedia.org/wiki/Hall_effect) current sensors alone. These units were built out of plastic wall boxes with the sensors held in series with the hot lines connecting to each plug.
|
||||
|
||||
The first iteration worked well, but was quite small, with only 16 outlets (4 boxes), which I quickly outgrew.
|
||||
|
||||
|
||||
|
||||
The second iteration was a fair bit larger, with 28 outlets (7 boxes), which was more than enough for my rack even now.
|
||||
|
||||
|
||||
|
||||
This design had a lot of downsides however:
|
||||
|
||||
1. In terms of monitoring, only getting the current was problematic. Current, measured in amperes, is only one part of the energy equation, and voltage and power factor are other important components which the ACS714 sensor does not provide. I was able to hack together a solution in my monitoring software by using the output voltage readings from my UPSes, but this wasn't satisfactory to me, especially given the slow speed of readings and the inaccuracy relative to a live device reading.
|
||||
|
||||
2. The sensors were, in my experience, quite unreliable. They were nearly impossible to calibrate and would sometimes report wildly inaccurate values due to interference. This was especially pronounced with low loads, since I needed to use 20A sensors for safety which have a correspondingly low threshold. Under 0.1A (about 12W) they were completely useless, and under 0.5A (about 60W) they were often +/- 15-20% out from a Kill-A-Watt's readings. Only at very high current values (>1.0A) were they accurate, and then only to about 1 decimal place, a fairly rough value.
|
||||
|
||||
3. The physical design of the PDU was cumbersome. Each box had to be wired in a very tight space with very tight tolerances on wire length, leading to many a scraped and cut finger. This was fine at the start, but connect 8 of these boxes together and the unit became cumbersome to work with. Maintenance was also a hassle for this reason. If a sensor died, which thankfully has not happened, replacing it would be a massive chore. And due to the through runs of the power busses, made out of normal 14-2 Romex wire, the boxes were permanently attached to each other, making disassembly tricky at best.
|
||||
|
||||
In setting out to design version 3 of the PDU, I wanted to solve all 3 issues, making something more robust and easier to service and maintain, as well as more accurate.
|
||||
|
||||
## PDU 3.0: Physical design
|
||||
|
||||
Solving issue 3 turned out to be fairly easy - the solution was a 3D Printer, specifically my new Ender 3 v2. Instead of using pre-made 3-gang plastic wall boxes, I could design individual "modules" one at a time, print their components, and then assemble them together using some sort of quick-release between them.
|
||||
|
||||
I began with a plan to create a CAD design of a full box, but ultimately this ended up going nowhere, not least due to my lack of experience (and patience!) with 3D modeling software. Instead, I was able to find two smaller components with which I could build out larger boxes: a 2-gang wallplate, and a 120mm "honeycomb" fan grill. These two components could easily be combined with a bit of superglue and electrical tape to form a 2-outlet cubic box which would hold all the wiring, the sensors, and the plugs, while providing ample room to work, as well as an open visual apperance to allow easy inspection of the internal components.
|
||||
|
||||
|
||||
|
||||
To connect the electrical portion of the modules together, I avoided the 1.0 and 2.0 method of using marette connectors to join multiple leads, and instead went with a busbar concept. I was able to find two designs of busbar: a dual-bus, 4-post version which would be used for the hot and neutral leads, and a raised 6-post version for the ground leads. This would greatly simplify the assembly by allowing me to use Y-connectors and securely screw down the leads, keeping everything very neat.
|
||||
|
||||
|
||||
|
||||
These busbars were then mounted on the bus sidepeace pictured above, to give a secure base to work off of. This piece alone was enough to assemble the core electrical components easily with plenty of working room.
|
||||
|
||||
|
||||
|
||||
Connecting the leads was then a trivial exercise of cutting exactly-length pieces of 14-gauge wire, stripping the right amount off each end, and bending them into position. Each sensor - we'll cover these in the next section - required 4 leads: two hot, in and out, and two neutral, in and out (though bridged internally), so all 4 leads met up in a level location towards the back of the module.
|
||||
|
||||
|
||||
|
||||
The final step was a method to connect the modules to each other. Rather than fixed, through wire, I settled on a relatively-recent innovation, which is very common in Europe but virtually unknown in North America: clamp-down connectors. These are absolutely fantastic for this purpose, able to handle a full 32A through them while being absolutely trivial to connect and disconnect. And it turned out that the clearances between modules were absolutely perfect for these. Thus, I could easily connect and disconnect modules during assembly or maintenance.
|
||||
|
||||
|
||||
|
||||
And *voilà*, a finished module, ready to accept the four sensor modules. I could then attach the other side plate and the back when I was ready to connect the modules and microcontroller.
|
||||
|
||||
|
||||
|
||||
The input section proved to be equally simple to assemble. I was able to find a set of combination IEC-13 input, fuse, and switch units from Amazon fairly easily. While technically rated only for 10A, I feel comfortable with this since each circuit should only ever be expected to run just under 10A load, and this is a derated value. I simply replaced the stock fuses with a 15A fast-blow variety for safety, and assembled a final segment to hold them. Input can now be handled by a simple IEC cord, along with switching, power control, and an indicator light, protected by a fuse should anything ever short out.
|
||||
|
||||
|
||||
|
||||
While I ultimately did need to permanently attach the modules physically to keep the movement from fatiguing the wires, breaking this connection in the future would simply be a matter of cutting some cable ties and breaking some glue bonds to replace the module. I could also extend the unit arbitrarily, with another module or two should I eventually need more than 32 ports.
|
||||
|
||||
The last step was assembling a cage for the sensor modules. One "flaw" in the design of the sensors I will mention below is that they float at line level, so an external enclosure was absolutely critical to avoid accidentally touching the modules.
|
||||
|
||||
## PDU 3.0: Sensors
|
||||
|
||||
As previously mentioned, the original PDU designs used a Hall effect sensor, which only reported current. But for version 3.0, I wanted something more feature-rich and robust. In scouring the Internet for a solution, I came across the perfect device: the HLW8012.
|
||||
|
||||
|
||||
|
||||
This module is able to output PWM signals on two interface pins that correspond to the active power (in Watts), and, via a select pin, the voltage or current passing through the sensor. Though there are some flaws with the design, specifically that the voltage is read from the neutral side and thus a large voltage drop in the load will provide bogus readings, and the fact that these units float at line level due to their design, these seemed like the perfect fit.
|
||||
|
||||
I was able to find a blog post by Xose Pérez, titled [The HLW8012 IC in the new Sonoff POW](https://tinkerman.cat/post/hlw8012-ic-new-sonoff-pow/) which went over the basics of how this sensor worked and how it can be used to make a DIY single-plug power monitor, similar to a Kill-A-Watt. He [even wrote an Arduino library for it](https://github.com/xoseperez/hlw8012)!
|
||||
|
||||
Armed with this knowledge, I ordered 48 (and later, due to an error, another 24) of the sensors from a Chinese seller called [ElectroDragon](https://www.electrodragon.com/product/energy-meter-hlw8012-breakout-board/) and got to work assembling the last component.
|
||||
|
||||
|
||||
|
||||
During my preliminary testing with an Aruino Uno, however, I found some issues with Xose's library - it was clearly not designed to work with more than one module at a time, so I had to come up with another solution. I also ran into an issue that plagued the 2.0 design: how to collect dozens of wires from dozens of sensors back to a central microcontroller to actually power and read data from them.
|
||||
|
||||
## PDU 3.0: Microcontrollers
|
||||
|
||||
My first idea was to use the Arduino Mega. This is a monstrous Arduino microcontroller with 54 digital I/Os, more than enough to handle 16 or 18 sensors per circuit, which was my goal - match or slightly excede the 32-port 2.0 design. But it had some fatal flaws: first, this is an 8-bit microcontroller which makes dealing with relatively-large integer and floating point values very cumbersome; second, the microcontroller CPU is very slow; and third and most important to my physical design, I would have to run a total of 20 wires from each module back up to the central Arduino board at the top, an exercise in frustration.
|
||||
|
||||
I continued to look for a good solution, when finally a discussion with a friend led to an excellent discovery: the STM32 "Black Pill" microcontroller. Forget an 8-bit 16MHz 54-I/O Arduino board; this tiny monster can do 16 I/Os with a 32-bit 100MHz ARM-based core, which is more than enough to read sensors on the order of microseconds. And it has a USB-C output!
|
||||
|
||||
|
||||
|
||||
But one microcontroller wouldn't cut it; I'd have a total of 3 leads (plus power) from each sensor, and due to the "float at line voltage" design of the sensors, needed separate microcontrollers for each circuit lest they short. So I would need more than just one. Thus I came up with an even better solution: why not use one STM32 for "each" module (really, one for each "side" of two modules)? This had two benefits; first, I could keep the "each module is separate" mentality even though the compute side; and second, the USB connections would make it trivial to run back to the central controller, a Raspberry Pi - I would just need a USB hub, USB isolator, and some relatively-short USB cables, instead of dozens and dozens of discrete leads. And the distance between each sensor and the microcontroller was small enough to use normal 6" jumper wires. A match made in heaven!
|
||||
|
||||
|
||||
|
||||
## PDU 3.0: Programming
|
||||
|
||||
With all the physical assembly completed and the units ready for live testing, I was able to get started with the programming aspect.
|
||||
|
||||
The first task was to calibrate the sensors. Looking through Xose's library code, I was able to determine that each sensor would need 3 multiplier values (one each for wattage, voltage, and amperage), which was then divided by the resulting PWM value to produce a usable reading. But how to find those multipliers?
|
||||
|
||||
To solve this problem, I used on of the free STM32's to build a sensor calibrator. The idea is quite simple: I would connect up the sensor to the calibrator, attach a known resistive load (a 40W and 60W lightbulb in parallel totaling ~100W of load), and connect everything to a no-name Kill-A-Watt to give me a reference. I could then enter the reference value the Kill-A-Watt showed, and let the calibrator read the modules and calculate the correct multiplier.
|
||||
|
||||
This process took a lot of iteration to get right, and in the end I settled on code that would run a large number of scans trying to determine the exact value that matched my input values. But it was worth the time, and the results turned out to be perfect - I was able to use the calibrator on each sensor to determine what their multiplier should be, and then store this for later use inside the live code on each microcontroller, giving me nearly to-the-watt accuracy.
|
||||
|
||||
```C++
|
||||
[calibrator code]
|
||||
```
|
||||
|
||||
|
||||
|
||||
With the calibration values in hand, I turned to writing code to handle the actual module microcontrollers. The code here ended up being extremely simple once I had the calibration: simply poll the PWM of each sensor in turn, calculate the output, then display it on the serial console for reading by the Raspberry Pi using JSON formatting. Note the `struct` for the sensor modules, which contain the individual multipiers found during the calibration step for that given module, as well as the identifier of each plug.
|
||||
|
||||
```C++
|
||||
[microcontroller code]
|
||||
```
|
||||
|
||||
The final step was to write the controller software for the Raspberry Pi side. This turned out to be a bit complex, due to needing to read from a total of 10 microcontrollers in quick succession.
|
||||
|
||||
[TBD]
|
||||
|
||||
```Python
|
||||
[pdusensord code]
|
||||
```
|
||||
|
||||
## Putting everything together
|
||||
|
||||
With all the programming and module assembly done, I could begin assembling the final PDU. Here is the final result:
|
||||
|
||||
|
||||
|
||||
I spent a few days load testing it with a resistive heater in my garage to be sure everything was in safe working order, and it passed all my tests. I then let it run with 3 servers attached for 3 full months to do a final burn-in, occasionally switching which outlets they were attached to, without incident.
|
||||
|
||||
The last step was final installation into my rack:
|
||||
|
||||
|
||||
|
||||
And the readings by my monitoring software are exactly what I wanted - accurate to the Watt, at least in theory:
|
||||
|
||||
|
||||
|
||||
I hope you found this interesting!
|
||||
83
content/en/posts/.drafts/my-take-on-free-software/index.md
Normal file
83
content/en/posts/.drafts/my-take-on-free-software/index.md
Normal file
@ -0,0 +1,83 @@
|
||||
+++
|
||||
|
||||
class = "post"
|
||||
date = "2024-02-17T00:00:00-05:00"
|
||||
tags = ["philosophy", "floss"]
|
||||
title = "My Opinions on Free and Libre Open Source Software"
|
||||
description = "Because trying to write them as replies never works"
|
||||
type = "post"
|
||||
weight = 1
|
||||
draft = true
|
||||
|
||||
+++
|
||||
|
||||
## Why Write This?
|
||||
|
||||
Over the years, I've been engaged in many arguments and debates about the nature of open source, especially *vis-a-vis* funding open source. Invariably, my position is apparently unclear to others in the debate, forcing me to expend literally thousands of words clarifying minutae and defeating strawmen.
|
||||
|
||||
As a leader of two projects that are inherently goverened by my philosophy on Free and Libre Open Source Software (hereafter, "FLOSS"), I feel it's important to get my actual opinions out in the open and in as direct and clear a manner as I possibly can. Hence, this blog post.
|
||||
|
||||
## Part One: What I Believe FLOSS "means" a.k.a. "The Philosophy of FLOSS"
|
||||
|
||||
"FLOSS" is a term I use very specifically, because it is a term that Richard Stallman, founder of the Free Software Foundation (FSF) and writer of the GNU General-Purpose License (GPL) suggests we use.
|
||||
|
||||
In terms of general philosophy, I agree with Mr. Stallman on a great number of points, though I do disagree on some.
|
||||
|
||||
To me, "FLOSS" is about two key things, which together make up and ethos and philosophy on software development.
|
||||
|
||||
### FLOSS is about ensuring users have rights
|
||||
|
||||
This part is is pretty self-explanatory, because it's what's covered explicitly in every conception of FLOSS, from the FSF's definition, to the Open Source Initiative (OSI) definition, to the Debian Free Software Guidelines (FSG).
|
||||
|
||||
Personally, I adhere to the FSF and GPL's 4 freedoms, and thus I reject - for myself - non-copyleft licenses.
|
||||
|
||||
> “Free software” means software that respects users' freedom and community. Roughly, it means that the users have the freedom to run, copy, distribute, study, change and improve the software. Thus, “free software” is a matter of liberty, not price. To understand the concept, you should think of “free” as in “free speech,” not as in “free beer.” We sometimes call it “libre software,” borrowing the French or Spanish word for “free” as in freedom, to show we do not mean the software is gratis.
|
||||
|
||||
> You may have paid money to get copies of a free program, or you may have obtained copies at no charge. But regardless of how you got your copies, you always have the freedom to copy and change the software, even to sell copies.
|
||||
|
||||
Now, as I'll discuss below, I have some disagreements with this definition when we begin to talk about "price". But those first two sentences are what's important here.
|
||||
|
||||
### FLOSS is a statement of altruism
|
||||
|
||||
This is the part that I think, if not makes me unique, at least makes me different than most people who write and release "open source" or other FLOSS software.
|
||||
|
||||
I believe that FLOSS software is a statement of altruism. It is about giving something to the world, to humanity, and to the computing community.
|
||||
|
||||
On it's face, this doesn't seem radical, but it is, and it almost completely informs my opinions on monitization and distribution that I'll discuss below. So it's a very important point to take in: to adhere to "FLOSS philosophy" means, to me, to have altruistic motives and actions.
|
||||
|
||||
## Part Two: Monetizing FLOSS done Wrong with "Open-core"
|
||||
|
||||
With my definition of "FLOSS Philosophy" out of the way, let's discuss monetization, starting with things I see as counter to said philosophy and thus intellectually dishonest or worse.
|
||||
|
||||
This blog post originally started as a treatise on Open-Core software, but without the philosophical underpinning, I found it very hard to explain why I felt the way I did about it.
|
||||
|
||||
For anyone unaware of what this term means, "open-core" software is software that is *nominally* FLOSS, but which hides some subset of actual code features behind a proprietary license and other restrictions. For a hypothetical example, consider a grocery list software program. If the program itself is free and open source, but the ability to, say, create lists longer than 50 entries or to create lists of electronics instead of groceries, is a proprietary, paid extension, this is "open-core" software.
|
||||
|
||||
Open-core is one of the most pervasive FLOSS monetization options. Countless pieces of software, from GitLab to CheckMK to MongoDB, are "open-core".
|
||||
|
||||
And I think this model is scummy, because it fundamentally violates the second part of the philosophy. How?
|
||||
|
||||
1. "Open-core" software is not about altruism. Sure, it may *seem* that way because *part* of it is FLOSS. But that other part is not, and thus, the *complete* software solution is not FLOSS
|
||||
|
||||
2. "Open-core" software is, almost *invariably*, marketed as FLOSS, becausee the social clout of FLOSS brings in contributors and users, building an "ecosystem" that is then monitized when...
|
||||
|
||||
3. The lines of all pieces of "open-core" software is arbitrary. Why 50 grocery items, and not 100? Why just groceries but not electronics? Why is the line drawn there, and not somewhere else? The very existence of such a line is arbitrary, as is its positioning. Thus, the software *as a whole* is not FLOSS because of arbitrary limits on its usage.
|
||||
|
||||
Now, some may argue that feature X is "only for enterprises and they should pay" or something similar. This is nonsense. It is not up to the *author* to decide that, it's up to the *user*. And by presenting an arbitrary line, the philosophical idea of altruism goes out the widow. There is nothing altruistic about proprietary software, and "open-core" software is just proprietary software with FLOSS marketing.
|
||||
|
||||
There is one last part of "open-core" software that I find particular egregious. By its nature, "open-core" software is contrary to a volunteer ethos and community-driven development. Consinder the grocery example above and a new contributor called Jon. Jon wants to add support in for listing clothing in addition to grocery items. He wants to exend this "FLOSS" software. Will his contribution even be accepted? After all, the "FLOSS" part is just for *groceries*, and electronics are hidden behind the paywall. Will Jon's merge request languish forever, be ignored, or be outright deleted? And if it's merged to the "FLOSS" portion of the software, the line becomes even more arbitrary.
|
||||
|
||||
## Part Three: Monetizing FLOSS done Wrong with "CLAs"
|
||||
|
||||
Contributor License Agreements or CLAs are incredibly common in "corporate" FLOSS software. They're usually marketed as "protecting" the community, when in fact they do anything but. The software license protects the community; the CLA allows the company to steal contributions at an arbitrary future date by changing the license at will.
|
||||
|
||||
I think it should be pretty obvious to anyone who adheres to the philosophy above why this too is scummy. Contributors make contributions under a particular license, and if that license is changed in the future, particularly to a propreitary license, those contributions are stolen from the world at large and divorced from the altruistic intentions of the contributor.
|
||||
|
||||
Now, not every project with a CLA will necessarily switch licenses in the future. The issue with CLAs is that they give the controlling interests the *option* to do so. And for how long can those interests be trusted, especially from a profit-driven corporate entity?
|
||||
|
||||
## Part Three: Monetizing FLOSS done Right with Employer-sponsored FLOSS
|
||||
|
||||
|
||||
|
||||
|
||||
## Part Four: My Thoughts on the Future of FLOSS
|
||||
@ -0,0 +1,119 @@
|
||||
+++
|
||||
|
||||
class = "post"
|
||||
date = "2019-11-14T19:00:00-05:00"
|
||||
tags = ["politics","union","administrator","developer"]
|
||||
title = "On Knowledge Workers and Unions"
|
||||
description = "or, On unions for developers and aministrators from a Marxist perspective"
|
||||
type = "post"
|
||||
draft = true
|
||||
weight = 1
|
||||
|
||||
+++
|
||||
|
||||
It's been quite a while since I made a post on here, and this one is not about tech itself, but on my opinions, politically-influenced, on Unions and my industry of DevOps. This post is heavily influenced by my own political views. I'm a Marxist - I subscribe to his Labour Theory of Value, his idea of Dialectical and Historical Materialism, and his ideas on the Class Relations of workers (proletarians) and owners (bourgeoisie) along with a few more obscure classes. I'll try to avoid filling this post with excessive leftist jargon in the hopes of not requiring much or any previous knowledge of leftistism, but some may still slip through and I'll try to define them in context. But ultimately I hope this post will inspire some alternative thoughts about unionization and professional relations within our industry to our benefit. I think labour relations are something the "IT Industry" needs to think about broadly as we expand in size, both to avoid selling ourselves short (literally) and repeating the mistakes of the past. I'm sure a lot of this could also be applicable to other fields, but I'm focusing heavily on my own field here, and I will refer to "knowledge workers" to respresent us and a few similar fields more generally and "IT industry" to refer to DevOps (and its two child fields, software development and systems administration) more specifically.
|
||||
|
||||
# On Knowledge Workers and Unions
|
||||
|
||||
I just finished reading [Erik Dietrich's fantastic post](https://daedtech.com/the-beggar-ceo-and-sucker-culture/) on what he calls the Beggar CEO and Sucker Culture. It draws on a previous post of his, [Defining the Corporate Herirarchy](https://daedtech.com/defining-the-corporate-hierarchy/) itself inspired by [a post by Venkatesh Rao](https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-or-the-office-according-to-the-office/) and the original cartoon by Hugh MacLeod. If you haven't read these posts, I'd definitely recommend reading them - the post by Rao in particular is more a series than a post, is incredibly long, but is incredibly important and has definitely influenced a lot about how I think of corporate culture.
|
||||
|
||||
In this article, Dietrich talks about an Ask Abby-style article in which (briefly) a CEO complains that her workers leave after their 9-5 and won't work long hours "like [she] does". And I agree with a lot of Dietrich's points here. But he is careful not to make this "political". He, for his own reasons, keeps the discussion purely in terms of existing Neoliberal Capitalism (the dominant form of Capitalism in the Western core since the early 1980's focused on tax cuts for the rich and "austerity", public service cuts for the working classes) without any class analysis. One commenter noticed this, the aptly-named "Unionist":
|
||||
|
||||
>Unionist: More than 2,000 words and not one of them is “union”. That’s the real problem.
|
||||
|
||||
The resulting comment chain was a fairly expected debate between a few pro-union (and a few seemingly leftist) commenters, some anti-unionist knowledge workers, and the author himself. One section of the chain struck me in particular:
|
||||
|
||||
>Eric Dietrich: I don’t think that collective bargaining is the cure for what ails a high-demand, knowledge work field. Reason being, I don’t think we need to accept the subordination that entails — I see the demand for and cost of software creating a future where we engage in a professional services model, like doctors and lawyers. Those professions don’t need to unionize because they control the game. So should we.
|
||||
|
||||
>Eric Boersma: Respectfully, I think this is a place where you’re letting your own skill set cloud your judgement of what’s possible. Everyone cannot just be consultants, dispensing code where it is valuable and proper, because that’s not a model that fits the needs of many or most programmers or businesses. Most programmers are quite bad at selling their own skill sets. Most poorly estimate what they’re capable of providing – many to the positive, some to the negative. Consultation very rarely provides space for effective on-the-job training, putting a higher workload onto individual employees to continue to grow their skills in useful directions throughout their career. Additionally, it’s telling that your two examples, lawyers and doctors, have significant and difficult profession entrance exams which gate people who are not capable of doing the job effectively from being able to claim the title as well as grueling early-career workloads.
|
||||
|
||||
>The vast majority of doctors do not work for themselves. The vast majority of lawyers do not work for themselves. The vast majority of developers will never work for themselves; all three of those groups can use the power that collective bargaining provides to effectively improve their work conditions. Swearing off unionization as a means of professional advancement is like becoming a programmer and swearing that you’re never going to use TCP/IP because you’ve heard bad things about it. You’re taking tools out of your toolbox before ever giving them a shot, and ignoring them even when they’re clearly the best tool for the job you’re trying to accomplish.
|
||||
|
||||
Dietrich's opinion reflects what I see as a trend in the IT industry away from meta-analysis of our own employment in a leftist lense. I think this is a very flawed, but common, understanding, and he uses an also-common-and-flawed comparison to two other well-discussed knowledge worker careers: doctors and lawyers. Boersma points out several quick examples of these flaws, but I think this deserves a more in-depth breakdown, because the issue of unions in the IT industry, and of knowledge workers more broadly, is woefully un-discussed and becomes more relevant with every new member joining the field.
|
||||
|
||||
## On "Knowledge Workers" - Doctors and Lawyers, a brief history
|
||||
|
||||
The first place to start would be on the comparison made between IT workers and two other extremely-well-cited knowledge worker fields: doctors and lawyers. It is extremely common to mention these two careers when discussing wages, compensation, and other employment-related matters. I hope for their sake that readers are generally aware of these two professions, if not the specifics of each, so I'll avoid discussing them in detail. But the comparisons involved when discussing these two professions almost invariably comes down to a few common elements that both professions share, at least in the Anglosphere (the "English-speaking West" of the US, UK, and the nations of the British Commonwealth, including my own Canada):
|
||||
|
||||
1. Doctors and Lawyers are generally considered to be "high class" jobs worthy of aspiration to.
|
||||
|
||||
1. Doctors and Lawyers, especially Senior (10+ years experience) members, are generally very well-paid, making "upper-middle-class wages".
|
||||
|
||||
1. Doctors and Lawyers generally work very long hours, upwards of 60+ per week.
|
||||
|
||||
1. Doctors and Lawyers are both extremely well-educated, requiring many years of schooling, practical "grunt-work" experience, as well as professional certification.
|
||||
|
||||
If you're an IT professional, or an Engineer, or an Architect, or one of several other knowledge fields, looking at this list, you'll probably notice that these traits are also generally shared by us. And this is not something I see as a flaw in Dietrich's argument. It's absolutely true that these professions, collectively, are something entirely different from what many would call "blue-collar" labour, the proletarians of Marxist thought. And throught the development of leftist though, a name was created to describe these workers: the "Professional and Managerial Class" (PMC). Normal western non-leftist though commonly calls this the "Middle Class", but that is a term devoid of meaningful analysis and hence I will not use it. I also exclude the "managerial" element of this class in my discussion here, partly due to the influence of Rao's opinions on corporate culture in the 21st centure, and also because to lump them together would hinder the analysis.
|
||||
|
||||
As the world has moved further into the 21st century, knowledge workers have come to dominate the discussions of the future of labour. This is after all what someone is implying when they say, usually to a worker who's job has been automated by machinery, "go back to school to get an education and a 'better job'". Education is an important component of PMC careers. But usually when this is said, the person saying it is *not* implying that the target should be one of these two specific jobs. Why is that?
|
||||
|
||||
First and foremost, being "high class" jobs really means little, and under capitalism usually means "very well-paid". So why are these two careers very well paid?
|
||||
|
||||
The most common answer is generally the other two points: "they work very long hours and deserve the high pay", and "they had to spend a lot of time and, without public-paid post-secondary education, money training in the job". But this doesn't tell the whole story.
|
||||
|
||||
The important thing is their *union*.
|
||||
|
||||
But, you may ask, what union? I don't mean "unions" as is traditionally thought of them here. What I mean is their professional standards organizations.
|
||||
|
||||
Doctors have medical schools and the AMA, ACP, etc.. Lawyers have law schools and the "bar" of their jurisdiction. PEng's have engineering schools and their local societies, in Ontario the OSPC. A similar story is true for almost every other PMC industry (except of course the managers).
|
||||
|
||||
These professions all have bodies that, while not focusing primarily, or even at all, on collective bargaining or the proletarian-versus-bourgeoisie element of employment, include an element of certification to the profession. In order to truly call yourself a Lawyer, legally, you must pass the bar. Or pass a medical school exam, followed by a residency. Or write a professional engineering certification. What these bodies do is ensure that these knowledge workers form an insular society, which is gatekept by the existing members of the organization in order to ensure a minimum bar of knowledge before a new member can work professionally.
|
||||
|
||||
This, I think, is what fundamentally separates PMC careers from "blue collar" careers like trades or service work, despite trades in particular superficially resembling this. Every one of these careers has a bar that must be crossed.
|
||||
|
||||
## On Traditional Unions - the why and how
|
||||
|
||||
With the PMC out of the way, we can discuss the main point of unions - their protection of workers from exploitation. All things that are usually associated with unions - collective bargaining for better compensation, workplace standards, protection of members from dismissal - all tie back into this point; they protect workers from exploitation by the ownership class.
|
||||
|
||||
The history of unions is long and depressing. Born out of the conditions of coal mines, steel foundries, and Dickensian sweatshops, they sought to organize workers together to fight against their exploitation. They were brutally, violently, suppressed time and again, but kept fighting until they won what are commonly consindered the hallmarks of modern employment: 8(-ish) hour days, fair pay, weekends, workplace health and safety regulations, and a plethora of other things modern workers take for granted.
|
||||
|
||||
But Western unions lost much of their power throughout the 20th century, as the protections they won became the norm, and as more dangerous, labourious, and low-skilled work was exported to poorer parts of the world. This culminated, I think, in one of the major blows against unions in modern history: the 1981 Air Traffic Controllers strike, where Ronald Reagan called the ATC union's bluff, and fired the entire union's membership and replaced them.
|
||||
|
||||
One of the biggest threats of unions has always been the idea that firing the entire union membership is impossible - if not because they would be easy to replace, then because they could literally occupy the factory, mine, or workplace and prevent others from working. But this was impossible in 1981. How do even 11000 people occupy hundreds of airports, guarded by millitarized law-enforcement officers behind razor-wire fences. It was an exceptional situation for sure, but the ripple effects were wide-reaching.
|
||||
|
||||
Union membership has been declining steadily since the 1980's, especially as neoliberalism became the norm. And unions have since developed a very unsavoury reputation - that they "keep bad employees employed", that they simply suck money from members to fund a union elite (a Labour Aristocracy), and that they're generally useless.
|
||||
|
||||
But of course, this has never been true and has been said of unions since their earliest days. The fact is unions have been a force for more good than bad, and that fixing these problems with unions is one of building class consciousness and solidarity, not insurmountable considering where unions started. Unions can be powerful if they're well organized, and this is visible even after four decades of attack.
|
||||
|
||||
## On the IT Industry's Lack of Organization
|
||||
|
||||
Despite them not being "unionized" in the normal sense, knowledge workers are still organized by their certification bodies. And "blue-collar" workers form traditional unions, which ensure all members are treated fairly.
|
||||
|
||||
But the IT industry lacks both of these. In fact, for the most part it lacks *any* coherent organization at all. And this is indeed a problem.
|
||||
|
||||
First, as a whole the IT industry does not have any sort of professional certification body or specific schooling. There may be optional certs, computer science or engineering courses, and a few professional *societies* like the IETF or the League of Professional System Administrators, but these are not binding organizations. Indeed, a large part of the appeal of the IT industry, especially software development, is that it requires no formal training or education to become a member, allowing those who have self-taught compete with even thir most well-educated colleagues.
|
||||
|
||||
Second, despite appearing very much like a traditional trade, the IT industry has almost no collective representation. Indeed, due to the silo'd nature of the field (as is, for instance Engineering more broadly) individual members of the industry may not see much at all in common with one another, with new subfields being created every day. This makes traditional unionization more difficult as well. After all, how many articles have been written about DevOps and how it breaks walls between System Administrators and Developers? Just tackling this one bridge, entirely within the professional sphere itself, has been a huge challenge. But this helps us.
|
||||
|
||||
Ultimately there are two sources to these issues that I can see:
|
||||
|
||||
1. Generally, IT industry professionals are not responsible for hiring their own, with some exceptions.
|
||||
|
||||
1. The culture of the IT industry, stretching back as far as the 1980's, has always favoured brash individualism over collective solidarity.
|
||||
|
||||
Each of these requires a bit to go into, but both fundamentally shape the lack of organization within the IT industry, as well as help identify the things we need to combat to improve this.
|
||||
|
||||
## On Hiring in IT - buzzwords, HR, and Startups
|
||||
|
||||
* common thing i see people complain about - HR hiring
|
||||
* buzzwords abound, filter by keywords
|
||||
* testing of employees is hard
|
||||
* finding good workers takes time
|
||||
* professional certifications are worthless
|
||||
|
||||
## On the Toxic Culture of IT
|
||||
|
||||
* High focus on individualism, low empathy
|
||||
* Rockstars, 10x
|
||||
* Nerd in-group (reference Dietrich article)
|
||||
* The good - FOSS
|
||||
* The good - DevOps, building connections
|
||||
|
||||
## Why bother with organization?
|
||||
|
||||
* good for people
|
||||
* good for tech! (DevOps, etc.)
|
||||
|
||||
## The ideal IT union
|
||||
|
||||
* descrie the ideal IT union
|
||||
79
content/en/posts/.drafts/open-core-is-cancer/index.md
Normal file
79
content/en/posts/.drafts/open-core-is-cancer/index.md
Normal file
@ -0,0 +1,79 @@
|
||||
---
|
||||
title: "Open Core Is Cancer"
|
||||
date: 2019-08-16T09:53:17-04:00
|
||||
draft: true
|
||||
---
|
||||
|
||||
# Open-Core software is cancer on the FOSS world
|
||||
|
||||
In December 2018, I started the Jellyfin project with a few other contributors. The saga is fairly long and complex, but the short version is: "FOSS" branded software closes off portions of their code, "FOSS" branded software does scummy things, "FOSS" branded software goes closed-source, "FOSS" community forks project, original authors call the FOSS community "moochers", "pirates", etc. "FOSS" community fork sees massive support and encouragement regardless. Clearly people like truly "FOSS" software.
|
||||
|
||||
Now, almost more than 2 years later, this saga has forced me to see something that I cannot unsee: this pattern is everywhere in the "FOSS" software world. It permeates projects, from large Enterprise-backed projects all the way down to small, single-developer projects. This mentality and software monetization paradigm is cancer on the FOSS world. It divides and drags down members of the community. It stifles contributors and users. And it is only getting worse.
|
||||
|
||||
In this post I hope to record my thoughts on this trend, as a sort-of manifesto for my ideas in developing FOSS software, such as my [PVC](https://github.com/parallelvirtualcluster) hypervisor manager project, and of course [Jellyfin](https://jellyfin.media).
|
||||
|
||||
## What is "Open-Core" software?
|
||||
|
||||
Before I can explain why I think Open-Core software is cancerous, it's helpful to have a definition for it. I use the following:
|
||||
|
||||
> Open-Core software is software that consists of at least two versions.
|
||||
> One version is fully FOSS-licensed, distributed, and maintained as FOSS software. It accepts contributions from community members, it is released under a FOSS license, and is generally Gratis. It is often called a "community edition", "free edition", or something of the sort.
|
||||
> The other version(s) are NOT FOSS-licensed. They are almost invariably fully closed-source, and are only available for a fee in binary format. They are often called the "enterprise edition", "paid edition", etc.
|
||||
> Most critially for the definition, and separating "Open-Core" from two closely-related projects that are simply "FOSS" and "Closed-source", is that the *the non-FOSS components extend the functionality of the FOSS component*. Or put another way, the non-FOSS components offer *features and functionality* that are explicitly missing from the FOSS component.
|
||||
|
||||
With this definition out of the way, we can begin to talk about the motivations for "Open-Core" software, and why this is destructive to the ethos of FOSS.
|
||||
|
||||
## Why would anyone choose "Open-Core" as their software model?
|
||||
|
||||
Money.
|
||||
|
||||
It's about the money.
|
||||
|
||||
No, really. It is widely accepted that FOSS software has a "monitization problem". That is, it's hard for programmers to get paid to write software which is given away 100% freely. And even I can conceded that this view has merit. Despite being an avowed Communist, I can understand completely that in a Capitalist economic system, payment for programmer time is important.
|
||||
|
||||
However, this sort of monitization is also the *easiest* form. It's absolutely trivial to hide functionality behind a paywall. After all, the code is the same. A developer simply has to not release part of the code as FOSS and suddenly, if that feature is high demand, they have a guaranteed revenue stream.
|
||||
|
||||
This is not the only method to ensure developer time is compensated, nor even is the requirement for compensation guaranteed. The Debian project is able to create a fully-functional (and in my opinion one of the best) operating systems in the world entirely through volunteer effort. Other projects are able to fund developer teams only through donations. These methods are not impossible. But they require work beyond just writing the code. Volunteer-only projects require non-monetary compensation to be worthwhile to the contributors (often in the form of self-motivated goals, and the sense of community the projects provide, important human motivators). And donation-based projects require active effort to ensure a steady stream of donations. But again, this is not impossible, and there are countless examples of 100% FOSS software which is able to succeed with these methods.
|
||||
|
||||
There is one more method: support. The model of RedHat, the most successful FOSS-based company in history. Provide the product for free, and support it for a fee. Clearly this method also works. But it requires even more effort and process than the donation-based approach. And unfortunately, projects that already use this method can quickly succumb to a desire to boost their monitization by going "open-core"; even RedHat has. the reason is quite simple: you need talent to monitize support, talent that is often very quick to leave for greener pastures. And you have to be able to support complex issues with the software in order to justify the costs. It is a balancing act, and one that is not always successful. But it certainly can be, at least until the Capitalist profit motive overrides moral considerations.
|
||||
|
||||
## Open-Core, by its very definition, disrespects the ideals of FOSS software
|
||||
|
||||
This is a pretty spicy take. And of course, I'm not qualified to speak for every FOSS developer, community member, or Richard Stallman out there. But let me break it down.
|
||||
|
||||
FOSS software, and specifically copyleft (GNU GPL), is built on the 4 key freedoms, laid out by the GNU project:
|
||||
|
||||
* The freedom to run the program as you wish, for any purpose (freedom 0).
|
||||
* The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
|
||||
* The freedom to redistribute copies so you can help others (freedom 2).
|
||||
* The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.
|
||||
|
||||
In the strictest sense, open-core seems to abide by the letter of these laws. After all, one could easily argue that "any given feature in the closed-edition is simply not FOSS, but the rest of the software is". But I think this is fundamentally missing the *point* of the four freedoms. They speak towards an ethos of collaboration, of sharing the softwrae fully in order to enable each user to have those freedoms.
|
||||
|
||||
And this is where I think open-core is insidious. If a user of a piece of software is only able to achieve 80% (for example) of the functionality of the software, under traditional FOSS software it behooves them to implement that remaining 20%. Hopefully, with the support of the community, those improvements can then be shared back to the remainder of the community, both to build on, and improve further. But if that remaining 20% is locked behind the non-FOSS portion of an open-core project, a conflict of interest arrises. Community members may want to implement that functionality for themselves, but attempts to contribute it back upstream will inevitably be rejected - the group running the software does not want that 20% being freely available under the FOSS license. This has a chilling effect on the community: seemingly arbitrary pieces of functionality can, at any time, be stripped out in favour of the non-FOSS component. And this in turn leads towards contributors second-guessing features - does X fall under the banner of Y which is a non-FOSS component? Am I even privy to the existence of the functionality? Will my contribution be rejected without explanation and possibly even deleted? Will I be threatened with "infringement" because I accidentally reimplemented non-FOSS code? This is also a clear conflict of interest with the ideals behind the four freedoms. By causing developers to question the nature of their contributions and where they fit in to the grand monitization scheme of the project, they really are having the second, third, and fourth freedoms compromised.
|
||||
|
||||
## Gratis users are left in the dark
|
||||
|
||||
So, the "Open-Core" model is harmful to developer effort and the spirit of cooperation in FOSS software communities. But it also harms users as well.
|
||||
|
||||
Motivations are often unclear and hidden. I'll admit, I've seen examples of Open-Core software that truly does respect their FOSS and Gratis sides. They release all the important features for free, and reserve a select few for their paid version. However, how can any user ever know where the line is drawn? Perhaps a feature that seems "high-end" to the developers is a critical feature for a very small operation. Or perhaps, in the inverse, a very heavy/large user has no use for any of the paid features. No one wins in this scenario.
|
||||
|
||||
"Open-Core" ultimately results in keeping users, and specifically the Gratis users, in the dark, always second-guessing what might happen next release. Always second guessing whether a "bug" is legitimate or a scummy attempt to force users to the non-FOSS versions. And this harms FOSS in general - after all, the project is branded FOSS, has a FOSS community, but if users see, for example, a big performance bug, and this bug is not present in the non-FOSS versions, it begs the questions "is this a priority to fix?" and "is this intentional?". Both of which harm the developers, as well as the FOSS community more broadly.
|
||||
|
||||
Further, it's often hard to tell whether projects are taking a "release-later" "open-core" position. With this sort of position, the software releases a feature as part of the non-FOSS product today, but intends to eventually release it as part of the FOSS project. But of course, if they announced they were doing this, it would cannibalize their attempt to monitize. So this fact is hidden from end-users, until the day the feature is released. Now, I do think that this method of "Open-Core" is probably the least offensive, since the features do eventually make it to the FOSS community. But this is also predicated on the good-will of those behind the project, and can change at any time. All of this leaving users in the dark.
|
||||
|
||||
## Monitization drives decisions
|
||||
|
||||
Beyond just leaving users in the dark, with "Open-Core" software, the attempt to monitize will, almost inevitibly, change the motivations of the project for the worse.
|
||||
|
||||
In a truly FOSS project, 100% set to the ideals of the 4 freedoms, every decision, every feature, exists to make the project better. It has to - otherwise, why would anyone contribute it? Perhaps it only scratches the itch of a single member of the project, but that's still a motivation to improve the project as a whole. And since every change is open to everyone, it can be adequetly critiqued and tweaked to be the best it can be.
|
||||
|
||||
But when monetization is the first goal and "Open-Core" is the solution, everything flips on its head. Suddenly the "FOSS" aspect is secondary to the moneymaking, and the ideological compromises stack up, one after another, until there is very little left of the freedoms and the project ultimately decides to go closed-source.
|
||||
|
||||
## Under Open-Core, FOSS is just marketing
|
||||
|
||||
Ultimately, under an "Open-Core" model, FOSS just becomes a marketing tool. A way to "hook" users who care about the freedoms of FOSS software into using a product, only to effectively extort them for payment to get 100% of the functionality. This is harmful.
|
||||
|
||||
## There must be a better way
|
||||
|
||||
I don't have all, or even a good, answer to the problem of FOSS monetization. Some models work for some companies or individuals, others don't, and [some people go nuclear](https://lwn.net/Articles/880809/). But as I've stated above, I think "Open-Core" is one of the worst ways to proceed, harming both developers and users in the process. It is an afront to the 4 freedoms in spirit, stifles innovation, and should be stopped.
|
||||
@ -0,0 +1,971 @@
|
||||
+++
|
||||
|
||||
class = "post"
|
||||
date = "2020-05-31T00:00:00-04:00"
|
||||
tags = ["systems administration", "development", "matrix"]
|
||||
title = "Building a scalable, redundant Matrix Homeserver"
|
||||
description = "Deploy an advanced, highly-scalable Matrix instance with split-workers and backends from scratch"
|
||||
type = "post"
|
||||
weight = 1
|
||||
draft = true
|
||||
|
||||
+++
|
||||
|
||||
## What is Matrix?
|
||||
|
||||
Matrix is, fundamentally, a combination of the best parts of IRC, XMPP, and Slack-like communication platforms (Discord, Mattermost, Rocketchat, etc.) built to modern standards. In the Matrix ecosystem, users can run their own server instances, called "homeservers", which then federate amongst themselves to create a "fediverse". It is thus fully distributed, allowing users to communicate with each other on their own terms, while providing all the features one would expect of a global chat system, such as large public rooms, as well as standard features of more modern platforms, like small private groups, direct messages, file uploads, and advanced integration and moderation features, such as bots. The reference homeserver application is called "Synapse", written in Python 3, and released under an Apache 2.0 license.
|
||||
|
||||
In this guide, I seek to provide a document detailing the full steps to deploy a highly-available, redundant, multi-worker Matrix instance, with a fully redundant PostgreSQL database and LDAP authentication and 3PID backend. For those of you who just want to run a quick-and-easy Matrix instance with few advanced features, this guide is probably not for you, and there are numerous guides out there for setting up basic Matrix Synapse instances instead.
|
||||
|
||||
Most of the concepts in this guide, as well as most of the configuration files given, can be adapted to a single-host but still split-worker instance instead, should the configuration below be deemed too complicated or excessive for your usecase. Be sure to carefully read this document and the Matrix documentation if you wish to do so, though most sections can be adapted verbatim.
|
||||
|
||||
## The problem with Synapse
|
||||
|
||||
The main issue with Synapse in its default configuration, as documented by the Matrix project themselves, is that it is single-threaded and non-redundant. Since a lot of actions inside Synapse require significant CPU resources, especially those related to federation, this can be a significant bottleneck. This is especially true in very large rooms, where there are potentially hundreds of joined users on multiple homeservers that all must be communicated to. Without tweaking, this can manifest as posts to large rooms taking an extrordanarily long time, upwards of 10 seconds, to send, as well as problems joining very large rooms for the first time (significant delays, timeouts, join failures, etc.).
|
||||
|
||||
Unfortunately, most homeserver users aren't running their instance on the fastest possible CPU, thus, the only solution to improve performance in this area is to somehow allow the Synapse process to use multiple threads. Luckily for us, Matrix Synapse, since about version 1.10, supports this via workers. Workers allow one to split various functions out of the main Synapse process, which then allows multi-threaded operation and thus, increased performance.
|
||||
|
||||
The configuration of workers [is discussed in the Synapse documentation](https://github.com/matrix-org/synapse/blob/master/docs/workers.md), however a number of details are glossed over or not mentioned completely. Thus, this blog post will outline some of the specific details involved in tuning workers for maximum performance.
|
||||
|
||||
## Step 1 - Prerequisites and planning
|
||||
|
||||
The system outlined in this guide is designed to provide a very scalable and redundant Matrix experience. To this end, the entire system is split up into multiple hosts. In most cases, these should be Virtual Machines running on at least 2 hypervisors for redundancy at the lower layers, though this is outside of the scope of this guide. For our purposes, we will assume that the VMs discussed below are already installed, configured, and operating.
|
||||
|
||||
The configuration outlines here makes use of a total of 14 VMs, with 6 distinct roles. Within each role, either 2 or 3 individual VMs are configured to provide redundancy. The roles can be roughly divided into two categories, frontends that expose services to users, and backends that expose databases to the frontend instances.
|
||||
|
||||
The full VM list, with an example naming convention where X is the host "ID" (e.g. 1, 2, etc.), is as follows:
|
||||
|
||||
Quantity Name Description
|
||||
--- --- ---
|
||||
2 flbX Frontend load balancers running HAProxy, handling incoming requests from clients and federated servers.
|
||||
2 rwX Riot Web instances under Nginx.
|
||||
3 hsX Matrix Synapse homeserver instances running the various workers.
|
||||
2 blbX Backend load balancers running HAProxy, handling database requests from the homeserver instances.
|
||||
3 mpgX PostgreSQL database instances running Patroni with Zookeeper.
|
||||
2 mldX OpenLDAP instances.
|
||||
|
||||
While this setup may seem like overkill, it is, aside from the homeserver instances, the minimum configuration possible while still providing fully redundancy. If redundancy is not desired, a smaller configuration, down to as little as one host, is possible, though this is not detailed below.
|
||||
|
||||
In addition to these 14 VMs, some sort of shared storage must be provided for the sharing of media files (e.g. uploaded files) between the homeservers. For the purpose of this guide, we assume that this is an NFS export at `/srv/matrix` from a system called `nas`. The configuration of redundant, shared storage is outside of the scope of this guide, and thus we will not discuss this beyond this paragraph, though prospective administrators of highly-available Matrix instances should consider this as well.
|
||||
|
||||
All the VMs mentioned above should be running the same operating system. I recommended Debian 10.X (Buster) here, both because it is the distribution I run myself, and also because it provides nearly all the required packages with minimal fuss. If you wish to use another distribution, you must adapt the commands and examples below to fit. Additionally, this guide expects that you are running the Systemd init system. This is not the place for continuing the seemingly-endless initsystem debate, but some advanced features of Systemd (such as template units) are used below and in the official Matrix documentation, so we expect this is the initsystem you are running, and you are on your own if you choose to use an alternative.
|
||||
|
||||
For networking purposes, it is sufficient to place all the above servers in a single RFC1918 network. Outbound NAT should be configured to allow all hosts to reach the internet, and a small number of ports should be permitted through a firewall towards the external load balancer VIP (virtual IP address). The following is an example IP configuration in the network `10.0.0.0/24` that can be used for this guide, though you may of course choose a different subnet and host IP allocation scheme if you wish. All these names should resolve in DNS, or be configured in `/etc/hosts` on all machines.
|
||||
|
||||
IP address Hostname Description
|
||||
--- --- ---
|
||||
10.0.0.1 gw NAT gateway and firewall, upstream router.
|
||||
10.0.0.2 blbvip Floating VIP for blbX instances.
|
||||
10.0.0.3 blb1 blbX host 1.
|
||||
10.0.0.4 blb2 blbX host 2.
|
||||
10.0.0.5 mpg1 mpgX host 1.
|
||||
10.0.0.6 mpg2 mpgX host 2.
|
||||
10.0.0.7 mpg3 mpgX host 3.
|
||||
10.0.0.8 mld1 mldX host 1.
|
||||
10.0.0.9 mld2 mldX host 2.
|
||||
10.0.0.10 flbvip Floating VIP for flbX instances.
|
||||
10.0.0.11 flb1 flbX host 1.
|
||||
10.0.0.12 flb2 flbX host 2.
|
||||
10.0.0.13 rw1 rwX host 1.
|
||||
10.0.0.14 rw2 rwX host 2.
|
||||
10.0.0.15 hs1 hsX host 1.
|
||||
10.0.0.16 hs2 hsX host 2.
|
||||
10.0.0.17 hs3 hsX host 3.
|
||||
|
||||
# Step 2 - Installing and configuring OpenLDAP instances
|
||||
|
||||
[OpenLDAP]() is a common LDAP server, which provides centralized user administration as well as the configuration of additional details in a user directory. Installing and configuring OpenLDAP is beyond the scope of this guide, though the Matrix Homeserver configurations below assume that this is operating and that all Matrix users are stored in the LDAP database. In our example configuration, there are 2 OpenLDAP instances running with replication (`syncrepl`) between them, which are then load-balanced in a multi-master fashion. Since no services below here will be performing writes to this database, this is fine. The administrator is expected to configure some sort of user management layer of their choosing (e.g. scripts, or a web-based frontend) for managing users, resetting passwords, etc.
|
||||
|
||||
While this short section may seem like a cop-out, this is an extensive topic with many potential caveats, and should thus have its own (future) post on this blog. Until then, I trust that the administrator is able to look up and configure this themselves. I include these references only to help guide the administrator towards full-stack redundancy and to explain why there are LDAP sections in the backend load balancer configurations.
|
||||
|
||||
# Step 3 - Installing and configuring Patroni instances
|
||||
|
||||
[Patroni](https://github.com/zalando/patroni) is a service manager for PostgreSQL which provides automated failover and replication support for a PostgreSQL database. Like OpenLDAP above, the configuration of Patroni is beyond the scope of this guide, and the configurations below assume that this is operating and already configured. In our example configuration, there are 3 Patroni instances, which is the minimum required for quorum among the members. As above, I do plan to document this in a future post, but until then, I recommend the administrator reference the Patroni documentation as well as [this other post on my blog](https://www.boniface.me/post/patroni-and-haproxy-agent-checks/) for details on setting up the Patroni instances.
|
||||
|
||||
# Step 4 - Installing and configuring backend load balancers
|
||||
|
||||
While I do not go into details in the previous two steps, this section details how to make use of a redundant pair of HAProxy instances to expose the redundant databases mentioned above to the Homeserver instances.
|
||||
|
||||
In order to provide a single entrypoint to the load balancers, the administrator should first install and configure Keepalived. The following `/etc/keepalived/keepalived.conf` configuration will set up the `blbvip` floating IP address between the two instances, while providing checking of the HAProxy instance health. This configuration below can be used on both proxy hosts, and inline comments provide additional clarification and information as well as indicating any changes required between the hosts.
|
||||
|
||||
```
|
||||
# Global configuration options.
|
||||
global_defs {
|
||||
# Use a dedicated IPv4 multicast group; adjust the last octet if this conflicts within your network.
|
||||
vrrp_mcast_group4 224.0.0.21
|
||||
|
||||
# Use VRRP version 3 in strict mode and with no iptables configuration.
|
||||
vrrp_version 3
|
||||
vrrp_strict
|
||||
vrrp_iptables
|
||||
}
|
||||
|
||||
# HAProxy check script, to ensure that this host will not become PRIMARY if HAProxy is not active.
|
||||
vrrp_script chk {
|
||||
script "/usr/bin/haproxyctl show info"
|
||||
interval 5
|
||||
rise 2
|
||||
fall 2
|
||||
}
|
||||
|
||||
# Primary IPv4 VIP configuration.
|
||||
vrrp_instance VIP_4 {
|
||||
# Initial state, MASTER on both hosts to ensure that at least one host becomes active immediately on boot.
|
||||
state MASTER
|
||||
|
||||
# Interface to place the VIP on; this is optional though still recommended on single-NIC machines; replace "ens2" with your actual NIC name.
|
||||
interface ens2
|
||||
|
||||
# A dedicated, unique virtual router ID for this cluster; adjust this if required.
|
||||
virtual_router_id 21
|
||||
|
||||
# The priority. Set to 200 for the primary (first) server, and to 100 for the secondary (second) server.
|
||||
priority 200
|
||||
|
||||
# The (list of) virtual IP address(es) with CIDR subnet mask for the "blbvip" host.
|
||||
virtual_ipaddress {
|
||||
10.0.0.2/24
|
||||
}
|
||||
|
||||
# Use the HAProxy check script for this VIP.
|
||||
track_script {
|
||||
chk
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Once the above configuration is installed at `/etc/keepalived/keepalived.conf`, restart the Keepalived service with `sudo systemctl restart keepalived` on each host. You should see the VIP become active on the first host.
|
||||
|
||||
The HAProxy configuration below can be used verbatim on both proxy hosts, and inline comments provide additional clarification and information to avoid breaking up the configuration snippit. This configuration makes use of an advanced feature for the Patroni hosts [which is detailed in another post on this blog](https://www.boniface.me/post/patroni-and-haproxy-agent-checks/), to ensure that only the active Patroni node is sent traffic and to avoid the other two database hosts from reporting `DOWN` state all the time.
|
||||
|
||||
```
|
||||
# Global settings - tune HAProxy for optimal performance, administration, and security.
|
||||
global
|
||||
# Send logs to the "local6" service on the local host, via an rsyslog UDP listener. Enable debug logging to log individual connections.
|
||||
log ip6-localhost:514 local6 debug
|
||||
log-send-hostname
|
||||
chroot /var/lib/haproxy
|
||||
pidfile /run/haproxy/haproxy.pid
|
||||
|
||||
# Use multi-threadded support (available with HAProxy 1.8+) for optimal performance in high-load situations. Adjust `nbthread` as needed for your host's core count (1/2 is optimal).
|
||||
nbproc 1
|
||||
nbthread 2
|
||||
|
||||
# Provide a stats socket for `hatop`
|
||||
stats socket /var/lib/haproxy/admin.sock mode 660 level admin process 1
|
||||
stats timeout 30s
|
||||
|
||||
# Run in daemon mode as the `haproxy` user
|
||||
daemon
|
||||
user haproxy
|
||||
group haproxy
|
||||
|
||||
# Set the global connection limit to 10000; this is certainly overkill but avoids needing to tweak this for larger instances.
|
||||
maxconn 10000
|
||||
|
||||
# Default settings - provide some default settings that are applicable to (most) of the listeners and backends below.
|
||||
defaults
|
||||
log global
|
||||
timeout connect 30s
|
||||
timeout client 15m
|
||||
timeout server 15m
|
||||
log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq %bi:%bp"
|
||||
|
||||
# Statistics listener with authentication - provides stats for the HAProxy instance via a WebUI (optional)
|
||||
userlist admin
|
||||
# WARNING - CHANGE ME TO A REAL PASSWORD OR A SHA512-hashed PASSWORD (with `password` instead of `insecure-password`). IF YOU USE `insecure-password`, MAKE SURE THIS CONFIGURATION IS NOT WORLD-READABLE.
|
||||
user admin insecure-password P4ssw0rd
|
||||
listen stats
|
||||
bind :::5555 v4v6
|
||||
mode http
|
||||
stats enable
|
||||
stats uri /
|
||||
stats hide-version
|
||||
stats refresh 10s
|
||||
stats show-node
|
||||
stats show-legends
|
||||
acl is_admin http_auth(admin)
|
||||
http-request auth realm "Admin access required" if !is_admin
|
||||
|
||||
# Stick-tables peers configuration
|
||||
peers keepalived-pair
|
||||
peer blb1 10.0.0.3:1023
|
||||
peer blb1 10.0.0.4:1023
|
||||
|
||||
# LDAP frontend
|
||||
frontend ldap
|
||||
bind :::389 v4v6
|
||||
maxconn 1000
|
||||
mode tcp
|
||||
option tcpka
|
||||
default_backend ldap
|
||||
|
||||
# PostgreSQL frontend
|
||||
frontend pgsql
|
||||
bind :::5432 v4v6
|
||||
maxconn 1000
|
||||
mode tcp
|
||||
option tcpka
|
||||
default_backend pgsql
|
||||
|
||||
# LDAP backend
|
||||
backend ldap
|
||||
mode tcp
|
||||
option tcpka
|
||||
balance leastconn
|
||||
server mld1 10.0.0.8:389 check inter 2000 maxconn 64
|
||||
server mld2 10.0.0.9:389 check inter 2000 maxconn 64
|
||||
|
||||
# PostgreSQL backend using agent check
|
||||
backend pgsql
|
||||
mode tcp
|
||||
option tcpka
|
||||
option httpchk OPTIONS /master
|
||||
http-check expect status 200
|
||||
server mpg1 10.0.0.5:5432 maxconn 1000 check agent-check agent-port 5555 inter 1s fall 2 rise 2 on-marked-down shutdown-sessions port 8008
|
||||
server mpg2 10.0.0.6:5432 maxconn 1000 check agent-check agent-port 5555 inter 1s fall 2 rise 2 on-marked-down shutdown-sessions port 8008
|
||||
server mpg3 10.0.0.7:5432 maxconn 1000 check agent-check agent-port 5555 inter 1s fall 2 rise 2 on-marked-down shutdown-sessions port 8008
|
||||
```
|
||||
|
||||
Once the above configurations are installed on each server, restart the HAProxy service with `sudo systemctl restart haproxy`. Use `sudo hatop -s /var/lib/haproxy/admin.sock` to view the status of the backends, and continue once all are running correctly.
|
||||
|
||||
## Step 5 - Install and configure Synapse instances
|
||||
|
||||
The core homeserver processes should be configured on all homeserver machines. There are numerous options but
|
||||
|
||||
|
||||
## Step 2 - Configure systemd units
|
||||
|
||||
The easiest way to set up workers is to use a template unit file with a series of individual worker configurations. A series of unit files are [provided within the Synapse documentation](https://github.com/matrix-org/synapse/tree/master/docs/systemd-with-workers), which can be used to set up template-based workers.
|
||||
|
||||
I decided to modify these somewhat, by replacing the configuration directory at `/etc/matrix-synapse/workers` with `/etc/matrix-synapse/worker.d`, but this is just a personal preference. If you're using official Debian packages (as I am), you will also need to adjust the path to the Python binary. I also adjust the description to be a little more consistent. The resulting template worker unit looks like this:
|
||||
|
||||
```
|
||||
[Unit]
|
||||
Description = Synapse Matrix worker %i
|
||||
PartOf = matrix-synapse.target
|
||||
|
||||
[Service]
|
||||
Type = notify
|
||||
NotifyAccess = main
|
||||
User = matrix-synapse
|
||||
WorkingDirectory = /var/lib/matrix-synapse
|
||||
EnvironmentFile = /etc/default/matrix-synapse
|
||||
ExecStart = /usr/bin/python3 -m synapse.app.generic_worker --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --config-path=/etc/matrix-synapse/worker.d/%i.yaml
|
||||
ExecReload = /bin/kill -HUP $MAINPID
|
||||
Restart = on-failure
|
||||
RestartSec = 3
|
||||
SyslogIdentifier = matrix-synapse-%i
|
||||
|
||||
[Install]
|
||||
WantedBy = matrix-synapse.target
|
||||
```
|
||||
|
||||
There is also a generic target unit that should be installed to provide a unified management point for both the primary Synapse process as well as the workers. After some similar tweaks, including adjusting the After condition to use `network-online.target` instead of `network.target`, the resulting file looks like this:
|
||||
|
||||
```
|
||||
[Unit]
|
||||
Description = Synapse Matrix homeserver target
|
||||
After = network-online.target
|
||||
|
||||
[Install]
|
||||
WantedBy = multi-user.target
|
||||
```
|
||||
|
||||
Install both of these units, as `matrix-synapse-worker@.service` and `matrix-synapse.target` respectively, to `/etc/systemd/system`, and run `sudo systemctl daemon-reload`.
|
||||
|
||||
Once the unit files are prepared, you can begin building each individual worker configuration.
|
||||
|
||||
## Step 3 - Configure the individual workers
|
||||
|
||||
Each worker is configured via an individual YAML configuration file, with our units under `/etc/matrix-synapse/worker.d`. By design, each worker makes use of `homeserver.yaml` for all global configuration values, then the individual worker configurations override specific settings for the particular worker. The [Synapse documentation on workers](https://github.com/matrix-org/synapse/blob/master/docs/workers.md) provides a good starting point, but some sections are vague, and thus this guide hopes to provide more detailed instructions and explanations.
|
||||
|
||||
Each worker is given a specific section below, which includes the full YAML configuration I use, as well as any notes about the configuration that are worth mentioning. They are provided in alphabetical order, rather than the order provided in the documentation above, for clarity.
|
||||
|
||||
For any worker which responds to REST, a port must be selected for the worker to listen on. The main homeserver runs by default on port 8008, and I have `ma1xd` running on port 8090, so I chose ports from 8091 to 8097 for the various REST workers in order to keep them in a consistent range.
|
||||
|
||||
Finally, the main homeserver must be configured with both TCP and HTTP replication listeners, to provide communication between the workers and the main process. For this I use the ports provided by the Matrix documentation above, 9092 and 9093, with the following configuration in the main `homeserver.yaml` `listeners` section:
|
||||
|
||||
```
|
||||
listeners:
|
||||
- port: 8008
|
||||
tls: false
|
||||
bind_addresses:
|
||||
- '::'
|
||||
type: http
|
||||
x_forwarded: true
|
||||
resources:
|
||||
- names: [client, webclient]
|
||||
compress: true
|
||||
- port: 9092
|
||||
bind_addresses:
|
||||
- '::'
|
||||
type: replication
|
||||
x_forwarded: true
|
||||
- port: 9093
|
||||
bind_addresses:
|
||||
- '::'
|
||||
type: http
|
||||
x_forwarded: true
|
||||
resources:
|
||||
- names: [replication]
|
||||
```
|
||||
|
||||
There are a couple adjustments here from the default configuration. First, the `federation` resource has been removed from the primary listener, since this is implemented as a worker below. TLS is disabled here, and `x_forwarded: true` is added to all 3 frontends, since this is handled by a reverse proxy, as discussed later in this guide. All three listeners use a global IPv6+IPv4 bind address of `::` so they will be accessible by other machines on the network, which is important for the final, multi-host setup. As noted in the Matrix documentation, *ensure that the replication ports are not publicly accessible*, since they are unauthenticated and unencrypted; I run these servers on an RFC1918 private network behind a firewall so this is secure, but you will need to provide some sort of firewall if your Synapse instance is directly available on the public Internet.
|
||||
|
||||
The configurations below show a hostname, `mlbvip`, for all instances of `worker_replication_host`. This will be explained and discussed further in the reverse proxy section. If you are only interested in running a "single-server" instance, you may use `localhost`, `127.0.0.1`, or `::1` here instead, as these ports will not managed by the reverse proxy in such a setup.
|
||||
|
||||
#### `appservice` worker (`/etc/matrix-synapse/worker.d/appservice.yaml`)
|
||||
|
||||
The `appservice` worker does not service REST endpoints, and thus has a minimal configuration.
|
||||
|
||||
```
|
||||
---
|
||||
worker_app: synapse.app.appservice
|
||||
|
||||
worker_replication_host: mlbvip
|
||||
worker_replication_port: 9092
|
||||
worker_replication_http_port: 9093
|
||||
```
|
||||
|
||||
Once the configuration is in place, enable the worker by running `sudo systemctl enable matrix-synapse-worker@appservice.service`. It will be started later in the process.
|
||||
|
||||
|
||||
#### `client_reader` worker (`/etc/matrix-synapse/worker.d/client_reader.yaml`)
|
||||
|
||||
The `client_reader` worker services REST endpoints, and thus has a listener section, with port 8091 chosen.
|
||||
|
||||
```
|
||||
---
|
||||
worker_app: synapse.app.client_reader
|
||||
|
||||
worker_replication_host: mlbvip
|
||||
worker_replication_port: 9092
|
||||
worker_replication_http_port: 9093
|
||||
|
||||
worker_listeners:
|
||||
- type: http
|
||||
port: 8091
|
||||
resources:
|
||||
- names:
|
||||
- client
|
||||
```
|
||||
|
||||
Once the configuration is in place, enable the worker by running `sudo systemctl enable matrix-synapse-worker@client_reader.service`. It will be started later in the process.
|
||||
|
||||
#### `event_creator` worker (`/etc/matrix-synapse/worker.d/event_creator.yaml`)
|
||||
|
||||
The `event_creator` worker services REST endpoints, and thus has a listener section, with port 8092 chosen.
|
||||
|
||||
```
|
||||
---
|
||||
worker_app: synapse.app.event_creator
|
||||
|
||||
worker_replication_host: mlbvip
|
||||
worker_replication_port: 9092
|
||||
worker_replication_http_port: 9093
|
||||
|
||||
worker_listeners:
|
||||
- type: http
|
||||
port: 8092
|
||||
resources:
|
||||
- names:
|
||||
- client
|
||||
```
|
||||
|
||||
Once the configuration is in place, enable the worker by running `sudo systemctl enable matrix-synapse-worker@event_creator.service`. It will be started later in the process.
|
||||
|
||||
#### `federation_reader` worker (`/etc/matrix-synapse/worker.d/federation_reader.yaml`)
|
||||
|
||||
The `federation_reader` worker services REST endpoints, and thus has a listener section, with port 8093 chosen. Note that this worker, in addition to a `client` resource, also provides a `federation` resource.
|
||||
|
||||
```
|
||||
---
|
||||
worker_app: synapse.app.federation_reader
|
||||
|
||||
worker_replication_host: mlbvip
|
||||
worker_replication_port: 9092
|
||||
worker_replication_http_port: 9093
|
||||
|
||||
worker_listeners:
|
||||
- type: http
|
||||
port: 8093
|
||||
resources:
|
||||
- names:
|
||||
- client
|
||||
- federation
|
||||
```
|
||||
|
||||
Once the configuration is in place, enable the worker by running `sudo systemctl enable matrix-synapse-worker@federation_reader.service`. It will be started later in the process.
|
||||
|
||||
#### `federation_sender` worker (`/etc/matrix-synapse/worker.d/federation_sender.yaml`)
|
||||
|
||||
The `federation_sender` worker does not service REST endpoints, and thus has a minimal configuration.
|
||||
|
||||
```
|
||||
---
|
||||
worker_app: synapse.app.federation_sender
|
||||
|
||||
worker_replication_host: mlbvip
|
||||
worker_replication_port: 9092
|
||||
worker_replication_http_port: 9093
|
||||
```
|
||||
|
||||
Once the configuration is in place, enable the worker by running `sudo systemctl enable matrix-synapse-worker@federation_sender.service`. It will be started later in the process.
|
||||
|
||||
#### `frontend_proxy` worker (`/etc/matrix-synapse/worker.d/frontend_proxy.yaml`)
|
||||
|
||||
The `frontend_proxy` worker services REST endpoints, and thus has a listener section, with port 8094 chosen. This worker has an additional configuration parameter, `worker_main_http_uri`, which allows the worker to direct requests back to the primary Synapse instance. Similar to the `worker_replication_host` value, this uses `mlbvip` in this example, and for "single-server" instances *must* be replaced with `localhost`, `127.0.0.1`, or `::1` instead, as this port will not managed by the reverse proxy in such a setup.
|
||||
|
||||
```
|
||||
---
|
||||
worker_app: synapse.app.frontend_proxy
|
||||
|
||||
worker_replication_host: mlbvip
|
||||
worker_replication_port: 9092
|
||||
worker_replication_http_port: 9093
|
||||
|
||||
worker_main_http_uri: http://mlbvip:8008
|
||||
|
||||
worker_listeners:
|
||||
- type: http
|
||||
port: 8094
|
||||
resources:
|
||||
- names:
|
||||
- client
|
||||
```
|
||||
|
||||
Once the configuration is in place, enable the worker by running `sudo systemctl enable matrix-synapse-worker@frontend_proxy.service`. It will be started later in the process.
|
||||
|
||||
#### `media_repository` worker (`/etc/matrix-synapse/worker.d/media_repository.yaml`)
|
||||
|
||||
The `media_repository` worker services REST endpoints, and thus has a listener section, with port 8095 chosen. Note that this worker, in addition to a `client` resource, also provides a `media` resource.
|
||||
|
||||
```
|
||||
---
|
||||
worker_app: synapse.app.media_repository
|
||||
|
||||
worker_replication_host: mlbvip
|
||||
worker_replication_port: 9092
|
||||
worker_replication_http_port: 9093
|
||||
|
||||
worker_listeners:
|
||||
- type: http
|
||||
port: 8095
|
||||
resources:
|
||||
- names:
|
||||
- client
|
||||
- media
|
||||
```
|
||||
|
||||
Once the configuration is in place, enable the worker by running `sudo systemctl enable matrix-synapse-worker@media_repository.service`. It will be started later in the process.
|
||||
|
||||
#### `pusher` worker (`/etc/matrix-synapse/worker.d/pusher.yaml`)
|
||||
|
||||
The `pusher` worker does not service REST endpoints, and thus has a minimal configuration.
|
||||
|
||||
```
|
||||
---
|
||||
worker_app: synapse.app.pusher
|
||||
|
||||
worker_replication_host: mlbvip
|
||||
worker_replication_port: 9092
|
||||
worker_replication_http_port: 9093
|
||||
```
|
||||
|
||||
Once the configuration is in place, enable the worker by running `sudo systemctl enable matrix-synapse-worker@pusher.service`. It will be started later in the process.
|
||||
|
||||
#### `synchrotron` worker (`/etc/matrix-synapse/worker.d/synchrotron.yaml`)
|
||||
|
||||
The `synchrotron` worker services REST endpoints, and thus has a listener section, with port 8096 chosen.
|
||||
|
||||
```
|
||||
---
|
||||
worker_app: synapse.app.synchrotron
|
||||
|
||||
worker_replication_host: mlbvip
|
||||
worker_replication_port: 9092
|
||||
worker_replication_http_port: 9093
|
||||
|
||||
worker_listeners:
|
||||
- type: http
|
||||
port: 8096
|
||||
resources:
|
||||
- names:
|
||||
- client
|
||||
```
|
||||
|
||||
Once the configuration is in place, enable the worker by running `sudo systemctl enable matrix-synapse-worker@synchrotron.service`. It will be started later in the process.
|
||||
|
||||
#### `user_dir` worker (`/etc/matrix-synapse/worker.d/user_dir.yaml`)
|
||||
|
||||
The `user_dir` worker services REST endpoints, and thus has a listener section, with port 8097 chosen.
|
||||
|
||||
```
|
||||
---
|
||||
worker_app: synapse.app.user_dir
|
||||
|
||||
worker_replication_host: mlbvip
|
||||
worker_replication_port: 9092
|
||||
worker_replication_http_port: 9093
|
||||
|
||||
worker_listeners:
|
||||
- type: http
|
||||
port: 8097
|
||||
resources:
|
||||
- names:
|
||||
- client
|
||||
```
|
||||
|
||||
Once the configuration is in place, enable the worker by running `sudo systemctl enable matrix-synapse-worker@user_dir.service`. It will be started later in the process.
|
||||
|
||||
## Step 4 - Riot instance
|
||||
|
||||
Riot Web is the reference frontend for Matrix instances, allowing a user to access Matrix via a web browser. Riot is an optional, but recommended, feature for your homeserver
|
||||
|
||||
## Step 5 - ma1sd instance
|
||||
|
||||
ma1sd is an optional component for Matrix, providing 3PID (e.g. email, phone number, etc.) lookup services for Matrix users. I use ma1sd with my Matrix instance for two main reasons: first, to map nice-looking user data such as full names to my Matrix users, and also as RESTful authentication provider to interface Matrix with my LDAP instance. For this guide, I assume that you already have an LDAP instance set up and that you are using it in this manner too.
|
||||
|
||||
## Step 6 - Reverse proxy
|
||||
|
||||
For this guide, HAProxy was selected as the reverse proxy of choice. This is mostly due to my familiarity with it, but also to a lesser degree for its more advanced functionality and, in my opinion, nicer configuration syntax. This section provides configuration for a "load-balanced", multi-server instance with an additional 2 slave worker servers and with separate proxy servers; a single-server instance with basic split workers can be made by removing the additional servers. This will allow the homeserver to grow to many dozens or even hundreds of users. In this setup, the load balancer is separated out onto a separate pair of servers, with a `keepalived` VIP (virtual IP address) shared between them. The name `mlbvip` should resolve to this IP, and all previous worker configurations should use this `mlbvip` hostname as the connection target for the replication directives. Both a reasonable `keepalived` configuration for the VIP and the HAProxy configuration are provided.
|
||||
|
||||
The two proxy hosts can be named as desired, in my case using the names `mlb1` and `mlb2`. These names must resolve in DNS, or be specified in `/etc/hosts` on both servers.
|
||||
|
||||
The Keepalived configuration below can be used on both proxy hosts, and inline comments provide additional clarification and information as well as indicating any changes required between the hosts. The VIP should be selected from the free IPs of your server subnet.
|
||||
|
||||
```
|
||||
# Global configuration options.
|
||||
global_defs {
|
||||
# Use a dedicated IPv6 multicast group; adjust the last octet if this conflicts within your network.
|
||||
vrrp_mcast_group4 224.0.0.21
|
||||
|
||||
# Use VRRP version 3 in strict mode and with no iptables configuration.
|
||||
vrrp_version 3
|
||||
vrrp_strict
|
||||
vrrp_iptables
|
||||
}
|
||||
|
||||
# HAProxy check script, to ensure that this host will not become PRIMARY if HAProxy is not active.
|
||||
vrrp_script chk {
|
||||
script "/usr/bin/haproxyctl show info"
|
||||
interval 5
|
||||
rise 2
|
||||
fall 2
|
||||
}
|
||||
|
||||
# Primary IPv4 VIP configuration.
|
||||
vrrp_instance VIP_4 {
|
||||
# Initial state, MASTER on both hosts to ensure that at least one host becomes active immediately on boot.
|
||||
state MASTER
|
||||
|
||||
# Interface to place the VIP on; this is optional though still recommended on single-NIC machines; replace "ens2" with your actual NIC name.
|
||||
interface ens2
|
||||
|
||||
# A dedicated, unique virtual router ID for this cluster; adjust this if required.
|
||||
virtual_router_id 21
|
||||
|
||||
# The priority. Set to 200 for the primary (first) server, and to 100 for the secondary (second) server.
|
||||
priority 200
|
||||
|
||||
# The (list of) virtual IP address(es) with CIDR subnet mask.
|
||||
virtual_ipaddress {
|
||||
10.0.0.10/24
|
||||
}
|
||||
|
||||
# Use the HAProxy check script for this VIP.
|
||||
track_script {
|
||||
chk
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Once the above configuration is installed at `/etc/keepalived/keepalived.conf`, restart the Keepalived service with `sudo systemctl restart keepalived` on each host. You should see the VIP become active on the first host.
|
||||
|
||||
The HAProxy configuration below can be used verbatim on both proxy hosts, and inline comments provide additional clarification and information to avoid breaking up the configuration snippit. In this example we use `peer` configuration to enable the use of `stick-tables` directives, which ensure that individual user sessions are synchronized between the HAProxy instances during failovers; with this setting, if the hostnames of the load balancers do not resolve, HAProxy will not start. Some additional, advanced features are used in several ACLs to ensure that, for instance, specific users and rooms are always directed to the same workers if possible, which is required by the individual workers as specified in [the Matrix documentation](https://github.com/matrix-org/synapse/blob/master/docs/workers.md).
|
||||
|
||||
```
|
||||
global
|
||||
# Send logs to the "local6" service on the local host, via an rsyslog UDP listener. Enable debug logging to log individual connections.
|
||||
log ip6-localhost:514 local6 debug
|
||||
log-send-hostname
|
||||
chroot /var/lib/haproxy
|
||||
pidfile /run/haproxy/haproxy.pid
|
||||
|
||||
# Use multi-threadded support (available with HAProxy 1.8+) for optimal performance in high-load situations. Adjust `nbthread` as needed for your host's core count (2-4 is optimal).
|
||||
nbproc 1
|
||||
nbthread 4
|
||||
|
||||
# Provide a stats socket for `hatop`
|
||||
stats socket /var/lib/haproxy/admin.sock mode 660 level admin process 1
|
||||
stats timeout 30s
|
||||
|
||||
# Run in daemon mode as the `haproxy` user
|
||||
daemon
|
||||
user haproxy
|
||||
group haproxy
|
||||
|
||||
# Set the global connection limit to 10000; this is certainly overkill but avoids needing to tweak this for larger instances.
|
||||
maxconn 10000
|
||||
|
||||
# Set default SSL configurations, including a modern highly-secure configuration requiring TLS1.2 client support.
|
||||
ca-base /etc/ssl/certs
|
||||
crt-base /etc/ssl/private
|
||||
tune.ssl.default-dh-param 2048
|
||||
ssl-default-bind-ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384
|
||||
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
|
||||
ssl-default-server-ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384
|
||||
ssl-default-server-options ssl-min-ver TLSv1.2 no-tls-tickets
|
||||
|
||||
defaults
|
||||
log global
|
||||
|
||||
option http-keep-alive
|
||||
option forwardfor except 127.0.0.0/8
|
||||
option redispatch
|
||||
option dontlognull
|
||||
option splice-auto
|
||||
option log-health-checks
|
||||
|
||||
default-server init-addr libc,last,none
|
||||
|
||||
timeout client 30s
|
||||
timeout connect 30s
|
||||
timeout server 300s
|
||||
timeout tunnel 3600s
|
||||
timeout http-keep-alive 60s
|
||||
timeout http-request 30s
|
||||
timeout queue 60s
|
||||
timeout tarpit 60s
|
||||
|
||||
peers keepalived-pair
|
||||
# Peers for site bl0
|
||||
peer mlb1.i.bonilan.net mlb1.i.bonilan.net:1023
|
||||
peer mlb2.i.bonilan.net mlb2.i.bonilan.net:1023
|
||||
|
||||
resolvers nsX
|
||||
nameserver ns1 10.101.0.61:53
|
||||
nameserver ns2 10.101.0.62:53
|
||||
|
||||
userlist admin
|
||||
user admin password MySuperSecretPassword123
|
||||
|
||||
listen stats
|
||||
bind :::5555 v4v6
|
||||
mode http
|
||||
stats enable
|
||||
stats uri /
|
||||
stats hide-version
|
||||
stats refresh 10s
|
||||
stats show-node
|
||||
stats show-legends
|
||||
acl is_admin http_auth(admin)
|
||||
http-request auth realm "Admin access" if !is_admin
|
||||
|
||||
frontend http
|
||||
bind :::80 v4v6
|
||||
mode http
|
||||
option httplog
|
||||
acl url_letsencrypt path_beg /.well-known/acme-challenge/
|
||||
use_backend letsencrypt if url_letsencrypt
|
||||
redirect scheme https if !url_letsencrypt !{ ssl_fc }
|
||||
|
||||
frontend https
|
||||
bind :::443 v4v6 ssl crt /etc/ssl/letsencrypt/ alpn h2,http/1.1
|
||||
bind :::8448 v4v6 ssl crt /etc/ssl/letsencrypt/ alpn h2,http/1.1
|
||||
mode http
|
||||
option httplog
|
||||
capture request header Host len 64
|
||||
|
||||
http-request set-header X-Forwarded-Proto https
|
||||
http-request add-header X-Forwarded-Host %[req.hdr(host)]
|
||||
http-request add-header X-Forwarded-Server %[req.hdr(host)]
|
||||
http-request add-header X-Forwarded-Port %[dst_port]
|
||||
|
||||
# Method ACLs
|
||||
acl http_method_get method GET
|
||||
|
||||
# Domain ACLs
|
||||
acl host_matrix hdr_dom(host) im.bonifacelabs.ca
|
||||
acl host_element hdr_dom(host) chat.bonifacelabs.ca
|
||||
|
||||
# URL ACLs
|
||||
# Sync requests
|
||||
acl url_workerX_stick-auth path_reg ^/_matrix/client/(r0|v3)/sync$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(api/v1|r0|v3)/events$
|
||||
acl url_workerX_stick-auth path_reg ^/_matrix/client/(api/v1|r0|v3)/initialSync$
|
||||
acl url_workerX_stick-auth path_reg ^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$
|
||||
|
||||
# Federation requests
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/v1/event/
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/v1/state/
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/v1/state_ids/
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/v1/backfill/
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/v1/get_missing_events/
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/v1/publicRooms
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/v1/query/
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/v1/make_join/
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/v1/make_leave/
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/(v1|v2)/send_join/
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/(v1|v2)/send_leave/
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/(v1|v2)/invite/
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/v1/event_auth/
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/v1/exchange_third_party_invite/
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/v1/user/devices/
|
||||
acl url_workerX_generic path_reg ^/_matrix/key/v2/query
|
||||
acl url_workerX_generic path_reg ^/_matrix/federation/v1/hierarchy/
|
||||
|
||||
# Inbound federation transaction request
|
||||
acl url_workerX_stick-src path_reg ^/_matrix/federation/v1/send/
|
||||
|
||||
# Client API requests
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/createRoom$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/publicRooms$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/joined_members$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/context/.*$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/members$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/v1/rooms/.*/hierarchy$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/unstable/org.matrix.msc2716/rooms/.*/batch_send$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/unstable/im.nheko.summary/rooms/.*/summary$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(r0|v3|unstable)/account/3pid$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(r0|v3|unstable)/account/whoami$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(r0|v3|unstable)/devices$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/versions$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/voip/turnServer$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/event/
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/joined_rooms$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/search$
|
||||
|
||||
# Encryption requests
|
||||
# Note that ^/_matrix/client/(r0|v3|unstable)/keys/upload/ requires `worker_main_http_uri`
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(r0|v3|unstable)/keys/query$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(r0|v3|unstable)/keys/changes$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(r0|v3|unstable)/keys/claim$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(r0|v3|unstable)/room_keys/
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(r0|v3|unstable)/keys/upload/
|
||||
|
||||
# Registration/login requests
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/login$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(r0|v3|unstable)/register$
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/v1/register/m.login.registration_token/validity$
|
||||
|
||||
# Event sending requests
|
||||
acl url_workerX_stick-path path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/redact
|
||||
acl url_workerX_stick-path path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/send
|
||||
acl url_workerX_stick-path path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state/
|
||||
acl url_workerX_stick-path path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$
|
||||
acl url_workerX_stick-path path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/join/
|
||||
acl url_workerX_stick-path path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/profile/
|
||||
|
||||
# User directory search requests
|
||||
acl url_workerX_generic path_reg ^/_matrix/client/(r0|v3|unstable)/user_directory/search$
|
||||
|
||||
# Pagination requests
|
||||
acl url_workerX_stick-path path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/messages$
|
||||
|
||||
# Push rules (GET-only)
|
||||
acl url_push-rules path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/pushrules/
|
||||
|
||||
# Directory worker endpoints
|
||||
acl url_directory-worker path_reg ^/_matrix/client/(r0|v3|unstable)/user_directory/search$
|
||||
|
||||
# Event persister endpoints
|
||||
acl url_stream-worker path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/typing
|
||||
acl url_stream-worker path_reg ^/_matrix/client/(r0|v3|unstable)/sendToDevice/
|
||||
acl url_stream-worker path_reg ^/_matrix/client/(r0|v3|unstable)/.*/tags
|
||||
acl url_stream-worker path_reg ^/_matrix/client/(r0|v3|unstable)/.*/account_data
|
||||
acl url_stream-worker path_reg ^/_matrix/client/(r0|v3|unstable)/rooms/.*/receipt
|
||||
acl url_stream-worker path_reg ^/_matrix/client/(r0|v3|unstable)/rooms/.*/read_markers
|
||||
acl url_stream-worker path_reg ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/
|
||||
|
||||
# Backend directors
|
||||
use_backend synapseX_worker_generic if host_matrix url_workerX_generic
|
||||
use_backend synapseX_worker_generic if host_matrix url_push-rules http_method_get
|
||||
use_backend synapseX_worker_stick-auth if host_matrix url_workerX_stick-auth
|
||||
use_backend synapseX_worker_stick-src if host_matrix url_workerX_stick-src
|
||||
use_backend synapseX_worker_stick-path if host_matrix url_workerX_stick-path
|
||||
use_backend synapse0_directory_worker if host_matrix url_directory-worker
|
||||
use_backend synapse0_stream_worker if host_matrix url_stream-worker
|
||||
|
||||
# Master workers (single-instance) - Federation media repository requests
|
||||
acl url_mediarepository path_reg ^/_matrix/media/
|
||||
acl url_mediarepository path_reg ^/_synapse/admin/v1/purge_media_cache$
|
||||
acl url_mediarepository path_reg ^/_synapse/admin/v1/room/.*/media.*$
|
||||
acl url_mediarepository path_reg ^/_synapse/admin/v1/user/.*/media.*$
|
||||
acl url_mediarepository path_reg ^/_synapse/admin/v1/media/.*$
|
||||
acl url_mediarepository path_reg ^/_synapse/admin/v1/quarantine_media/.*$
|
||||
acl url_mediarepository path_reg ^/_synapse/admin/v1/users/.*/media$
|
||||
use_backend synapse0_media_repository if host_matrix url_mediarepository
|
||||
|
||||
# MXISD/MA1SD worker
|
||||
acl url_ma1sd path_reg ^/_matrix/client/(api/v1|r0|unstable)/user_directory
|
||||
acl url_ma1sd path_reg ^/_matrix/client/(api/v1|r0|unstable)/login
|
||||
acl url_ma1sd path_reg ^/_matrix/identity
|
||||
use_backend synapse0_ma1sd if host_matrix url_ma1sd
|
||||
|
||||
# Webhook service
|
||||
acl url_webhook path_reg ^/webhook
|
||||
use_backend synapse0_webhook if host_matrix url_webhook
|
||||
|
||||
# .well-known configs
|
||||
acl url_wellknown path_reg ^/.well-known/matrix
|
||||
use_backend elementX_http if host_matrix url_wellknown
|
||||
|
||||
# Catchall Matrix and RElement
|
||||
use_backend synapse0_master if host_matrix
|
||||
use_backend elementX_http if host_element
|
||||
|
||||
# Default to Riot
|
||||
default_backend elementX_http
|
||||
|
||||
frontend ma1sd_http
|
||||
bind :::8090 v4v6
|
||||
mode http
|
||||
option httplog
|
||||
use_backend synapse0_ma1sd
|
||||
|
||||
backend letsencrypt
|
||||
mode http
|
||||
server elbvip.i.bonilan.net elbvip.i.bonilan.net:80 resolvers nsX resolve-prefer ipv4
|
||||
|
||||
backend elementX_http
|
||||
mode http
|
||||
balance leastconn
|
||||
option httpchk GET /index.html
|
||||
# Force users (by source IP) to visit the same backend server
|
||||
stick-table type ipv6 size 5000k peers keepalived-pair expire 72h
|
||||
stick on src
|
||||
errorfile 500 /etc/haproxy/sorryserver.http
|
||||
errorfile 502 /etc/haproxy/sorryserver.http
|
||||
errorfile 503 /etc/haproxy/sorryserver.http
|
||||
errorfile 504 /etc/haproxy/sorryserver.http
|
||||
server element1 element1.i.bonilan.net:80 resolvers nsX resolve-prefer ipv4 check inter 5000 cookie element1.i.bonilan.net
|
||||
server element2 element2.i.bonilan.net:80 resolvers nsX resolve-prefer ipv4 check inter 5000 cookie element2.i.bonilan.net
|
||||
|
||||
backend synapse0_master
|
||||
mode http
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
retries 0
|
||||
errorfile 500 /etc/haproxy/sorryserver.http
|
||||
errorfile 502 /etc/haproxy/sorryserver.http
|
||||
errorfile 503 /etc/haproxy/sorryserver.http
|
||||
errorfile 504 /etc/haproxy/sorryserver.http
|
||||
server synapse0.i.bonilan.net synapse0.i.bonilan.net:8008 resolvers nsX resolve-prefer ipv4 check inter 5000 backup
|
||||
|
||||
backend synapse0_directory_worker
|
||||
mode http
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
retries 0
|
||||
errorfile 500 /etc/haproxy/sorryserver.http
|
||||
errorfile 502 /etc/haproxy/sorryserver.http
|
||||
errorfile 503 /etc/haproxy/sorryserver.http
|
||||
errorfile 504 /etc/haproxy/sorryserver.http
|
||||
server synapse0.i.bonilan.net synapse0.i.bonilan.net:8033 resolvers nsX resolve-prefer ipv4 check inter 5000 backup
|
||||
|
||||
backend synapse0_stream_worker
|
||||
mode http
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
retries 0
|
||||
errorfile 500 /etc/haproxy/sorryserver.http
|
||||
errorfile 502 /etc/haproxy/sorryserver.http
|
||||
errorfile 503 /etc/haproxy/sorryserver.http
|
||||
errorfile 504 /etc/haproxy/sorryserver.http
|
||||
server synapse0.i.bonilan.net synapse0.i.bonilan.net:8035 resolvers nsX resolve-prefer ipv4 check inter 5000 backup
|
||||
|
||||
backend synapse0_media_repository
|
||||
mode http
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
retries 0
|
||||
errorfile 500 /etc/haproxy/sorryserver.http
|
||||
errorfile 502 /etc/haproxy/sorryserver.http
|
||||
errorfile 503 /etc/haproxy/sorryserver.http
|
||||
errorfile 504 /etc/haproxy/sorryserver.http
|
||||
server synapse0.i.bonilan.net synapse0.i.bonilan.net:8095 resolvers nsX resolve-prefer ipv4 check inter 5000 backup
|
||||
|
||||
backend synapse0_ma1sd
|
||||
mode http
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
errorfile 500 /etc/haproxy/sorryserver.http
|
||||
errorfile 502 /etc/haproxy/sorryserver.http
|
||||
errorfile 503 /etc/haproxy/sorryserver.http
|
||||
errorfile 504 /etc/haproxy/sorryserver.http
|
||||
server synapse0.i.bonilan.net synapse0.i.bonilan.net:8090 resolvers nsX resolve-prefer ipv4 check inter 5000
|
||||
|
||||
backend synapse0_webhook
|
||||
mode http
|
||||
balance roundrobin
|
||||
option httpchk GET /
|
||||
server synapse0.i.bonilan.net synapse0.i.bonilan.net:4785 resolvers nsX resolve-prefer ipv4 check inter 5000 backup
|
||||
|
||||
backend synapseX_worker_generic
|
||||
mode http
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
errorfile 500 /etc/haproxy/sorryserver.http
|
||||
errorfile 502 /etc/haproxy/sorryserver.http
|
||||
errorfile 503 /etc/haproxy/sorryserver.http
|
||||
errorfile 504 /etc/haproxy/sorryserver.http
|
||||
server synapse1.i.bonilan.net synapse1.i.bonilan.net:8030 resolvers nsX resolve-prefer ipv4 check inter 5000
|
||||
server synapse2.i.bonilan.net synapse2.i.bonilan.net:8030 resolvers nsX resolve-prefer ipv4 check inter 5000
|
||||
|
||||
backend synapseX_worker_stick-auth
|
||||
mode http
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
# Force users (by Authorization header) to visit the same backend server
|
||||
stick-table type string len 1024 size 5000k peers keepalived-pair expire 72h
|
||||
stick on hdr(Authorization)
|
||||
errorfile 500 /etc/haproxy/sorryserver.http
|
||||
errorfile 502 /etc/haproxy/sorryserver.http
|
||||
errorfile 503 /etc/haproxy/sorryserver.http
|
||||
errorfile 504 /etc/haproxy/sorryserver.http
|
||||
server synapse1.i.bonilan.net synapse1.i.bonilan.net:8030 resolvers nsX resolve-prefer ipv4 check inter 5000
|
||||
server synapse2.i.bonilan.net synapse2.i.bonilan.net:8030 resolvers nsX resolve-prefer ipv4 check inter 5000
|
||||
|
||||
backend synapseX_worker_stick-path
|
||||
mode http
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
# Force users to visit the same backend server
|
||||
stick-table type string len 1024 size 5000k peers keepalived-pair expire 72h
|
||||
stick on path,word(5,/) if { path_reg ^/_matrix/client/(r0|unstable)/rooms }
|
||||
stick on path,word(6,/) if { path_reg ^/_matrix/client/api/v1/rooms }
|
||||
stick on path
|
||||
errorfile 500 /etc/haproxy/sorryserver.http
|
||||
errorfile 502 /etc/haproxy/sorryserver.http
|
||||
errorfile 503 /etc/haproxy/sorryserver.http
|
||||
errorfile 504 /etc/haproxy/sorryserver.http
|
||||
server synapse1.i.bonilan.net synapse1.i.bonilan.net:8030 resolvers nsX resolve-prefer ipv4 check inter 5000
|
||||
server synapse2.i.bonilan.net synapse2.i.bonilan.net:8030 resolvers nsX resolve-prefer ipv4 check inter 5000
|
||||
|
||||
backend synapseX_worker_stick-src
|
||||
mode http
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
# Force users (by source IP) to visit the same backend server
|
||||
stick-table type ipv6 size 5000k peers keepalived-pair expire 72h
|
||||
stick on src
|
||||
errorfile 500 /etc/haproxy/sorryserver.http
|
||||
errorfile 502 /etc/haproxy/sorryserver.http
|
||||
errorfile 503 /etc/haproxy/sorryserver.http
|
||||
errorfile 504 /etc/haproxy/sorryserver.http
|
||||
server synapse1.i.bonilan.net synapse1.i.bonilan.net:8030 resolvers nsX resolve-prefer ipv4 check inter 5000
|
||||
server synapse2.i.bonilan.net synapse2.i.bonilan.net:8030 resolvers nsX resolve-prefer ipv4 check inter 5000
|
||||
```
|
||||
|
||||
Once the above configurations are installed on each server, restart the HAProxy service with `sudo systemctl restart haproxy`. You will now have access to the various endpoints on ports 443 and 8448 with a redirection from port 80 to port 443 to enforce SSL from clients.
|
||||
|
||||
## Final steps
|
||||
|
||||
Now that your proxy is running, test connectivity to your servers. For Riot, visit the bare VIP IP or the Riot subdomain. For Matrix, visit the Matrix subdomain. In both cases, ensure that the page loads properly. Finally, use the [Matirx Homserver Federation Tester](https://federationtester.matrix.org/) to verify that Federation is correctly configured for your Homserver.
|
||||
|
||||
Congratulations, you now have a fully-configured, multi-worker and, if configured, load-balanced Matrix instance capable of handling many dozens or hundreds of users with optimal performance!
|
||||
|
||||
If you have any feedback about this post, including corrections, please contact me - you can find me in the [`#synapse:matrix.org`](https://matrix.to/#/!mjbDjyNsRXndKLkHIe:matrix.org) Matrix room, or via email!
|
||||
86
content/en/posts/.drafts/state-of-the-servers-2022/index.md
Normal file
86
content/en/posts/.drafts/state-of-the-servers-2022/index.md
Normal file
@ -0,0 +1,86 @@
|
||||
+++
|
||||
date = "2022-11-01T00:00:00-05:00"
|
||||
tags = ["systems administration", "pvc","ceph","homelab","servers","networking"]
|
||||
title = "State of the Servers 2022"
|
||||
description = "A complete writeup of my homeproduction system as of end-2022, its 10th anniversary"
|
||||
type = "post"
|
||||
weight = 1
|
||||
draft = true
|
||||
+++
|
||||
|
||||
My home lab/production datacentre is my main hobby. While I have others, over the past 10 years I've definitely spent more time on it than anything else. From humble beginnings I've built a system to rival many small-to-medium enterprises (SMEs) or ISPs in my basement, providing me highly redundant and stable services both for my Internet presence, and for learning.
|
||||
|
||||
While I've written about parts of the setup in the past, I don't think I've ever done a complete and thorough writeup of every piece of the system, what went into my choices (spoiler: mostly cost) and design, and how it all fits together. This post is my attempt to rectify that.
|
||||
|
||||
For most of its first 8 years, the system was constantly changing even month-to-month as I obtained new parts, tinkered away, and just generally broke things for fun. But over COVID, while working from home, and with money being tight, as well as my maturity as a senior systems architect, things really stabilized, and it's only changed in a few minor ways since 2020. I have big plans for 2023, but for right now things have been stable for long enough to be able to really dig into all the parts, as well as hint at my future plans.
|
||||
|
||||
So if you dare, please join me on a virtual tour of my "homeproduction" system, the monster in my basement.
|
||||
|
||||
## Part One: A Brief History
|
||||
|
||||
My homelab journey started over 16 years ago while still in high school. At the time I was a serious computer enthusiast, and had more than enough spare parts to build a few home servers. Between then and finishing my college program (Network Engineering and Security Analyst at Mohawk College in Hamilton, Ontario) in late 2012, I went through a variety of setups that were almost exclusively based on single servers with storage, some sort of hypervisor, and just for tinkering and media storage.
|
||||
|
||||
When I started my career in earnest in January 2013, I finally had the disposable income to buy my first real server: a used Dell C6100 with 4 blade nodes. This system formed the basis of my lab for the next 6 years, and is still running today in a colo providing live functions for me.
|
||||
|
||||
My first few iterations tended to focus on a pair of Xen servers for virtualization and a separate ZFS server for storage, while also going through various combinations of routers including Mikrotiks trying to find something that would solve my endless WiFi issues. At this time I was running at most a dozen or so VMs with some core functionality for Internet presence, but nothing too fancy - it was primarily a learning tool. At one point I also tried doing a dual-primary DRBD setup for VM disks, but this went about as well as you might expect (not well at all), so I went back to a ZFS array for ZVOLs. I was also using `bcfg2` for configuration management. Basically, I had fully mirrored what I used and deployed at work built from the ground up, and it gave me some seriously in-depth knowledge of these tools that were crucial to my later role.
|
||||
|
||||
|
||||
|
||||
Around this time I was also finally stabilizing on a pretty consistent set of systems, and a rumored change to Google's terms for hosted domains prompted me to move one of my first major production services into my home system: email. I can safely say that, having now run email at home for 7 years, it works plenty fine if you take the proper care.
|
||||
|
||||
In early 2016 I discovered two critical new things for the management of my systems: Ansible and Ceph. At first, I was using Ansible mostly for ad-hoc tasks, but I quickly started putting together a set of roles to replace bcfg2 as my primary configuration management tool. While declarative configuration management is nice and all, I liked the flexibility of a more procedural, imperitive system, especially when creating new roles, and it gave me a lot of power to automate complex program deployments that were impossible in bcfg2. By the end of the year I had fully moved over to Ansible for configuration management. I also started using `git` to track my configuration around this time, so this is the earliest period I still have records of, though I might wish to forget it...
|
||||
|
||||
|
||||
|
||||
Ceph was the real game-changer for me though. For most of the previous 2 years I had been immensely frustrated with my storage host being a single point of failure in my network: if it needed a kernel update, *everything* had to go down. I had looked into some of the more esoteric "enterprise" solutions like multipath SAS and redundant disk arrays, but cost, space, and power requirements kept me from going that route. Then a good friend introduced me to Ceph which he had been playing with at his workplace. Suddenly I could take 3 generic servers (which he, newly married, was happy to provide due to wife-acceptance-factor reasons) and build a redudant and scalable storage cluster that could tolerate single-host failures. At the time Ceph was a lot more primitive than it is today, forcing some uncommon solutions - using ZFS as an underlying filestore for instance to ensure corrupt data wouldn't be replicated. But this same cluster still serves me now after many years of tweaking and adjusting, having grown from just a dozen 3TB drives to over 20 8TB and 14TB drives and 168TB of raw space. At this time, getting actual file storage on Ceph was hard, due to the immaturity of CephFS at this point, and the hack solution of an XFS array in-VM with a dozen 1TB stripes was fraught with issues, but it worked well enough for long enough for CephFS to mature and for me to move to it for bulk data storage.
|
||||
|
||||
The next major change to my hypervisor stack came in mid-2016. In addition to a job change that introduced me to a lot of new technologies, at that point I was really feeling limited by Xen's interface and lack of any sort of batteries, so I looked into alteratives. At this time I looked into ProxMox, but was not at all impressed with its performance, reliability, or featureset; that opinion has not changed since. So I decided on one of my first daring plans: to switch from Xen to KVM+Libvirt, and use Corosync and Pacemaker to manage my VMs, with shared storage provided by the Ceph cluster.
|
||||
|
||||
By the end of 2016 I had also finally solved my WiFi problem, using a nice bonus to purchase a pair of Ubiquiti UAP-LR's which were, in addition to their strong signal, capable of proper roaming, finally allowing me to actually cover my entire house with usable WiFi. And a year later I upgraded this to a pair of UAP-AC Pro's for even better speed, keeping one of the UAP-LR's as a separate home automation network. I also moved my routing from the previous Mikrotik Routerboards I was using to pfSense, and bought myself a 10-Gigabit switch to upgrade the connectivity of all of the servers, which overnight nearly doubled the performance of my storage cluster. I also purchased several more servers around this time, first to experiment with, and then to replace my now-aging C6100.
|
||||
|
||||
2017 was a year of home automation and routing. I purchased my first set of Belkin WeMo switches, finally set up HomeAssistant, and got to work automating many of my lights, including a [custom voice controller system](https://www.boniface.me/self-hosted-voice-control/). Early in the year I also decided to abandon my long-serving ISP-provided static IP block and move to a new setup. While I liked the services and control it gave me, being DSL on 50 year old lines, the actual Internet performance was abysmal, and I wanted WAN redundancy. So I set up a remote dedicated server with a static IP block routed to it, then piped this back to my home using OpenVPN tunnels load-balanced over my now-redundant DSL and Cable Internet connections, proving both resiliency as well as a more "official" online presence. Later in the year, after discussing with a few coworkers, I invested in a proper colocation, abandoned the dedicated server, and used my now-freed and frustrating C6100 as a redundant pair of remote routers, with a pfSense pair on the home side.
|
||||
|
||||
In early 2018, the major drawbacks of Corosync and Pacemaker were rearing their ugly heads more and more often: any attempt to restart the service would trash my VM cluster, which had grown to about 30 VMs running many more service by this point. ProxMox still sort of sucked, and OpenStack was nigh-incomprehensable to a single mere wizard like myself. What I wanted was Nutanix, but even most SME's can't afford that. So, I started building my own, named [PVC or Parallel Virtual Cluster](https://docs.parallelvirtualcluster.org). It wasn't that ambitious at first: I just wanted a replacement to Corosync+Pacemaker which would actually preserve state properly, using Zookeeper as the state management backend. Over time I slowly added more functionality to it, and a major breakthrough came in late 2018 when I left the "new" job and returned to my "old" job, bringing this project with me, and impressing my managers with its potential to replace their aging Xen-based platform (on which I based my original homelab design, ironically enough; student became teacher). By early 2020 I had it deployed in production at 2 ISPs, and today have it deployed at 9, plus two in-house clusters, with several more on the way. I discuss PVC in more detail later.
|
||||
|
||||
In late 2018, I finally grew fed up with pfSense. The short of it is, nothing config-wise in pfSense is static: events like "the WAN 1 interface went down" would trigger PHP scripts which would regenerate and reload dozens of services, meaning that trivialities like WAN failovers would take up to 30 seconds. Frustrated, I decided to abandon pfSense entirely and replaced my routers with custom-built FreeBSD boxes in line with the remote routers at the colocation. This setup proved invaluable going forward: 1-second failure detection and seamless failover have been instrumental in keeping a 99+% uptime on my *home* system.
|
||||
|
||||
2019 was a fairly quiet year, with some minor upgrades here and there, with the occasional server replacement to help keep power usage down. And by early 2020, most of the current system had fallen into place. While the number of VMs fluctuates month to month still, the core set is about 40 that are always running, across 3 hypervisor hosts running PVC, with bulk data on the 3 Ceph nodes. 2 routers on each side provided redundant connectivity, and after an unfortunate complication with my TPIA cable provider, in 2022 I moved to a business-class Gigabit cable connection, and added a Fixed-Wireless connection in addition to the existing DSL, bringing me to 3 Internet connections. There's no kill like overkill.
|
||||
|
||||
## Part Two: The Rack
|
||||
|
||||
|
||||
|
||||
|
||||
The rack itself is built primarily of 2x4's and wood paneling. Originally, as seen above, I had used Lack tables, but due to the heat output I wanted to contain the heat and try to vent it somewhere useful, or at least not as obtrusive. This went through several iterations, and after scouring for enclosed racks around me to no avail, in ~2017 I took what is now a common refrain for me and built my own.
|
||||
|
||||
The primary consturction uses 6 ~6-foot 2x4 risers, which are connected at the top and bottom by horizontal 2x4's to form a stable frame. Heavy castors are present on the bottom below each riser to allow for (relatively) easy movement of the rack around the room as needed, for instance for maintnance or enhancements. The bottom front section also features further horizonal 2x4's to form a base for the heavy UPSes discussed in the next section.
|
||||
|
||||
The actual servers sit on pieces of angle iron cut to approximately 3 feet, which bridge the first and second sets of risers on each side, and secured by 2 heavy screws and washers on each riser. This provides an extremely stable support for even the heaviest servers I have, and allows for fairly easy maintenance without having to deal with traditional rails and their mounting points.
|
||||
|
||||
The outside of the entire rack is covered by thin veneer paneling to trap heat inside in a controlled way. On the left side, the back section forms a door which can be opened to provide access to the backs of the servers, cabling, and power connections.
|
||||
|
||||
I've gone through several airflow configurations to try to keep both the rack itself, and the room it's in, cooler. First, I attempted to directly exhaust the hot air out the adjoining window, but this was too prone to seasonal temperature variation to be useful. I then attempted to route the heat out the side of the rack to the front where it could be cooled by an air conditioner, but this proved ineffective as well. Finally, I moved to a simple, physics-powered solution whereby the top 6 inches of the rack is used to direct hot air, via a set of 4 fans in 2 pairs, towards the top front of the rack and out into the room; this solution works very well to keep the inside temperature of the rack to a relatively reasonable 35 degrees Celsius.
|
||||
|
||||
## Part Three: Cooling and Power
|
||||
|
||||
Continuing on from the rack exhaust, cooling inside the room is provided by a standalone portable 12000BTU air conditioner from Toshiba.
|
||||
|
||||
## Part Four: The Internet Connections and Colocation
|
||||
|
||||
## Part Five: The Network
|
||||
|
||||
## Part Six: The Bulk Storage Cluster and Backups
|
||||
|
||||
## Part Seven: The Hypervisor Cluster
|
||||
|
||||
## Part Eight: The "Infrastructure" VMs
|
||||
|
||||
## Part Nine: The Matrix/Synapse VMs
|
||||
|
||||
## Part Ten: The Other VMs
|
||||
|
||||
## Part Eleven: Diagrams!
|
||||
|
||||
## Part Twelve: A Video Tour
|
||||
|
||||
## Part Thirteen: The Future of the System
|
||||
Reference in New Issue
Block a user