TL; DR: Powering platforms like Netflix or your favorite gaming system, FreeBSD is a robust, open-source operating system relied on by millions of sites to serve web pages. Even Mac OSX is based, in part, on FreeBSD. With over 400 volunteer devs at its helm, the FreeBSD Project is a dynamic force that puts the control in administrators’ hands. Project Committer Allan Jude has been a fan of the project since 2002. He discusses the system’s internal structure and the feature functionality that drew him in — contributing to its documentation and then to its development — and what he’s working on now, more than a decade later.
“So I made this change and now it’s everywhere…”
Imagine if the code you worked on today or the pull request you submitted yesterday were to get committed and affect millions of individuals’ lives — their PlayStation or their Netflix — and the millions of sites relying on their operating system to help serve web pages.
“It’s definitely humbling… and a little strange,” Allan Jude admitted.
When Allan started his own company hosting websites for video streaming, FreeBSD was the only operating system he had previously used with other hosts. Based on his experience and comfort with it, he trusted the system with the future of his budding business.
A decade later, the former-SysAdmin went to a conference focused on the open-source operating system, where he ran into some of the folks on its documentation team. “They inspired me,” he told our team in a recent chat. He began writing documentation but soon wanted to contribute improvements beyond the docs.
Today, Allan sits as a FreeBSD Project Committer. It’s rare that you get to chat with someone involved with a massive-scale open-source project like this — rare and awesome.
Two Huge Benefits of FreeBSD from the SysAdmin Point of View
We live in a college town with tons of aspiring web developers and SysAdmins funneled out of the University of Florida. Many of them are destined to go the Linux route simply because the name is more familiar. They see it everywhere. Yet the reality, according to Allan, is that FreeBSD “can do pretty much everything that Linux can do — and most of the things it does better.”
The fact that Allan’s involvement in the project’s development stems from his personal experience with the system begs the question: Why THIS OS? “Because of the way it’s organized,” he said, and it seems that declaration includes the internal structure and internal team.
1. Code Organization Meticulously Maintained & Documented
The FreeBSD Project functions like an extremely well-organized world all its own. Allan explained the environment: “There’s a documentation page that explains how the file system’s laid out and everything has a place and it always goes in that place.”
Especially with so many hands in the pot working on this project, it would be easy for order to unravel. Allan explained that in many other applications, developers structure the code however they feel makes sense, and that varies from dev to dev. “With FreeBSD, there’s always a place for it [whatever it is] and it always goes there,” he concluded.
To maintain this order, FreeBSD has a Style Guide that is to be followed meticulously. Allan told us that the tight-knit team serves as a source of accountability for one another. “It’s a fairly close-knit group of people. If you put something in the wrong place, someone will say, ‘Hey, that actually goes over here,’ and it gets corrected.”
As the dev team continues to grow, they evangelize the proper FreeBSD style with a mentorship system. Each new dev that’s onboarded is paired with a mentor to ensure they’re trained in accordance with said guidelines. “When you join FreeBSD you’re assigned a mentor for a while, so that means that everything that you actually contribute is reviewed by them first,” Allan explained, “and if anything is wrong with it, it’s their fault — not yours — which really helps a lot in the very beginning when you’re just getting started.” No kidding. Can you imagine just diving in and taking on the fate of someone’s beloved gaming system with your first pull request on a massive project like this?
2. Distribution Packages with Painless Versioning & Choice of 25,000 Ports
Another standout factor is that FreeBSD does packages differently — totally maintained by their team with easy versioning for you. FreeBSD allows you to take control of your packages when setting up and maintaining your system, and they offer a “rolling release” cycle with 25,000 different ports from which to choose.
“You don’t have to go hunting around adding external repos that are managed by who knows who,” Allan said, explaining one of his favorite perks of FreeBSD. “It’s all in one FreeBSD repository package,” and when you want to switch, say from PHP 5 to PHP 7, “it’s just a matter of telling [FreeBSD]” to upgrade and it’s done.
With the added benefit of FreeBSD Jails — the containers that allow you to store your site’s dev and production versions separately on a single machine — you can actually make the switch on a trial basis. There’s no need to spin up new machines or alter the production environment.
Allan’s Recent Project Spotlight: Boot Environments
So what’s a day in the life of a dev on the FreeBSD Project like? Allan told us that most folks working on the project are doing so as hobbyists or as part-time work in conjunction with their “day jobs.” He said that often times he’ll be working on a project for said day job, and it isn’t as easy as he believes it should be, or it doesn’t work the way he wants it to, or he gets inspired by the What Ifs: “It’d be great if it had this extra feature,” he verbalized his train of thought. “I’ll start working on that, and, once I have a general idea of how I want to do it, I publish to the mailing list and get feedback.” With a responsive community just an inbox away and a thorough code review to follow, Allan and fellow devs have the support system they need to push fresh contributions into the FreeBSD release cycle.
“I think the most recent thing that I did that a lot of people were waiting for is a feature we call Boot Environments,” Allan told us. “If you’re using ZFS, you can take a snapshot of your system before you install an upgrade and then, if that upgrade doesn’t work out, you can roll back to what the system was like before — but it only affects the system files.” In other words, your home directories and database files are unaffected while the operating system is rolled back to the point in which you were happy with it.
“Previously, you had to use a tool in the operating system to switch between them,” Allan said. “After an update, if the system wouldn’t boot, that wouldn’t really help.” To combat this frustration, Allan made the feature more usable for all by adding an option to select your Boot Environment in the menu upon first bootup. As an added bonus, the feature is out in front of more users who wouldn’t necessarily have known it existed beforehand. “Now they see that option every time it comes up, so they’ll go look up the documentation on how to use it,” Allan concluded. Giving the people what they want and even what they may not know they want — Well done, Allan.
The Testing Protocol, Using ZFS as an Awesome Example
For those of you who don’t know, ZFS, or the Z File System, is the new wave of file systems, in that it’s actually more than just a file system. It’s an open-sourced project initially developed by Sun Microsystems™ that combines volume management and traditional file system features, while recognizing the overall structure of the storage disks involved. It overcomes many former problems found in other file systems, which could only recognize a single disk (even when placed on top of an advanced RAID configuration of hardware).
Naturally, we asked Allan how this changed the game for his own business as a video streaming company. “Now that we’re getting close to a petabyte of storage, other file systems aren’t really an option anymore,” he explained, matter-of-factly. The major selling points of ZFS for Allan’s business are the snapshots and replication. “When you’re talking about tens of terabytes of files for some customers, other ways of backing up don’t quite work anymore, so having the block-level replication really improves things for us a lot,” he said, noting that being able to maintain separate file systems for each customer is really nice as well. With ZFS, they’re able to manage customer files as a discrete unit, with what’s called a data set — and if someone skips out on the bill, they can remove that customer’s singular snapshot of storage without affecting other paying users.
Incorporating New File Systems into an Operating System is No Small Feat
Okay devs — take a moment to think through the process of integrating a new file system into an operating system. Mmm, who doesn’t like a nice daunting project to take up weeks and weeks? Brace yourselves now: This beast of a feat was actually tackled and overcome by one Polish developer by the name of Paweł Jakub Dawidek — and he did it in a mere 21 days.
“He worked around 18 hours a day,” Allan said, probably envisioning our jaws dropping through the speakerphone. “He wasn’t working that way on purpose; it was more that it was just so exciting and interesting that he couldn’t stop thinking about it or stop working on it.”
That attitude is representative of the feelings felt throughout the FreeBSD development team, I think. Every time we asked questions like, “How often do you work on the project?” Allan’s responses were along the lines of, “Well, sometimes I can’t get to it throughout the week, and other times I GET to spend a WHOLE WEEKEND.” Simply put, Allan said, “This is the fun stuff.”
Testing is Tricky and Rigorous, so FreeBSD Came Out with the Universe Build
Further complicating things, ZFS has its own upstream repo with its own rules and testing requirements: e.g., a pull request has to be in a ZFS-running operating system like FreeBSD or Linux and used in production for a minimum of three months before actually being committed.
“The ZFS repo has to be in production quality at all times,” Allan elaborated. “FreeBSD itself has a test suite, a set of regression tests, and then continuous integration with Jenkins.” The trick there is the multiple platform conundrum. You make a commit, the code is seemingly perfect, but then you realize you only know how it runs on your x86 hardware. Maybe its functionality on ARM is a different story. Rut ro, Raggy.
“We have this concept called the Universe Build where you build FreeBSD for every different processor environment that’s supported,” Allan shared. That way no matter how well it works on your system, you also know it’ll work for everyone else.
As is to be expected, Allan told us the Universe Build can take a while to develop to perfection — “especially if you don’t have all the hardware, but the FreeBSD Foundation provides servers to the developers of the project.” So thankfully, they’ve got access to really big servers, some 36 cores, and a boat load of RAM. “If you had to do a Universe Build on your laptop, it might take a day and a half,” Allan added, with a laugh.
What Might Slip Through the Cracks is Immediately Caught By Jenkins
Jenkins, a beloved Java-based automation server that’s been open-sourced to support the development world in large-scale builds, is FreeBSD’s automation software of choice. That’s got to offer some comfort to these guys actively working on a project that’s so ingrained into the fabric of millions of people’s lives. Allan told us that if anything goes awry, “Jenkins will notice.” If you were a part of the recent change that’s being tested, if you so much as contributed a tiny commit, you’ll receive an email: Hey, this is broken; you might want to check and make sure it’s not your fault.
Between Jenkins and the Core Development team using the dev version that’s not yet shipped, Allan told us it’s rare that issues aren’t noticed and rectified pretty quickly.
The Power to Serve — Join the FreeBSD Force
Before joining the 400-some-odd developers and their fellow contributors in a shared mission to strengthen and evolve the powerful open-source computer OS, FreeBSD, Allan was a SysAdmin. He’d dabbled in PHP, Perl, and shell scripting, but it wasn’t until recently that he took on C.
“An important take-away is that you don’t have to be a major developer with tons of experience to make a difference in the project,” Allan said — and the difference that devs like Allan are making is incredible. If you too want to submit the commit that contributes to the project relied on by millions of web servers, there are plenty of ways to get involved!
We’re especially talking to SysAdmins here, as Allan noted that they are the main users of FreeBSD. “Having more SysAdmins involved in the actual build of the system means we can offer the tools they’re looking for — designed the way a SysAdmin would want them designed, not necessarily the way a developer would think makes the most sense,” Allan added.
If you’re currently using FreeBSD and come across an opportunity to make a feature easier or more functional — start working to make it so! “We can take contributions from anyone who wants to offer them,” Allan concluded.
Want to get involved, but not quite at that level, skill-wise or availability-wise, just yet? Check out the FreeBSD wiki’s JuniorJobs page for the latest list of little tasks that need doing.