The Fascinating Story of ibiblio: From Their FTP Roots to Today’s Digital Repository of Public Information and Open-Source Software

TL; DR: As one of the first websites in the world, the predecessor of ibiblio set the stage for the ultimate “collection of collections.” An online public library supported by the University of North Carolina, ibiblio provides a range of publicly available information and programs spanning music, art, history, science, politics, and open-source software. The digitial archive is committed to public information and open-source communities, and the team behind it collaborates with some of the country’s most powerful hardware and software companies. Led by Director Paul Jones, ibiblio continues to preserve history while adapting to new technology standards and innovations.

Paul Jones is the definition of an early adopter. The University of North Carolina professor wrote email programs in the 1980s, before quitting email altogether in 2011. He started using Java in 1995, the year it was released. He hosted an FTP site before the World Wide Web existed and was among the first in the world to create a website.

That website combined his three passions and academic homes — computer science, journalism, and library sciences — and has since grown into ibiblio, a massive and diverse digital repository of public information and open-source software.

Hosting everything from Linux distributions and the world’s largest free text archive Project Gutenberg to etree’s live music sharing community and a database of slaves in Louisiana, ibiblio makes more than 1,500 resources available to the world.

“The journalism side thinks you should be able to find out about anything, and library sciences thinks you should be able to have access to it for self-improvement,” Paul said. “It’s not hard to say that if you have any of that in your background and you have the means to actualize ways to make that happen, that’s an opportunity and a gift that you should do.”

Paul ran the internationally accessed FTP site for five years and watched network information discovery and retrieval tools gain popularity. He was interested in “helping connect astronomers with shared images, humanists with shared texts, all different kinds of things,” so Paul wrote a grant proposal to Sun Microsystems in 1992 to create web hosting for free information.

“Part of the deal with Sun was to find and recruit interesting content, and we did that by finding people who were being frankly kicked off their university or research servers because their stuff was too interesting to too many people, and they couldn’t deal with the capacity issues.”

The original version of ibiblio, called SunSITE, contained access to worldwide library catalogs, documents from the National Institutes of Health, and even recipes from usenet.

1. Firsthand Account of the Soviet Union Collapse

When asked to put 24-plus years of ibiblio into perspective, Paul points to the multitude of big events that have played out on the site. As the Soviet Union began falling in the early ’90s, SunSITE turned into an unlikely source of information about a failed coup attempt.

“You couldn’t see it on television, but there was a Russian hacker sitting across the street typing in English about everything that was happening,” Paul said. “It was pretty incredible.”

2. Documenting Presidential Speeches and Materials From Clinton White House

The SunSITE team gathered and archived President Bill Clinton’s speeches before being chosen to help the White House with their first web initiatives to document his speeches, press briefings, and related materials during the presidency — ibiblio even archived the communications between Paul and the president-elect’s office.

3. WXYC Sends Radio Broadcasts Worldwide With First Internet Simulcast

Paul and SunSITE also helped the University of North Carolina’s student radio station, WXYC, become the first traditional radio station to broadcast on the Internet.

“We were moderately famous enough that we even became a Jeopardy! answer,” Paul said. “We invented streaming radio in the early ’90s, and they’re still doing it. Everybody does it now, though, so it’s a little bit different.”

Through the decades, Paul said ibiblio is transforming from a library into more of a historical archive. “Originally, it was about getting a lot of new stuff in there, and now it’s about keeping useful things in a productive way that people can interact with in context,” he said.

Webpages from before 1999 aren’t much of a problem because they’re flat, static pages, according to Paul. The challenge comes with preserving database-driven, interactive sites built after 1999 such as a searchable database with information on people enslaved in Louisiana.

“That database probably will not be updated or changed, but it uses a customized version of MySQL,” Paul said. “You want technology to get better every time you move forward, but each time you do, databases and complex websites are endangered.”

Preserving the functionality of older technologies is an issue that Paul said “hasn’t been paid the right attention to.” Some of ibiblio’s resources go stagnant when development money runs out, or the people maintaining the sites quit, retire, or die.

“You want to stop development on them and stop all the software environments and presets, but you want to keep it accessible and secure,” he said, adding that virtualization and containers might lead to a possible solution.

“Our largest challenge at the moment is to think long term about how to keep the collections available in context, not just as flattened webpages,” he said. “We want to keep them living, accessible, and secure, but not necessarily updated. That’s the philosophical challenge that reflects a technical problem.”

The ibiblio resources make up the largest collection of data leaving the university network. Even though the ibiblio servers receive 12 million to 16 million requests for information each day, the university’s large bandwidth needs mean that ibiblio doesn’t have to worry about scaling to meet traffic demands.

“As long as students watch Netflix, I’ll have plenty of bandwidth,” Paul said with a laugh. “They like bandwidth coming in, but I like it going out.”

Paul and his teams have virtualized all of ibiblio’s machines and rely on university support for infrastructure support. Because of that assistance, Paul runs ibiblio with a very small staff — himself, one full-time employee, and two working students.

“You can do something like this without too much if you start early,” he said. “Even when we were working on White House stuff, I think we only had 12 people.”

Starting With Sun Microsystems: Grant Gets ibiblio Predecessor Going

A key component in the 1992 grant from Sun Microsystems that established SunSITE and ibiblio was to distribute free and open software, according to Paul. Drivers and device catalogs for SunOS, the precursor to the Solaris operating system, became among the first open-source software available on SunSITE. Sun, which sponsored the collection for eight years, eventually replicated SunSITE through 40 countries.

Central Place for Archiving, Documenting, and Distributing Linux

Around the same time, ibiblio began accepting and redistributing code bundles for their Linux archives. Popular distributions, such as Damn Small Linux and Puppy Linux, found an early home at ibiblio before the site stopped accepting new submissions in the mid-2000s. By then, the Linux collection comprised more than 5,000 files and had grown to 12GB.

“We have been one of the central places from which you can download Linux,” Paul said. “We were an early, early Linux proponent.”

Fostering Developer Communities and Growing With Red Hat

Paul fortunately connected with another budding open-source software company in the early ’90s, when they only had four employees. Now, Red Hat is a multinational open-source software company with more than 9,000 employees valued at more than $14.2 billion.

“Red Hat went from being edgy to being revolutionary, to being tried, trusted, and tested,” said Paul, who still has personal relationships with the four original employees. “They’re the only people who have successfully monetized Linux at that level. They manage to own a distro, brand it strongly, keep it open, and yet, make a profit and provide the right kinds of services and support. A lot of people have tried.”

After 25 years of watching the World Wide Web emerge, evolve, and explode, Paul still marvels at all the innovations he’s seen — as well as how quickly they become commonplace.

“The web was static at first, and then you suddenly had moving images, then streaming audio and streaming video,” he said. “The cool part is the part that was cool now seems so mundane.”

For example, he has watched a computer shrink from something that barely fit under his desk to a machine that fits in his hand and goes wherever he does.

Paul sees a similar transition happening with virtualization and container technology, which he hopes will greatly impact the preservation of ibiblio.

“I started working with this about four years ago, so I feel this is heading in the right direction,” he said. “I think we’ll have a mundane solution that everyone will think is normal. Successful technology is normal but not very flashy.”

Paul compares the evolution of technology to early cars, radios, and televisions — when they came out, everyone wanted to learn how to work on them. Eventually, though, you just want it to work. “You don’t care about the specs of your TV once it becomes mundane,” he said. “It should just work and fade into the background. Our goal in technology is to be mundane.”