My own private ISP

August 25, 1999

Setting up your own Internet service provider can sometimes be the only way to get satisfactory Internet access options. In case you should want to give it a try, here’s a case study: my own experience.

Over the past year or two, thousands of computer users have been flocking to open source operating systems such as Linux, FreeBSD, and OpenBSD. A mere five years ago, however, the thought of a “free” operating system in the corporate environment was virtually unheard of. While designing the network topology for one of my first clients, it took months to convince them to allow Linux workstations into their Sun environment.

Now that open source is finally getting the recognition that it deserves, many corporations are starting to integrate “free” products into their networks. Other companies, mostly home businesses and geek startups, rely solely on open source software to create their magic. Sosik-Hamor Networks is 100% open source with the exception of a Cisco router and a few Macintosh workstations.

Bandwidth requirements

About six months ago my wife Kelly and I decided we had outgrown our current Internet connection and it was time to rethink our plans for the future. I was currently a UNIX systems administrator for Lucent Microelectronics but was looking for more of a challenge. I was also doing some freelance consultation and Web design and we started to question the reliability and security of using our 500 Kbps cablemodem to connect to our colocated Linux Web server.

Kelly and I started discussing different bandwidth options in our price range. ISDN was outrageously expensive, ADSL wasn’t available yet, and the new breed of cable modems being released by our ISP were going to be DHCP-only, which was not an option for our home network. Then, as if to fully realize every geek’s dream, Kelly said, “We’re already paying an arm and a leg for the colocated server … how much is a T1?” So, Sosik-Hamor Networks was born.

Installing the T1

After shopping around for bandwidth with local ISPs and some of the larger telcos, we started running into problems. Since we’re located in the middle of nowhere and our local telco is a monopoly, we had extremely limited options. Our local telco was either unable or unwilling to bring in a co-op line from an external provider, so we were forced to go with them for our T1. With this experience, we found out that physical location is one of the most important things to consider when putting together a business that will require high-speed access. Make sure that your local telco can handle a high-speed line from any ISP of your choice.

Because the sales representative couldn’t comprehend why a home business would need a T1, I was greeted with much suspicion. It took over four months just to get a price quote and another month before the fiber was run from the telco to our street. On top of that, every step of the installation was met with hostility from the ISP due to the fact that I took a very direct approach after being blown off for five months: “Give meservice or I’ll sue you for not allowing me to choose an alternate provider.” Although blunt and hostile, a contract was in my hands within two hours and fiber was dropped into the basement a week later.

The final price tag for fiber installation and ISP setup for the 950 foot fiber run was $2,500 total and $970/month for a full 1.544MBps T1. Telco circuit charges and ISP bandwidth fees are all covered under the monthly charge, which is an incredible deal compared to the $8,000 installation and $3,400/month quote I was getting from some other ISPs in the area.

Network planning

During the wait for the T1, Kelly and I came up with a detailed network topology map and decided exactly what hardware and software would be required to put together an inexpensive and upgradable network that could be modified with minimal service interruptions.

  • Cisco networking equipment will be used exclusively.
  • The DMZ outside the firewall must be switched and SNMP-aware.
  • The LAN inside the firewall will eventually be switched and SNMP-aware.
  • All software must be 100% open source.
  • OpenBSD will be used exclusively outside the firewall.
  • Linux will be used exclusively inside the firewall.
  • Macintoshes will be used exclusively for project development.
  • The internal file server must be AppleTalk or AppleShare capable.
  • A secure auditing workstation will sit between the DMZ switch and the DMZ ethernet port on the router.

After taking stock of our current hardware, we then compiled a list of what we owned and what we needed. All of the purchased hardware was chosen because outstanding deals had been found.

Available hardware:

  • SPARCstation 2, 64MB RAM, 1.2GB HDD
  • SPARCstation 1+, 32MB RAM, 540MB HDD
  • AMD K6/233, 96MB RAM, 4.6GB and 5.2GB HDD
  • AMD K6/266, 128MB RAM, 7.2GB HDD
  • IBM Aptiva P166MMX, 64MB RAM, 3.5GB and 25GB HDD
  • Team Internet 486dx2/66, 64MB RAM, 1.2GB HDD
  • Apple iMac G3/266, 160MB RAM, 6.2GB HDD
  • Apple PowerMacintosh G3/400, 144MB RAM, 9.2GB HDD
  • Miscellaneous m68k Macintoshes
  • MaxTech 24-port Unmanaged Hub
  • 2 Addtron 8-port Unmanaged Hubs

Hardware to purchase:

  • Cisco 2611 router
  • WIC-1DSU-T1 integrated DSU/CSU
  • Kalpana EPS-2015 RS managed switch
  • 19″ wallmount telco rack
  • 8′ steel equipment rack

Next, we started distributing the machines. The AMD systems and SPARCstations would become OpenBSD servers in the DMZ and the IBM and Apple systems would become Linux and Mac OS 8.6 production boxes on the internal LAN. Linux was chosen for the IBM Aptiva because we not only needed a file server but also a workstation-style installation with the X-Window System to run X applications such as xload from the servers in the DMZ. The final Team Internet machine became an OpenBSD security and auditing workstation to keep track of traffic and the little gremlins that tend to creep into networks.

Getting online

When shopping around for Cisco hardware, I ran across a friend on #cisco on EFNet Internet Relay Chat. He gave me the pros and cons of each Cisco router and put together a great deal on a new Cisco 2611 with integrated WIC-1DSU-T1 DSU/CSU for $2,500. I later ordered a 32MB RAM upgrade from Crucial Technology for $70 to bring the router up to 40MB.

Now that we had a router, we needed to pick up a switch for the DMZ. Switching was absolutely required because sniffing would be an issue with any colocated servers. Since we only needed a switch to protect against sniffing and wouldn’t need cutting-edge network management features for a while, we tracked down some surplus Kalpana switches and an EPS-2015 RS for $125.

After months of fighting with our ISP, the fiber was finally dropped into the basement and the fiber patch panel and MUX were installed on top of the router and switch on our 19 inch wallmount rack. A few hours later, we were up and running with a minimal configuration and pinging the ISP. The installation tech left us on our own and we started router configuration for the DMZ, firewall, and NAT. Another half hour and our Macs behind the firewall could see the outside world.

Setting up core services

Since the SPARCstations are extremely slow at the command line but work great for months on end at menial tasks, they became the dedicated “core” servers that would take care of tasks such as DNS, outbound Web proxy, and outbound e-mail. We also decided that these two servers would be completely hardened with virtually no services whatsoever. Water would be configured as a bastion nameserver while earth would only handle DNS and outbound Web proxy and e-mail.

Required software:

earth.shn.nu

The SPARCstation 2 became “earth,” the primary server that takes care of all core operations at Sosik-Hamor Networks. This system was the first to be installed because it had an external CD-ROM drive that could be used for FTP installs for other machines. OpenBSD 2.5 boot floppies were downloaded from ftp.openbsd.org and we started an FTP install on earth.

Total installation took around 45 minutes from start to finish, including downloading the approximately 250MB full distribution over the T1. Since the FTP installation method decompresses and installs files on the fly, like Linux, no scratch disk is required. Once finished, an obscure and long root passwd was chosen and the machine was rebooted into single user mode. All services except FTP and Daytime were disabled in inetd.conf and all daemons except named were disabled in rc.conf. Even portmap was disabled because the server would never be used in an open file server environment. The machine was rebooted and brought up on the Net.

With the machine up and running, the first software to be installed was SSH. To get up and running quickly, the pre-built SSH binary package was downloaded from ftp.openbsd.org and installed using pkg_add. The system was now accessible from the outside world, so I started up a few SSH sessions from my Macintosh.

The full i386 and SPARC OpenBSD 2.5 distributions were downloaded from ftp.openbsd.org and put in /home/ftp/pub/openbsd so we could immediately start installing on the other three servers. Anonymous FTP was configured and temporarily enabled for this task, but would be disabled once the other installations were finished. Although anonymous FTP is not a direct security hole, it is one more port to tempt a potential attacker.

Qmail was downloaded from www.qmail.org and installed from source because there hasn’t been a direct qmail port for OpenBSD yet. Compiling qmail took a while because of the slow processor, but installation and configuration was painless. TCP wrappers were configured so qmail could only relay e-mail from the DMZ and firewall networks, and then set up as a secondary MX to queue e-mail for any other domains on the network should a primary mail server go down.

Squid was then installed for the outbound Web proxy to help hide the identity of machines in use behind the firewall. Configured with a 100MB disk cache and maximum privacy features enabled, Squid caches often accessed Web pages and hides the USER_AGENT and USER_REFERER strings so make it more difficult for Web servers to track movement from page to page. As with qmail, Squid only allows connections from within Sosik-Hamor Networks.

Forward and reverse nameserver zone files were then created from the default templates Sosik-Hamor Networks’ primary domains. Zone transfers were disabled except for other nameservers in the DMZ, which makes it difficult for an attacker to download the forward and reverse maps of the network.

With earth up and running, other machines were brought online over the course of the next two or three days.

water.shn.nu

Because water, the SPARCstation 1+, would be a bastion nameserver, only minimal packages would be required. Compilers and other niceties were not installed because any patches and upgrades that were needed could be mirrored from earth. The only other package installed was SSH to allow for secure remote logins.

The installation process for water was virtually the same as earth except, instead of wasting bandwidth over the T1, OpenBSD was installed via FTP from earth. After installation was finished and the machine was brought up in single user mode, everything was commented out of inetd.conf and disabled in rc.conf except for named and sshd.

Although water was configured as a secondary nameserver to pull zone transfers from earth, we decided to list it first in the zone files and with InterNIC/NuNIC. That way most DNS requests would come into water first and keep earth free for other duties. Even though DNS isn’t a very CPU or network intensive task, a nameserver that handles 100 or 200 domains needs to be carefully configured.

Setting up Web services

Most large Web hosting companies run multiprocessor PII and PIII machines to handle their high workload. Since we were just starting out and our old colocated P90 Linux server with 64MB RAM handled 50,000 hits a day without breaking a sweat, we figured that, until we got busy, our two existing AMD K6 systems would be perfect for our personal and client Web servers. Both K6 systems were basically the same, but we opted to use the K6/233 for fire and the more powerful K6/266 for wind.

fire.shn.nu

Fire was built first because we needed a machine to act as our testbed and production server for our corporate and project Web sites. All new configurations and software are tested on fire before going live on wind, and most interactive development packages from the ports tree are installed because fire is also used as the primary staff shell server.

All OpenBSD packages were installed via FTP from earth and all non-essential services were disabled in single user mode before bringing the system up on the Net. Because this was going to be a staff production system, quite a few services were left enabled and installed.

As with all of our systems, SSH was the first software to get installed to allow for secure remote access. Qmail was then installed and configured for virtual domain support with no relaying. With this configuration, qmail only allows inbound email or outbound email originating from localhost. Also, e-mail for each domain is handled by its own individual account and users have full configuration over aliases and forwarding without root intervention.

MySQL was then installed from the ports tree for all Web sites that need database interaction. A simple make; make install in the ports tree took care of everything and the MySQL server was up and running. The only additional change made to the default configuration was to create a mysql user, change the ownership of /var/db/mysql from root to mysql and make safe_mysqld launch as the user mysql user from rc.local.

A database is useless without a bridge to get data to and from the Web server, so PHP3 was installed. This required recompiling Apache from scratch in /usr/src/usr.sbin/httpd, so the Configuration file was modified to add in all the extra modules that we needed, ./Configure was run and then the PHP3 module was installed into the Apache source tree. Other various modules were added as well and then Apache was recompiled and dropped into /usr/sbin. Once compiled, suexec was also compiled and dropped into the Apache sbin directory to allow for secure execution of CGI binaries.

Once all of the servers were set up, niceties such as Emacs and Pine were installed from the ports tree. Deciding which applications get installed is simply a matter of user preference, so take a look through /usr/ports and do a make install for anything that looks interesting. Pre-built binaries are also available from ftp.openbsd.org. Smaller applications are just fine to run from precompiled binary packages, but larger applications that need tuning (Apache, MySQL, etc.) should be compiled from the ports tree on the system they will be running on.

wind.shn.nu

Wind is an exact mirror of fire and acts as the clients server. The only difference between wind and fire is that miscellaneous applications may get installed at each client’s request (IRC scripts, etc.).

akasha.shn.nu

Akasha is the watchful eye that makes sure all is well out in the DMZ. It sits on a hub between the DMZ switch and the DMZ ethernet port on the router and constantly runs a sniffer and network analyzer to look out for “interesting” traffic. The main purpose for this is to keep track of per-MAC address accounting and to look for denial of service attacks or other nasty packets that may come across the fiber. Depending on the requirements at the moment, akasha may be the Team Internet 486dx2/66 running OpenBSD or a Macintosh IIcx running Mac OS 7.5.5.

Setting up the internal network

Now that the DMZ was set up and ready to go, the systems on the LAN inside the firewall needed to be reloaded and configured.

socks.shn.nu

Socks was chosen to be the internal file server and Red Hat Linux 6.0 was installed. Of the two disks, the 3.5GB was used as the boot disk with /home for home directories and the 25GB data disk was mounted as one huge 23.5GB partition under /home/warehouse01. As the need arises, more 25GB (or larger) disks will be installed as /home/warehouse02, etc.

Because of the large drives, the entire Red Hat distribution was installed and unused services were disabled for security. Making the jump from Red Hat Linux 5.0 to 6.0 was quite a large step, and I wanted to see everything 6.0 had to offer. So, not only was socks configured as a file server, but also as a user workstation to play with the new window managers that have become available. Even though virtually all Web site development is done using Adobe GoLive under Mac OS, the Linux workstation is used for most systems administration tasks.

To serve our internal Macintosh G3 workstations, socks also runs Netatalk, the UNIX implementation of the AppleTalk protocol. File transfer speeds over the internal 10BaseT LAN are surprisingly fast, but we will be moving to switched 100BaseT eventually to help speed things up. 10BaseT seems fast until you routinely start opening up 20MB Photoshop files for editing. All other services on socks are set up almost exactly like fire with Apache, MySQL, PHP3, etc.

To tie everything together, a Belkin OmniView 6-port KVM switch was used to control akasha, fire, wind and socks from the same keyboard, mouse, and monitor. All machines sit on an 8 foot vented steel rack, and an air conditioner keeps the machine temperature at 72 degrees. The entire NOC can be controlled from the keyboard and monitor hooked up to the OmniView.

DIY: Do It Yourself!

Overall, setting up an Internet resource provider isn’t that expensive. A mixture of OpenBSD and Linux can make older workstations into perfect servers and keep initial startup costs extremely low. Using existing hardware and 100% open source software, we kept our startup costs under $8,000 and — even using brand new systems — easily stayed under $15,000. Also, by dropping fiber into a home business, your home office becomes a business expense with the added benefit of Mb speeds at home!

Sean Sosik-Hamor is an Alpha Geek and systems administrator for Lucent Technologies. For open source IT, Sean also hosts the Apache help forum.

URLs for original article: