drink the sweet feeling of the colour zero

Basic Linux Bandwidth Shaping

Tags: , , , ,

This post is largely for my own personal reference.

Bandwidth shaping has traditionally been very difficult.  To truly understand it you must know a fair amount about networking.  The tools are somewhat arcane.  Fortunately, some folks have given us all a leg up by greatly simplifying the process.  It is still far from “easy peasy,” but it is also no longer the black art it once was.

I have recently had cause to configure bandwidth shaping on my edge Linux routers.  For this task I have made use of the Hierarchy Token Bucket (HTB) queuing that is part and parcel of modern Linux kernels.  I set up the HTB init script on my edge Linux routers, and need to document exactly how I installed the whole thing.  This includes the HTB Webmin module, which I use as a visual reference to tell me if I have configured the HTB text files properly.

This setup involves the following:
– HTB Init Script http://sourceforge.net/projects/htbinit/
– HTB Webmin Module http://sehier.fr/webmin-htb/
– Webmin http://www.webmin.com/
– CentOS 5.5 http://centos.org/

This setup presumes the following:

– You have setup and configured a CentOS 5.5 system
– You have configured the networking on this system to serve as a network router
– You have installed Webmin
– You have properly configured the firewall to allow access to Webmin (default port: 10000)
– You have a basic working knowledge of Linux commands (wget, chmod, chown and similar.)
– You have a basic working knowledge of Webmin (how to add modules and navigate the GUI.)

Step 1:  Install the htb.init script.

Wget my modified HTB Init script (http://www.trevorpott.com/downloads/htb/htb.init) into /etc/init.d.  Chmod it 0755 and chown root:root.  This is necessary to use my modified version because of an incompatibility in the original script’s use of the “find” command.  (-maxdepth was improperly positioned and blew up under CentOS 5.5)

The original unmodified script is available on sourceforge here.

Step 2:  Install the Webmin HTB module.

Webmin –> Webmin Configuration –> Webmin Modules
Select “third party module from” and enter http://sehier.fr/webmin-htb/webmin-htb.tar.gz
(Note: also cached here: http://www.trevorpott.com/downloads/htb/webmin-htb.tar.gz)
Select “Install Module”.

This will properly create the Webmin module directory under /etc/webmin, as well as register the module with Webmin itself.  Sadly, as this is not a .wbm, there are some bugs.

Wget the .tar.gz into /etc/webmin/ and tun “tar -zxvf” against it.  This will unpack the files into the /etc/webmin/htb directory with all the proper permissions.  You can now delete the webmin.htb.tar.gz file.

Step 3:  Setting up the config files.

Create the directory /etc/sysconfig/htb.  The directory should be chmoded to 0755 and chowned root:root.  This directory houses the files that the htb.init script will use to configure htb on your system.  A sample configuration is provided here: http://www.trevorpott.com/downloads/htb/archive.tgz

An explanation of how these files work is provided below.

Step 4: Install the DAG::Tree PERL library.

Enter the following (without the quotes) into the command line: “cpan -i Tree::DAG_Node”.  When it asks if you would like to build manually, enter “no.”

Step 5: Check the configuration.

To check your config using Webmin do the following:
Webmin –> Networking –> Hierarchy Token Bucket queuing.  This module will allow you to configure HTB as well as provides alerts if there are misconfigurations.

To check the configuration using the command line do the following:
/etc/init.d/htb.init compile

Step 6:  Start the service.

If you are satisfied with the configuration, then in Webmin do the following:
System –> Bootup and Shutdown
Select “htb.init” and click “start now and on boot.”

How the config files work:

The configuration for the HTB init script is picked up both from the naming of the config files and thier content.  To really understand how HTB works you should read the user manual.  The significantly dumbed down version is as follows:

HTB provides a method to provide bandwidth shaping using your Linux box.  You set up a “root” class that contains the total bandwidth you wish to allocate for one group of IP addresses or ports.  You then create classes which are subordinate to this root class.  Each of these classes can be (and indeed should be) guaranteed a minimum amount of bandwidth.  All classes subordinate to the root can also be configured with a ceiling.  The ceiling parameter is the maximum bandwidth that class can consume.

Should the root class have extra unconsumed bandwidth available, (because one of the other subordinate classes is not consuming it’s full allotment,) then any subordinate classes requiring bandwidth above any beyond their minimum guaranteed amount will be able to “borrow” bandwidth from another class subordinate to the same root.

Subordinate classes can for example be an IP address, a subnet or a specific class of traffic such as “all traffic to port 80.”

A working real world example (as per the provided archive.tgz) is thus:

I have a small Linux box configured with two network interfaces: eth0 and eth1. My provider offers me a 100Mbit pipe, however they charge me based on throughput usage rather than total bandwidth consumption. Thanks to this, I wish to limit my total possible throughput consumption to 15Mbit symmetrical.

For the purposes of this demo file, I have changed all the IP addresses involved to be on the /27 subnet. In the real world the subnet in use has externally addressable addresses as I am using this system to shape throughput to a subnet provided me by my ISP.

eth0 is the interface going out to my ISP.
eth1 is the interface on which my ISP-delegated subnet can be found.

By shaping traffic on the eth0 interface I can control the speed of traffic flowing from my provisioned subnet to my ISP. (Upstream traffic.)

By Shaping traffic on the eth1 interface I can control the speed of traffic flowing from my ISP to my provisioned subnet. (Downstream traffic.)

The file eth0 contains one line: default=91. This tells HTB that the default class for all unclassified traffic on eth0 will be 91. The file I have setup to define this is eth0-2:91.default_up

The file eth0-2.root_up defines the root class for eth0. The root class is the number 2. The HTB init script infers this from the filename. Everything before the dash (eth0) is the interface. Everything after the dash but before the period (2) is the class. Everything after the period (root_up) is the “friendly name” of the class.

Looking at the default file we see that is has a colon. As with the root class, everything before the dash (eth0) is the interface. Everything after the dash but before the period (2) is the class. Since this class has a colon (2:91), the script will parse this as being “class 91, subordinate to class 2.” Everything after the period (default_up) is the “friendly name” of the class.

You will notice also several files with “friendly names” consisting of three numbers. These “friendly names” are simply the last octet of the IP address that rule is defining bandwidth shaping for.

The logical hierarchy defined by the filenames and their contents is as follows:

-eth0: all unclassified traffic will use class 91
–class 2 (root_up): Maximum throughput of 15Mbit.
—class 11 (137): Guaranteed 1Mbit, Ceiling of 15Mbit, SRC
—class 21 (157): Guaranteed 5Mbit, Ceiling of 15Mbit, SRC
—class 31 (133): Guaranteed 1300Kbit, Ceiling of 15Mbit, SRC
—class 41 (132): Guaranteed 600Kbit, Ceiling of 15Mbit, SRC
—class 51 (158): Guaranteed 600Kbit, Ceiling of 15Mbit, SRC
—class 61 (144): Guaranteed 4Mbit, Ceiling of 15Mbit, SRC
—class 71 (136): Guaranteed 500Kbit, Ceiling of 15Mbit, SRC
—class 91 (default_up): Guaranteed 2Mbit, Ceiling of 15Mbit, Burst 15k

-eth1: all unclassified traffic will use class 91
–class 2 (root): Maximum throughput of 15Mbit. Burst in 15k increments.
—class 10 (137): Guaranteed 4Mbit, Ceiling of 15Mbit, DEST
—class 20 (157): Guaranteed 5Mbit, Ceiling of 15Mbit, DEST
—class 30 (133): Guaranteed 1300Kbit, Ceiling of 15Mbit, DEST
—class 40 (132): Guaranteed 600Kbit, Ceiling of 15Mbit, DEST
—class 50 (158): Guaranteed 600Kbit, Ceiling of 15Mbit, DEST
—class 60 (144): Guaranteed 1Mbit, Ceiling of 15Mbit, DEST
—class 70 (136): Guaranteed 500Kbit, Ceiling of 15Mbit, DEST
—class 90 (default): Guaranteed 2Mbit, Ceiling of 15Mbit

An example based on this configuration is that of an FTP server located at It has a guaranteed 1Mbit up and 4Mbit down. (It is mostly used for other people to send files up to us.) It can however receive or send information at up to 15Mbit, should none of the other systems on the subnet be consuming their allotments.

The rate limits for each network card are set at 15Mbit total. That rate limit will affect both upstream and downstream traffic on each NIC. While I am only defining upstream caps on my eth0 NIC and downstream caps on my eth1 NIC, this configuration effectively limits my system to 15Mbit half duplex. This is by design. I want to be able to send at 15Mbit upstream or receive information at 15Mbit upstream, but I also do not want my combined upstream and downstream to surpass 15Mbit. It is a quirk of how I am billed (95th percentile of half-duplex consumed throughput.)

Additionally, of a possible usable 29 IP addresses in this subnet, only 7 are explicitly defined in the bandwidth shaping rules above. Any servers located on other IPs within the subnet would fall under the “default” rule. This allows me to do three important things:

1) Guarantee traffic to specific computers within my subnet.
2) Cap the total bandwidth consumed by all computers to 15Mbit.
3) Force all systems not explicitly defined to obtain throughput by contention.

There you have it:  a dumbed down overview of a very basic HTB shaping setup using the HTB.init script

Linux Routers Gone Wild (Introduction)

Tags: , , , , ,

I have recently embarked upon a difficult professional journey.  The larger part of this journey is in fact an attempt to secure my network and slowly, inexorably retire as much Microsoft software from service as possible.  The reasons for this lie largely in the complexity of Microsoft licensing; I am often beset by so many IT projects that it is honestly a nightmare attempting to comprehend the plethora of licensing options and caveats.  Trying to make sure our company remains in compliance is itself almost a full time job.

The solution to this is simple: cut back on as much Microsoft software as is humanely possible.  There are naturally some fairly enormous barriers to this concept.  The first being that there is simply no way we are (ever) going to be able to ditch Microsoft on the desktop.  There is simply too much industry-specific software we are totally reliant on for this to be anything but a midsummer night’s dream.  To manage these desktops, I need a directory and something that will handle group policy like templates.  After much searching and pondering the simple reality is that Microsoft’s Active Directory is the best bang for my buck in this department, and so there is no reason to abandon it.  (I should state for the record however that Novell’s offerings an unbelievably close second.)

The second obstacle is not a hardware or software limitation, but rather one of wetware.  The wetware, (which will remain nameless,) ultimately responsible for accepting or rejecting my various schemes and proposals consists of two units.  The first unit is logical, rational and driven by nothing more than sounds business rationale.  If you can make a solid business case for something, one of the two decision making wetware units will be easily won over.  Unfortunately, this wetware unit has exceptionally limited IT knowledge; when my recommendations clash with those of the second decision making wetware unit, issues can arise.

The second decision making unit in question is rather less approachable than the first.  Though remarkably intelligent, this unit remains deeply wedded to all things Microsoft and has what I consider to be an incredibly dangerous fascination with whatever happens to be the newest technology of the day.  As a born and bred technology geek, I truly understand the “gee whiz” factor shiny new kit can bring.  As someone who goes to work a loyal company man and puts aside everything except my job, I can’t and won’t let my employer risk the business on untested or questionable gear.  (Let several someone elses walk through that minefield first.)

The wetware obstacles to reducing the corporate Microsoft overhead, (and with it the licensing burdens) are thusly formidable.  An unfortunate amount of my job has devolved into simply playing the politics necessary to be allowed to implement the right solutions for our requirements and budget.  In many cases I actually have to purchase and implement first, and then inform people of it later if I feel it corporately critical that a project be accomplished without being tied up with internal political infighting and negotiations for six to eight months.

This year I am removing Microsoft’s Internet and Security Acceleration (ISA) Server from my organisation.  Like the use of LAMP webservers, LACS e-mail sanitisation servers and the slow introduction of Linux fileserver, this is the story of nibbling at the edges of a Microsoft network with tactical implementations of Linux systems.

© 2009 drink the sweet feeling of the colour zero. All Rights Reserved.

This blog is powered by the Wordpress platform and beach rentals.