Iptables total bandwidth limiting issues

Hi,
I want to achieve the following on a 100mbit dedicated server.
traffic coming from Nginx on port 81: max 50 mbit usage
all other traffic can use all they like.

I created 3 different classes with tc qdisk via the following ssh commands

tc qdisc del dev eth0 root

tc qdisc add dev eth0 root handle 1: htb default 1 r2q 160

tc class add dev eth0 parent 1: classid 1:1 htb rate 105000kbit burst 1k
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 105000kbit ceil 105000kbit burst 1k
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 50000kbit ceil 50000kbit burst 1k

tc qdisc add dev eth0 parent 1:2 handle 2: sfq perturb 10
tc qdisc add dev eth0 parent 1:3 handle 3: sfq perturb 10

Code (markup):

This goes without problems. I then want to apply this to Iptables where it goes wrong

iptables -t mangle -A OUTPUT -o eth0 -j CLASSIFY --set-class 1:2
iptables -t mangle -A OUTPUT -o eth0 -m tcp -p tcp --dport 81 -j CLASSIFY --set-class 1:3

Code (markup):

when entering these commands it does not do anything; port 81 still uses the full 100 mbit from the server. the output from iptables -L is:

serveur:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
           all  --  anywhere             anywhere

Code (markup):

I have hardly any understanding of tc and iptables so I have no idea what is wrong... I hope someone can figure out what's wrong

Additional info:

fast:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP qlen 100
    link/ether 00:1c:c0:* brd ff:ff:ff:*
    inet 76.73.*/29 brd 76.73.* scope global eth0
    inet 76.73.*/29 brd 76.73.* scope global secondary eth0:0
    inet 76.73.*/29 brd 76.73.* scope global secondary eth0:1
    inet 76.73.*/29 brd 76.73.* scope global secondary eth0:2
    inet 76.73.*/29 brd 76.73.* scope global secondary eth0:3
    inet6 fe80::21c:*/64 scope link
       valid_lft forever preferred_lft forever

Code (markup):

 

It seems that your iptables doesn't show up in the iptables -L listing. Do you get any error message?

 

fast:~# tc qdisc del dev eth0 root
fast:~# tc qdisc add dev eth0 root handle 1: htb default 1 r2q 160
fast:~# tc class add dev eth0 parent 1: classid 1:1 htb rate 105000kbit burst 1k
fast:~# tc class add dev eth0 parent 1:1 classid 1:2 htb rate 105000kbit ceil 10                                                                                                             5000kbit burst 1k
fast:~# tc class add dev eth0 parent 1:1 classid 1:3 htb rate 50000kbit ceil 500                                                                                                             00kbit burst 1k
fast:~# tc qdisc add dev eth0 parent 1:2 handle 2: sfq perturb 10
fast:~# tc qdisc add dev eth0 parent 1:3 handle 3: sfq perturb 10
fast:~# iptables -t mangle -A OUTPUT -o eth0 -j CLASSIFY --set-class 1:2
fast:~# iptables -t mangle -A OUTPUT -o eth0 -m tcp -p tcp --dport 81 -j CLASSIF                                                                                                             Y --set-class 1:3
fast:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
           all  --  anywhere             anywhere

fast:~#

Code (markup):

no error message

iptables v1.4.2


When I try to make a new chain I get this:

fast:~# iptables -N NGINX
fast:~# iptables -L NGINX
Chain NGINX (0 references)
target     prot opt source               destination
fast:~# iptables -t mangle -A NGINX -o eth0 -j CLASSIFY --set-class 1:2
iptables: No chain/target/match by that name
fast:~# iptables -P NGINX ACCEPT
iptables: Bad built-in chain name

Code (markup):

 

Hi, the error was that it should have been --sport 81 (I think)
I used a script i found here:
http://www.faqs.org/docs/Linux-HOWTO/ADSL-Bandwidth-Management-HOWTO.html

and modified it to my needs, it works on my server.

#!/bin/bash
#
# myshaper - DSL/Cable modem outbound traffic shaper and prioritizer.
#            Based on the ADSL/Cable wondershaper (www.lartc.org)
#
# Written by Dan Singletary (8/7/02)
#
# NOTE!! - This script assumes your kernel has been patched with the
#          appropriate HTB queue and IMQ patches available here:
#          (subnote: future kernels may not require patching)
#
#       http://luxik.cdi.cz/~devik/qos/htb/
#       http://luxik.cdi.cz/~patrick/imq/
#
# Configuration options for myshaper:
#  DEV    - set to ethX that connects to DSL/Cable Modem
#  RATEUP - set this to slightly lower than your
#           outbound bandwidth on the DSL/Cable Modem.
#           I have a 1500/128 DSL line and setting
#           RATEUP=90 works well for my 128kbps upstream.
#           However, your mileage may vary.
#  RATEDN - set this to slightly lower than your
#           inbound bandwidth on the DSL/Cable Modem.
#
#
#  Theory on using imq to "shape" inbound traffic:
#
#     It's impossible to directly limit the rate of data that will
#  be sent to you by other hosts on the internet.  In order to shape
#  the inbound traffic rate, we have to rely on the congestion avoidance
#  algorithms in TCP.  Because of this, WE CAN ONLY ATTEMPT TO SHAPE
#  INBOUND TRAFFIC ON TCP CONNECTIONS.  This means that any traffic that
#  is not tcp should be placed in the high-prio class, since dropping
#  a non-tcp packet will most likely result in a retransmit which will
#  do nothing but unnecessarily consume bandwidth.  
#     We attempt to shape inbound TCP traffic by dropping tcp packets
#  when they overflow the HTB queue which will only pass them on at
#  a certain rate (RATEDN) which is slightly lower than the actual
#  capability of the inbound device.  By dropping TCP packets that
#  are over-rate, we are simulating the same packets getting dropped
#  due to a queue-overflow on our ISP's side.  The advantage of this
#  is that our ISP's queue will never fill because TCP will slow it's
#  transmission rate in response to the dropped packets in the assumption
#  that it has filled the ISP's queue, when in reality it has not.
#     The advantage of using a priority-based queuing discipline is
#  that we can specifically choose NOT to drop certain types of packets
#  that we place in the higher priority buckets (ssh, telnet, etc).  This
#  is because packets will always be dequeued from the lowest priority class
#  with the stipulation that packets will still be dequeued from every
#  class fairly at a minimum rate (in this script, each bucket will deliver
#  at least it's fair share of 1/7 of the bandwidth).  
#
#  Reiterating main points:
#   * Dropping a tcp packet on a connection will lead to a slower rate
#     of reception for that connection due to the congestion avoidance algorithm.
#   * We gain nothing from dropping non-TCP packets.  In fact, if they
#     were important they would probably be retransmitted anyways so we want to
#     try to never drop these packets.  This means that saturated TCP connections
#     will not negatively effect protocols that don't have a built-in retransmit like TCP.
#   * Slowing down incoming TCP connections such that the total inbound rate is less
#     than the true capability of the device (ADSL/Cable Modem) SHOULD result in little
#     to no packets being queued on the ISP's side (DSLAM, cable concentrator, etc).  Since
#     these ISP queues have been observed to queue 4 seconds of data at 1500Kbps or 6 megabits
#     of data, having no packets queued there will mean lower latency.
#
#  Caveats (questions posed before testing):
#   * Will limiting inbound traffic in this fashion result in poor bulk TCP performance?
#     - Preliminary answer is no!  Seems that by prioritizing ACK packets (small <64b)
#       we maximize throughput by not wasting bandwidth on retransmitted packets
#       that we already have.
#   

# NOTE: The following configuration works well for my 
# setup: 100/100M FDC box and 100/100M ovh box

DEV=eth0
RATEUP=101000
LIMITUP=75000

# 
# End Configuration Options
#

if [ "$1" = "status" ]
then
        echo "[qdisc]"
        tc -s qdisc show dev $DEV
#        tc -s qdisc show dev imq0
        echo "[class]"
        tc -s class show dev $DEV
#        tc -s class show dev imq0
        echo "[filter]"
        tc -s filter show dev $DEV
#        tc -s filter show dev imq0
        echo "[iptables]"
        iptables -t mangle -L MYSHAPER-OUT -v -x 2> /dev/null
#        iptables -t mangle -L MYSHAPER-IN -v -x 2> /dev/null
        exit
fi

# Reset everything to a known state (cleared)
tc qdisc del dev $DEV root    2> /dev/null > /dev/null
#tc qdisc del dev imq0 root 2> /dev/null > /dev/null
iptables -t mangle -D POSTROUTING -o $DEV -j MYSHAPER-OUT 2> /dev/null > /dev/null
iptables -t mangle -F MYSHAPER-OUT 2> /dev/null > /dev/null
iptables -t mangle -X MYSHAPER-OUT 2> /dev/null > /dev/null
#iptables -t mangle -D PREROUTING -i $DEV -j MYSHAPER-IN 2> /dev/null > /dev/null
#iptables -t mangle -F MYSHAPER-IN 2> /dev/null > /dev/null
#iptables -t mangle -X MYSHAPER-IN 2> /dev/null > /dev/null
#ip link set imq0 down 2> /dev/null > /dev/null
#rmmod imq 2> /dev/null > /dev/null

if [ "$1" = "stop" ] 
then 
        echo "Shaping removed on $DEV."
        exit
fi

###########################################################
#
# Outbound Shaping (limits total bandwidth to RATEUP)

# set queue size to give latency of about 2 seconds on low-prio packets
#ip link set dev $DEV qlen 30

# changes mtu on the outbound device.  Lowering the mtu will result
# in lower latency but will also cause slightly lower throughput due 
# to IP and TCP protocol overhead.
#ip link set dev $DEV mtu 1000

# add HTB root qdisc
tc qdisc add dev $DEV root handle 1: htb default 26

# add main rate limit classes
tc class add dev $DEV parent 1: classid 1:1 htb rate ${RATEUP}kbit

# add leaf classes - We grant each class at LEAST it's "fair share" of bandwidth.
#                    this way no class will ever be starved by another class.  Each
#                    class is also permitted to consume all of the available bandwidth
#                    if no other classes are in use.
tc class add dev $DEV parent 1:1 classid 1:19 htb rate 50kbit ceil ${RATEUP}kbit prio 0
tc class add dev $DEV parent 1:1 classid 1:20 htb rate 15000kbit ceil ${RATEUP}kbit prio 1
tc class add dev $DEV parent 1:1 classid 1:21 htb rate 15000kbit ceil ${LIMITUP}kbit prio 2
tc class add dev $DEV parent 1:1 classid 1:26 htb rate 50kbit ceil ${RATEUP}kbit prio 3


# attach qdisc to leaf classes - here we at SFQ to each priority class.  SFQ insures that
#                                within each class connections will be treated (almost) fairly.
tc qdisc add dev $DEV parent 1:19 handle 19: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $DEV parent 1:21 handle 21: sfq perturb 10
tc qdisc add dev $DEV parent 1:26 handle 26: sfq perturb 10

# filter traffic into classes by fwmark - here we direct traffic into priority class according to
#                                         the fwmark set on the packet (we set fwmark with iptables
#                                         later).  Note that above we've set the default priority
#                                         class to 1:26 so unmarked packets (or packets marked with
#                                         unfamiliar IDs) will be defaulted to the lowest priority
#                                         class.
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 19 fw flowid 1:19
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 20 fw flowid 1:20
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 21 fw flowid 1:21
tc filter add dev $DEV parent 1:0 prio 0 protocol ip handle 26 fw flowid 1:26

# add MYSHAPER-OUT chain to the mangle table in iptables - this sets up the table we'll use
#                                                      to filter and mark packets.
iptables -t mangle -N MYSHAPER-OUT
iptables -t mangle -I POSTROUTING -o $DEV -j MYSHAPER-OUT

# add fwmark entries to classify different types of traffic - Set fwmark from 20-26 according to
#                                                             desired class. 20 is highest prio.
iptables -t mangle -A MYSHAPER-OUT -p tcp --dport ssh -j MARK --set-mark 19    # secure shell
iptables -t mangle -A MYSHAPER-OUT -p tcp --sport ssh -j MARK --set-mark 19    # secure shell
iptables -t mangle -A MYSHAPER-OUT -p tcp --sport http -j MARK --set-mark 20   # Local web server
iptables -t mangle -A MYSHAPER-OUT -p tcp --dport http -j MARK --set-mark 20   # Local web server
iptables -t mangle -A MYSHAPER-OUT -p tcp --sport 81 -j MARK --set-mark 21   # Local web server
iptables -t mangle -A MYSHAPER-OUT -p tcp --dport 81 -j MARK --set-mark 21   # Local web server
iptables -t mangle -A MYSHAPER-OUT -m mark --mark 0 -j MARK --set-mark 26      # redundant- mark any unmarked packets as 26 (low prio)

# Done with outbound shaping
#
####################################################

echo "Outbound shaping added to $DEV.  Rate: ${RATEUP}Kbit/sec."

Code (markup):

 

Bumping this thread with a new question.

How to do this if a server runs with 'pfifo_fast' in stead of 'htb'? I wanted to apply this to another server from ovh but it errors at the qdisc add.

I tried googling and found out this this specific is not possible with pfifo_fast so I want to ask if anyone knows if I can put port 80 at least on a higher priority or has another solution for this