Snort mailing list archives
ids-load-balancing-HOWTO
From: "Jeremy M. Guthrie" <jeremy.guthrie () berbee com>
Date: Wed, 27 Jul 2005 15:43:14 -0500
I have created a new Howto for creating a multi-gigabit-per second IDS load
balancer/distributor. Technically the limits are on buses and CPUS, so if
you have enough bus speed and CPU... then you could use 10gbps cards. I am
in the process of trying to get this posted to the Linux documentation
project for the official online copy.
I see this in direct competition to Top Layer, Foundry, and Gigamon. I'm sure
there are a few others as well.
Hopefully this makes a pretty big splash!
Special shout out to Chris Gerg on this.
----------------------------------------------------------------
ids-load-balancing-HOWTO
Jeremy M. Guthrie
jeremy.guthrie () berbee com
July 2005
Version 1.0
Jeremy M. Guthrie
7-27-2005
-----------------------------------------------------------------------------
Table of Contents
1. Terms
0.1. The Example Problem
2. Possible Solutions
3. The Linux Answer
4. Performance Tuning
4.1. General Limits
4.2. Instrumentation
4.2.1. rtstat or lnstat in the land of iproute2
4.2.2 Important /proc files for your reference.
4.2.2.1. /proc/net/softnet_stat
4.2.3 CPUs, Memory, and Interrupts & the Rules To Follow
4.2.3.1 CPUs should only following one NIC.
4.2.3.2 Taskset is your friend.
4.3 Tune your routing!
4.3.1 Adjust one kernel parameter and reboot.
4.3.2 Adjust /proc/sys/net/... to tune your routing
4.3.3 Basic scripts
4.3.3.1 Watcherrors
4.3.3.2 Policy Routing Control Script
4.3.4 A Quick Word on SNMP
5. Copyright
5.1. GNU Free Documentation License
5.2. PREAMBLE
5.3. APPLICABILITY AND DEFINITIONS
5.4. VERBATIM COPYING
5.5. COPYING IN QUANTITY
5.6. MODIFICATIONS
5.7. COMBINING DOCUMENTS
5.8. COLLECTIONS OF DOCUMENTS
5.9. AGGREGATION WITH INDEPENDENT WORKS
5.10. TRANSLATION
5.11. TERMINATION
5.12. FUTURE REVISIONS OF THIS LICENSE
5.13. How to use this License for your documents
-----------------------------------------------------------------------------
0. Credit and Notes About the Author
First and foremost this document would not be possible if it were not for
Jim Leu, Robert Olsson, Stephen Hemminger, David Miller, Jesse Brandeburg,
Chris Gerg, and Jon Vanderhill. All of these people were instrumental in
providing code, feedback, resources, or other critical input to help get
this document and system where it is today.
I myself am a network engineer. I have been building or managing networks
since 1992. I have been a Linux advocate since late 1995. It is my
primary OS for what I do. I am not a kernel developer though I have
written other software packages. I make no claim that I am the final
authority on systems I reference within. I am providing descriptions of
the bits and pieces based on existing documentation or discussions I have
had with other persons. IOW, if you see a blatant error or a feature
omission, it is not on purpose and would love the feedback to correct this
documentation.
-----------------------------------------------------------------------------
0.1. Terms
EBTables: A Layer-2 filtering technology in the Linux V2.6 kernel.
Linear Search: The process of one-by-one searching through the hash
collision buckets.
L2CC: Layer Two Cross Connect: A software patch created by Jim Leu that
takes all input frames from an ethernet controller and NATs the
destination MAC address to be that of some host on an outbound
port.
L2XC: Another name for the L2CC software.
Innovation: "the introduction of something new" as defined by
Merriam-Webster Online(www.m-w.com).
MAC-Munger: Another name for the L2CC software.
Policy-Router: A router which routes because of a dictated policy rather
than by normal IPV4 per-hop-behavior. ie. route all port 80
out my DSL link, all SMTP traffic out my cable link vs. default
route traffic via my cable.
SPAN: Switch port analyzer. Think of it as a port which mirrors traffic
off of a VLAN or switchport out another port. ie. mirror all
all traffic going in/out of the Firewall's inside interface to the
SPAN port.
-----------------------------------------------------------------------------
1. The Example Problem
You are a large business with a lot of internet bandwidth. Your IDS is
capable of dealing with some subset of your traffic volume but it cannot
handle it by itself. It also would cost too much money to buy a bigger,
faster, IDS. You want to buy the same model as you already have. The two
IDSes together can handle the volume but now you have to figure out how to
divide the traffic up between the two.
Sounds simple doesn't it?
Example bandwidth, 800 megabit per second. Assume you are using Cisco IDS
4255 where each IDS lists at $25,000 a piece. A 4255 is supposed to be
capable of 600mbps. Remember, you want the IDS to see all of the data in
a flow, not just one half! So you cannot just break up the flow without
some thought.
-----------------------------------------------------------------------------
2. Possible Solutions
F5, Radware, and Top Layer networks all make boxes that will split data
appart. All of these are commercial products that help load balance
flows for IDSs or traffic management. They are not cheap and in some
cases are budget busters.
The example network hardware I reference in here will be Cisco platforms
as that is what I am most familiar with.
-----------------------------------------------------------------------------
3. The Linux Answer
The good news is that Linux has an answer with L2CC/EBTables and Policy
Routing. The two main components run in the kernel so they operate
securely. I will detail scaling the solution upward as it should be able
to meet the needs of most any bandwidth.
The Linux solution identified herein has been dubbed by Chris Gerg as the
(I(DS)^2). However I will to the whole system as the IDS load balancer
here after.
The concept is simple:
(or EBTables)
+---------------+ +---+
| Catalyst 6509 |-[1000SX-SPAN]->| l | +---------------+
+---------------+ | 2 |-[1000SX]->| Policy Router |
+---------------+ | c | +-+-----+-------+
| Catalyst 6509 |-[1000SX-SPAN]->| c | |
+---------------+ +---+ |[1000SX]
v
+-------------------+
| Catalyst 3750 Gig |
+---+------------+--+
| |
|<--1000TX-->|
+---+---+ |
| IDS A | |
+-------+ |
+---+---+
| IDS A |
+-------+
There is a highly available network with two enterprise class switches
front-ending the network. Each switch hands off a SPAN port to the Layer
Two Cross Connect(l2cc/l2xc) host AKA EBTables Host. The SPAN data is an
identical copy of the data from the ports they mirror.
-
If the destination MAC address of an example packet is 05:05:05:03:03:03,
then we to NAT this to an address of the policy router's outside interface
MAC address. Why? Normal routers will not route data unless the layer
two destination address matches their own MAC address. The l2cc host
changes/L2-NATs the destination MAC address to be that of the Policy
Router's 'outside' interface. When the data arrives at the policy
router, it looks in its ip routing ruleset to determine what 'table' to
use to forward the incoming packets. Once a table is found and selected,
that table is then used to define the per-hop-behavior at that point in
time for that packet.
3.1. Further Example Details
Time to add yet another layer of detail to the example. We will assume
we want to run Snort or NTOP against all traffic coming through our
network. In this example we will assume that we have 1.0.0.0/16 assigned
to us. We also know that we have a pretty good distribution of traffic
such that 1.0.0.0/17 gets about 350mbps and 1.0.128.0/17 gets about 450
mbps.
What we also know is that because of the network that traffic to/from
both /17 subnets comes in over both SPANs but is NOT duplicate traffic.
This can because of things like per-packet or per-flow routing decisions
made by downstream equipment.
3.2. Needed Software Updates & Other Requirements
You will want to make sure you have the latest tools for the job. I have
only ever worked this solution using Intel Gig NICs. Others should work
but I have never tested them.
Software to get:
1. Latest Intel E1000 drivers
http://sourceforge.net/projects/e1000/
2. Latest IPRoute2 utilities:
http://www.policyrouting.org/
3. Latest L2CC software:
http://mpls-linux.sf.net/
4. A V2.4 & V2.6 Linux Kernel:
http://www.kernel.org/
L2CC requires V2.4 and you should use V2.6 for your policy router.
5. Schedutils
http://tech9.net/rml/schedutils/
6. EBTables
http://ebtables.sourceforge.net/
As for other requirements....
I HIGHLY recommend a multi-CPU box. Hyperthread has shown advantages and
works well in our implementation. I typically assign one CPU per NIC on
either the EBTables box and/or Policy Router. Your mileage will vary but
use common sense. If you have a dual 3.2 ghz P4 w/ Hyperthreading then
that will have enough horse power to handle large data volumes. In some
cases you could easily assign more than one NIC to a CPU.
3.3. Notes About Limitations
Every piece of hardware or software has limits. Keep this in mind when
deploying your system. I will show you where counters are instrumented
and generically how to manage them. Like any good system administrator,
you will have to manage your system.
3.4. It's All About the 'IP'
ifconfig, netstat, and other old-style utilities are being phased out
slowly. Familiarize yourself with the 'ip' utility from the iproute2
package as it replaces the prior listed programs. The iproute2 package
includes other programs that will help in providing access to critical
performance information. 'lnstat' and 'rtstat' can be used to poll
routing performance information from your kernel DEPENDING on which
kernel release you are running.
3.5. Other assumptions
Going forward it will be assumed that you have the appropriate kernels
and/or features installed unless we talk explicitly about a feature.
3.6 L2CC vs EBTables & L2-NAT
L2CC was the only option available when this document was first written
but now EBTables is available. EBTables was beaten up a quite bit in the
May 2005 Networld+Interop in Vegas. EBTables proved to be a very
reliable bit of software. L2CC has more burn in time for my organization
but we are in the process of migrating. This document will provide
examples for both L2CC & EBTables.
3.6.1 L2CC Config
The example L2CC host has three Gig NICs. eth0-1 are gathering SPAN data
while eth2 is the output port. The Policy router's eth0 MAC address is
01:01:01:10:10:10.
The L2CC host will need the L2CC patch applied against the V2.4 kernel.
From there do the following:
make menuconfig
Select "Networking Options"
Compile in "Layer 2 Cross Connectr (EXPERIMENTAL)", do not build as a
module.
Rebuild your kernel and reboot.
#add two entries, one for each NIC
l2cc -a -i eth0 -o eth2 -m 01:01:01:10:10:10
l2cc -a -i eth1 -o eth2 -m 01:01:01:10:10:10
#delete two entries, one for each NIC
l2cc -d -i eth0 -o eth2 -m 01:01:01:10:10:10
l2cc -d -i eth1 -o eth2 -m 01:01:01:10:10:10
*WARNING* Test that your configuration is working using TCPDump in an
ISOLATED network.
In the above example, I should be able to run three instances of TCPDump
and see that data coming in eth0 & eth1 is having its MAC address NAT'd
when being transmitted out eth2.
3.6.2 EBTables & L2-NAT
The example EBTables host has three Gig NICs. eth0-1 are gathering SPAN
data while eth2 is the output port. The Policy router's eth0 MAC address
is 01:01:01:10:10:10.
EBTables will require a Linux host running with a V2.6 kernel.
To build your Linux kernel with EBTables support:
make menuconfig
Select "Device Drivers"
Select "Networking Support"
Select "Networking Options"
Compile in "802.1d Ethernet Bridging"
Select "Network packet filtering"
Select "Bridge: Netfilter Configuration"
Select "Ethernet Bridge tables (ebtables) support"
Select "ebt: nat table support"
Select "ebt: dnat target support"
Select "ebt: snat target support"
Rebuild your kernel and reboot.
#prep your interfaces
ifconfig eth0 up
ifconfig eth1 up
ifconfig eth2 up
#create the br0 interface
brctl addbr br0
#turn off spanning-tree
brctl stp br0 off
#add interfaces to the br0 broadcast domain
brctl addif br0 eth0
brctl addif br0 eth1
brctl addif br0 eth2
#Prep ebtables
ebtables -F INPUT
ebtables -F OUTPUT
ebtables -F FORWARD
ebtables -t nat -F PREROUTING
#NAT all incoming data on eth0 to 00:11:25:8c:8c:37
ebtables -t nat -A PREROUTING -I eth0 -j dnat -to-destination
00:11:25:8c:8c:37
#NAT all incoming data on eth1 to 00:11:25:8c:8c:37
ebtables -t nat -A PREROUTING -I eth1 -j dnat -to-destination
00:11:25:8c:8c:37
#Tell EBTables to route/bridge data destined to 00:11:25:8c:8c:37 out eth2
ebtables -A OUTPUT -o eth2 -d 00:11:25:8c:8c:37 -j ACCEPT
*WARNING* Test that your configuration is working using TCPDump in an
ISOLATED network.
In the above example, I should be able to run three instances of TCPDump
and see that data coming in eth0 & eth1 is having its MAC address NAT'd
when being transmitted out eth2.
3.7. The Policy Router Config
I am going to assume you have the appropriate iproute2 package for your
kernel. You may need a newer version of the iproute2 utilities.
ie. V2.6.9 kernel
strace -f rtstat ...
--snip--
open("/proc/net/rt_cache_stat", O_RDONLY) = -1 ENOENT (No such file or
directory)
--snip--
In this case I would need the latest iproute2 code to use the
/proc/net/stat/rt_cache instead. In fact, rtstat may also be called
'lnstat'.
*ASSUMPTION* This policy router config assumes there are no overlapping
subnets. Overlapping subnets mean you must order your policy rules
appropriately for them to work. Overlapping IP examples: 192.168.0.0/16
& 192.168.0.0/24
The policy router is made up of several components. The policy router
uses rules and tables. Rules are used to classify which traffic belongs
to which table. This is where the policy in policy routing comes from.
You define what policies you want to implement.
Tables are routing tables used to device the per-hop-behavior for the
packet being routed via that table. Shortly you will see how the
combination of rules with tables are combined to split traffic.
In our example, we want to split traffic of two /17s to the two sensors.
With that in mind we will add rules to do the actual policy mapping.
Eth0 is our input device while eth1 is our output device. Eth0 will have
an IP address of 10.0.0.1/32. An IP address is required otherwise the
Linux kernel will not policy-route for the interface. Eth1 will have an
IP address of 10.0.1.1/24. Sensor 1 will have an IP address of
10.0.1.10, Sensor 2 will have an IP address of 10.0.1.11.
Routing at high speed means any hiccup, EVEN SMALL ones, result in lost
packets as recieve rings for network cards can be overrun quickly. Thus
we have to minimize any hiccups. One hiccup is ARP. Sensors NEVER
transmit and we will always have data to send to them. You will see
several changes we will make to account for this.
#First, turn off IP forwarding before we configure routing
echo 0 > /proc/sys/net/ipv4/ip_forward
#Add static ARP entries as we should ALWAYS know what MAC
#address to associate with our Sensor IPS
arp -s 10.0.1.10 00:02:50:98:DC:1C
arp -s 10.0.1.11 00:02:50:A1:5D:5A
#send any traffic to/from 1.0.0.0/17 to table 15
ip rule add type unicast dev eth0 from 1.0.0.0/17 table 15
ip rule add type unicast dev eth0 to 1.0.0.0/17 table 15
#send any traffic to/from 1.0.128.0/17 to table 16
ip rule add type unicast dev eth0 from 1.0.128.0/17 table 16
ip rule add type unicast dev eth0 to 1.0.128.0/17 table 16
#Tell policy routing code that the only path in table 15 is via Sensor 1!
ip route add default via 10.1.0.10 dev eth1 table 15
#Tell policy routing code that the only path in table 16 is via Sensor 2!
ip route add default via 10.1.0.11 dev eth1 table 16
#When done with our changes, flush the cache
ip route flush cache
#Lastly, turn on IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward
This is all you actually 'have to do' to enable a working system. There
are however other changes you SHOULD make to help with performance.
For one, if you left IP forwarding on and blew away the policy routing
table, all traffic would then follow your normal default route on the
box!!!!!!
*WARNING*
Let's complicate our policy router by adding another interface,
eth3->192.168.10.5 with a default gw to a firewall so we can remotely
manage the system. In our example, we had 800mbps heading towards our
policy router. If we turn off policy routing but left IP forwarding on,
the Linux host will try to forward >>>>800mbps<<<< of traffic towards the
firewall on the 192.168.10.0/24 network thereby killing it. Okay?!
Follow? If not, re-read till you do.
Hint: If your firewall has a 100mbps interface and you fire 800mbps at
it, the firewall will stop working because you will overload its
interface with bandwidth.
There are then two ways to protect yourself:
A) Always turn off IP forwarding before making ANY changes to the policy
router.
B) Turn on iptables filtering. Here is a quick an dirty example of an
IPtables filter to apply to this example host:
#The only data that will be allowed in or out of eth3 will be traffic
#to or from 192.168.10.5. So even if you accidentally leave IP forwarding
#on, you can trust that IP Tables it stemming the flow from burrying your
#gateway/firewall.
iptables -F
iptables -A FORWARD -s 192.168.10.5 -o eth3 -j ACCEPT
iptables -A FORWARD -o eth0 -j DROP
iptables -A OUTPUT -s 192.168.10.5 -o eth3 -j ACCEPT
iptables -A OUTPUT -o eth0 -j DROP
The switches will need some adjustments to make sure that the switch
knows exactly where the sensors are. If the sensors never transmit data
on their ports then the switches turn act as hubs which is exactly what
we don't want.
#3750 config:
interface GigabitEthernet1/0/1
description Eth0 of Sensor 1
switchport access vlan 100
switchport trunk native vlan 100
switchport trunk allowed vlan none
switchport mode access
switchport nonegotiate
load-interval 30
no cdp enable
!
interface GigabitEthernet1/0/2
description Eth0 of Sensor 2
switchport access vlan 100
switchport trunk native vlan 100
switchport trunk allowed vlan none
switchport mode access
switchport nonegotiate
load-interval 30
no cdp enable
!
interface GigabitEthernet1/0/28
description Eth1 of Policy Router
switchport access vlan 100
switchport trunk native vlan 100
switchport trunk allowed vlan none
switchport mode access
switchport nonegotiate
load-interval 30
no cdp enable
!
mac-address-table static 0002.5098.DC1C vlan 100 interface
GigabitEthernet1/0/1
mac-address-table static 0002.50A1.5D5A vlan 100 interface
GigabitEthernet1/0/2
That's it? Right? Well, sure. If your system is blazing fast and does
not require any tuning. This assumes your system defaults are adequate.
That may not be the case. The rest of this document will be dedicated to
discussing eliminating the bottlenecks.
-----------------------------------------------------------------------------
4. Performance Tuning
Jumping back to a prior statement, you need to be aware of the limits of
the policy routing system. I will list them out here and we will discuss
how to go about addressing them. Some are easy to update, others are
what they are.
4.1. General Limits
Kernel Limits:
Max # of Policy Routing Rules: 32768 - some are reserved
Max # of Policy Routing Tables: 256 - some are reserved
Max # of Interfaces in V2.6 Kernel: 4096
Max # of Interfaces in V2.4 Kernel: 256
IP Route Hash Limits:
Card Limits:
Intel EtherExpress RX/TX Buffer Count: 256 packets
Max RX/TX Buffer Count: 4096 packets
4.2. Instrumentation
Knowing that any system is running well can take a bit to figure out.
There will be a few places that we concentrate on monitoring. Memory,
CPU utilization, interrupt distribution, network stack drops, network
card drops, # of routes, garbage collection, routing packets per second,
# of hash entries, and others.
4.2.1. rtstat or lnstat in the land of iproute2
Rtstat and lnstat are two tools used to watch routing activity within the
Linux kernel. Kernels prior to approximately V2.6.9 will use rtstat.
After 2.6.9 iproute2 uses lnstat to gather routing detail. You can tell
if your kernel works with rtstat or lnstat by trying the existing install
of rtstat. Both tools pull data from /proc, it is a question of which
one.
ie.
[plato jguthrie 10:29am]~-> rtstat
fopen: No such file or directory
4.2.2 Important /proc files for your reference.
#Data on each route cache entries/hashes
/proc/net/rt_cache
#Aggregate statistics on route cache entries/hashes
/proc/net/stat/rt_cache
#General information about the network stack on a per-cpu basis
/proc/net/softnet_stat
We will examine each of the files to examine what they can tell us.
4.2.2.1. /proc/net/softnet_stat
cat /proc/net/softnet_stat
00000130 00000034 00000000 00000000 00000000 00000000 00000000 00000000
00000000
00000150 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000
The basic idea behind softnet_stat is that we use it as a way to tell us
if the Kernel itself is dropping packets. The second field in the list
of nine is the packet drop count. You can see in this example that we
have dropped 0x34 packets. Look into using NAPI or other network card
features to possibly relieve CPU overhead.
4.2.3 CPUs, Memory, and Interrupts & the Rules To Follow
As with any rules here, all can be affected by your budget.
1. Assume that each CPU on your box will be handling only one NIC's
interrupts.
2. Assume that you will be using 'taskset' to keep non-kernel routing
functions assigned to a specific CPU.
3. EBTables uses little memory, you can skimp here
4. Policy Routing Chews Memory, you cannot skimp here - minimum 1
Gigabyt of RAM ***HIGHLY*** recommended
4.2.3.1 CPUs should only following one NIC.
If you look at the output below you can see that CPU0 is taking the
interrupts for eth3. CPU1 is taking interrupts for eth2 & eth0.
Optimising any system relies on keep thrashing to a minimum. As a result
I highly recommend disable IRQ Balancing.
make menuconfig for your kernel config
Select "Processor type and features"
Disable "Enable kernel irq balancing"
Rebuild your kernel and reboot.
You will have to poke around /proc to set which CPU an interrupt binds
to. Here is what was used to set the interrupt/CPU bindings down below:
echo 01 > /proc/irq/18/smp_affinity
echo 02 > /proc/irq/20/smp_affinity
The value used is expressed in powers of two. ie. CPU3 would actually be
04.
cat /proc/interrupts
CPU0 CPU1
0: 3184569581 1789102599 IO-APIC-edge timer
1: 1005 218 IO-APIC-edge i8042
7: 0 0 IO-APIC-level ohci_hcd
8: 1 1 IO-APIC-edge rtc
12: 122 74 IO-APIC-edge i8042
14: 2 0 IO-APIC-edge ide0
18: 995373697 5139 IO-APIC-level eth3
20: 2 1378253801 IO-APIC-level eth2
27: 7542100 9352305 IO-APIC-level eth0
28: 4150402 13187680 IO-APIC-level aic7xxx
30: 0 0 IO-APIC-level acpi
NMI: 0 0
LOC: 679927478 679903506
ERR: 0
MIS: 0
4.2.3.2 Taskset is your friend.
Taskset allows an administrator to bind a software process to a specific
processor on a box. By using taskset you help cut down on CPU cache
thrashing. If your hosts will be running SNMP daemons, snort, etc, then
you will want to bind snort to the least used CPU. You want to keep your
base load balancing system predictable.
4.3 Tune your routing!
The Linux kernel defaults for routing work great in a lot of situations
and unfortunately this is not one of them. You will find that the Linux
kernel will take some tuning to get the performance you want.
The kernel counts on route hash entries to track existing conversations.
One route hash entry is used per host-host communication. ie.
Entry 1: 192.168.1.1 -> 192.168.2.2
Entry 2: 192.168.2.4 -> 192.168.7.5
Imagine HOW MANY of these you might have given your volumes of traffic.
Also imagine that the kernel has NO WAY to tell when a conversation is
over. The kernel is not following TCP/UDP conversations thus entries age
out of existence. You will need to conduct performance testing to
confirm how well your environment runs. Just be warned that the more
route hash entries you run, the more RAM the kernel WILL use.
Example 'free' from your Policy router:
/proc/sys/net/ipv4/route/gc_thresh: 786432
total used free shared buffers cached
Mem: 1034088 1007112 26976 0 310840 217220
-/+ buffers/cache: 479052 555036
Swap: 1028120 0 1028120
4.3.1 Adjust one kernel parameter and reboot.
You will need to bump up the maximum number of supported route hash
entries the kernel supports. I recommend setting this rather high and
using another parameter to set your ceiling. Add the following to your
boot config kernel parameters:
rhash_entries=2400000
4.3.2 Adjust /proc/sys/net/... to tune your routing
The Linux kernel uses six parameters to adjust how it handles managing
the routing hashes, collisions, and aging.
gc_elesticity can best be described as the average bucket depth the
kernel will accept before it starts expiring route hash entries. This
will help maintain the upper limit of active routes.
echo 8 > /proc/sys/net/ipv4/route/gc_elasticity
I had limited success playing with these next two entries seeing as I
could find little information on the effect of either one.
echo 60 > /proc/sys/net/ipv4/route/gc_interval
echo 0 > /proc/sys/net/ipv4/route/gc_min_interval
gc_thresh is another limiting factor in controlling how much RAM your
policy routing will eat up. This number cannot be greater than the
rhash_entries kernel parameter. As a rule of thumb, set your
rhash_entries parameter REALLY high(mine is 2.4million) and control your
running limit with gc_thresh.
echo 1048576 > /proc/sys/net/ipv4/route/gc_thresh
This parameter needs better kernel docs.
echo 300 > /proc/sys/net/ipv4/route/gc_timeout
The secret_interval instructs the kernel how often to blow away ALL route
hash entries regardless of how new/old they are. In our environment this
is generally bad. The CPU will be busy rebuilding thousands of entries
per second every time the cache is cleared. However we set this to run
once a day to keep memory leaks at bay(though we've never had one).
echo 86400 > /proc/sys/net/ipv4/route/secret_interval
4.3.3 Basic scripts
These aren't perfect but then again, what is....
4.3.3.1 Watcherrors
#!/bin/tcsh
set interval=15
set argc=`echo $argv | wc -w | tr -s " " "\t" | cut -f2`
if ( $argc > 0 ) then
set interval=$argv[1]
endif
set stats=`ifconfig eth3 | egrep 'RX packets:' | tr -s ": " "\t" | cut -f4,6`
set interrupts=`cat /proc/interrupts | egrep "eth[23]" | tr -s " " "\t" | cut
-f 3,4`
while ( 1 )
sleep $interval
set newstats=`ifconfig eth3 | egrep 'RX packets:' | tr -s ": " "\t" |
cut -f4,6`
set packets=`expr $newstats[1] - $stats[1]`
set errors=`expr $newstats[2] - $stats[2]`
set percentage=`expr $errors "*" 10000 / $packets`
set packetspersec=`expr $packets / $interval `
set date=`date "+%m/%d/%y %H:%M:%S"`
set entries=`cat /proc/net/stat/rt_cache | tr -s " " "\t" | cut -f 1 |
head -n 2 | tail -n 1`
set newinterrupts=`cat /proc/interrupts | egrep "eth[23]" | tr -s " "
"\t" | cut -f 3,4`
set eth3int=`expr $newinterrupts[1] - $interrupts[1]`
set eth2int=`expr $newinterrupts[4] - $interrupts[4]`
echo "$date entries: $entries Pkts: $packets Err: $errors PPS:
$packetspersec Drop %: 0.$percentage% Eth3RXInt: $eth3int Eth2TXInt:
$eth2int"
set stats=( $newstats )
set interrupts=( $newinterrupts )
end
4.3.3.2 Policy Routing Control Script
This script assumes that you will have two files, a policy route file,
and policy rule file.
The ROUTE file should have the following format:
[ip of next hop] [outgoing interface] [table #]
Example 'routefile' contents:
10.0.1.10 eth2 31
10.0.1.11 eth2 32
The RULE file should have the following format:
[CIDR BLOCK] [table #]
Example 'rulefile' contents:
172.30.0.0/24 31
172.16.0.0/22 32
Data to/from 172.30.0.0/24 would be sent to sensor 10.0.1.10. Data
to/from 172.16.0.0/22 would be sent to sensor 10.0.1.11.
#POLICY V1.0 script
#!/bin/tcsh
set policyrulefile=/opt/bin/policyrules
set policyroutefile=/opt/bin/policyroutes
set argc=`echo $argv | wc -w | tr -s " " "\t" | cut -f2`
if ( $argc < 3 ) then
echo Policy V1.00
echo "Usage: policy [add|delete|show] [routefile] [rulefile]"
exit
endif
set command=`echo $1 | tr "[a-z]" "[A-Z"`
if ( ( $command == "ADD" ) || ( $command == "DELETE" ) ) then
if ( ! -e $argv[2] ) then
echo Route file $argv[2] does not exist
exit
endif
if ( ! -e $argv[3] ) then
echo Rule file $argv[3] does not exist
exit
endif
endif
set policyroutefile=$argv[2]
set policyrulefile=$argv[3]
set policyrulecount=`egrep "[0-9]\/[0-9]" $policyrulefile | wc -l | tr -s " "
"\t" | cut -f2`
set policyrules=`egrep "[0-9]\/[0-9]" $policyrulefile | tr -s " " "\t"`
set policyroutecount=`egrep "[0-9] eth" $policyroutefile | wc -w | tr -s
" " "\t" | cut -f2`
set policyroutes=`egrep "[0-9] eth" $policyroutefile | tr -s " " "\t"`
if ( $command == "ADD" ) then
echo -n "Turning up..."
set alternate=0
foreach policyrule ($policyrules )
if ( $alternate ) then
set alternate=0
set table=$policyrule
# echo "Adding rule: $range $table"
echo -n "."
/sbin/ip rule add type unicast dev eth3 from $range
table $table
/sbin/ip rule add type unicast dev eth3 to $range
table $table
else
set alternate=1
set range=$policyrule
endif
end
set loop=0
while ( $loop != $policyroutecount )
set loop=`expr $loop + 1`
set gw=$policyroutes[$loop]
set loop=`expr $loop + 1`
set device=$policyroutes[$loop]
set loop=`expr $loop + 1`
set table=$policyroutes[$loop]
#echo "Adding default route: $gw $device $table"
echo -n "+"
/sbin/ip route add default via $gw dev $device table $table
end
/sbin/ip route flush cache
echo 1 > /proc/sys/net/ipv4/ip_forward
echo ""
endif
if ( $command == "DELETE" ) then
echo -n "Turning down..."
echo 0 > /proc/sys/net/ipv4/ip_forward
set alternate=0
foreach policyrule ($policyrules )
if ( $alternate ) then
set alternate=0
set table=$policyrule
#echo "Deleting rule: $range $table"
echo -n "."
/sbin/ip rule delete type unicast dev eth3 from $range
table $table
/sbin/ip rule delete type unicast dev eth3 to $range
table $table
else
set alternate=1
set range=$policyrule
endif
end
set loop=0
while ( $loop != $policyroutecount )
set loop=`expr $loop + 1`
set gw=$policyroutes[$loop]
set loop=`expr $loop + 1`
set device=$policyroutes[$loop]
set loop=`expr $loop + 1`
set table=$policyroutes[$loop]
#echo "Deleting default route: $gw $device $table"
echo -n "+"
/sbin/ip route delete default via $gw dev $device table $table
end
/sbin/ip route flush cache
echo ""
endif
if ( $command == "SHOW" ) then
set tables=`cat $policyroutefile | egrep "[0-9]" | tr -s " " "\t" |
cut -f3 | sort -u`
ip rule list
foreach table ($tables)
ip route list table $table
end
endif
-----------------------------------------------------------------------------
4.3.4 A Quick Word on SNMP
So you want to watch your system's interface usage with SNMP 'eh? Let me
guess though, you're having a problem? You never see traffic go over
120mbps? That's beccause you're polling w/ Integer32s which roll over at
120mbps after five minutes. You need to use SNMP V2c AND at least Net-SNMP
v5.2.1 for 64 bit counter support. Once you've installed the latest
net-snmp, you should be able to:
snmpwalk -v2c -On -c COMMUNITY localhost ifHCInOctets
This SHOULD return data IF you have things setup correctly!
-----------------------------------------------------------------------------
5. Copyright
Copyright © 2005 Jeremy M. Guthrie
Permission is granted to copy, distribute and/or modify this document under
the terms of the GNU Free Documentation License, Version 1.1 or any later
version published by the Free Software Foundation; with no Invariant
Sections, no Front-Cover Texts and no Back-Cover Texts. A copy of the license
is included in the section entitled "GNU Free Documentation License".
-----------------------------------------------------------------------------
5.1. GNU Free Documentation License
Version 1.1, March 2000
Copyright (C) 2000 Free Software Foundation, Inc. 59 Temple Place, Suite
330, Boston, MA 02111-1307 USA Everyone is permitted to copy and
distribute verbatim copies of this license document, but changing it is
not allowed.
-----------------------------------------------------------------------------
5.2. PREAMBLE
The purpose of this License is to make a manual, textbook, or other written
document "free" in the sense of freedom: to assure everyone the effective
freedom to copy and redistribute it, with or without modifying it, either
commercially or noncommercially. Secondarily, this License preserves for the
author and publisher a way to get credit for their work, while not being
considered responsible for modifications made by others.
This License is a kind of "copyleft", which means that derivative works of
the document must themselves be free in the same sense. It complements the
GNU General Public License, which is a copyleft license designed for free
software.
We have designed this License in order to use it for manuals for free
software, because free software needs free documentation: a free program
should come with manuals providing the same freedoms that the software does.
But this License is not limited to software manuals; it can be used for any
textual work, regardless of subject matter or whether it is published as a
printed book. We recommend this License principally for works whose purpose
is instruction or reference.
-----------------------------------------------------------------------------
5.3. APPLICABILITY AND DEFINITIONS
This License applies to any manual or other work that contains a notice
placed by the copyright holder saying it can be distributed under the terms
of this License. The "Document", below, refers to any such manual or work.
Any member of the public is a licensee, and is addressed as "you".
A "Modified Version" of the Document means any work containing the Document
or a portion of it, either copied verbatim, or with modifications and/or
translated into another language.
A "Secondary Section" is a named appendix or a front-matter section of the
Document that deals exclusively with the relationship of the publishers or
authors of the Document to the Document's overall subject (or to related
matters) and contains nothing that could fall directly within that overall
subject. (For example, if the Document is in part a textbook of mathematics,
a Secondary Section may not explain any mathematics.) The relationship could
be a matter of historical connection with the subject or with related
matters, or of legal, commercial, philosophical, ethical or political
position regarding them.
The "Invariant Sections" are certain Secondary Sections whose titles are
designated, as being those of Invariant Sections, in the notice that says
that the Document is released under this License.
The "Cover Texts" are certain short passages of text that are listed, as
Front-Cover Texts or Back-Cover Texts, in the notice that says that the
Document is released under this License.
A "Transparent" copy of the Document means a machine-readable copy,
represented in a format whose specification is available to the general
public, whose contents can be viewed and edited directly and
straightforwardly with generic text editors or (for images composed of
pixels) generic paint programs or (for drawings) some widely available
drawing editor, and that is suitable for input to text formatters or for
automatic translation to a variety of formats suitable for input to text
formatters. A copy made in an otherwise Transparent file format whose markup
has been designed to thwart or discourage subsequent modification by readers
is not Transparent. A copy that is not "Transparent" is called "Opaque".
Examples of suitable formats for Transparent copies include plain ASCII
without markup, Texinfo input format, LaTeX input format, SGML or XML using a
publicly available DTD, and standard-conforming simple HTML designed for
human modification. Opaque formats include PostScript, PDF, proprietary
formats that can be read and edited only by proprietary word processors, SGML
or XML for which the DTD and/or processing tools are not generally available,
and the machine-generated HTML produced by some word processors for output
purposes only.
The "Title Page" means, for a printed book, the title page itself, plus such
following pages as are needed to hold, legibly, the material this License
requires to appear in the title page. For works in formats which do not have
any title page as such, "Title Page" means the text near the most prominent
appearance of the work's title, preceding the beginning of the body of the
text.
-----------------------------------------------------------------------------
5.4. VERBATIM COPYING
You may copy and distribute the Document in any medium, either commercially
or noncommercially, provided that this License, the copyright notices, and
the license notice saying this License applies to the Document are reproduced
in all copies, and that you add no other conditions whatsoever to those of
this License. You may not use technical measures to obstruct or control the
reading or further copying of the copies you make or distribute. However, you
may accept compensation in exchange for copies. If you distribute a large
enough number of copies you must also follow the conditions in section 3.
You may also lend copies, under the same conditions stated above, and you may
publicly display copies.
-----------------------------------------------------------------------------
5.5. COPYING IN QUANTITY
If you publish printed copies of the Document numbering more than 100, and
the Document's license notice requires Cover Texts, you must enclose the
copies in covers that carry, clearly and legibly, all these Cover Texts:
Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover.
Both covers must also clearly and legibly identify you as the publisher of
these copies. The front cover must present the full title with all words of
the title equally prominent and visible. You may add other material on the
covers in addition. Copying with changes limited to the covers, as long as
they preserve the title of the Document and satisfy these conditions, can be
treated as verbatim copying in other respects.
If the required texts for either cover are too voluminous to fit legibly, you
should put the first ones listed (as many as fit reasonably) on the actual
cover, and continue the rest onto adjacent pages.
If you publish or distribute Opaque copies of the Document numbering more
than 100, you must either include a machine-readable Transparent copy along
with each Opaque copy, or state in or with each Opaque copy a
publicly-accessible computer-network location containing a complete
Transparent copy of the Document, free of added material, which the general
network-using public has access to download anonymously at no charge using
public-standard network protocols. If you use the latter option, you must
take reasonably prudent steps, when you begin distribution of Opaque copies
in quantity, to ensure that this Transparent copy will remain thus accessible
at the stated location until at least one year after the last time you
distribute an Opaque copy (directly or through your agents or retailers) of
that edition to the public.
It is requested, but not required, that you contact the authors of the
Document well before redistributing any large number of copies, to give them
a chance to provide you with an updated version of the Document.
-----------------------------------------------------------------------------
5.6. MODIFICATIONS
You may copy and distribute a Modified Version of the Document under the
conditions of sections 2 and 3 above, provided that you release the Modified
Version under precisely this License, with the Modified Version filling the
role of the Document, thus licensing distribution and modification of the
Modified Version to whoever possesses a copy of it. In addition, you must do
these things in the Modified Version:
A. Use in the Title Page (and on the covers, if any) a title distinct from
that of the Document, and from those of previous versions (which should,
if there were any, be listed in the History section of the Document). You
may use the same title as a previous version if the original publisher of
that version gives permission.
B. List on the Title Page, as authors, one or more persons or entities
responsible for authorship of the modifications in the Modified Version,
together with at least five of the principal authors of the Document (all
of its principal authors, if it has less than five).
C. State on the Title page the name of the publisher of the Modified
Version, as the publisher.
D. Preserve all the copyright notices of the Document.
E. Add an appropriate copyright notice for your modifications adjacent to
the other copyright notices.
F. Include, immediately after the copyright notices, a license notice giving
the public permission to use the Modified Version under the terms of this
License, in the form shown in the Addendum below.
G. Preserve in that license notice the full lists of Invariant Sections and
required Cover Texts given in the Document's license notice.
H. Include an unaltered copy of this License.
I. Preserve the section entitled "History", and its title, and add to it an
item stating at least the title, year, new authors, and publisher of the
Modified Version as given on the Title Page. If there is no section
entitled "History" in the Document, create one stating the title, year,
authors, and publisher of the Document as given on its Title Page, then
add an item describing the Modified Version as stated in the previous
sentence.
J. Preserve the network location, if any, given in the Document for public
access to a Transparent copy of the Document, and likewise the network
locations given in the Document for previous versions it was based on.
These may be placed in the "History" section. You may omit a network
location for a work that was published at least four years before the
Document itself, or if the original publisher of the version it refers to
gives permission.
K. In any section entitled "Acknowledgements" or "Dedications", preserve the
section's title, and preserve in the section all the substance and tone
of each of the contributor acknowledgements and/or dedications given
therein.
L. Preserve all the Invariant Sections of the Document, unaltered in their
text and in their titles. Section numbers or the equivalent are not
considered part of the section titles.
M. Delete any section entitled "Endorsements". Such a section may not be
included in the Modified Version.
N. Do not retitle any existing section as "Endorsements" or to conflict in
title with any Invariant Section.
If the Modified Version includes new front-matter sections or appendices that
qualify as Secondary Sections and contain no material copied from the
Document, you may at your option designate some or all of these sections as
invariant. To do this, add their titles to the list of Invariant Sections in
the Modified Version's license notice. These titles must be distinct from any
other section titles.
You may add a section entitled "Endorsements", provided it contains nothing
but endorsements of your Modified Version by various parties--for example,
statements of peer review or that the text has been approved by an
organization as the authoritative definition of a standard.
You may add a passage of up to five words as a Front-Cover Text, and a
passage of up to 25 words as a Back-Cover Text, to the end of the list of
Cover Texts in the Modified Version. Only one passage of Front-Cover Text and
one of Back-Cover Text may be added by (or through arrangements made by) any
one entity. If the Document already includes a cover text for the same cover,
previously added by you or by arrangement made by the same entity you are
acting on behalf of, you may not add another; but you may replace the old
one, on explicit permission from the previous publisher that added the old
one.
The author(s) and publisher(s) of the Document do not by this License give
permission to use their names for publicity for or to assert or imply
endorsement of any Modified Version.
-----------------------------------------------------------------------------
5.7. COMBINING DOCUMENTS
You may combine the Document with other documents released under this
License, under the terms defined in section 4 above for modified versions,
provided that you include in the combination all of the Invariant Sections of
all of the original documents, unmodified, and list them all as Invariant
Sections of your combined work in its license notice.
The combined work need only contain one copy of this License, and multiple
identical Invariant Sections may be replaced with a single copy. If there are
multiple Invariant Sections with the same name but different contents, make
the title of each such section unique by adding at the end of it, in
parentheses, the name of the original author or publisher of that section if
known, or else a unique number. Make the same adjustment to the section
titles in the list of Invariant Sections in the license notice of the
combined work.
In the combination, you must combine any sections entitled "History" in the
various original documents, forming one section entitled "History"; likewise
combine any sections entitled "Acknowledgements", and any sections entitled
"Dedications". You must delete all sections entitled "Endorsements."
-----------------------------------------------------------------------------
5.8. COLLECTIONS OF DOCUMENTS
You may make a collection consisting of the Document and other documents
released under this License, and replace the individual copies of this
License in the various documents with a single copy that is included in the
collection, provided that you follow the rules of this License for verbatim
copying of each of the documents in all other respects.
You may extract a single document from such a collection, and distribute it
individually under this License, provided you insert a copy of this License
into the extracted document, and follow this License in all other respects
regarding verbatim copying of that document.
-----------------------------------------------------------------------------
5.9. AGGREGATION WITH INDEPENDENT WORKS
A compilation of the Document or its derivatives with other separate and
independent documents or works, in or on a volume of a storage or
distribution medium, does not as a whole count as a Modified Version of the
Document, provided no compilation copyright is claimed for the compilation.
Such a compilation is called an "aggregate", and this License does not apply
to the other self-contained works thus compiled with the Document, on account
of their being thus compiled, if they are not themselves derivative works of
the Document.
If the Cover Text requirement of section 3 is applicable to these copies of
the Document, then if the Document is less than one quarter of the entire
aggregate, the Document's Cover Texts may be placed on covers that surround
only the Document within the aggregate. Otherwise they must appear on covers
around the whole aggregate.
-----------------------------------------------------------------------------
5.10. TRANSLATION
Translation is considered a kind of modification, so you may distribute
translations of the Document under the terms of section 4. Replacing
Invariant Sections with translations requires special permission from their
copyright holders, but you may include translations of some or all Invariant
Sections in addition to the original versions of these Invariant Sections.
You may include a translation of this License provided that you also include
the original English version of this License. In case of a disagreement
between the translation and the original English version of this License, the
original English version will prevail.
-----------------------------------------------------------------------------
5.11. TERMINATION
You may not copy, modify, sublicense, or distribute the Document except as
expressly provided for under this License. Any other attempt to copy, modify,
sublicense or distribute the Document is void, and will automatically
terminate your rights under this License. However, parties who have received
copies, or rights, from you under this License will not have their licenses
terminated so long as such parties remain in full compliance.
-----------------------------------------------------------------------------
5.12. FUTURE REVISIONS OF THIS LICENSE
The Free Software Foundation may publish new, revised versions of the GNU
Free Documentation License from time to time. Such new versions will be
similar in spirit to the present version, but may differ in detail to address
new problems or concerns. See [http://www.gnu.org/copyleft/] http://
www.gnu.org/copyleft/.
Each version of the License is given a distinguishing version number. If the
Document specifies that a particular numbered version of this License "or any
later version" applies to it, you have the option of following the terms and
conditions either of that specified version or of any later version that has
been published (not as a draft) by the Free Software Foundation. If the
Document does not specify a version number of this License, you may choose
any version ever published (not as a draft) by the Free Software Foundation.
-----------------------------------------------------------------------------
5.13. How to use this License for your documents
To use this License in a document you have written, include a copy of the
License in the document and put the following copyright and license notices
just after the title page:
Copyright (c) YEAR YOUR NAME. Permission is granted to copy, distribute
and/or modify this document under the terms of the GNU Free Documentation
License, Version 1.1 or any later version published by the Free Software
Foundation; with the Invariant Sections being LIST THEIR TITLES, with the
Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST. A
copy of the license is included in the section entitled "GNU Free
Documentation License".
If you have no Invariant Sections, write "with no Invariant Sections" instead
of saying which ones are invariant. If you have no Front-Cover Texts, write
"no Front-Cover Texts" instead of "Front-Cover Texts being LIST"; likewise
for Back-Cover Texts.
If your document contains nontrivial examples of program code, we recommend
releasing these examples in parallel under your choice of free software
license, such as the GNU General Public License, to permit their use in free
software.
-------------------------------------------------
--
--------------------------------------------------
Jeremy M. Guthrie jeremy.guthrie () berbee com
Senior Network Engineer Phone: 608-298-1061
Berbee Fax: 608-288-3007
5520 Research Park Drive NOC: 608-298-1102
Madison, WI 53711
Attachment:
_bin
Description:
Current thread:
- ids-load-balancing-HOWTO Jeremy M. Guthrie (Jul 27)
- <Possible follow-ups>
- Re: ids-load-balancing-HOWTO Richard Bejtlich (Jul 28)