Bind8:
    nsupdate <--- very important this makes dynamic updates possible

This software won't work with the nsupdate from BIND9 due to the fact that they no longer read updates from file, I will patch the software asap to use DNS::Updates a dynamic DNS update module for Perl. In the meantime you can use BIND9 with the nsupdate from BIND8

Installing failover:
Instaling failover is as simple as copying the failover-server and failover-monitor to /usr/local/failover or any other directory you wish and copying failover-monitor.config and failover.config to /etc this location is mandatory.

Configuring failover:
The configuration of failover-server is done by editting /etc/failover.config and the configuration of the failover-monitor is done by editting the failover-monitor.config.

The failover-server should run on a DNS server and on every clusternode. The failover server listens for service restarts and DNS updates send by the failover-monitor software.

The failover-monitor tests the clusternodes for their service health. If a service fails then the failover-monitor will send a request to the failover-server that runs on the DNS server to remove the associated record. Then a restart of that particular service is invoked if the service starts to reply again, then failover-monitor will send a DNS add request to the failover-server that runs on the DNS server to reinsert the associated A record. When the service doesn't react to the restart then the record stays out untill the problem is fixed.

Configuring the failover-monitor:
The failover-monitor is the active system that checks for the health of clusternodes. It does so by sending requests to the servers ip address and waiting for the right respons from the server. The failover-monitor should run on 1 or more dedicated systems (486 is more then enough) or on the DNS server.
This service is configured by editting the /etc/failover-monitor.config file.
Here is a listing of such a file:

###-------This Section is for the failover-server-------------------------
###The server listens default on port 2223. The server can be told by the
###failover-monitor wich service it needs to restart

##Server password to remotetly restart services
password = mypasswd

##Server port on wich remote server is running
clusternode_server_port = 2224

###--------This Section is for the failover-monitor service-------------------
###The failover-monitor tests the if both interfaces respond to pings if not
###it wil take over the ip address of the server wich it tests
###It also probes the important services and asks for a port forward if one fails

##Address to whom all messages should be send
email = root@phonax.com

##Add the 2 entries of the bind8 DNS servers wich take care of the failover
##The failover-server should be running on these DNS SERVERS!!!
bind8_dns_server1 = 192.168.0.1
bind8_dns_server2 = 192.168.0.2

##Servers + Services to test
##check-serverx = ip_address|hostname|service_name|start_stop_script|request_to_send|response_to_except
check_server1 = 192.168.0.1|www.mydomain.com|http|apache|GET /robots.txt HTTP/1.0|httpd is working
check_server2 = 192.168.0.2|www.mydomain.com|http|apache|GET /robots.txt HTTP/1.0|httpd is working
check_server7 = 192.168.0.3|www.mydomain.com|http|fasttrack|GET /robots.txt HTTP/1.0|httpd is working
check_server3 = 192.168.0.1|smtp.mydomain.com|smtp|postfix|QUIT|220
check_server4 = 192.168.0.2|smtp.mydomain.com|smtp|sendmail|QUIT|220
check_server5 = 192.168.0.4|pop.mydomain.com|pop3|inetd|QUIT|OK
check_server6 = 192.168.0.5|pop.mydomain.com|pop3|inetd|QUIT|OK

Explenation of failover-monitor.config
password = the password which the the failover-server expects in order to start an action.
clusternode_serverport = the portnumber on wich ALL the failover-server's listen.
email = is the address to whom the messages will be sent for failing servers and restarts.
bind8_dns_server1 = the primairy DNS server it's IP address on wich a failover-server MUST run to except DNS update requests.
bind8_dns_server2 = the (optional) secondary DNS server it's IP address on wich a failover-server MUST run to except DNS update requests.

check_serverx is a more complex line, it infact is a "script" to check several TCP services.
It is devided in 5 sections seperated by a pipeline sign.

The first section is the ip address to test the service (we connect via ip to test the services not via hostname).
The second section is the Round Robin hostname for that service. When a test fails this hostname and the ipaddress pair will be removed from the DNS.
The third section is the service name as it is stored in the services file, we determine portnumber by checking the services file.
The fourth section is the start stop script without path wich will be used to try and restart a service when it fails.
The fith section is the request without \r\n that the failover-monitor sends to the server.
The sixth section is the reply whe expect (this may be a small fragment)

In the failover-monitor.config above we can see that there is a 2 nodecluster running: http, smtp and pop3
on both nodes and a third node that does http with a fasttrack server. You can also see that you can a different ammount of servers for each service.
The check_server1 checks a webserver on 192.168.0.1 with the name www.mydomain.com when it fails the script:
/etc/init.d/apache stop /etc/init.d/apache start is runned and a check of a life sign is done again. If it still fails
we consider it to be dead and we send a request to the bind8_dns_server1 and bind8_dns_server2 to delete the
record: www.mydomain.com A 192.168.0.1

After a service on a node has failed it will still be tested to see if it came backonline. If it does, then the recorded is put back in both DNS servers.

The third and fourth entries test whether the smtp servers on 192.168.0.1 and 192.168.0.2 are still running, a request QUIT will be send and we expect an 220 back from the server. As we can see here, the services that run don't have to be the same software or version. This provides you with more security against Exploits and DoS attacks. On 192.168.0.1 postfix is running and on 192.168.0.2 sendmail is running. This could be a smart thing to do, so that you don't both systems won't fall for the same exploit. (infact you should consider to bann sendmail anyways :) )

The fifth and sixth test we do is a nasty one (one that might fail the test often). It is the pop3 service. Pop3 doesn't like to be called often and sometimes it goes in a 10 minute pause. This comes due to the fact that pop3 is started from the inetd if pop3 fails we restart the inetd. It is often NOT necessary to check services invoked by inetd but you can do it.
The amount of test-rules are not limited.

Configuring the failover-server:
The failover-server is relativly simple to configure:
###-------This Section is for the failover-server-------------------------
###The server listens default on port 2223. The server can be told by the
###failover-monitor wich service it needs to restart
###THIS FILE SHOULD RESIDE IN /etc !!!

##Server password to remotetly restart services
password = mypasswd

##Server port on wich server will listen
local_server_port = 2224

##Address to whom all messages should be send
email = root@phonax.com

##Path where to find the system V startup scripts for the services
scripts_path = /etc/init.d

###--------This Section is for the failover-monitor service-------------------
###The failover-monitor tests the if both interfaces respond to pings if not
###it wil take over the ip address of the server wich it tests
###It also probes the important services and asks for a port forward if one fails

##Directory where to find the failover DNS nsupdate scripts
failover_directory = /usr/local/failover

password = the password is the password wich we will expect to receive from an failover-monitor
local_server_port = the port on wich the server will listen
email = the email address to wich messages can be mailed.
scripts_path = the path were the start stop scripts reside for the services that the failover-monitor test on BSD derived systems I advice you to make a directory /etc/init.d and in this directory you link all the start stop scripts.
failover_directory = the directory were we can find the normal-dns file (the normal-dns file is the file in wich the Round Robin A Records are put) after restarting a monitor, this normal-dns file will be read in to the DNS servers. I use this to create a standard situation with all the DNS records in them. A Dynamic DNS Update will explicitly delete a record.

DNS configuration for failover:
The DNS server is the most important part for the Loadsharing failover. It relies upon BIND 8.x and the nsupdate tool that comes with Bind8. The master principal for the failover is Round Robin DNS when a host fails a record is removed from the Round Robin Pool.
If you are completely new to DNS then please also read my section on DNS

You can suffice with 1 DNS server and the second one being your slave/secondary DNS server. However even with the new Notify specification in BIND 8 it still can take up to 30/60 seconds before a zone is transferred. This could cause out of date DNS records between the master and the slave server. In order to remove the update latency, I advice you to setup 2 BIND 8 servers that both are in practice primary DNS servers for the domains that you host, for the internet 1 is master and the other is secondary (depending on wich of the two servers you advertise as the primary DNS). Failover will send update queries to both DNS servers synchronously so the updates are done almost simultanously on both servers.
YOU STILL COULD CHOOSE FOR 1 MASTER SERVER AND 1 SECONDARY SERVER, SPECIALLY WHEN YOUR DNS SERVERS ARE NOT PART OF THE CLUSTERNODES.

The amount of changes that you have to make to the named.conf are very small.
You need to add the following entry in each zone file:
allow-update { localhost; };
Perhaps you need to change the SOA header to make the server that used to be a slave server be a primary again. (This is as simple as changing the SOA header to tell the server that it is the primary name server.

This is not ALWAYS necessarry with all BIND versions, I found out that my FreeBSD 3.3 system did not update his DNS because it checked wich DNS server is the official primary server, I told its zone file that he is the primary DNS server off the phonax.com domain asswel, then nsupdate on FreeBSD posted update queries to it self instead of to the other DNS server.

The zone files stay the same as you nomally have. In order to make a basic loadbalancing scheme for www is as easy as adding several A records for www like this:

www 1 IN A 192.168.0.1
          1 IN A 192.168.0.2

Loadbalancing the receiving receiving SMTP:
DNS provides the means of loadbalancing SMTP email receiving by adding several MX records with the same priority number, like this:

mydomain.com IN MX 10 clusternode1.mydomain.com.
                        IN MX 10 clusternode2.mydomain.com.

DNS example for named.conf for mydomain.com
options {
                directory "/var/named";
                forwarders {ip_dns1_isp; ip_dns2_isp;};
            };

zone "." {
                type hint;
                file "named.ca";
              };

zone "0.0.127.in-addr.arpa" {
                  type master;
                  file "named.local";
                };

zone "mydomain.com" {
                  type master;
                  file "db.mydomain.com";
                };

zone "0.168.192.in-addr.arpa" {
                  type master;
                  file "db.0.168.192";
                };

DNS example for the domain mydomain.com file name = db.mydomain.com the loadsharing is done for pop, smtp, www.

@ IN SOA ns1.mydomain.com. root.ns1.mydomain.com.(
                2000030101 ; Serial
                28800 ; refresh
                140 ; Retry
                3600000 ; Expire
                86400 ) ; Minimum

        IN NS ns1.mydomain.com.
        IN NS ns2.mydomain.com.

@     IN MX 10 clusternode1.mydomain.com.
        IN MX 10 clusternode2.mydomain.com.

; The individual hosts in the cluster:
clusternode1 IN A 192.168.0.1
clusternode2 IN A 192.168.0.2
clusternode3 IN A 192.168.0.3
clusternode4 IN A 192.168.0.4
clusternode5 IN A 192.168.0.5

; The 2 monitoring nodes monitor1 IN A 192.168.0.100
monitor2 IN A 192.168.0.101

;Hosts that are not part of a clustergroup
ns1 IN A 192.168.0.10
ns2 IN A 192.168.0.11

;The loadsharing done with Round Robin DNS
www 1 IN A 192.168.0.1
www 1 IN A 192.168.0.2
www 1 IN A 192.168.0.3

;Load sharing for sendning email
smtp 1 IN A 192.168.0.1
smtp 1 IN A 192.168.0.2

;Load sharing for pop3 serrver
pop 1 IN A 192.168.0.4
pop 1 IN A 192.168.0.5

DNS example reveressed lookup for the 192.168.0.x network file name = db.0.168.192
@ IN SOA ns1.mydomain.com. root.ns1.mydomain.com.(
                2000030101 ; Serial
                28800 ; refresh
                140 ; Retry
                3600000 ; Expire
                86400 ) ; Minimum

IN NS ns1.mydomain.com.
IN NS ns2.mydomain.com.

1 IN PTR clusternode1.mydomain.com.
2 IN PTR clusternode2.mydomain.com.
3 IN PTR clusternode3.mydomain.com.
4 IN PTR clusternode4.mydomain.com.
5 IN PTR clusternode5.mydomain.com.
10 IN PTR ns1.mydomain.com.
11 IN PTR ns2.mydomain.com.
100 IN PTR monitor1.mydomain.com.
101 IN PTR monitor2.mydoamin.com.

Now that the DNS has been setup for loadsharing, we must configure the normal-dns file in the failover directory, wich is used to reset the DNS into it's "normal" situation. This is done since a nsupdate querie is permanent not temporarily!!!!. You could also leave the normal-dns blank, I however like a reset situation that is known So the normal-dns file in failover directory (IS VERY IMPORTANT, SINCE WE USE IT AS A DEFAULT) you can live without it but it is hard to set your zone back to it's starting position after updates have taken place. (unless you have your own thorough documentation :) ):

You also need to add these records in the following matter in to the normal-dns file in the failover directory:
update add www.domainname.com 1 A 192.168.0.1
update add www.domainname.com 1 A 192.168.0.2
update add www.domainname.com 1 A 192.168.0.3
update add smtp.domainname.com 1 A 192.168.0.1
update add smtp.domainname.com 1 A 192.168.0.2
update add pop.domainname.com 1 A 192.168.0.4
update add pop.domainname.com 1 A 192.168.0.5


(The two blank lines are necessary since nsupdate needs 1 to process the requests).
As you can see only the records that are used in Loadsharing are added in the normal-dns, since only these records will be removed by a request from the failover-monitor to the failover-server running on a DNS.

In this example you can see that failover has the ability that other "failover" servers don't, that is to use many different servers (even cross-platform that all can run different versions of services and these services can be monitored explicitly.