Managing a Distributed Software Library Using LUDE

Charles Gagnon

Figure 1 | Figure 2

Few sites use only one flavor of UNIX. Even if they use the UNIX from a single vendor, they rarely have only one release installed at one time. Commercial software vendors usually issue new releases or patches every six or eight months, and freely distributed software and applications are often updated more frequently. These elements can make managing a distributed software library a challenge for system administrators.

I recently implemented a software repository using a combination of LUDE 2.1(*), a freely available tool from the University of Montreal, and the automounter, now available on most UNIX platforms. This solution eases management and maintainance of multiple versions of public domain software, as well as commercial applications. It also makes testing and upgrading to newer releases a more uniform and documentable process, and allows upgrades to be done by one system administrator or multiple administrators working in concert.

LUDE is a public domain application allowing the compilation and the installation of all software packages. It organizes the packages such that each software package is kept in a separate subdirectory, but local copies of the binaries and man pages are available without editing the $PATH and $MANPATH environment variables. In addition, more than one release of a package can co-exist during transitions.

The LUDE/automounter solution also makes all the applications on the server available from one NFS-mounted /usr/local directory, solving the problem of managing long $PATH variables on countless machines.

Preparing The File Server

To build your repository, you will need one server to hold all the applications locally. This NFS server will also share the files for the rest of the community. The needed disk space naturally depends on how many applications will be installed. I strongly recommend using a volume manager so that file systems can be resized as necessary. A previous implementation was done using an Ultra2 from Sun Microsystems running Solaris 2.6, augmented by a RAID5 array from Box Hill. This server easily could have been a PowerPC running AIX or a Pentium-based server running Linux. For our example, let's call the file server "repository".

First, create the directories needed under /export. (I like the /export structure because it seems logical to me, but it could be anywhere):

/export/local/SunOS5.5.1/
/export/local/SunOS5.6/
/export/local/AIX4.2/
/export/local/AIX3.2/
/export/local/Linux2.0.31/
/export/soft/

I created one directory per architecture/OS release in my environment. The names are usually the concatenation of uname -s and uname -r making it easy to obtain with a script. To mount the directories, I use the automounter and my NIS automount maps. For example, on your NIS server, make sure the auto.master map contains:

/- auto.direct

Add the following lines to your auto.direct map:

/usr/local   -rw,soft   repository:/export/local/${OSNAME}${OSREL}/
/usr/soft    -rw,soft   repository:/export/soft

These two directories will be know as the "depot" (/usr/soft) and the "public" (/usr/local) directories. I'll explain their exact role when I discuss LUDE.

Some UNIX flavors, like Solaris, support the ${OSNAME} and ${OSREL} variables "out of the box". For other implementations of UNIX, such as AIX or SunOS 4.1.x, you will need to restart the automounter with -D arguments:

/usr/sbin/automountd -D OSNAME=AIX -D OSREL=4.2

Since Linux doesn't support the direct (/-) mounts with the automounter, I mounted all of them under /apps using this extra line in the auto.master:

/apps auto.linux

The auto.linux map should look exactly like the auto.direct map for the other machines:

local   -rw,soft   repository:/export/local/${OSNAME}${OSREL}/
soft    -rw,soft   repository:/export/soft

The directories /apps/local and /apps/soft will later need to be manually linked to /usr/local and /usr/soft, respectively. It's a dirty workaround but it works. Once this setup is complete, you should be able to see the following mounts on a client or the repository machine:

sun_client# mount
repository:/export/local/SunOS5.5.1/ on /usr/local
repository:/export/soft on /usr/soft
aix_client# mount
repository:/export/local/AIX4.2/ on /usr/local
repository:/export/soft on /usr/soft
linux_client# mount
repository:/export/local/Linux2.0.31/ on /apps/local
repository:/export/soft on /apps/soft

Downloading and Installing LUDE

LUDE is built around Perl5. You will therefore need to download and install the latest copy of Perl5 if it isn't already on your system. You can download the latest Perl sources from ftp.perl.org, by downloading the latest .tar.gz file located in the /pub/perl/CPAN/src/ directory. A multitude of other sites also offer sources and pre-compiled binary versions of Perl5. Installing and configuring Perl5 is beyond the scope of this article, but I recommend you follow the Perl literature for that task.

You can retrieve the latest copy of LUDE form the University of Montreal ftp server (ftp.iro.umontreal.ca/pub/lude/). The file is called lude.tar.gz, and the latest release, when this article was written, was 2.1.22. Perl and LUDE are the only files you should have to download to build your software library.

Then, extract the LUDE archive in a temporary location on your file server:

$ zcat lude.tar.gz | tar -xvf      or
$ tar -zxvf lude.tar.gz

Once the sources are extracted, run the "configure" script for LUDE. The script does several things, including checking for the presence of specific files, trying to bootstrap Perl5 if necessary, localizing your LUDE installation for your specific site, and creating makefiles. This script will also ask several questions. To make your task easier, you should prepare your answers beforehand. Note that the script will provide default answers.

• What are the other classes that the current machine can be identified with? (list of classes separated by blanks)

  The classes in LUDE define the different architectures LUDE will have to work with. In our example environment, we will create different classes. The class name should match the names of your public directories in /export. It is not a requirement, but it makes everything clearer. I don't usually use LUDE's default class names, because I find them too long. The classes in our example will be:

  SunOS5.6
  SunOS5.5.1
  Linux2.0.31
  AIX4.2
  AIX3.2
  

  Next, the "configure" script will ask for the various paths needed for the installation:

• What is the default public tree you want to use for your LUDE installation? [/usr/local]

  The public tree will be the shared directory containing all the packages. Each "class" will have its own copy of the public tree. It's important here to answer with /usr/local and not /export/SunOS5.6, because the LUDE commands and functions will be executed from any client on our network. /usr/local should be the correct directory (/export/${OSNAME}${OSREL}) on all clients.

• What is the default server root directory you want to use for your LUDE installation? [/usr/local/server]

  I don't currently use the server feature of LUDE, so I would leave this answer as the default.

• What is the default depot you want to use for your LUDE installation? [/usr/soft]

  The depot will contain all the LUDE packages. I will explain later the role of the depot. I usually choose /usr/soft as my depot. Again, answer /usr/soft and not /export/soft for the same reasons as before.

• In which depot should this release of LUDE be installed? [/usr/soft]

This indicates where to install the LUDE files. It should be the same as the previous question.

At the end of the configuration, the script will ask whether make install should be run for you. Since we want to do some modifications, I recommend answering "no" at this time. The second to last step will be to personalize some of LUDE files. I normally modify the file "ludeguess" located in src/bin of the LUDE built tree. This script is used to guess the class for a machine. It uses a combination of uname commands to determine the class. I normally modify this file to return the proper class names. For SunOS machines, I replace the line [88]:

echo sparc_sun_solaris2`echo ${UNAME_RELEASE}|sed -e 's/[^.]*//'`

with:

echo "`uname -s``uname -r`"

This echo statement should work for most platforms, but you can pretty much decide what the class name will look like for each platform and release of the OS. Finally, run make install to finish your LUDE installation. This last step will create the required directories and install all the required LUDE files.

How LUDE Works

Here's a short overview of the way LUDE handles things and how it can make life easier. In a LUDE environment, the files will never be installed in /usr/local directly. All your files should reside in the /usr/soft structure. You will also have one subdirectory per package, and all of those will be created automatically:

/usr/soft/procmail-3.11
/usr/soft/procmail-3.09
/usr/soft/maker-5.5

Each package directory will contain a compiling space (src) per class (plus one for the originals) and a binary space (run) per class. Figure 1 illustrates this directory structure for a "typical" package in LUDE.

LUDE provides a space to compile and install software packages for a multitude of platforms. It also makes these packages easier to manage, because everything related to one application resides in one directory.

The command format for LUDE is:

lude --{some_function} --package {package_name}

Let's take a look at the major LUDE commands:

create - The lude --create is the option used to create a new package when installing new software. One command has to be issued for each architecture for which the package will be installed. For example:

lude --create --package procmail-3.11 --class SunOS5.6
lude --create --package procmail-3.11 --class AIX4.2

This function creates the directory structure in which the package will live.

duplicate - Once the original sources are in place, the duplicate option will make a copy of the source for a certain class.

unduplicate - Undo the package duplication. LUDE removes all files that were not changed during the compilation process.

release - Once a package is ready and compiled, all the locks are removed using the release option. File locking is important in LUDE, because people could work on the same package from different machines.

install - The most important function of LUDE. Once the package is compiled and installed, the administrator will be able to create links from everything under the "run" directory of LUDE directly to the public directory [/usr/local] on you repository.

uninstall - Remove the links in the public directory.

remove - The remove option will remove all trace of the package both in the public directory [/usr/local] and in the software depot [/usr/soft].

The RCS Example

Let's look at an example of using the LUDE environment to manage the installation of RCS. I'll explain the installation process of RCS within the LUDE environment step-by-step. For the example, I will assume LUDE was installed properly, and the automounter setup explained earlier was completed.

Building RCS for SunOS5.6 and AIX4.2

First, we need to access a machine that runs SunOS5.6 and mounts the LUDE directories (/usr/local and /usr/soft). Note that the only step where you need a machine of the right architecture is for the compilation, otherwise, any machine with NFS write access to /usr/local and /usr/soft will do. The first step is to create the package directories:

$ cd /usr/soft
$ lude --create --pack rcs-5.7

Since I'm on a SunOS5.6 machine, the guessed class is already SunOS5.6. But if wanted to create the same package for AIX, I would need to explicitly declare the class:

$ lude --create --pack rcs-5.7 --class AIX4.2
$ ls -l
[...]

drwxr-xr-x   5 charlesg css           512 Sep 23  1997 rcs-5.7/
[...]

That rcs-5.7 directory contains the src and the run subdirectories needed to compile and install the RCS binaries and man pages. Now let's extract the sources. The {package-name}/src/orig is there to extract the original sources, as downloaded from the Net:

$ cd /usr/soft/rcs-5.7/src/orig
$ tar -zxf /tmp/rcs-5.7.tar.gz

All the sources were extracted in the rcs-5.7/src/orig directory. So, we'll make one copy for the SunOS compile and one copy for the AIX compile. We use the duplicate option of LUDE to achieve that:

$ lude --duplicate --pack rcs-5.7 [--class SunOS5.6]
$ lude --duplicate --pack rcs-5.7 --class AIX4.2

Now we have two copies of the sources, the SunOS version is in rcs-5.7/src/default/SunOS5.6, and the AIX version in rcs-5.7/src/default/AIX4.2. Let's configure and compile the SunOS version first.

$ cd /usr/soft/rcs-5.7/src/default/SunOS5.6/

At that point, follow the instructions for the various packages. The packages normally contain a README or an INSTALL file with all the necessary instructions. The only thing you will need to modify is the target directory or the directory where everything will be installed. In the case of RCS, I used the --prefix option of the "configure" script.

$ ./configure --prefix=/usr/soft/rcs-5.7/run/default/SunOS5.6
$ make all
$ make install

All my RCS binaries, libraries, and man pages are now compiled and installed under /usr/soft/rcs-5.7/run/default/SunOS5.6. Later, I'll explain why the users won't have to know that path.

For the AIX platform, you will need access to a machine capable of compiling for AIX. The procedure stays the same otherwise.

Once the package is built and installed, we can clean up and unduplicate all the "unchanged" files from the src directory. The only files kept are the original sources and those modified during the SunOS5.6 or the AIX4.2 build. That feature allows the administrator to recompile any application easily. Next time the lude --duplicate is run, the configuration of the last build will still be there for all the platforms supported in your environment. If you frequently have to make minor changes to a compiled piece of code, not counting the actual release revisions, LUDE will definitely be helpful for you.

To clean up our installation:

$ make clean
$ cd /usr/soft
$ lude --unduplicate --pack rcs-5.7 [--class SunOS5.6]

The last step is the actual installation of the package. This is where all the static links are created between the "run" environment of LUDE and the public directory. One simple command will create all the links:

$ lude --install --package rcs-5.7 [--class SunOS5.6]

Once the install is done, the application is available from /usr/local/bin and its man pages from /usr/local/man. The application physically lives under the run/default/SunOS5.6 directory, but it is effectively linked in the /usr/local for the correct platform. This simple step makes accessing the software easier for the users and also eases management of the $PATH variable for the administrators. Figure 2 gives an overview of the structure, including the /usr/soft (depot) and the /usr/local (public) directories. If you look closely at the example after the lude --install, you will find that:

sun_client$ which rcs
/usr/local/bin/rcs
sun_client$ ls -l /usr/local/bin/rcs
rcs@ --> /usr/soft/rcs-5.7/run/default/SunOS5.6/bin/rcs*

aix_client$ which rcs
/usr/local/bin/rcs
aix_client$ ls -l /usr/local/bin/rcs
rcs@ --> /usr/soft/rcs-5.7/run/default/AIX4.2/bin/rcs*

Other Tricks

Once this static link process is well understood, many interesting things can be done. I used this same process to manage commercial applications and pre-built software packages. I simply skip the use of the package/src structure, and copy or install the files directly in the run/default structure. Once my files are copied, I can install the application using:

$ lude --install --package package_name

and links will be created for me. In my current environment, for example, FrameMaker was installed using the same technique:

$ cd /usr/soft/maker-5.5/run/default/5.6


$ ls -l
drwxr-xr-x   4 charlesg css    512 Mar  9 18:20 Maker-5.5/
drwxr-xr-x   2 charlesg css    512 Mar  9 18:25 bin/

Once linked using lude --install, users use /usr/local/bin/maker to launch FrameMaker, and the $FMHOME is set to /usr/local/Maker-5.5 for everyone.

Changing versions and upgrading software becomes a truly trivial task. The "run" environment of LUDE can also be used as a testing environment for new releases. The files can be installed in package/run/default/ and fully tested before being linked in /usr/local to replace the old version.

This process can make managing packages easier because the administrator is able to build new versions of the software and install them only when fully tested and ready for production. To install a release of RCS, the administrator would simply run:

$ lude --uninstall --package rcs-5.6
$ lude --install --package rcs-5.7

If a mistake was made while upgrading to rcs-5.7, the administrator could, for example:

$ lude --uninstall --package rcs-5.7
$ lude --install --package rcs-5.6

That wouldn't really delete or move any files, but it would simply change the static links in /usr/local.

References

LUDE2 Documentation at:

http://www.iro.umontreal.ca/contrib/lude/lude2_toc.html

Managing NFS and NIS. Hal Stern and Mike Loukides. 1991. O'Reilly & Associates.

LUDE was developed at the University of Montreal Computer Sciences Department. More info is available from www.iro.umontreal.ca/contrib/lude/.

About the Author

Charles Gagnon has a degree in Electrical Engineering but is now working as a Consultant for Collective Technologies. Currently working on site at Bell Atlantic, he supports a heterogeneous UNIX environment. Collective Technologies specializes in systems management consulting, helping clients manage their complex, distributed client/server computing environments. Charles can be reached at: charlesg@colltech.com.