From: Jeff Lightner (jlightner@water.com)
Date: Mon Dec 13 2004 - 10:50:16 EST
Resolution to my "Unix" problem was:
Verify the issue first!
The problem turned out to have nothing to do with file transfers at all.
The 5 hours was number crunching on the AS400 itself. In fact after direct
discussion with the end user it turned out no file transfer is even being
done. Since this was all AS400 there is nothing we can do on the "Unix"
side.
However there were responses and my own research so I'll include them for
the archives.
ORIGINAL QUESTION:
Would someone be willing to let me know what they are doing to optimize file
transfer from AS400 to HP-UX (or even Windows)?
Also I'd like to know what compression and/or encryption techniques might be
available for this.
We're currently doing FTP from AS400 which is taking about 5 hours so they'd
like to speed this up. I've already verified there is an AS400 version of
gzip (www.gzip.org <http://www.gzip.org/> ). There is apparently also a
commercial product call RevZIP from Seasoft (www.seasoft.com/rev-zip.asp
<http://www.seasoft.com/rev-zip.asp> ).
The gzip site says there is no encryption available for gzip so one would
need to use something like pgp but I didn't see a version of that for AS400.
The archives had a few tidbits about FTP from AS400 but none seemed to
address it in detail.
ADDITIONAL COMMENTS:
First I'll mention that there is a software called Connect:Direct from
Sterling Commerce (previously called NDM at another site where I worked on
it) that does file transfers with compression and encryption/security.
Looking at their site
http://www.sterlingcommerce.com/solutions/connectdirect/unix/
<http://www.sterlingcommerce.com/solutions/connectdirect/unix/> I saw they
also have another product called FTP+ that may be worthwhile for
investigation. Connect:Direct requires software to be installed on both
sides - I didn't investigate FTP+.
Second a key limitation in the description of the AS400 available gzip was
noted by our AS400 admin when he was asked to look at its README. It
specifically states that it works only for IFS (filesystem) but NOT for
"Database" (which is DB2 like according to him). Since most of what our
AS400 does is Database gzip probably wouldn't have helped had this been a
file transfer issue. He did confirm it is possible to dump from the
Database side to the IFS side internal to the AS400 but raised concerns
about both disk space and processing cycles to do that.
Third some of the suggestions regarded ssh/scp/sftp. I did look briefly for
an AS400 port of these but didn't find one.
RESPONSES:
John Adams wrote:
We're not transferring anything that takes more than about half an hour at
most.
We usually use gzip to compress first, then do the ftp put, then uncompress
after. The compress and send are separate tasks, as (obviously) is the
uncompress. That way, both the compress and send can have a backlog.
As far as encrypting, since all our transfers are in-house, we don't worry
about it. I would suggest that, if you really want to encrypt it, you
should be able to implement the crypt command on both sides easily enough.
If you can't find a straight implementation for AS-400, you can always try
to use the BSD source code (C language) and implement it yourself. It's a
simple rotor style encryption, but it might be good enough for your
purposes.
Allan Mariller wrote:
It might be possible to get an ssh server or client for the AS400 - that way
you'll cover both compression and encryption in one move. You can use scp os
sftp which is really just scp with an ftp like interface. Something that may
be even more efficient if you could pull it off, would be to have ssh and
rsync on both servers. rsync uses ssh as a transport protocol so it benefits
from the encryption and compression, but rsync will gain performance when
copying files where some already exist on both servers. Only new or changed
files are sent, and I'm fairly sure that if the same file exists in both
places, only the differences are sent to make the destination file the same
as the source. There is also a switch to tell rsync to delete any files that
no longer exist to keep both servers fully in sync.
Dan Zucker wrote:
check out softlink's fastcopy.
Jim McDonald wrote:
Got perl on the AS400?
If yes try do it yourself in perl
encryption example mail -
http://www.webreference.com/perl/tutorial/16/3.html
compression - also you can use perl to compress data, it uses the zlib
library. zlib has been ported to os400
Corné Beerse wrote:
>From a unix point of view, this are 2 different things which require 2
different tools.
gzip will suit your needs for compression, it is available at both ends.
For encryption, I'd switch from ftp to the file-transfer stuff that is
available
with `ssh`: http://www.openssh.com/ <http://www.openssh.com/> ,
http://hpux.its.tudelft.nl/hppd/hpux/Networking/Admin/openssh-3.9p1/
<http://hpux.its.tudelft.nl/hppd/hpux/Networking/Admin/openssh-3.9p1/> .
I do know there is a setting in some ftp-servers that can compress on the
fly:
see a file or directory and `ftp-get` it with an additional ".gz" and it
will be compressed. Decompression at the receiving end still is manual. I do
not know if that can also be used for encryption on the fly.
An other way for encryption is to use something like vpn and/or tunneling.
This tunneling is also possible with the ssh-toolchain.
To setup a tunnel and/or vpn connection, it is also possible to use an other
machine close to the AS400 that does the tunneling or such and uses some
kind of port-forwarding to the AS400.
Rick Jones wrote:
Adding compression would conceivably reduce the data transfered, but will
_definitely_ increase the CPU demand. If one or the other system has high
CPU util already, you may go from the frying pan into the fire.
Especially if you add encryption. If you want to encrypt, scp (part of
OpenSSH) is probably the easiest way to go. For UX it may be on your OS
media, or you can probably get one from software.hp.com
Bill Hassell wrote:
I would focus in on the network first. gzip is standard on HP-UX. I would
use SSH (if the
AS400 has it available) rather than PGP since I don't think it encrypts
during the transfer
but requires a separate pass to do the encryption.
Start with computing the actual data transfer rate. Is this a WAN? If so,
there are a very
large number of variables, some of which can't be helped (slow replies) or
need tuning
on both sides. Get a copy of netperf for HP-UX http://www.netperf.org/
<http://www.netperf.org/> so you can get
some good stats and ideally, whether the AS400 is slowing the transfer rate
or perhaps
switch settings and gateway throughput problems.
CLOSE:
Thanks to all who responded.
My apologies if I've left anyone out.
Jeffrey C. Lightner
Unix Systems Administrator
DS Waters of North America
678-486-3516
--
---> Please post QUESTIONS and SUMMARIES only!! <---
To subscribe/unsubscribe to this list, contact majordomo@dutchworks.nl
Name: hpux-admin@dutchworks.nl Owner: owner-hpux-admin@dutchworks.nl
Archives: ftp.dutchworks.nl:/pub/digests/hpux-admin (FTP, browse only)
http://www.dutchworks.nl/htbin/hpsysadmin (Web, browse & search)
This archive was generated by hypermail 2.1.7 : Sat Apr 12 2008 - 11:02:44 EDT