From: Bill Thompson (bill.thompson@GOODYEAR.COM)
Date: Thu Apr 10 2003 - 09:43:36 EDT
Obviously none of these suggestions will work.
In every case your going to get a "parameter list is too long" error which
is exactly what John was trying to avoid...
-> tar -cf FILE.tar `find . -name "*.txt"`
ksh: /usr/bin/tar: 0403-027 The parameter list is too long.
-> tar -cf FILE.TAR *.txt
ksh: /usr/bin/tar: 0403-027 The parameter list is too long.
-> tar -cf FILE.TAR */*.txt
ksh: /usr/bin/tar: 0403-027 The parameter list is too long.
etc., etc.
The problem is John is trying to tar up 30,000 individual files.
However, John, I believe you are doing it in the most efficient manner
possible using tar. All of my references show the same method you are using
- create a file list using find and then use that list with tar.
One way to increase the speed is to use cpio instead of tar. It accepts a
list of files on stdin so you could try something like this:
find . -name "*.txt" | cpio -ocB >$CPIOFILE
I found this can be four to five times faster than using tar.
Bill Thompson
Sr UNIX Systems Administrator
The Goodyear Tire & Rubber Co.
Contains Confidential and/or Proprietary Information
May Not Be Copied or Disseminated Without Express Consent of The Goodyear
Tire & Rubber Company.
AIX-L Archives: http://marc.theaimsgroup.com/?l=aix-l&r=1&w=2
----- Original Message -----
From: "Ian Smith" <ian.smith@COMPUTING-SERVICES.OXFORD.AC.UK>
Newsgroups: bit.listserv.aix-l
To: <aix-l@Princeton.EDU>
Sent: Thursday, April 10, 2003 6:42 AM
Subject: Re: speeding up tar
> John,
>
> You can try the following - the tar reads the output of find on STDIN :
>
> tar -cf FILE.TAR `find . -name "*.txt"`
>
> this should be quicker but only works, however, if none of the files have
> spaces in them.
>
> If all the files are under one directory then you can just use:
>
> tar -cf FILE.TAR *.txt
>
> If all the files are under one directory and it's next level
sub-directories
> you can use:
>
> tar -cf FILE.TAR *.txt */*.txt
>
> I don't know how to code the file spec for files in directories 2 or more
> levels under the current dir though. Anybody ?
>
> However, if I was tarring up 30000 files I'd want to make just a cursory
> check at least, that all the files I thought were archived, had indeed
been
> archived. In which instance I'd probably do a tar -tvf FILE.TAR | wc -l
> and compare that with the find command piped through wc -l. Probably not
> all that quick in the end !
>
> -------------------------------------------------------------------
> Ian Smith
> Oxford University Computing Services, Oxford, UK.
> -------------------------------------------------------------------
>
>
>
>
>
>
>
> ~>MIME-Version: 1.0
> ~>X-Spam-Level: *
> ~>X-Spam-KB: http://www.Princeton.EDU/spam
> ~>X-Spam-Report: CARRIAGE_RETURNS,EXCHANGE_SERVER,SPAM_PHRASE_00_01
> ~>Date: Thu, 10 Apr 2003 11:04:43 +0100
> ~>From: John Dunn <john.dunn@SEFAS.CO.UK>
> ~>Subject: speeding up tar
> ~>X-To: AIX List <AIX-L@PUCC.PRINCETON.EDU>
> ~>To: aix-l@Princeton.EDU
> ~>
> ~>I have a script that tars up lots of files....up to 30,000
> ~>
> ~>Because of the number of files and the need to avoid the "arg list too
long"
> ~>problem, I am running a find command to list the files
> ~>
> ~>find . -name "*.txt" > tarlist
> ~>
> ~>tar -cf$TARFILE -L tarlist
> ~>
> ~>I am wondering whether using the find and then the -L method is
relatively
> ~>slow and whether avoiding using a list would speed things up.
> ~>
> ~>How can I tar up 30000 files without using find and -L??
> ~>
> ~>John
>
This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 22:16:44 EDT