Wednesday, November 30, 2011

Perl Power - Sequential Iteration in shell scripts

Lets say you want to process a bunch of files containing a numeric sequence like "inventory000297.txt" through "inventory000302.txt". You would like to print the following:

inventory000297.txt
inventory000298.txt
inventory000299.txt
inventory000300.txt
inventory000301.txt
inventory000302.txt

I did not see a simple way to do this using a plain shell script, but adding some perl is a different story...

for F in $(perl -e 'foreach $num (297 .. 302) {printf("%s%06i%s\n","inventory",$num,".txt");}')
do
    echo ${F}
done

Notes to keep in mind:
  • $( ) contains the perl one-liner that generates the output to print
  • The perl script is between the single quotes
  • The foreach loop block is within braces { }
  • The numerical range is in parenthesis ( 297 .. 302 ), change this as necessary
  • The printf has 3 format specifiers:

    %s corresponds with "inventory", change prefix as necessary

    %06i corresponds with $num, a fixed width 6 digit integer, change as necessary

    %s corresponds with the ".txt", change extension as necessary






Adjust the size of your sequence number - if 15 digits is needed you would use %015i for the format. That leading 0 says fill in with leading zeros on output to match the specified width.

Enjoy!

Monday, August 1, 2011

Perl One-liners - Deal with the "too many files" issue

In UNIX, sometimes you end up getting a "too many files" error when using the "cp, mv, or rm" commands if there are thousands of files in a given directory. The usual workaround is to utilize the "find" command.

Without the newer (GNU) find versions, like I don't have... I have to resort to using this type of syntax to display files only in the current directory (I don't want it to descend in to sub-directories). Let say there are tons of "xml" files in the current directory.

#Example using find that will count the number of xml files
find . \( ! -name . -prune -name "*.xml" \) -type f -print | wc -l

Here is an alternative way to count the files in Perl:

perl -e '@a=<*.xml>;map {print "$_\n";} @a;' | wc -l
#Even quicker version
perl -e '@a=<*.xml>;printf("%s\n", scalar(@a));'

The caveat for the Perl version is that you would need enough memory to hold the names of the files in the array @a .

Display all Standard Perl Modules available

Here is a quick way to see all names of the Standard Perl Modules that are installed on your system:

perl -MFile::Find=find -MFile::Spec::Functions -Tlw -e 'find { wanted => sub { print canonpath $_ if /\.pm\z/ }, no_chdir => 1 }, @INC'