Thursday, May 31, 2012

Perl Power - Remove files fast

Perl Squirrel has been very busy collecting nuts ( about 1.5 million ) over the past years and his store room was about to burst. If the nuts were actually files in a directory, how would Nugget get rid of them fast? Nugget wants to keep only the nuts that were collected in the past 90 days ( they would taste better ). First get a count of the files that would get removed using the Unix find command and mtime option to count files older than 90 days...

cd \perlsquirrel\storeroom find . \( ! -name . -prune "*nuts.*" \) -type f -mtime +90 -print | wc -l
Now lets say the number of files returned was 1 million. In order to remove these, Nugget did the following...
cd \perlsquirrel\storeroom find . \( ! -name . -prune "*nuts.*" \) -type f -mtime +90 -print | perl -lne unlink
This removed 1 million files in about 9 minutes.
Note: This find command prevents descending into subdiretories.
Perl Squirrel advises...measure twice...cut once.

Sunday, February 5, 2012

Source Code Search Browser for Command-line

Here is a handy utility ( viewsrc.sh ) for searching through your existing code base for code examples.  Let say you are developing some new code and you need to add a feature but you can not quite remember the syntax you used before.   No problem...without even having to leave the code you are in you can search for that example.

Syntax:  viewsrc.sh  <searchstring>  <fileExtension>
Where: searchstring is a regular expression (one accepted by "egrep"
and fileExtension is the file extension of the files to look at... for example... "pl" or "c" or "sh" or "py" or "html", ... you get the idea.

Features:
  • Search-word can be a simple regular expression
  • Search within files of certain file extensions
  • The search is from the current directory and also within sub-directories
  • Search is not case-sensitive
  • Every search match results in the full contents of the matching file being appended to one temporary file.
  • The temporary file is opened with vi/vim in "view" read-only mode.
  • The cursor is placed at the first occurrence of the search-word.
  • Press "n" for next match, or "N" for previous match.
  • The filename where each match occurred, is placed at the top of each matching file that was appended to the temporary file.
  • No changes are made to original code, only to a temporary file.
  • The advantage of the "viewsrc.sh" utility vs grep is that you see the full context of the match by seeing the entire source file.
  • When you find a match you like, just go backwards in the file until you see the source file name, if you want to know which file the match occurred in.
Example 1:
You would like to find examples ofPerl programs that use the "foreach" keyword

viewsrc.sh foreach pl

You will be placed in view mode, and if there are any matches, just press "n" to go to the next match, "N" for previous match, and when you are done, just quit like you would in vi/vim ...

:q!

Example 2:
You are editing some Bash code...


vi mycode.sh

Without leaving your editing session, search for example code containing the word "case"


:!viewsrc.sh case sh

Press 'n' to view next occurrance of the word "case".
Then exit with ":q!".

Note:  If you wanted to you can search multiple times in succession.  For every search you do you would have to do a ":q!" to return to the previous environment.

Here is the source for Bash Shell:


#!/bin/bash
#Script:  viewsrc.sh
#Purpose: This script is used to search through the current directory
# and down for any files that contain the search word (param 1) within files
# with the given file extension (param 2).
#
# Every file that contains the searched for word is
# concatenated into one temp file and viewed with the cursor on the
# first occurence.
# Press "n" to see the next occurance of the searchword.
# You exit with the command    :q!
#
# Example:
# You are editing a file in vi and you want to see korn shell code examples
# using the word "grep".
# To view this examples do the following:
#    1)  Press ESC to make sure you are not in edit mode
#
#    2)     :!viewsrc.ksh grep ksh  <press enter>
#
#    3)  A read-only temporary file will be displayed with the cursor on
#        the first match of the searchword you searched for.
#
#    4)     n    to see the next match if one exists.
#           N    to see the previous match
#
#    5)  To quit the view session enter the following:
#           :q   <press enter>  or
#           :q!  <press enter>
#
#    6)  You will see  "[Hit return to continue]"
#                       <press enter>
#
#    7)  You are back where you started.
#
#    Note:  Step 2 can be done multiple times to drill down into
#           procedure calls.  Just do Step 5 to go back a level.
#           Press CTRL-G to verify what file you are currently in
#           (whether it is your original file or the temp file)
#
#    Note:  If you don't get a match you will see something like:
#           /tmp/srch.030130_095607" [Read only] No lines in the buffer
#                   Do step 5 to go back to your original file.

#Function Name :  fullpath()
#Purpose     : Accepts a filename
#              Returns the full pathname of the file
#-----------------------------------------------------------------
# NOTE:  fullpath function works relative to the current directory.
#        Either the full path or the basename will work ok.
#-----------------------------------------------------------------

#Function to construct a full pathname for the given filename
fullpath ()
{
s=$i                                    #s is a string containing the filename
ts=`echo $s  | awk '{print substr($0,1,1)}'`    #ts is the 1st char of string
if [[ "$ts" = "/" ]]                    #Is leading char a slash
then
  #No change is necessary because it is already a full path
  echo $s
else
  #Tack on current directory without the leading ./
  ts=`echo $s  | awk '{print substr($0,1,2)}'`          #1st two char of string
  if [[ "$ts" = "./" ]]                 # does it start with "./"
  then
    # Replace the dot with the current working directory and tack on
    # the string from the second character to the end of the string
    s=$(echo `pwd``echo $s | awk '{printf ("%s", substr($0,2))}'`)
    echo $s
  else
    #Just a filename was given so build a string from the current working
    #directory, a slash and then the filename
    s=$(echo `pwd`/`echo $s | awk '{printf ("%s", substr($0,1))}'`)
    echo $s
  fi
fi
}


#- - - MAIN SCRIPT STARTS HERE - - -
USAGE="\n\nUsage:  viewsrc.ksh searchword FileExtension\n"
if (($# != 2))                  #One parameter required
then
    echo $USAGE
    exit 1
fi

SEARCHWORD=$1

# Declare Variables
TIMESTAMP=`date +%y%m%d_%H%M%S`
MATCHFILES=/tmp/temp.$TIMESTAMP
SEARCHFILE=/tmp/srch.$TIMESTAMP
DIRFILE=/tmp/dirfile.$TIMESTAMP

#If the searchfile already exists remove it
if [[ -f $SEARCHFILE ]]
then
   rm $SEARCHFILE
fi

#Create an empty file
touch $SEARCHFILE

#If the dirfile already exists remove it
if [[ -f $DIRFILE ]]
then
   rm $DIRFILE
fi

#Create an empty file
touch $DIRFILE

# Find Files That Contain The Searchword in various subdirectories
find . -name "*.${2}" -print > $DIRFILE

#Note:  in between the brackets is a space and a tab character
cat $DIRFILE | xargs grep -i -l "${1}" > $MATCHFILES

for i in `cat $MATCHFILES`
do
    # Consolidate The Files That Contain The Searchword
    echo "===========================================" >> $SEARCHFILE
    FP=$(fullpath ${i})
    echo "File: `hostname`:${FP}" >> $SEARCHFILE
    echo "===========================================" >> $SEARCHFILE
    cat $i >> $SEARCHFILE
done

# Display the SEARCHFILE at the first Searchword
EXINIT='set ic'
export EXINIT
view +/$SEARCHWORD/ $SEARCHFILE

# Cleanup Temporary Files
rm $MATCHFILES
rm $SEARCHFILE
rm $DIRFILE

TIPS:

  • To search on more than one word, put a period between words.
    Like....      
    viewsrc.ksh Searching.on.four.words sh
  • Searching on special characters like brackets would need to be backslash escaped
    Like....
    viewsrc.sh if.\[\[ sh


Enjoy!

Wednesday, November 30, 2011

Perl Power - Sequential Iteration in shell scripts

Lets say you want to process a bunch of files containing a numeric sequence like "inventory000297.txt" through "inventory000302.txt". You would like to print the following:

inventory000297.txt
inventory000298.txt
inventory000299.txt
inventory000300.txt
inventory000301.txt
inventory000302.txt

I did not see a simple way to do this using a plain shell script, but adding some perl is a different story...

for F in $(perl -e 'foreach $num (297 .. 302) {printf("%s%06i%s\n","inventory",$num,".txt");}')
do
    echo ${F}
done

Notes to keep in mind:
  • $( ) contains the perl one-liner that generates the output to print
  • The perl script is between the single quotes
  • The foreach loop block is within braces { }
  • The numerical range is in parenthesis ( 297 .. 302 ), change this as necessary
  • The printf has 3 format specifiers:

    %s corresponds with "inventory", change prefix as necessary

    %06i corresponds with $num, a fixed width 6 digit integer, change as necessary

    %s corresponds with the ".txt", change extension as necessary






Adjust the size of your sequence number - if 15 digits is needed you would use %015i for the format. That leading 0 says fill in with leading zeros on output to match the specified width.

Enjoy!

Monday, August 1, 2011

Perl One-liners - Deal with the "too many files" issue

In UNIX, sometimes you end up getting a "too many files" error when using the "cp, mv, or rm" commands if there are thousands of files in a given directory. The usual workaround is to utilize the "find" command.

Without the newer (GNU) find versions, like I don't have... I have to resort to using this type of syntax to display files only in the current directory (I don't want it to descend in to sub-directories). Let say there are tons of "xml" files in the current directory.

#Example using find that will count the number of xml files
find . \( ! -name . -prune -name "*.xml" \) -type f -print | wc -l

Here is an alternative way to count the files in Perl:

perl -e '@a=<*.xml>;map {print "$_\n";} @a;' | wc -l
#Even quicker version
perl -e '@a=<*.xml>;printf("%s\n", scalar(@a));'

The caveat for the Perl version is that you would need enough memory to hold the names of the files in the array @a .

Display all Standard Perl Modules available

Here is a quick way to see all names of the Standard Perl Modules that are installed on your system:

perl -MFile::Find=find -MFile::Spec::Functions -Tlw -e 'find { wanted => sub { print canonpath $_ if /\.pm\z/ }, no_chdir => 1 }, @INC'

Saturday, December 25, 2010

Count Each ASCII Character Occurance in a Given File

Ever get strange characters in a file and want to know what they are? Using this code you can see what characters would be causing the issue. The normal printing characters are ASCII 32 through 126, so any characters outside this range might be causing you trouble. Consult an online ASCII chart and look for the decimal values to verify the output from this tool.


#!/usr/bin/env perl -w
#Script:  ord_frequency.pl
#Frequency of ASCII characters in file or standard input
#this can read from stdin or from a passed filename
#Reads file one character at a time in binary mode
#gets frequency of the ordinal value of each character.
#Display frequency distribution of ordinal values.
#----------------------------------------------------------
$file = shift;                          #optional file
$file = '-' unless defined $file;       #use STDIN if no file given
open FILE, "$file" or die $!;           #Open file
my %freq;
binmode FILE;                           #set binary mode
my ($buf, $data, $n);
while (($n = read FILE, $data, 1) != 0) {   #read 1 char at a time
   $freq{ord($data)}++;                 #record ordinal frequency
}
close(FILE);
#Display frequency Sorted Numerically by ordinal value
printf("%12i: %s\n", $freq{$_}, $_) foreach sort { $a <=> $b; } keys %freq;

Enjoy!

Open Multiple Text Files With One Click using ActiveState Perl

This is a convenient way of opening many text files at once in an editor that supports multiple tabs.  For those that use a text editor for taking notes, logs, work files, journaling, etc. and often reference these files frequently this technique is very easy to use.

Requirements are:
  • An editor that supports multiple files open at once in separate tabs.
  • ActiveState Perl installed (and in your PATH)
  • The sourcecode below
  • setup a Windows Shortcut to the perl code
In the code below you will see that it is configured to open 4 files when "log.pl" runs. 
The files are:
YYYY_MM.txt (each month it will create a new file)
JOURNAL.txt
INFO.txt
TIPS.txt

You can change the names and the number of files you wish to open and the script will build the appropriate command line to run (see lines 20-23 below).

Make sure you specify the correct editor you are using... In this case it is using TextPad.exe, but it can use other editors if you specify the correct path to your editor (see line 14 below).

Source Code:
#!/usr/bin/perl
#Script : log.pl
#Purpose: To open multiple files at once in the editor that supports tabs
#Requirements: ActivePerl (in your path)
#Create a shortcut for this script
# Target: path to your script
# Start in: directory your script is in
use POSIX qw(strftime);

$scnt = 0; #string count
@strings = (); #array of strings
$cmd = ""; #command to run

$strings[$scnt++] = "C:\\Progra~1\\TextPa~1\\TextPad.exe"; #Use Textpad editor
#$strings[$scnt++] = "C:\\Progra~1\\EditPl~1\\editplus.exe"; #Use Editplus editor

#--------------------------------------------------------------------
# FILES YOU ENTER HERE WILL BE OPENED IN SEPARATE TABS IN YOUR EDITOR
#--------------------------------------------------------------------
$strings[$scnt++] = strftime("c:\\dir1\\notes\\%Y_%m.txt", localtime);
$strings[$scnt++] = "c:\\dir1\\notes\\JOURNAL.txt";
$strings[$scnt++] = "c:\\dir1\\notes\\INFO.txt";
$strings[$scnt++] = "c:\\dir1\\notes\\TIPS.txt";

foreach $str (@strings) { #Build Command string
$cmd .= chr(34) . $str . chr(34) . " ";
}

system ($cmd); #Open files in editor
  

Enjoy!