FIND+GREP replacement for PowerShell

It's pretty common in UNIX world to use find in conjunction with grep for searching filesystem tree for something. Would it be searching for specific class in source code tree:

$ find -name \*.cpp -o -name \*.hpp -exec grep -Hb class {} \;

Or just searching for some word in your notes, doesn't matter, but it is really common. So what do we have in PowerShell world to substitute those commands?

First of all, let's try to replace grep. For this we have cmdlet called Select-String, or sls (an alias). Let's try something simple:

$ sls class *.hpp

CustomTimeEdit.hpp:5:class CustomTimeEdit : public QTimeEdit {
DaySelecter.hpp:12: class Model : public QAbstractItemModel
DaySelecter.hpp:43: class View : public QTreeView
[...]

For some reason, documentation got this messed up, saying path should go first, and pattern second, but anyway.

What if we want to search files, whose names match several patterns (for example, cpp and hpp files):

$ sls DaySelector *.hpp,*.cpp

DaySelecter.hpp:1:#ifndef __DAYSELECTER_HPP__
DaySelecter.hpp:2:#define __DAYSELECTER_HPP__
DaySelecter.hpp:10:namespace DaySelecter {
[...]

What if we want to match several patterns in those files?

$ sls DaySelector,MainWindow *.hpp,*.cpp


DaySelecter.hpp:1:#ifndef __DAYSELECTER_HPP__
DaySelecter.hpp:2:#define __DAYSELECTER_HPP__
DaySelecter.hpp:10:namespace DaySelecter {
MainWindow.hpp:1:#ifndef __MAINWINDOW_HPP__
MainWindow.hpp:2:#define __MAINWINDOW_HPP__
MainWindow.hpp:4:#include
[...]

The fact is, for both pattern and path Select-String accepts arrays (which in PowerShell are specified with comma operator).

Now you remember PowerShell operates with objects, not text, yes? So let's see, what those objects are:

$ sls DaySelector *.hpp,*.cpp | gm

TypeName: Microsoft.PowerShell.Commands.MatchInfo

Name MemberType Definition
---- ---------- ----------
Equals Method bool Equals(System.Object obj)
GetHashCode Method int GetHashCode()
GetType Method type GetType()
RelativePath Method string RelativePath(string directory)
ToString Method string ToString(), string ToString(string directory)
Context Property Microsoft.PowerShell.Commands.MatchInfoContext Context {get;set;}
Filename Property System.String Filename {get;}
IgnoreCase Property System.Boolean IgnoreCase {get;set;}
Line Property System.String Line {get;set;}
LineNumber Property System.Int32 LineNumber {get;set;}
Matches Property System.Text.RegularExpressions.Match[] Matches {get;set;}
Path Property System.String Path {get;set;}
Pattern Property System.String Pattern {get;set;}

Not just we can look at the output, we can cut the pieces of output with ease (no more sed scripts for command output parsing!), we can analyze output easily. That's pretty exciting, I think. That's the future of command-line. But I'm off the road.

I think, you've got the idea. With sls you can perform regexp matching (default behaviour), simple matching (-SimpleMatch), search for just first match in file, ignoring all others (-List), or, on the contrary, search for all matches, even if several are present on the same line (-AllMatches), do negative matching (-NotMatch). You can also obtain results in context of several lines forward and backward (-Context option), pretty like diff -u does. What's more, you can even specify different encoding, though this list, for some reason, is far from complete: it contains only Unicode encodings, ANSI and OEM, which is sufficient in most cases, but the list could have been more diverse.

Now what about searching the tree. In PowerShell, we use Get-ChildItem, or ls. I don't know full powers of UNIX ls, but PowerShell ls is pretty powerful thing. We can find all files in the tree, matching specific patterns:

$ ls -r -inc *.cpp,*.hpp

Or we can find files, not matching pattern:

$ ls -r -ex *.hpp~

Or we can even do some crazy stuff, matching some files and then
excluding some files, not matching specific pattern:

$ ls -r -inc *.cpp,*.hpp -ex DaySelecter*

If you want more control over matching, you can process matching files with Where-Object block -- remember? PowerShell operates on objects, not text:

$ ls -r -inc *.cpp,*.hpp -ex DaySelecter* | ? { $_.IsReadOnly }

Ok, now, how we can use powers of ls with sls? The answer is, sls can accept files not only from command line, but from pipe as well, so we can just pipe both commands together, and get the result desired:

$ ls -r -inc *.cpp,*.hpp -ex *DaySelecter* | sls DaySelecter

MainWindow.cpp:26: connect(viewDaySelecter->selectionModel(),
MainWindow.cpp:38: viewDaySelecter->setDiary(diaryModel);

Now that's some powers we need!

Managing Oracle Forms CVS tree under Powershell

At work, I have to deal with Oracle Forms, which itself is a tool for creating data access forms in a visual way. This tool stores forms as a binary files (.fmb's), so managing them under ordinary version control system like CVS is a pain, because I can't see what changes were made to form (if any). In spite of that, CVS is my only available option.

Fortunately, Oracle Developer Suite has a tool, frmf2xml which can be used to convert binary FMB file to XML, so, gluing it all together (cvs, frmf2xml and ordinary diff utility) with PowerShell, I have written a script, Compare-CvsForms, which is much like cvs diff command, but works with theese binary forms. Actually, I've also written vdiff.py script, based on difflib, which is much more convenient for comparing such a messy data (XML, huh).

So, here is the Compare-CvsForms.ps1 script (I've put it to my PowerShell profile directory):

Param ([string]$formsFile)

if (cvs status $formsFile |
select-string 'Status: (Locally Modified|Needs Checkout|Needs Patch)')
{
# store working copy in another place
$local:temp_fmb = [System.IO.Path]::ChangeExtension(
 [System.IO.Path]::GetTempFileName(), "fmb")
$local:temp_xml = [System.IO.Path]::ChangeExtension(
 [System.IO.Path]::GetTempFileName(), "xml")
Move-Item -force $formsFile $local:temp_fmb
# produce xml of a working copy
frmf2xml $local:temp_fmb | out-null
if ($LastExitCode -ne 0) {
Echo "ERROR: Cannot convert working copy of form to XML!"
Move-Item -Force $local:temp_fmb $formsFile
return
}
# get repositary copy
cvs update 2>&1 | out-null
#produce xml of a repositary copy
frmf2xml $formsFile | out-null
if ($LastExitCode -ne 0) {
Echo "ERROR: Cannot convert CVS copy of form to XML!"
Move-Item -Force $local:temp_fmb $formsFile
Remove-Item -Force $xml_from
return
}
# compare xmls
$xml_from=$formsFile.substring(0, $formsFile.length-4) + "_fmb.xml"
$xml_to=$local:temp_fmb.substring(0, $local:temp_fmb.length-4) + "_fmb.xml"
vdiff.py -context $xml_from $xml_to
if ($LastExitCode -eq 0) {
$formsFile.toUpper() + ": Working copy does not differ from repositary."
}
Move-Item -Force $local:temp_fmb $formsFile
Remove-Item -Force $xml_to,$xml_from
} else {
$formsFile.toUpper() + ": Working copy is up to date."
}

And vdiff.py (pretty straightforward) script (surely, you should have Python installed, if not - you could just as easily replace this script with ordinary diff utility):

#!/usr/bin/python
import os
import sys
import difflib
import tempfile

program_name = os.path.splitext(os.path.basename(os.path.sys.argv[0]))[0].upper()
browser = "c:\\Program Files\\Internet Explorer\\iexplore.exe"

if len(sys.argv) < 3:
  print "\nUSAGE:\n\n    %s FROM-FILE TO-FILE\n" % program_name
  sys.exit(2)

if sys.argv[1] == "-context":
  context = True
  from_fn = sys.argv[2]
  to_fn = sys.argv[3]
else:
  context = False
  from_fn = sys.argv[1]
  to_fn = sys.argv[2]

try:
  from_lines = open(from_fn).readlines()
except IOError:
  print "%s: Error while reading file -- %2" % (program_name, from_fn)
  sys.exit(1)

try:
  to_lines = open(to_fn).readlines()
except IOError:
  print "%s: Error while reading file -- %2" % (program_name, to_fn)
  sys.exit(1)

differs = False
for x in difflib.context_diff(from_lines, to_lines):
  differs = True
  break

if differs:
  diff = difflib.HtmlDiff(wrapcolumn=70).make_file(from_lines, to_lines, from_fn, to_fn, context)
  diff = diff.replace("<head>",
                      '<head><meta http-equiv="Content-Type" content="text/html; charset=utf-8">')
  tmp = tempfile.mkstemp(".html")
  os.write(tmp[0], diff)
  os.close(tmp[0])
  os.spawnl(os.P_WAIT, browser, ' file://' + tmp[1] + '')
  sys.exit(1)
else:
  sys.exit(0)

Scripting with PowerShell actually is a great deal of fun :)

A little correction to Process-File function

Once after processing my C++ source code tree with Process-File my sources became uncompilable. The problem was caused by redirection operator `>', which transformed source files from plain-old ASCII to Unicode. So there's corrected version which uses Out-File cmdlet instead with encoding specified explicitly:

function Process-File([scriptblock]$script, 
                      $filename, 
                      $encoding = "ascii") {
    $local:temp = New-TempFileName
    Get-Content -Encoding $encoding $filename | `
          Foreach $script | `
          out-file -encoding $encoding $local:temp
    Move-Item -force $local:temp $filename
}

Process-File function

Some time ago while scripting (either bash or PowerShell) I've found myself using some pattern too often. It can be expressed as

1. take a file;
2. process every line in it;
3. save file back.
   

E.g. `sed -i' provides this kind of functionality. This was kind of boring, because you can't just write "cat file.txt | do-something > file.txt", and if you really could, this was too long anyway. So I wrote function which does just that -- takes file, processes it with some code (specified as an argument), and writes it back. Here it goes:

function Process-File([scriptblock]$script, $filename) {
    $local:temp = [System.IO.Path]::GetTempFileName()
    Get-Content $filename | Foreach $script > $local:temp
    Move-Item -force $local:temp $filename
}

Here some testing code:

PS C:\Documents and Settings\andrey> cat test.txt
PS C:\Documents and Settings\andrey> echo "abcabc" > test.txt
PS C:\Documents and Settings\andrey> cat test.txt
abcabc
PS C:\Documents and Settings\andrey> Process-File { $_ -replace 'a','b' } test.txt
PS C:\Documents and Settings\andrey> cat test.txt
bbcbbc
PS C:\Documents and Settings\andrey>

PowerShell is just wonderfull.