With small steps...

FIND+GREP replacement for PowerShell

2010-05-05T07:38:00.000-07:00

It's pretty common in UNIX world to use find in conjunction with grep for searching filesystem tree for something. Would it be searching for specific class in source code tree:

$ find -name \*.cpp -o -name \*.hpp -exec grep -Hb class {} \;

Or just searching for some word in your notes, doesn't matter, but it is really common. So what do we have in PowerShell world to substitute those commands?

First of all, let's try to replace grep. For this we have cmdlet called Select-String, or sls (an alias). Let's try something simple:

$ sls class *.hpp CustomTimeEdit.hpp:5:class CustomTimeEdit : public QTimeEdit { DaySelecter.hpp:12: class Model : public QAbstractItemModel DaySelecter.hpp:43: class View : public QTreeView [...]

For some reason, documentation got this messed up, saying path should go first, and pattern second, but anyway.

What if we want to search files, whose names match several patterns (for example, cpp and hpp files):

$ sls DaySelector *.hpp,*.cpp DaySelecter.hpp:1:#ifndef __DAYSELECTER_HPP__ DaySelecter.hpp:2:#define __DAYSELECTER_HPP__ DaySelecter.hpp:10:namespace DaySelecter { [...]

What if we want to match several patterns in those files?

$ sls DaySelector,MainWindow *.hpp,*.cpp DaySelecter.hpp:1:#ifndef __DAYSELECTER_HPP__ DaySelecter.hpp:2:#define __DAYSELECTER_HPP__ DaySelecter.hpp:10:namespace DaySelecter { MainWindow.hpp:1:#ifndef __MAINWINDOW_HPP__ MainWindow.hpp:2:#define __MAINWINDOW_HPP__ MainWindow.hpp:4:#include [...]

The fact is, for both pattern and path Select-String accepts arrays (which in PowerShell are specified with comma operator).

Now you remember PowerShell operates with objects, not text, yes? So let's see, what those objects are:

$ sls DaySelector *.hpp,*.cpp | gm TypeName: Microsoft.PowerShell.Commands.MatchInfo Name MemberType Definition ---- ---------- ---------- Equals Method bool Equals(System.Object obj) GetHashCode Method int GetHashCode() GetType Method type GetType() RelativePath Method string RelativePath(string directory) ToString Method string ToString(), string ToString(string directory) Context Property Microsoft.PowerShell.Commands.MatchInfoContext Context {get;set;} Filename Property System.String Filename {get;} IgnoreCase Property System.Boolean IgnoreCase {get;set;} Line Property System.String Line {get;set;} LineNumber Property System.Int32 LineNumber {get;set;} Matches Property System.Text.RegularExpressions.Match[] Matches {get;set;} Path Property System.String Path {get;set;} Pattern Property System.String Pattern {get;set;}

Not just we can look at the output, we can cut the pieces of output with ease (no more sed scripts for command output parsing!), we can analyze output easily. That's pretty exciting, I think. That's the future of command-line. But I'm off the road.

I think, you've got the idea. With sls you can perform regexp matching (default behaviour), simple matching (-SimpleMatch), search for just first match in file, ignoring all others (-List), or, on the contrary, search for all matches, even if several are present on the same line (-AllMatches), do negative matching (-NotMatch). You can also obtain results in context of several lines forward and backward (-Context option), pretty like diff -u does. What's more, you can even specify different encoding, though this list, for some reason, is far from complete: it contains only Unicode encodings, ANSI and OEM, which is sufficient in most cases, but the list could have been more diverse.

Now what about searching the tree. In PowerShell, we use Get-ChildItem, or ls. I don't know full powers of UNIX ls, but PowerShell ls is pretty powerful thing. We can find all files in the tree, matching specific patterns:

$ ls -r -inc *.cpp,*.hpp

Or we can find files, not matching pattern:

$ ls -r -ex *.hpp~

Or we can even do some crazy stuff, matching some files and then
excluding some files, not matching specific pattern:

$ ls -r -inc *.cpp,*.hpp -ex DaySelecter*

If you want more control over matching, you can process matching files with Where-Object block -- remember? PowerShell operates on objects, not text:

$ ls -r -inc *.cpp,*.hpp -ex DaySelecter* | ? { $_.IsReadOnly }

Ok, now, how we can use powers of ls with sls? The answer is, sls can accept files not only from command line, but from pipe as well, so we can just pipe both commands together, and get the result desired:

$ ls -r -inc *.cpp,*.hpp -ex *DaySelecter* | sls DaySelecter MainWindow.cpp:26: connect(viewDaySelecter->selectionModel(), MainWindow.cpp:38: viewDaySelecter->setDiary(diaryModel);

Now that's some powers we need!

Configuring GNU Emacs 23 + Ispell + Russian Dictionary + Windows

2010-03-31T23:48:00.000-07:00

Emacs ispell package is capable of running both Ispell and Aspell, but
as far as I've tried Aspell, it gives some nonsensical results. Maybe this
was due to some misconfiguration, but I wasn't eager to sort it out.

So I went with Ispell. Configuring it is also a bit of obscure process, so
I thought it would be good to have an installation guide step by step:

Download Ispell from here.

You should have Cygwin installed on your machine. Let's say, it is in folder c:\Programs\Cygwin.

Unpack aforementioned Ispell to your Cygwin directory.

Go to c:\Programs\Cygwin\usr\local\bin and copy ispell.exe to c:\Programs\Cygwin\bin. This is needed to run ispell directly from Windows, bypassing Cygwin shells.

Download russian dictionaries from here and unpack them to some temporary folder. Let's say it would be d:\temp.

Run Cygwin shell and perform the following commands (modify them appropriately):


cd /cygdrive/d/temp
make win
mv russian.hash russianw.hash
mv russian.aff russianw.aff
mv russianw.* /usr/local/lib

From now on, you sould be able to check files encoded in windows-1251 charset directly under Cygwin, like that:


LANG=ru_RU ispell -drussianw file.txt

Or, under cmd.exe:


set LANG=ru_RU.CP1251
c:\Programs\Cygwin\bin\ispell.exe -drussianw file.txt

Now all we need to do is to configure Emacs correctly. Add the following lines to your dotemacs:


(setq ispell-program-name "c:\\Programs\\cygwin\\bin\\ispell.exe")
(setq ispell-dictionary "russianw")

This should do the task. Now restart emacs and try running ispell on some file.

Disable arrow keys in Emacs

2008-07-04T10:01:00.000-07:00

So for this habit is ineradicable by will and my hands a little ache after hour or two of code surfing, I've decided just to disable this keys in my .emacs. Here we go:


(global-unset-key [(up)])
(global-unset-key [(down)])
(global-unset-key [(left)])
(global-unset-key [(right)])
(global-unset-key [(prior)])
(global-unset-key [(next)])
(global-unset-key [(home)])
(global-unset-key [(next)])

Про C++

2007-11-03T00:52:00.000-07:00

Плюсы -- подлый язык. Когда пишешь на C -- ставишь во главу стола производительность, когда пишешь на чем-то более высокоуровневом (будь то Python, или, прости господи, Lisp) -- пляшешь от ясного выражения концепций в конструкциях языка.

Когда же пишешь на плюсах, все время пытаешься усидеть на двух стульях разом (что вернуть -- список или итератор, или указатель? список чего? указателей или объектов? уф... и т.д. и т.п.) :( Может, с опытом это пройдет.

Managing Oracle Forms CVS tree under Powershell

2007-09-27T10:51:00.000-07:00

At work, I have to deal with Oracle Forms, which itself is a tool for creating data access forms in a visual way. This tool stores forms as a binary files (.fmb's), so managing them under ordinary version control system like CVS is a pain, because I can't see what changes were made to form (if any). In spite of that, CVS is my only available option.

Fortunately, Oracle Developer Suite has a tool, frmf2xml which can be used to convert binary FMB file to XML, so, gluing it all together (cvs, frmf2xml and ordinary diff utility) with PowerShell, I have written a script, Compare-CvsForms, which is much like cvs diff command, but works with theese binary forms. Actually, I've also written vdiff.py script, based on difflib, which is much more convenient for comparing such a messy data (XML, huh).

So, here is the Compare-CvsForms.ps1 script (I've put it to my PowerShell profile directory):


Param ([string]$formsFile)

if (cvs status $formsFile |
select-string 'Status: (Locally Modified|Needs Checkout|Needs Patch)')
{
# store working copy in another place
$local:temp_fmb = [System.IO.Path]::ChangeExtension(
 [System.IO.Path]::GetTempFileName(), "fmb")
$local:temp_xml = [System.IO.Path]::ChangeExtension(
 [System.IO.Path]::GetTempFileName(), "xml")
Move-Item -force $formsFile $local:temp_fmb
# produce xml of a working copy
frmf2xml $local:temp_fmb | out-null
if ($LastExitCode -ne 0) {
Echo "ERROR: Cannot convert working copy of form to XML!"
Move-Item -Force $local:temp_fmb $formsFile
return
}
# get repositary copy
cvs update 2>&1 | out-null
#produce xml of a repositary copy
frmf2xml $formsFile | out-null
if ($LastExitCode -ne 0) {
Echo "ERROR: Cannot convert CVS copy of form to XML!"
Move-Item -Force $local:temp_fmb $formsFile
Remove-Item -Force $xml_from
return
}
# compare xmls
$xml_from=$formsFile.substring(0, $formsFile.length-4) + "_fmb.xml"
$xml_to=$local:temp_fmb.substring(0, $local:temp_fmb.length-4) + "_fmb.xml"
vdiff.py -context $xml_from $xml_to
if ($LastExitCode -eq 0) {
$formsFile.toUpper() + ": Working copy does not differ from repositary."
}
Move-Item -Force $local:temp_fmb $formsFile
Remove-Item -Force $xml_to,$xml_from
} else {
$formsFile.toUpper() + ": Working copy is up to date."
}

And vdiff.py (pretty straightforward) script (surely, you should have Python installed, if not - you could just as easily replace this script with ordinary diff utility):


#!/usr/bin/python
import os
import sys
import difflib
import tempfile

program_name = os.path.splitext(os.path.basename(os.path.sys.argv[0]))[0].upper()
browser = "c:\\Program Files\\Internet Explorer\\iexplore.exe"

if len(sys.argv) < 3:
  print "\nUSAGE:\n\n    %s FROM-FILE TO-FILE\n" % program_name
  sys.exit(2)

if sys.argv[1] == "-context":
  context = True
  from_fn = sys.argv[2]
  to_fn = sys.argv[3]
else:
  context = False
  from_fn = sys.argv[1]
  to_fn = sys.argv[2]

try:
  from_lines = open(from_fn).readlines()
except IOError:
  print "%s: Error while reading file -- %2" % (program_name, from_fn)
  sys.exit(1)

try:
  to_lines = open(to_fn).readlines()
except IOError:
  print "%s: Error while reading file -- %2" % (program_name, to_fn)
  sys.exit(1)

differs = False
for x in difflib.context_diff(from_lines, to_lines):
  differs = True
  break

if differs:
  diff = difflib.HtmlDiff(wrapcolumn=70).make_file(from_lines, to_lines, from_fn, to_fn, context)
  diff = diff.replace("<head>",
                      '<head><meta http-equiv="Content-Type" content="text/html; charset=utf-8">')
  tmp = tempfile.mkstemp(".html")
  os.write(tmp[0], diff)
  os.close(tmp[0])
  os.spawnl(os.P_WAIT, browser, ' file://' + tmp[1] + '')
  sys.exit(1)
else:
  sys.exit(0)

Scripting with PowerShell actually is a great deal of fun :)

A little correction to Process-File function

2007-06-15T10:42:00.000-07:00

Once after processing my C++ source code tree with Process-File my sources became uncompilable. The problem was caused by redirection operator `>', which transformed source files from plain-old ASCII to Unicode. So there's corrected version which uses Out-File cmdlet instead with encoding specified explicitly:


function Process-File([scriptblock]$script, 
                      $filename, 
                      $encoding = "ascii") {
    $local:temp = New-TempFileName
    Get-Content -Encoding $encoding $filename | `
          Foreach $script | `
          out-file -encoding $encoding $local:temp
    Move-Item -force $local:temp $filename
}

Proper conversion of Mueller dictionary to Unicode

2007-04-28T01:11:00.000-07:00

Note: this article is a (bad) translation of my article on this subject written in russian.

This article describes more proper than Ubuntu (and maybe Debian) current package offers conversion of a Mueller dictionary to Unicode (UTF-8) with consequent conversion to dict format.
There is some reasons for doing this job:

Existing conversion from Mueller to dict has a few first dictionary articles corrupted (the ones that describe dictionary terms). I suppose this was caused by executing conversion script (to-dict) in the wrong locale. It should be executed in KOI8-R, not UTF-8.
More to that, original dictionary contains not just plain KOI8-R text, but SilIPA-encoded word transcriptions too. These transcriptions should be handled separately with "SilIPA-to-Unicode IPA Extensions" conversion.
More to that, the script to-dict itself contains a little bug, which caused some words starting with non-alphabetical character (dictionary terms, other words starting with dash or apostroph -- like 'cause, -shaped) to be excluded from dictionary index at all!

So I've decided to convert dictionary by myself. If you want to convert dictionary using my method, you'll have to download my script correct-translation.pl and corrected script to-dict (corrections -- script text converted to UTF-8 and options --allchars --utf8 in dictfmt call).

So, lets start. First of all, create temporary directory and put there original dictionary and those scripts:

~$ mkdir /tmp/mutf
~$ cd /tmp/mutf/
/tmp/mutf$ wget http://uj2.h15.ru/mutf/correct-translation.pl
...
/tmp/mutf$ wget http://uj2.h15.ru/mutf/to-dict
...
/tmp/mutf$ wget http://some.where/Mueller7GPL.koi
...
/tmp/mutf$ chmod 755 correct-translation.pl to-dict
/tmp/mutf$ ls -l
итого 5608
-rwxr-xr-x  1 andrey andrey     531 2005-07-06 16:20 correct-translation.pl
-rw-r--r--  1 andrey andrey 5714222 2005-07-06 16:23 Mueller7GPL.koi
-rwxr-xr-x  1 andrey andrey    6302 2005-07-06 16:22 to-dict
/tmp/mutf$

Note: sorry, I can't put this scripts to some more "permanent" storage, but I can mail them to you if you can't download them.
Now, we're going to translate dictionary to UTF-8. Firstly, we need to convert dictionary using iconv and then, we need to correct transcriptions with correct-translation.pl:

/tmp/mutf$ cat Mueller7GPL.koi | iconv -f koi8-r -t utf-8 | \
> ./correct-translation.pl > Mueller7GPL.utf && rm -f Mueller7GPL.koi
/tmp/mutf$

Now we will convert dictionary to dict format:

/tmp/mutf$ ./to-dict --src-data Mueller7GPL.utf mueller7.data
Writing the header of mueller7.data..
Formatting data (Mueller7GPL.utf -> mueller7.data)..
.
/tmp/mutf$ ./to-dict --data-dict mueller7.data mueller7 && rm -f mueller7.data
dictfmt: mueller7.data -> mueller7.dict and mueller7.index..
46185 headwords
Compressing mueller7.dict..
.
/tmp/mutf$

Now we just need to install dictionary:

/tmp/mutf$ cp mueller7.* /usr/share/dictd/
/tmp/mutf$ dictdconfig -w &&amp; (killall dictd; dictd)

Resulting article can look like this:

Integrating Emacs kill-ring and Windows clipboard

2007-02-12T09:02:00.000-08:00

Since I've started using Emacs on Windows, there was one little problem irritating my mind everyday -- by default, GNU Emacs 22 separates kill-ring and Windows clipboard contents, making it REALLY hard to use clipboard -- you can paste text by only pressing mouse-3 (the wheel) on editor window, and you can copy text only by selecting it with your mouse -- no handy C-y, no integration with kill ring.

So I did a little "investigation". There are two variables controlling mentioned integration -- interprogram-cut-function and interprogram-paste-function (described in emacs online documentation, which see). First of which holds function used to send killed text to clipboard, and the second one holds function used to get fresh clipboard text back from clipboard and put it to kill-ring. More to that, file w32-fns.el in your emacs site-lisp tree contains the next (as well as definitions for those functions -- x-select-text and x-get-selection-value):

(setq interprogram-cut-function 'x-select-text)
(setq interprogram-paste-function 'x-get-selection-value)

But for some reason this file isn't loaded on Emacs startup. So, what should we do is to add some loading code to our dotemacs file (do not forget to wrap-n-protect it with some kind of Win32-only conditional -- `(Windows body ...)' macro in my case):

(Windows
(load "w32-fns"))

Voila! Now text from clipboard goes right to the kill-ring and vice versa.

Process-File function

2007-01-07T07:42:00.000-08:00

Some time ago while scripting (either bash or PowerShell) I've found myself using some pattern too often. It can be expressed as

take a file;
process every line in it;
save file back.

E.g. `sed -i' provides this kind of functionality. This was kind of boring, because you can't just write "cat file.txt | do-something > file.txt", and if you really could, this was too long anyway. So I wrote function which does just that -- takes file, processes it with some code (specified as an argument), and writes it back. Here it goes:

function Process-File([scriptblock]$script, $filename) {
    $local:temp = [System.IO.Path]::GetTempFileName()
    Get-Content $filename | Foreach $script > $local:temp
    Move-Item -force $local:temp $filename
}

Here some testing code:

PS C:\Documents and Settings\andrey> cat test.txt
PS C:\Documents and Settings\andrey> echo "abcabc" > test.txt
PS C:\Documents and Settings\andrey> cat test.txt
abcabc
PS C:\Documents and Settings\andrey> Process-File { $_ -replace 'a','b' } test.txt
PS C:\Documents and Settings\andrey> cat test.txt
bbcbbc
PS C:\Documents and Settings\andrey>

PowerShell is just wonderfull.