Saturday, August 16, 2014

A small sample Linux/Unix Program for a Delayed Look at Processes

This is a demonstration of a way to customize a Unix/Linux tool a little bit.

The problem is to use the ps command to find a misbehaving process. The misbehaving process is running as the user "nobody":
ps wwaux | grep noboody
But the thing that makes the problem especially difficult is that the process is misbehaving when the system is shutting down, and all the users have been logged off.

So you need to put your call to ps in the shutdown scripts. But you don't really know where, so you need to repeatedly call it until the system shuts down:

for (( ct = 0; ct < 100; ++ct )) 
do ps wwaux | grep nobody;
done
But you really need the call to ps to be running concurrently with your shutdown process. What you want is a tool that calls ps repeatedly, so you can call it like this from your shutdown script:
/usr/local/bin/findps nobody &
But it would be nice to be able to tell it a maximum number of times to repeat:
/usr/local/bin/findps nobody 100 &
And, to help you find the output, you would like to put the output somewhere separate from the system's usual shutdown chatter:
/usr/local/bin/findps nobody 100 > /var/log/foundps.log &
Here's one way to do it in C:

==================================
/* findps.c,
// a Q&D program to repeatedly look for something in the output of ps wwaux.
// Written by and copyright Joel Rees, Amagasaki, Japan, August 2014.
//
// Permission to use granted if the following three conditions are all met:
// Don't try to steal my copyrights.
// Don't blame me if it gets stuck or goes boom.
// Do use it for good purposes, by your definition of good.
*/


#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <unistd.h>

/*
# Compile it like this:

cc -Wall -o findps findps.c

# ----------
# Test it like this:

./findps nobody

# ----------
# Install it something like this:

chmod o-rwx findps
sudo cp -i findps /usr/local/bin/
sudo chown root:root /usr/local/bin/findps
sudo touch /var/log/foundps.log

# To catch the culprit running as nobody
# and holding up your shutdown processes,
# call it in your shutdown script something like this:

/usr/local/bin/findps nobody 100 > /var/log/foundps.log &

# See also
man ps
# and look for where to find the numeric process id.
# If it gets out of control,
# getting another terminal session should help. Then
man kill
# Also, the return key and
# ctrl-s
# may be useful. But
# ctrl-z
# might not be so useful. But
man bg
# anyway. Or, I mean,
man sh
# and remember to search with the "/" command:
# /bg
# /job
# and so forth.

# An equivalent shell script, without parameter checks, etc. might look like this:
# ------------
#! /bin/bash
lim=$2
for (( ct = 0; ct < lim; ++ct )) ; do ps wwaux | grep $1; sleep 1; done
# ------------
*/


#define BIGCOMMANDSZ 1024


int main( int argc, char * argv[] )
{
    char cmdbuff[ BIGCOMMANDSZ + 1];
    int limit = 10; /* Maximum times through the loop. */

    if ( argc < 2 )
    {    fprintf( stderr, "Usage: %s <search-term> [<limit-count>]\n", argv[ 0 ] );
        fprintf( stderr, "\tDefault limit count is %d\n", limit );
        return EXIT_FAILURE;
    }
    if ( argc > 2 )
    {    limit = (int) strtol( argv[ 2 ], NULL, 10 );
    }
    cmdbuff[ BIGCOMMANDSZ ] = '\0';
    strncpy( cmdbuff, "ps wwaux | grep ", BIGCOMMANDSZ );
    strncat( cmdbuff, argv[ 1 ], BIGCOMMANDSZ );
    for ( ;limit > 0; --limit )
    {    system( cmdbuff );
        sleep( 1 );    /* So it doesn't run away too fast. */
    }
    return EXIT_SUCCESS;
}
==================================

That's basically as I posted it to the debian-user list.

Expanding on the shellscript version idea, but leaving out most of the comments:

==================================
#! /bin/bash

# Don't forget to do this:

# chmod ug+x,o-rwx findps.sh

# Note that bash may be heavier than you want to run at shutdown time.

argc=$#
limit=10

if ((argc<1))
then
  echo "Usage $0  <search-term> [<limit-count>]"
  echo "    Default limit count is $limit"
  exit
fi

if ((argc>1)) ; then limit=$2 ; fi

for (( ct = 0; ct < limit; ++ct ))
do
  ps wwaux | grep $1
  sleep 1
done
==================================

Comparing the weight of the C version and the bash version, the C version should be much lighter weight, and more appropriate to running at shutdown time.

Friday, May 30, 2014

Why checksums? -- チェックサムのワケ

If you go to download an app, say, the binary for The GIMP, you may find both the very large file containing the application binary and the very small checksum file.

アプリ(ここではザギンプを例にします。)をダウンロードしようと、バイナリの巨大なファイルとともに、チェックサムの極小さなファイルも見受ける場合があります。

Specifically, for the MSWindows application binary of The GIMP, as I type this, you would
  1. navigate to the list of mirrors
  2. select a mirror somewhat close to you, 
  3. maybe click on a "gimp" link on the mirror, 
  4. then find the link for the latest version (2.8, as of this writing), 
  5. click on the "windows" version directory there, 
  6. and finally you see the link for the downloadable binary, gimp-2.8.10-setup.exe (for version 2.8).
詳しく言えば、MSウィンドーズ用のザギンプのアプリケーションバイナリなら、現時点は
  1. ミラのリストに移動してから
  2. 自分に近いミラの一つを選んで
  3. 場合によって、更にミラの中の "gimp" のリンクをクイックして
  4. 最新バージョン(現時点は 2.8 )のリンクを探して
  5. その中の "windows" バージョンのリンクをクリックして
  6. やっとダウンロード可のバイナリの(2.8版の場合は) gimp-2.8.10-setup.exe を見つける。
If you are in Japan, maybe you arrived at the the ring server or the u-aizu server.

日本にいらっしゃる場合はもしかして ring のサーバもしくは u-aizu のサーバにたどり着いたかも知れません。

And you also see the file gimp-2.8.10-setup.exe.md5. What is that? It's very small, and if you open it with a text editor, all it contains is something strange like
c14fc68cd6b89313da3266210bc73b1f *gimp-2.8.10-setup.exe
It's the checksum for the MSWindows version download. (Yes, this was the actual checksum as of 26 April 2014.)

そして、gimp-2.8.10-setup.exe.md5 と言うファイルも見つけるでしょう。何なのでしょう?ホンマに小さいし、開こうと
c14fc68cd6b89313da3266210bc73b1f *gimp-2.8.10-setup.exe
のような妙な文字しか入っていない。これはMSウィンドーズバージョンのダウンロードのチェックサムです。(実際の 2014年 4月 26日の時点の本当のチェックサムです。)

So, what's it good for?

では、何の役に立ちますか?

There are two kinds of problems that can occur when you download something. If either occurs, you end up with a bad copy.  Several decades back, we used simple checksums to give us a hint about such errors. The server would add up the numeric bytes of the download and post the sum. The person downloading would also add them up and check the answer. If they didn't match, we knew we had a bad copy.

ダウンロードする際に起きたりする2種類の問題がある。どちらも起きたら、結局、利用しない方が良い複製分が手に入ってしまいます。数十年前の話ですが、そんなダウンロード失敗のヒントとして、単純型のチェックサムを活躍させました。サーバはダウンロード分のバイト数値の合計を足して行って、その合計を報せてくれるのです。ダウンロードする側も足して行って、ダウンロード済みの合計を照らし合わせました。等しくなければ、複製分が不当だったことが判りました。

And that's why it's called a checksum. It's a sum for checking.

チェックサムの意味は「照らし合わせ用の合計」です。

The checksums we used back then were very short (two or four hexadecimal digits, like "c1" or c14f") among other problems. They weren't even trying to be anywhere close to perfect.

その当時使っていたチェックサムは短かい(「C1」や「C14F」の様に2桁か4桁の)ものだったし、他の問題もありました。我々は理想的なチェックサムを考えていませんでした。

So, if the checksums matched, we didn't really know we had a good copy, but we had more confidence in the copy than we otherwise would have had.

等しい場合は、妥当な、利用可の複製分だと、本当に判った訳ではなかった。ただ、何もないよりは信頼できると思ったのです。

As you see above, modern checksums are much longer. They also use techniques derived from cryptography, so that copy errors or substitution by bad guys is much more likely to produce a bad checksum.

上記の例にお解りになると思いますが、現代風のチェックサムは昔よりもずっと長くなっています。その上に、暗号技術を借りて、複製の失敗も、悪奴の不正置き換えもチェックサム異常を起こす確率がはるかに上回っています。

By the way, there are several ways to check the checksum.

チャックサムを確認するには幾つかのやり方があります。

If you are using a typical Linux or BSD distribution (debian is the one I'm using right now.), you can calculate the checksum of the file you downloaded by typing one of the following command lines in a shell:
  • For MD5 checksums: md5sum gimp-2.8.10-setup.exe
  • For SHA1 checksums: sha1sum gimp-2.8.10-setup.exe
  • For SHA256 checksums: sha256sum gimp-2.8.10-setup.exe
大概のリナックス及び BSD 類のオペレーティングシステムの配布を活躍している場合(ボクは現在、デビアンを利用しています。)は以下のようなコマンドをシェルのコマンド行に打ち込むことでチェックサムを計算できます。
  • MD5 チェックサムに: md5sum gimp-2.8.10-setup.exe
  • SHA1 チェックサムに: sha1sum gimp-2.8.10-setup.exe
  • SHA256 チェックサムに: sha256sum gimp-2.8.10-setup.exe
You are wondering why you would want to download an app installer for MSWindows apps on a Linux or BSD box? As far as I know, Microsoft doesn't give you any good built-in ways to check these. So you end up with a chicken-and-egg problem about which to install first, which I need to blog about sometime. But you can download the gimp or whatever app on your Linux box and check the checksum there, and then copy it to your MSWindows box.

どうしてMSウィンドーズ用のアプリのインストーラを Linux/BSD器でダウンロードするかが気になりますか?ボクのわかっている限り、マイクロソフト社のオペレーティングシステムには、出荷のまま、確認用の道具を含めているわけではありません。したがって、鶏と卵の問題になります。この問題についてまた投稿しますが、 Linux器のお持ちの場合、その機械でダウンロードして確認すればいいでしょう。そしてMSウィンドーズ器に持っていけます。

GNU Privacy Guard (gnupg or gpg), Pretty Good Privacy (GPG), and other OpenPGP-compliant tools are available for many OSses, including MSWindows. In the case of GNU Pricacy Guard, the command lines look like this:
  • For MD5 checksums: gpg --print-md md5 gimp-2.8.10-setup.exe
  • For SHA1 checksums: gpg --print-md sha1 gimp-2.8.10-setup.exe
  • For SHA256 checksums: gpg --print-md sha256 gimp-2.8.10-setup.exe
And there is a graphical front-end for gnupg for MSWindows, called gpg4win.

OpenSSL also has commands for checking checksums:
  • For MD5 checksums: openssl dgst md5 md5 gimp-2.8.10-setup.exe
  • For SHA1 checksums: openssl dgst sha1 gimp-2.8.10-setup.exe
  • For SHA256 checksums: openssl dgst sha256 gimp-2.8.10-setup.exe

There should be a question nagging you right now.

This is great for random download errors, but what about back-doored apps put up by the bad guys? If the bad guy can put a back-doored app up on a mirror, she/he can also put up a valid checksum for the back-doored app.

Think outside the box. If the bad guys get into one server, how many other servers are they likely to get into? And how about the project server?

So what you do is get the checksum from the server you downloaded the app from, then get the checksum from the project's primary server and compare. Then, just to be sure, get a copy of the checksum from a mirror or two far away. And, if the checksums have been posted to the project's mailing list(s),  The more mirrors you check, the less likely it will be that the bad guys own every server you check.

If one of the checksums doesn't match, don't install the app. Ask about the checksums on the project's mail list. Wait until things become clear. Be patient. If you don't get a good answer, give up on that app.

What if the bad guys have broken into the project's primary server? Or if they have entirely taken over the project? Or if the project's developers have gone to the dark side? Well, that's game over if you install that app.

This is another good reason for finding the mailing lists for the app and reading a lot of posts. You'll have better basis to judge if you've become familiar with the people involved.

In brief:

  • check the checksums, 
  • and don't rely on just one source of information.