The problem is to use the ps command to find a misbehaving process. The misbehaving process is running as the user "nobody":
ps wwaux | grep noboodyBut the thing that makes the problem especially difficult is that the process is misbehaving when the system is shutting down, and all the users have been logged off.
So you need to put your call to ps in the shutdown scripts. But you don't really know where, so you need to repeatedly call it until the system shuts down:
for (( ct = 0; ct < 100; ++ct ))But you really need the call to ps to be running concurrently with your shutdown process. What you want is a tool that calls ps repeatedly, so you can call it like this from your shutdown script:
do ps wwaux | grep nobody;
done
/usr/local/bin/findps nobody &But it would be nice to be able to tell it a maximum number of times to repeat:
/usr/local/bin/findps nobody 100 &And, to help you find the output, you would like to put the output somewhere separate from the system's usual shutdown chatter:
/usr/local/bin/findps nobody 100 > /var/log/foundps.log &Here's one way to do it in C:
==================================
/* findps.c,
// a Q&D program to repeatedly look for something in the output of ps wwaux.
// Written by and copyright Joel Rees, Amagasaki, Japan, August 2014.
//
// Permission to use granted if the following three conditions are all met:
// Don't try to steal my copyrights.
// Don't blame me if it gets stuck or goes boom.
// Do use it for good purposes, by your definition of good.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
/*
# Compile it like this:
cc -Wall -o findps findps.c
# ----------
# Test it like this:
./findps nobody
# ----------
# Install it something like this:
chmod o-rwx findps
sudo cp -i findps /usr/local/bin/
sudo chown root:root /usr/local/bin/findps
sudo touch /var/log/foundps.log
# To catch the culprit running as nobody
# and holding up your shutdown processes,
# call it in your shutdown script something like this:
/usr/local/bin/findps nobody 100 > /var/log/foundps.log &
# See also
man ps
# and look for where to find the numeric process id.
# If it gets out of control,
# getting another terminal session should help. Then
man kill
# Also, the return key and
# ctrl-s
# may be useful. But
# ctrl-z
# might not be so useful. But
man bg
# anyway. Or, I mean,
man sh
# and remember to search with the "/" command:
# /bg
# /job
# and so forth.
# An equivalent shell script, without parameter checks, etc. might look like this:
# ------------
#! /bin/bash
lim=$2
for (( ct = 0; ct < lim; ++ct )) ; do ps wwaux | grep $1; sleep 1; done
# ------------
*/
#define BIGCOMMANDSZ 1024
int main( int argc, char * argv[] )
{
char cmdbuff[ BIGCOMMANDSZ + 1];
int limit = 10; /* Maximum times through the loop. */
if ( argc < 2 )
{ fprintf( stderr, "Usage: %s <search-term> [<limit-count>]\n", argv[ 0 ] );
fprintf( stderr, "\tDefault limit count is %d\n", limit );
return EXIT_FAILURE;
}
if ( argc > 2 )
{ limit = (int) strtol( argv[ 2 ], NULL, 10 );
}
cmdbuff[ BIGCOMMANDSZ ] = '\0';
strncpy( cmdbuff, "ps wwaux | grep ", BIGCOMMANDSZ );
strncat( cmdbuff, argv[ 1 ], BIGCOMMANDSZ );
for ( ;limit > 0; --limit )
{ system( cmdbuff );
sleep( 1 ); /* So it doesn't run away too fast. */
}
return EXIT_SUCCESS;
}
==================================
That's basically as I posted it to the debian-user list.
Expanding on the shellscript version idea, but leaving out most of the comments:
==================================
#! /bin/bash
# Don't forget to do this:
# chmod ug+x,o-rwx findps.sh
# Note that bash may be heavier than you want to run at shutdown time.
argc=$#
limit=10
if ((argc<1))
then
echo "Usage $0 <search-term> [<limit-count>]"
echo " Default limit count is $limit"
exit
fi
if ((argc>1)) ; then limit=$2 ; fi
for (( ct = 0; ct < limit; ++ct ))
do
ps wwaux | grep $1
sleep 1
done
==================================
Comparing the weight of the C version and the bash version, the C version should be much lighter weight, and more appropriate to running at shutdown time.
