Running Process Environment Troubleshooting many software problems involves checking a running process. You normally check how much CPU or memory the process is using. On some systems you can also check its I/O usage, number of file handles or file descriptors, signal despositions, etc. In this note, I'll talk about the environment of a running process, something that people rarely look at but that proves very helpful in troubleshooting. We're probably familiar with shell environment variables. For instance, in Bourne or its derivative shells, $PS1 defines first level prompt; in DOS, %PATHEXT% defines the file name extensions for excutable files; many applications have their own variables (Oracle has $ORACLE_HOME etc.). In fact, any running process including a shell has various environment variables associated with it. The behavior of the process is modified by them. Suppose you have multiple installations of Oracle or Java VMs on the same machine, whether you can launch your Oracle tool or Java program may depend on where your Oracle Home is set to or where JRE is launched from. So how do you find the environment for a running process? It depends on the OS. * Solaris /usr/ucb/ps eww or in Solaris 9 and up pargs -e Here, e for environment and ww for extra wide (see `man -s 1b ps'). Note that with the first command, any user can look at any process before Solaris 9. Beginning with Solaris 9, or more precisely, Solaris with the relevant security patch, you can only look at the environment of your own process or if you're root. Solaris 10 pargs has the same restriction, or security enhancement as Sun would call it. See http://sunsolve.sun.com/search/document.do?assetkey=1-26-102215-1 One more way, quite involved though, is by using mdb. See http://blog.curthread.org/posts/solaris/mdb/display_envp.html Also see the paragraph labeled IMPORTANT below. * Linux ps eww cat /proc//environ Must run as process owner or root. The second command returns all strings jumbled together, which can be piped to a filter to separate: cat /proc//environ | perl -pne '$a=chr(0);s/$a/\n/g' Also see the paragraph labeled IMPORTANT below. * Windows In Process Explorer (available from sysinternals.com), double click the process, go to Environment tab. If the string is too long, you can highlight and copy and paste into Notepad. * Other OSes All BSD's and BSD "derivatives" accept `ps eww ' command. Failing that, see if you have gdb installed. If so, `gdb path_to_process_executable ' and enter: set variable $p = (char **)environ while *$p print *$p++ end (Note that gdb `show environment' does NOT show the attached process environment.) Without gdb, create a process core image, run strings on the core file, and find the environment variables usually toward the end of the output (possibly not very readable). gcore is available on most UNIXes. If not, send to the process a signal whose disposition is dumping core. ********** IMPORTANT ********** On UNIX and Linux, what I call running process environment is not technically accurate. It's really the environment at the time the process was created (fork() and exec()). If the process later changes its environment variables (create, unset, change value), the above method won't detect the change. (On Solaris, even the mdb method won't help.) Windows does not suffer from this problem. ********** IMPORTANT ********** If you see incorrect environment setting, correct it by referring to the documentation of the application software. If the documentation has no such information, the general solution is to start the process from a shell or batch file where the correct environment is pre-set. Suppose a UNIX or Linux process XYZ has $PATH set to mistakenly include /usr/ccs. Simply type export PATH= XYZ to launch the application Sometimes the problem is that you can't start XYZ and the solution is the same. That sounds trivial to a UNIX/Linux user, but may not be so to a Windows user. Most Windows programs are started from Start | Programs | ... How do you control the environment? It's the same answer. When you navigate to the program from Start | Programs | ..., right click to go to Properties | Shortcut (second tab), use the long strings in Target and also "Start in" (if not empty) to manually start the program. Go to DOS (Start | Run | cmd | OK). Type cd /d <"Start in" from above> set = <"Target" string> Here is the environment variable you wish to change. Make sure there's no space around "=". Many times I have to manually change %PATH% to make it much cleaner as in this example: set path=c:\windows\system32;c:\Program Files\Oracle\jre\1.3.1\bin Sometimes I explicitly set %TNS_ADMIN% to some path I want to troubleshoot Oracle TNS connection problem. One practical example of solving a problem this way is in section "Can't launch Net Manager and Config Assistant" of http://yong321.freeshell.org/oranotes/LdapInsteadOfTnsnames.txt For Windows NT Users: To find the "Start in" and Target for a shortcut in Start | Programs | ..., navigate to C: | Winnt (%SYSTEMROOT%) | Profiles | | Start Menu | Programs | ... | Properties. Obviously, you can put the three DOS commands I listed above in a batch file (XYZ.bat, e.g.) in order to save typing next time. You can also change the environment setting in Control Panel | System. But the approach outlined here avoids globally changing the environment which other programs unfamiliar to you may rely on. Note that although a messed-up path is a common cause of application problems, it's not the only one. Even for paths, the environment variable to correct may not be $PATH on UNIX/Linux; it may be $LD_LIBRARY_PATH or $SHLIB_PATH, $CLASSPATH, or even without the word "PATH" such as $LD_PRELOAD. On Windows, however, DLLs are searched in the order of %PATH%, just like EXE files; Windows has no $LD_LIBRARY_PATH equivalent. Windows has one more complication and this probably explains some of the mysteries that sometimes only a reboot can solve a problem. The above method only works for interactive processes. Windows services are a very different type of processes. Even if you change the starting shell environment, or the setting from Control Panel | System | Advanced | Environment Variables, a stopped and restarted service still uses the old environment. Let's say you notice that Microsoft SQL Server jobs are failing and Process Explorer shows that the SQLAgent.exe process seems to have incorrect %PATH%. You also find that you can execute a DTS task interactively (not by scheduling a job) if you take the above approach (run mmc.exe /c ... in a clean %PATH% DOS shell) but fail without cleaning up the shell. The problem with the restarted service even after Environment Variables are changed in Control Panel is that a service is always a child of SERVICES.EXE, which is a child of WINLOGON.EXE, which is a child of SMSS.EXE, which is a child of System. (NT doesn't have SMSS.EXE) None of these processes can be killed without crashing the server. SERVICES.EXE uses all environment variables set in Control Panel. Since these processes exist and never exit since server bootup, they still take the old environment setting after you change it in Control Panel. The only way to change their running process environment (for individual services to inherit) is by reboot after you properly set it in Control Panel. Yong Huang 2005-2007 SEE ALSO How to Check the Environment Variables for an Oracle Process (https://metalink2.oracle.com/metalink/plsql/showdoc?db=NOT&id=373303.1)