Troubleshooting And Monitor Linux Processes With Proc And Ps


In Linux OS, ‘/proc’ is a virtual filesystem which doesn’t contain real files but runtime system information.

Some of the files under /proc/PID/ (Process ID) are helpful to investigate process related issues.

1.1 Common ways of finding a PID are:

ps aux (or -elf) | grep -i some_program_name

lsof -nPi:some_port_number

check pid file under /var/run/program_name/

1.2 Common files which would be helpful to investigate an issue:

1. /proc/PID/ulimits

‘ulimits’ shows all limits set for the specified PID.

Checking this file would be useful when issue might be misconfigured or unset of some particular limit

2. /proc/PID/io

By checking numbers in this file a couple of times periodically would be useful to check if this process is actually reading/writing or not and if it’s busy due to I/O More detail

3. /proc/PID/environ

This shows environment variable set to this process. Especially useful to check Java process’s actual CLASSPATH variable

cat /proc/PID/environ | tr '\0' '\n'

Above shows the values with nicer format, for example:

[root@node2 ~]# cat /proc/5670/environ | tr '\0' '\n' | grep ^CLASSPATH

4. /proc/PID/fd

‘fd’ is a directory which contains links to actual files.

This is useful to check which file is actually used for this process (can do same with ‘lsof’), and to check how many file descriptor has been used.

For example:

[root@node2 ~]# ls -l /proc/5670/fd | grep hadoop-common
lr-x------ 1 mapred hadoop 64 Sep  8 08:32 79 -> /usr/hdp/
lr-x------ 1 mapred hadoop 64 Sep  8 08:32 83 -> /usr/hdp/
[root@node2 ~]# ls -l /proc/5670/fd | wc -l

5. /proc/PID/status

This file contains the specified PID’s overview status which contains memory usage

2 PS : Get total memory and cpu used by a user

 ps -U [user] -o user,pid,cmd,%cpu,%mem | awk -v date1="$(date +"%s")" '{print date1, $0}'
comments powered by Disqus