Performance monitoring with Sysdig

The Sysdig software seems highly geared towards containers like Docker but it works just as well on bare metal. The software is scriptable much like dtrace is.

In order to work with this utility we must first install it.

Download the following playbook, set the correct IP address in the ansible hosts file and run it:

git clone https://gitlab.com/youvegotmoxie/sysdig-ansible.git
cd sysdig-ansible/
$ ansible-playbook -u root playbook.yml

Using sysdig:

See the top processes in terms of network bandwidth usage:

# sysdig -c topprocs_net
Bytes Process PID
——————————————————————————–
71.88KB php-fpm 2865
71.70KB memcached 1774
760B nginx 2148
68B sshd 4608

List processes that are using files by highest first, use ctrl+c to display statistics after the command is run:
# sysdig -c fdcount_by proc.name “fd.type=file”
lfd – processin 211
leechprotect 186
httpd 92
service 19
sh 11
lfd 10
sed 6
cat 5
python2.7 5
grep 5
nscd 4
basename 4
tailwatchd 3
env 3
consoletype 2
sshd 2
bash 2

Show all incoming nginx connections:
# sysdig -p”%proc.name %fd.name” “evt.type=accept and proc.name=nginx”
nginx 67.23.232.183:28198->192.34.58.244:https
nginx 67.23.232.183:redstorm_join->192.34.58.244:https
nginx 67.23.232.183:20087->192.34.58.244:https
nginx 67.23.232.183:24520->192.34.58.244:https
nginx 67.23.232.183:8980->192.34.58.244:https
nginx 67.23.232.183:31640->192.34.58.244:https

See the top files in terms of read+write bytes:
# sysdig -c topfiles_bytes
Bytes Filename
——————————————————————————–
63.97KB /var/www/html/blog/wp-includes/script-loader.php
32.00KB /var/mysqltmp/#sql_792_0.MAD
31.98KB /var/www/html/blog/wp-includes/class-wp-term-query.php
20.12KB /var/www/html/blog/wp-content/w3tc-config/master.php
17.50KB /var/mysqltmp/#sql_792_0.MAI
8.00KB /var/www/html/blog/wp-content/themes/screenr/style.css
2.00KB /proc/interrupts
1.18KB /proc/stat
817B /var/log/nginx/blog.donthurt.us-access_log
165B /dev/ptmx

 

Show where a process is spending the most time:
# sysdig -c topfiles_time proc.name=exim
Time Filename
——————————————————————————–
566us /var/spool/exim/db/wait-remote_smtp
240us /etc/pki/tls/cert.pem
113us /var/spool/exim/input/1cmS7T-0001Jg-NB-J
80us /var/log/exim/main.log
51us /var/spool/exim/msglog/1cmS7T-0001Jg-NB
43us /var/spool/exim/db/retry
41us /proc/meminfo
26us /home/mike/Maildir/tmp/1489181440.H114196P5075.centoscloud.donthurt.us
24us /etc/services
18us /var/spool/exim/input/1cmS7T-0001Jg-NB-D

 

Display file I/O calls that take longer than 1ms to complete:
sysdig -c fileslower 1
evt.datetime proc.name evt.type LATENCY(ms) fd.name
———————– ———— ——– ———— —————————————–
2017-03-10 16:32:20.318 rsync read 6 /usr/src/kernels/3.10.0-514.10.2.el7.x86_64/include/linux/fcdevice.h
2017-03-10 16:32:20.324 rsync read 5 /usr/src/kernels/3.10.0-514.10.2.el7.x86_64/include/linux/fd.h
2017-03-10 16:32:23.588 rsync read 1 /usr/src/kernels/3.10.0-514.10.2.el7.x86_64/include/linux/firmware.h
2017-03-10 16:32:27.453 rsync read 5 /usr/src/kernels/3.10.0-514.10.2.el7.x86_64/include/linux/frame.h

All sysdig filters can be saved to a log file for later viewing:
# sysdig -c fileslower 1 -w trace.scap

They can be read using the following command, replace the filter as needed depending on the output captured:
# sysdig -c fileslower 1 -r trace.scap

  • Sysdig also comes with a bunch of pre-made filters which they call chisels.
  • You can get a list of all filters using sysdig -cl
  • You can get information on a filter using the following syntax:
    # sysdig -i $chisel_name
    # sysdig -i bottlenecks

Category: Performance
———————
bottlenecks Slowest system calls

Lists the 10 system calls that took the longest to return during the capture interval.
Args:
(None)

To run a filter use the -c flag:
# sysdig -c spy_users
5552 17:59:36 root) /usr/bin/id -un
5552 17:59:36 root) /usr/bin/hostname
5552 17:59:36 root) /bin/sh /usr/libexec/grepconf.sh -c
5552 17:59:36 root) grep -qsi ^COLOR.*none /etc/GREP_COLORS
5552 17:59:36 root) /usr/bin/tty -s
5552 17:59:36 root) /usr/bin/tput colors
5552 17:59:36 root) /usr/bin/dircolors –sh /etc/DIR_COLORS
5552 17:59:36 root) /usr/bin/grep -qi ^COLOR.*none /etc/DIR_COLORS
5552 17:59:36 root) /usr/bin/id -u
5552 17:59:37 root) ls –color=auto