Sidekiq 3 Memory Killer

mem_graph

My Sidekiq 3 installation seams to be very hungry in respect of memory. If you dig through the Net you'll find a lot of similar reports.

For e.g. the unicorn web worker a memory-killer script exists which should prevent running into Madland.

So I thought I take an example and do a similar thing for the sidekiq workers. If you give them time and send a graceful kill signal there shouldn't be any data-loss either.

#!/bin/bash

# check if script is already running
SELFCHECK=$(ps -eo pid,command |grep -v grep |grep $0 |wc -l)
if [ "$SELFCHECK" -gt 2 ]; then
  echo "$0 is already running!";
  exit 1;
fi;

NAPTIME=120
MAXRSS=$(( 4200 * 1024 )); # 4.2GB
FOUND=$(ps -eo rss,pid,command |grep sidekiq |grep -v grep |grep -v $0);

while read -r line; do
  echo -n "$line ...";
  USAGE=$(echo $line |cut -d' ' -f1);
  PID=$(echo $line |cut -d' ' -f2);

  if [ "$USAGE" -gt $MAXRSS ]; then
    echo -e " \033[0;31mrestart\033[0m";
    kill -SIGUSR1 $PID;
    # wait before we forget about the graceful stuff
    sleep $NAPTIME;
    kill -SIGKILL $PID;
    # prevent killing all worker at once
    sleep $(( $NAPTIME / 2 ));
  else
    echo -e " \033[0;32mok\033[0m";
  fi;
done <<< "$FOUND"

(GitHub Gist)

Currently it starts killing if it exceeds 4GB but you can adjust it by touching following line: MAXRSS=$(( 4000 * 1024 )); # 4GB

But there is .. so to speak .. light at the end of the tunnel. They improved sidekiq massively in respect of memory consumption. In a similar installation I use Sidekiq 4 and for this I don't need any nasty killer scripts :)

Cheers,
Lukas