My Sidekiq 3 installation seams to be very hungry in respect of memory. If you dig through the Net you'll find a lot of similar reports.
For e.g. the unicorn web worker a memory-killer script exists which should prevent running into Madland.
So I thought I take an example and do a similar thing for the sidekiq workers. If you give them time and send a graceful kill signal there shouldn't be any data-loss either.
#!/bin/bash # check if script is already running SELFCHECK=$(ps -eo pid,command |grep -v grep |grep $0 |wc -l) if [ "$SELFCHECK" -gt 2 ]; then echo "$0 is already running!"; exit 1; fi; NAPTIME=120 MAXRSS=$(( 4200 * 1024 )); # 4.2GB FOUND=$(ps -eo rss,pid,command |grep sidekiq |grep -v grep |grep -v $0); while read -r line; do echo -n "$line ..."; USAGE=$(echo $line |cut -d' ' -f1); PID=$(echo $line |cut -d' ' -f2); if [ "$USAGE" -gt $MAXRSS ]; then echo -e " \033[0;31mrestart\033[0m"; kill -SIGUSR1 $PID; # wait before we forget about the graceful stuff sleep $NAPTIME; kill -SIGKILL $PID; # prevent killing all worker at once sleep $(( $NAPTIME / 2 )); else echo -e " \033[0;32mok\033[0m"; fi; done <<< "$FOUND"
Currently it starts killing if it exceeds 4GB but you can adjust it by touching following line:
MAXRSS=$(( 4000 * 1024 )); # 4GB
But there is .. so to speak .. light at the end of the tunnel. They improved sidekiq massively in respect of memory consumption. In a similar installation I use Sidekiq 4 and for this I don't need any nasty killer scripts :)