- supervisor is in the epel repository.
- python-setuptools is optional only if superlance is to be installed, superlance is an monitoring alert built for supervisor.
yum -y install epel-release yum install -y supervisor python-setuptools easy_install superlance
The following creates the configuration for one node of dse cassandra. Please note cassandra is started with -f because any process to be controlled by supervisor needs to be running in the foreground.
cat >> /etc/supervisord.d/dse1.ini << EOF [program:dse] command=/opt/dse/bin/dse cassandra -f ; the program (relative uses PATH, can take args) process_name=%(program_name)s ; process_name expr (default %(program_name)s) numprocs=1 ; number of processes copies to start (def 1) directory=/opt/dse/ ; directory to cwd to before exec (def no cwd) priority=1 ; the relative start priority (default 999); autorestart=true ; retstart at unexpected quit (default: true) startsecs=30 ; number of secs prog must stay running (def. 1) ;startretries=3 ; max # of serial start failures (default 3) ;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2) ;stopsignal=QUIT ; signal used to kill process (default TERM) ;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10) user=cassandra ; setuid to this UNIX account to run the program redirect_stderr=true ; redirect proc stderr to stdout (default false) stdout_logfile=/var/log/dse/%(program_name)s-stdout.log ; stdout log path, NONE for none; default AUTO EOF
supervisor's configuration file is in /etc/supervisord.conf, the following are a minimal configuration to uncomment. inet_http_server section is to enable the web admin user interface, this should be either disabled in production or enable user and password security. The include section will take all .ini configuration under /etc/supervisord.d/, to allow application team/user to manage these configuration, ownership or permission should be assigned properly in this directory or configuration files.
[unix_http_server] file=/var/run/supervisor/supervisor.sock ; (the path to the socket file) [inet_http_server] ; inet (TCP) server disabled by default port=0.0.0.0:9001 ; (ip_address:port specifier, *:port for all iface) [supervisorctl] serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL for a unix socket [include] files = supervisord.d/*.ini
One of the potential issue running DSE by supervisor is that supervisor inherit the ulimits of the process from systemctl regardless what the system ulimit sets to in /etc/security/limits.*. supervisor offers the following configuration options but it does not cover other parameters such as memlick, stack, rss and etc. this could create issues such as when using DSE's dsefs. minfds=100000 ; (min. avail startup file descriptors;default 1024) minprocs=32768 ; (min. avail process descriptors;default 200)
The workaround of this issue, we have to adjust the systemctl limits by the following
mkdir -p /etc/systemd/system/supervisord.service.d/ cat > /etc/systemd/system/supervisord.service.d/filelimit.conf << EOF [Service] LimitNOFILE=1048576 LimitNPROC=600000 LimitMEMLOCK=infinity LimitFSIZE=infinity LimitDATA=infinity LimitRSS=infinity LimitSTACK=infinity LimitCPU= infinity LimitAS=infinity LimitLOCKS=infinity LimitSIGPENDING=infinity LimitMSGQUEUE=infinity EOF systemctl daemon-reload systemd-delta --type=extended
restart supervisor:
systemctl restart supervisor
user with necessary sudo access could control DSE by supervisor:
sudo supervisorctl status sudo supervisorctl start dse sudo supervisorctl stop dse sudo supervisorctl restart dse
Unix admin have to configure the following: allow permission to service account owner to manage /etc/supervisord.d/
chmod -R svc_cassandra:fleet /etc/supervisord.d
provide sudo access in /etc/sudoers.d/cassandra
svc_cassandra ALL=NOPASSWD: /usr/bin/systemctl * supervisord svc_cassandra ALL=NOPASSWD: /usr/bin/supervisorctl *
