Jupyter Kernel & DataStax Enterprise Spark Integration

Install DSE from tarball

tar xpvf dse-4.8.15-bin.tar.gz

These hadoop config files needs to be copied from the target DSE environment

$DSE_HOME/resources/hadoop/conf
dse-mapred-default.xml
dse-core-default.xml

Create the following temp work directories for spark

Note all users use the notebook must have write permission to the var/lib/spark, group writable permission is recommended

mkdir -p /opt/dse4/var/lib/spark/worker
mkdir -p /opt/dse4/var/lib/spark/rdd
chown -R root:fleet /opt/dse4/var/lib/spark
chmod g+w -R /opt/dse4/var/lib/spark

The following environment variables needs to be set to the DSE home

/opt/dse4/resources/spark/conf/spark-env.sh

export SPARK_WORKER_DIR="/opt/dse4/var/lib/spark/worker"
export SPARK_LOCAL_DIRS="/opt/dse4/var/lib/spark/rdd"

DataStax Enterprise Cassandra & supervisord Integration

• supervisor is in the epel repository.
• python-setuptools is optional only if superlance is to be installed, superlance is an monitoring alert built for supervisor.

  yum -y install epel-release
  yum install -y supervisor python-setuptools
  easy_install superlance
  

The following creates the configuration for one node of dse cassandra. Please note cassandra is started with -f because any process to be controlled by supervisor needs to be running in the foreground.

cat >> /etc/supervisord.d/dse1.ini << EOF
[program:dse]
command=/opt/dse/bin/dse cassandra -f               ; the program (relative uses PATH, can take args)
process_name=%(program_name)s  ; process_name expr (default %(program_name)s)
numprocs=1                     ; number of processes copies to start (def 1)
directory=/opt/dse/            ; directory to cwd to before exec (def no cwd)
priority=1                     ; the relative start priority (default 999);
autorestart=true              ; retstart at unexpected quit (default: true)
startsecs=30                    ; number of secs prog must stay running (def. 1)
;startretries=3                ; max # of serial start failures (default 3)
;exitcodes=0,2                 ; 'expected' exit codes for process (default 0,2)
;stopsignal=QUIT               ; signal used to kill process (default TERM)
;stopwaitsecs=10               ; max num secs to wait b4 SIGKILL (default 10)
user=cassandra                 ; setuid to this UNIX account to run the program
redirect_stderr=true          ; redirect proc stderr to stdout (default false)
stdout_logfile=/var/log/dse/%(program_name)s-stdout.log        ; stdout log path, NONE for none; default AUTO
EOF

supervisor's configuration file is in /etc/supervisord.conf, the following are a minimal configuration to uncomment. inet_http_server section is to enable the web admin user interface, this should be either disabled in production or enable user and password security. The include section will take all .ini configuration under /etc/supervisord.d/, to allow application team/user to manage these configuration, ownership or permission should be assigned properly in this directory or configuration files.

[unix_http_server]
file=/var/run/supervisor/supervisor.sock   ; (the path to the socket file)

[inet_http_server]         ; inet (TCP) server disabled by default
port=0.0.0.0:9001        ; (ip_address:port specifier, *:port for all iface)

[supervisorctl]
serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL  for a unix socket

[include]
files = supervisord.d/*.ini

One of the potential issue running DSE by supervisor is that supervisor inherit the ulimits of the process from systemctl regardless what the system ulimit sets to in /etc/security/limits.*. supervisor offers the following configuration options but it does not cover other parameters such as memlick, stack, rss and etc. this could create issues such as when using DSE's dsefs. minfds=100000 ; (min. avail startup file descriptors;default 1024) minprocs=32768 ; (min. avail process descriptors;default 200)

The workaround of this issue, we have to adjust the systemctl limits by the following

mkdir -p /etc/systemd/system/supervisord.service.d/
cat > /etc/systemd/system/supervisord.service.d/filelimit.conf << EOF
[Service]
LimitNOFILE=1048576
LimitNPROC=600000
LimitMEMLOCK=infinity
LimitFSIZE=infinity
LimitDATA=infinity
LimitRSS=infinity
LimitSTACK=infinity
LimitCPU= infinity
LimitAS=infinity
LimitLOCKS=infinity
LimitSIGPENDING=infinity
LimitMSGQUEUE=infinity
EOF

systemctl daemon-reload
systemd-delta --type=extended

restart supervisor:

systemctl restart supervisor

user with necessary sudo access could control DSE by supervisor:

sudo supervisorctl status
sudo supervisorctl start dse
sudo supervisorctl stop dse
sudo supervisorctl restart dse

Unix admin have to configure the following: allow permission to service account owner to manage /etc/supervisord.d/

chmod -R svc_cassandra:fleet /etc/supervisord.d

provide sudo access in /etc/sudoers.d/cassandra

svc_cassandra           ALL=NOPASSWD: /usr/bin/systemctl * supervisord
svc_cassandra           ALL=NOPASSWD: /usr/bin/supervisorctl *

[http://supervisord.org/configuration.html]