Optimizing System Resources and HySDS for Death Valley conditions

1 Mozart Instance
- 1.1 Kernel parameters: sysctl
- 1.2 RabbitMQ
- 1.3 Redis
- 1.4 Elasticsearch
- 1.5 Celery

Mozart Instance

r5.2xlarge: 64 GB RAM, 8 vCPUs, 10 Gb network

Kernel parameters: sysctl

/usr/lib/sysctl.d/mozart.conf:

# Redis and Elasticsearch
#recycle sockets quickly
net.ipv4.tcp_tw_recycle=1

# backlog setting
net.ipv4.tcp_max_syn_backlog=102400

#Max allowed with this kernel
# up the number of connections per port
net.core.somaxconn=65535

# Redis
# allow background save under low memory conditions
vm.overcommit_memory=1

# Rabbitmq
# VERY important to reuse ports in TCP_WAIT
net.ipv4.tcp_tw_reuse = 1

#Increase the tcp-time-wait buckets pool size
net.ipv4.tcp_max_tw_buckets = 360000
# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 2500

# max file descriptor
fs.file-max = 1000000

# Added to reduce the time spent on terminated connections
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time=30
net.ipv4.tcp_keepalive_intvl=10
net.ipv4.tcp_keepalive_probes=4

# Ports dedicated to clients from this server
net.ipv4.ip_local_port_range = 30000 65535

/usr/lib/sysctl.d/elasticsearch.conf:

# Elasticsearch
vm.max_map_count=262144

Running sysctl -a shows:

RabbitMQ

/etc/rabbitmq/rabbitmq.config:

[
  { kernel,
    [
      { inet_default_connect_options, [{nodelay, true}]},
      { inet_default_listen_options,  [{nodelay, true}]}
    ]},
  { rabbit,
    [
      { loopback_users, [] },
      { vm_memory_high_watermark, 1 },
      { tcp_listen_options, [
                              { backlog, 102400 },
                              { nodelay, true },
                              {linger,        {true,0}},
                              { exit_on_close, false },
                              { keepalive, true },
                              {sndbuf,        32768},
                              {recbuf,        32768}
                           ]
      }
    ]
  }
].

/etc/default/rabbitmq-server (NOTE: not used since rabbitmq-server is started up by systemd):

/etc/systemd/system/rabbitmq-server.service.d/limits.conf:

Running rabbitmqctl status should show:

Redis

/etc/systemd/system/redis.service.d/limit.conf:

/etc/redis.conf:

Elasticsearch

/etc/systemd/sytem/elasticsearch.service.d/elasticsearch.conf:

/etc/sysconfig/elasticsearch:

/etc/elasticsearch/elasticsearch.yml:

Celery

The pertinent celeryconfig settings are: