Manage service groups in multi-server environment

Use the simple tool s2s to start/stop and manage multi-component services (service groups) in a multi-server environment. 

The s2s tool is especially applicable for censhare installations, for any services which are used via virtual IP addresses and services, which move between servers but need their own virtual disk from a SAN or NAS.

All service groups are configured in this environment and run on the assigned server. Functions exist to start, stop, status service groups and check and distribute configuration. Administration can be done from any but only from a single host at one time. Configuration is symmetric. Software and configuration information is distributed without special effort.

The basic script is written in bash for maximal compatibility at server startup and shutdown.

Installation

Get s2s from GIT and copy to server (e.g. at /tmp)

• Get s2s-install-update.tar.gz.

• Copy via scp to server.

scp ~/Downloads/scripts_censhare_s2s-install-update.tar.gz root@<server>:/tmp/

Installation of - s2s-install-update.tar.gz is required (e.g. at /tmp)

• Change to /etc, unpack, and run init config:

  CODE

  cd /etc && tar -xzf /tmp/scripts_censhare_s2s-install-update.tar.gz && ./s2s/bin/s2s init

Update of - s2s-install-update.tar.gz is required (e.g. at /tmp)

• Backup, reinstall, and move configs back.

  CODE

  mv /etc/s2s /etc/s2s_saved cd /etc && tar -xzf /tmp/s2s-install-update.tar.gz 
  cp /etc/s2s_saved/sysconfig/* /etc/s2s/sysconfig/ 
  cp /etc/s2s_saved/init.d/* /etc/s2s/init.d/ 
  cp /etc/s2s_saved/config /etc/s2s/ 
  cp /etc/s2s_saved/status /etc/s2s/

Integrate s2s into the run levels

• Solaris:

  CODE

  cd /etc && ln -s /usr/bin/s2s rc3.d/S50s2s; 
  cd /etc && for i in 0 1 2 S; do ln -s /usr/bin/s2s rc$i.d/K36s2s; done

• Linux:

Use Runlevel Editor

Configuration

Configuration Example

• Add second host to s2s

  CODE

  s2s addhost scripting-app-srv
  

• Create service group for censhare/Oracle

  CODE

  s2s addsg css 
  s2s addsg ora

• Add simple service to sg css/ora

  CODE

  s2s addss css 
  s2s addss ora

Create service script

• To build a startup script for a new service, copy s2s/init.tpl/aa_startup.template to a meaningful named text in same directory and prepare a default config template based on s2s/sysconfig.tpl/aa_config.template with the same name plus .tpl extension.

• Read the comments in startup script and have a look at the scripts already installed.

• Add the necessary code.

• If external config files are required, store them in /etc/s2s/ tree, preferable in /etc/sysconfig/, and manually make symlink references on all servers to the new s2s-location. All files below /etc/s2s/ are synced to all servers on s2s config changes or if requested with s2s -f sync - source is always the machine from witch s2s is called. There is no check whether the config file on another server is newer!

• Change config files manually only at server with admin right! Always do a s2s -f sync on any config changes before testing.

• These scripts are useable with or without s2s. Without s2s, pseudo comments are very helpful to insert service into SuSE/Novell runlevel framework.

Concept

Documentation

• s2s -h

Glossary

• Host group (hg): All hosts (servers) which may run one or more service groups within this environment constitute the host group.

• Service group (sg): A service group consists of some simple services and runs on one server. It could only be completely moved to another one. Multiple service groups may run on different servers within the host group and maybe moved independently.

• Simple service (ss): A simple service supplies a facility. It's possible to check for appropriate configuration, startup, shutdown and get a status from.

Concept

• Service groups may move from server to server, simple services do not. They are bound to their group.

• Configuration is held common (symmetric) over all servers in the host group.

• Administration may be done from any participating server - but the right to admin is restricted to one server at a time.

• The current state, which service group is dedicated to run on which server, is held in a serialized config file. Servers with an outdated config have to be resynced before executing any local task (check and handle outdated config has to be done).

• Not included is functionality to synchronize data for simple services. If a service group moves from one to another host, the admin is responsible for adequate configuration. Hooks for pre- and post-movements are available.

• There are no special precautions to avoid and handle split-brain conditions - be careful working in situations when not all servers are online.

Filesystem structure

• Configuration is held in /etc/s2s and below. This path is called $S2S_BASEDIR and is derived from main command name s2s. If that should be changed to allow the use of a second independent host group, set this variable externally and use another command name.

• Below the base directory, there are some files and directories:

  • bak/: Holds versioned backup of s2s.

  • config: main configuration file

  • config.bak: main configuration file backup

  • status: main status file, serialized

  • .lock: flag file contains hostname for current admin access

  • .locked: flag file to inhibit concurrent admin access

  • init.d/: contains for startup scripts (maybe links into ../init.tpl)

  • init.tpl/: associated templates

  • log/: Log files where s2s calls are written for every host.

  • sysconfig/: configuration files for startup scripts

  • sysconfig.tpl/: associated templates

  • bin/: scripts for additional use, ie movement of service groups

• Generally, startup scripts and configuration files are identical overall servers. But if there is a file with a hostname extension, for example, mount.css-db, then on server css-db this script is used for start/stop/status of the mount.

• Startup scripts in init.d get a well-defined $CONFIG_FILE variable containing a script to be sourced

Local files

Put local files (e.g. Helios licenses or Oracle Listener files with hostname inside) in /etc/s2s_local and link them.

Configuration

Due to limitations of bash variable naming: Use simple names for service groups and simple services.

Start with a letter followed by letters, digits and '_' as the only allowed special char (EBNF: id := letter {(letter|digit|'_')}.).