ArsDigita Server Architecture Setup Instructions

by Tracy Adams and Greg Haverkamp, a companion to ArsDigita Server Architecture

bug reports: acs-bugs@arsdigita.com
philosophy: ArsDigita Server Architecture

This document describes the installation procedures for a server conforming to the ArsDigita Server Architecture.

Pre requisites

This document assumes you have a UNIX server preconfigured with the following:

Oracle 8 (or higher)
AOLserver
ArsDigita Oracle driver for AOLserver
A working MTA (Mail Transfer Agent) that is willing to relay from all IP addresses on the machine.

Checklist for a properly installed ArsDigita Server Architecture

Web site responding
Nightly database exports succeeding
Machine backups are being performed
KeepAlive sends alerts and restarts server when the virtual server is down or can't reach the database
Uptime sends alerts when outside connectivity to machine fails
Staff server is properly installed
Reporte is producing log reports
WatchDog is monitoring server log
Cassandrix is monitoring disk space conditions
Cassandracle is monitoring the Oracle server (not necessary if you use Cassandracle inside ACS)
MTA Monitor is monitoring the mail server
Development server is being utilized

Installation steps

All ArsDigita services are given a unique name. Throughout the rest of this document, we will be using service_name as the name of the service. You will want to replace instances of service_name with the real name of the service you're creating.

Decide on the physical machine for your web service. Your team leader should know this.
Setup a system account for your use when working on the system.
This requires root access to the machine. One team member should be in charge of root commands.
- Connect to the machine as root using ssh:
  ssh -l root machine_name
- Using useradd, create your account with primary group arsdigit and secondary group dba
  useradd -g arsdigit -G dba -m username
  (see man useradd for more details about this step.)
- Select a password for your account
  passwd username
Prepare Oracle for ACS
- Connect to oracle using svrmgrl
  Note: you must be in the dba group to use svrmgrl (this should have been accomplished in step 2.)
  $ svrmgrl Oracle Server Manager Release 3.1.5.0.0 - Production (c) Copyright 1997, Oracle Corporation. All Rights Reserved. Oracle8i Enterprise Edition Release 8.1.5.0.0 - Production With the Partitioning and Java options PL/SQL Release 8.1.5.0.0 - Production SVRMGR> connect internal Connected.
- Find where the tablespaces are stored
  SVRMGR> select file_name from dba_data_files;
  Example results:
  /ora8/m01/app/oracle/oradata/ora8/system01.dbf /ora8/m01/app/oracle/oradata/ora8/oemrep01.dbf /ora8/m01/app/oracle/oradata/ora8/rbs01.dbf /ora8/m01/app/oracle/oradata/ora8/temp01.dbf /ora8/m01/app/oracle/oradata/ora8/users01.dbf /ora8/m01/app/oracle/oradata/ora8/indx01.dbf /ora8/m01/app/oracle/oradata/ora8/drsys01.dbf /ora8/m02/oradata/ora8/jsc01.dbf
- Using the above output, you should determine where to store your tablespace. As a general rule, you'll want to store your tablespace on a mount point (the m0n subdirectories in the above listing) separate from the Oracle system data files. In this example, we'll use /ora8/m02/oradata/ora8/.
- Create a tablespace for the service. It is important that the tablespace can autoextend. This allows the tablespace's storage capacity to grow as the size of the data grows. We set the pctincrease to be a very low value so that our extents won't grow geometrically. We do not set it to 0 at the tablespace level because this would affect Oracle's ability to automatically coalesce free space in the tablespace.
```
SVRMGR> create tablespace service_name datafile '/ora8/m02/oradata/ora8/service_name01.dbf' size
50m autoextend on default storage ( pctincrease 1);
```
- Create a database user for this service. Give the user access to the tablespace and rights to connect. (Substitute database_password with the appropriate password.)
```
SVRMGR> create user service_name identified by database_password default tablespace service_name
temporary tablespace temp quota unlimited on service_name;

SVRMGR> grant connect, resource, ctxapp, javasyspriv to service_name;

SVRMGR> revoke unlimited tablespace from service_name;

SVRMGR> alter user service_name quota unlimited on service_name;
```
- Run "sqlplus service-name" to make sure your Oracle user works correctly. (Very bad things can happen to Oracle if AOLServer repeated tries to connect with a misconfigured Oracle account).
- Transfer data: If you are moving the service from one location to another, export code for the old database and import into the new.
  To export: exp service_name/database_password file=foo.dmp consistent=y full=y
  To import:
```
imp service_name/database_password file=foo.dmp [fromuser=olduser touser=service_name]
```
- Should it become necessary to rebuild a tablespace from scratch, you can use the drop user command with the cascade option. This command will drop the user and every database object the user owns.
  drop user service_name cascade
  If this does not work because svrmgrl "cannot drop a user that is currently connected", make sure to kill the AOLserver using this user. If it still does not work, do:
  select username, sid, serial# from v$session where username='service_name';
  and then
  alter system kill session 'sid,serial#';
  where sid and serial# are replaced with the corresponding values for the open session.
  Use with caution!
  If you feel the need to delete everything related to the service, you can also issue the following:
```
drop tablespace service_name including contents ;
```

Set up nightly Oracle exports

Find out if oracle exports are running on the machine.
crontab -l | grep export-oracle
You should see a line similar to the following:
0 23 * * * /usr/local/sbin/export-oracle
If you don't see it, you should add it:
EDITOR=emacs crontab -e

Make sure /usr/local/sbin/export-oracle has the correct environmental variables.

Compare the settings in /usr/local/sbin/export-oracle with:

# su -l oracle
Sun Microsystems Inc.   SunOS 5.6       Generic August 1997
$ env
HOME=/home/oracle
HZ=
LD_LIBRARY_PATH=/ora8/m01/app/oracle/product/8.1.5/lib:/usr/lib:/lib:/usr/openwin/lib
LOGNAME=oracle
ORACLE_BASE=/ora8/m01/app/oracle
ORACLE_HOME=/ora8/m01/app/oracle/product/8.1.5
ORACLE_SID=ora8
ORACLE_TERM=vt100
ORAENV_ASK=NO
PATH=/ora8/m01/app/oracle/product/8.1.5/bin:/ora8/m01/app/oracle/product/8.1.5/bin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/ccs/bin:/usr/ucb:/usr/dt/bin:/usr/openwin/bin:/usr/local/bin:/usr/sbin
SHELL=/bin/sh
TERM=vt100
TZ=US/Eastern

Choose a location for the dump files and modify /usr/local/sbin/export-oracle accordingly. Your complete export_oracle file should look something like this:

#!/bin/sh
HOME=/home/oracle
HZ=
LD_LIBRARY_PATH=/ora8/m01/app/oracle/product/8.1.5/lib:/usr/lib:/lib:/usr/openwin/lib
LOGNAME=oracle
ORACLE_BASE=/ora8/m01/app/oracle
ORACLE_HOME=/ora8/m01/app/oracle/product/8.1.5
ORACLE_SID=ora8
ORACLE_TERM=vt100
ORAENV_ASK=NO
PATH=/ora8/m01/app/oracle/product/8.1.5/bin:/ora8/m01/app/oracle/product/8.1.5/bin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/ccs/bin:/usr/ucb:/usr/dt/bin:/usr/openwin/bin:/usr/local/bin:/usr/sbin
SHELL=/bin/sh
TERM=vt100
TZ=US/Eastern

exportdir=/export_root/oracle-exports
file=$exportdir/oraexport-service_name.dmp
mv -f $file.gz $file.old.gz
exp service_name/database_password file=$file owner=service_name consistent=Y
gzip $file

Set up the Arsdigita Community System on your virtual server
- Allot an IP address
  1. Find out which of the system's IP addresses aren't being used by another service (ignoring localhost (127.0.0.1):
    netstat -in | awk '{ print $4 }' | xargs -i% sh -c 'if [ `grep % /home/nsadmin/*.ini > /dev/null` $? -eq "1" ]; then echo %; fi'
  2. Select the first of the available IP addresses for your service. Once you create the appropriate .ini file with the IP address, you should notify your syadmin that you have claimed this IP. (ArsDigitans should do this by adding a ticket to the sysadmin ticket tracker on the intranet.)
- Using service-name.ini as a template, follow the steps under Installing a virtual server. Follow the Arsdigita Community System where it says to install your service.
- To prepare for high volume.
  - Make sure your AOLServer is configured to use all the machines processors. For multi-processor Solaris boxes, include the following block in the /home/nsadmin/service-name.ini file:
```
[ns/threads]
; use more than 1 processor (Solaris)
SystemScope=on
```
    Note: If you have a single processor machine, SystemScope=off is more efficient.
  - If you use adp pages, configure AOLServer to store parsed versions.
```
[ns/server/service_name/modules]
...
nscache=nscache.so


[ns/server/service_name/module/nscache]
...
CacheADP=on
```
  - It is tempting to increase the MaxThreads parameter to get more performance. Increasing this beyond 50 causes more thrashing or more pages waiting for database handles. (AOL does some tuning on this parameter and generally settles on numbers between 8 and 75).
  - It is tempting to increase the number of database handles in your main pool. Increasing beyond 8-12 can do more harm than good. The Oracle client library running inside the AOLServer process does not handle large number of connections efficiently. (It uses locks instead of more efficient multi-threading techniques).
  - Look for handle-hogging pages. These usually appear frequently and with long wait times on the bottom of monitor.
    On handle-hogging pages (ie, intermedia searching), you don't want to tie up all your Oracle handles and deny service to everyone else. On these pages, you want to:
    - Obtain the handle as late as possible and release the handle as early as possible.
    - If possible, use only 1 handle from the pool.
    - Refrain from sending email when holding a handle.
    - Limit the number of times this page can run simultaneously. For example:
      if ![ad_return_if_another_copy_is_running 6] { ns_log Notice "foo out of handles" return }
  - If you have done the above and are still limited by db handles, you a limited by the ability of Oracle's client library to manage connections. Symptoms:
    - monitor shows that all db-backed pages are backing up
    - Oracle is not overloaded. The nsd processes might be taking a large percentage of the machine's load or the CPU usage might not be high.
    You can still get more hits out of the machine by
    - Adding another AOLServer process pointed at the same page root.
    - Configure DNS to do a round robin.
  - Host graphics on a separate machine.
  - Consider caching graphics
```
[ns/server/service_name/modules]
...
nscache=nscache.so


[ns/server/service_name/module/nscache]
...
CacheStatic=on
```
    Careful!: On a db-backed web site, you want to make sure the entire database is stored in RAM. Caching graphics can be counterproductive if it interferes with query performance.
  - Use multiple machines (see ArsDigita Server Architecture)
If you're running the ecommerce module, complete the Setup portion of Ecommerce Technical Details
Set up Keepalive.
Keepalive is a virtual server located on the same physical machine as your service. It watches your service, notifies you of problems, and restarts it should it go down. All the services on the same machine share the same keepalive.
If keepalive does not exist on your machine, you will have to install another virtual server on your machine.
- Keepalive listens to port 1997. Because this should not conflict with any other service, you may re-use an IP address used by another service.
- Follow the directions for Installing a virtual server. Use keepalive.ini as a template. Since keepalive does not use Oracle, you can run /home/nsadmin/bin/nsd instead of /home/nsadmin/bin/nsd-oracle
- To install the service, follow the Keepalive installation instructions.
Once the Keepalive service is running, you need to test it:
- Test that keepalive is monitoring your service by moving /web/service_name/SYSTEM/dbtest
  mv /web/service_name/SYSTEM/dbtest /web/serice_name/SYSTEM/dbtest.moved
- Fix /web/service_name/SYSTEM/dbtest by moving it back
  mv /web/service_name/SYSTEM/dbtest.moved /web/serice_name/SYSTEM/dbtest
Sign up for Uptime
If the machine on which your service runs is down, the keepalive service on your machine will be down as well. Uptime resides on a separate server and sends alerts when your server can not be reached. You should use the forms at Uptime to register alerts for the URL http://service_domain/SYSTEM/uptime.txt. You'll want alerts to be sent to the following:
- All the people involved with your service
- noc@arsdigita.com
You should break your montoring page to make sure Uptime sends an alert. Then return the page to normal.
Set up Glassroom to maintain the information required to run the service and the ticket tracker.
The Glassroom for the service will be on a separate machine and has a domain staff.service_name domain. The ticket tracker and glassroom modules should be run on this server. Glassroom is built on top of an ACS installation.
Put the domain name for the main and staff servers in the nameservers. One team member should be in charge of nameserver updates.
Set up Watchdog to monitor your logs for errors.
Watchdog is virtual server located on the same machine as your service. It watches your log files and will notify you of tcl errors.
If watchdog does not exist on your machine, you will have to install another virtual server on your machine:
- Watchdog listens to port 1998. Because this should not conflict with any other service, you may re-use an IP address used by another service.
- Follow the directions for installing a virtual server. Use watchdog.ini as a template. Since watchdog does not use Oracle, you can run /home/nsadmin/bin/nsd instead of /home/nsadmin/bin/nsd-oracle
- To install the service, follow the Watchdog installation instructions
If Watchdog is installed on your machine, add your service:
- Look at /home/nsadmin/watchdog.ini to locate the domain and port for Watchdog on this machine
- Visit Watchdog to add your service
- Create some tcl bugs and verify that you receive notifications. You can force Watchdog to act immediate by accessing test under the Watchdog server.
Set up Reporte to generate server log reports.
- If Reporte does not exist for your server, use the Installation Guide
- Install a new service to Report
Set up Rollover, which will keep the size of the error logs under control.
- Follow the directions for installing a virtual server to create a virtual server for Rollover.
- Configure /web/rollover/tcl/rollover-list-of-systems as appropriate for your installation.
Set up Cassandrix to montor the Unix machine the server sits on. Rolf insists this will be done shortly.
Set up Cassandracle to monitor the Oracle tranactions your website performs. http://arsdigita.com/free-tools/cassandracle
- Follow the directions for installing a virtual server to create a virtual server for Cassandracle.
- Create a cassandracle user.
- Grant the cassandracle user connect, resource and dba privileges.
  grant connect, resource, dba to cassandracle
Set up ArsDigita MTA monitor - this email monitor. Branimir is working on this.
Development server
- Run the development server on the same machine/ip unless there is a reason not to. Use port 8080 for http, port 8443 for https.
- Set up an Oracle user (if needed)
- Set up another virtual server. This development server should be at the same IP as the running web service.

Installing a virtual server

Create /home/nsadmin/service_name.ini owned by nsadmin. Template ini files are listed in the above directions. Make the following substitutions as appropriate:
- If your virtual server uses the database, substitute the service's database password with database_password.
- Substitute the email of the person in charge of the server for service_email
- Substitute the service's IP address for service_ip_address
- Substitute the service's domain (e.g., photo.net) for service_domain
- If you do not want to use ftp on your server, delete the [ns/server/service_name/module/nsftp] section and the line nsftp=nsftp.so in the [ns/server/service_name/modules] section.
Prepare the servers/service_name. This directory and all the files must be owned by nsadmin. You can do this via one of the folowing methods:
- Copy another service's directory or
- Copy the AOLserver template files
```
cp -pr /home/nsadmin/modules/nssetup/template1/ /home/nsadmin/servers/service_name
```

Note: If you start with the template, the nsadmin password for the service will be blank and you will have to aquire cert.pem and key.pem from another location.

Install your service in /web/service_name

Execute /home/nsadmin/bin/nsd-oracle -c /home/nsadmin/servername.ini to test out your server setup. View your error log to make sure the service is starting without any problems. Attempt to connect to the service from a web browser.

Log into the /NS/Admin pages. Assign an nsadmin password if it is blank.

Ensure that your service automatically starts on boot (or any other time the service dies):

This requires root and deals with sensitive files that can break every service on the machine. One member from a team should be in charge of this step.

Open /etc/inittab for editing.
Copy an existing nsd line for a web service. If one doesn't exist, use this example as a template:
nss:234:respawn:/home/nsadmin/bin/nsd-oracle -i -c /home/nsadmin/service_name.ini
Make sure the first field is unique.
Important: Make sure there is a newline at the end of the file. If there is not a newline at the end of the file, the system may suffer catastrophic failures.
Use /sbin/init q to start the server.
Important: See if it works by killing your nsd process. The process should be restarted by the system. If this doesn't work, you have probabably destroyed all the services on the machine.

gregh@arsdigita.com

teadams@mit.edu