POST INSTALLATION STEPS
Once you are done with your <EM><STRONG>GEM</STRONG></EM> installation, there are a variety of post-installation tasks . This checklist is designed to help you with these tasks.
There are outstanding issues with chmod() and dos newline removal that impact Hybrid environments. Specifically,
you must run configure.pl on unix AFTER you run configure.pl on windows. This is because there is no way to
get windows to use just a standard newline instead of CR/newline (so all files will have a ^M in them if you run on windows last).
You should also run the Batch File creation on UNIX last. If you ran your Win32 install after your Unix install, go to your unix server and run <code>perl upgrade.pl</code> to fix these issues.
The password files are located in <font color=red>conf/*_password.dat</font> and are hand editable.
sybase_passwords.dat
oracle_password.dat
pc_password.dat
unix_passwords.dat
sqlsvr_password.dat
All configuration files are self documented. An archive of your old configuration files is kept in <font color=red>conf/saved_files</font>
If you add new databases or systems, recreate & reschedule your Batch Scripts. Most scripts work against all servers in the configuration files, but some scripts work on a specific database (specifically the ones with the Database name in the file name).
One of the most horrific tasks of the windows DBA is disk space cleanup. Something is *always* running out of space... <EM><STRONG>GEM</STRONG></EM> automates this task for you.
The file <font color=red>conf/Win32_Cleanup.dat</font> contains a list of disk space Cleanup instructions for the windows cleanup batch job. It is suggested that you customize this file to include your tran/archive log directories, your application log directories, and any other directories that fill up on your systems. A best practice is to add rows to this file over time, every time the windows disk space monitor spots a directory that is filling up. The lines in this file conform to the arguments of ADMIN_SCRIPTS/bin/purge_old_files.pl, and include --hours and --days to identify the keep times, --dir to identify the directory to purge, and either --compress=<compress program> or --del (to identify what to do with the old files). See the syntax of purge_old_files.pl for other options.
# FILENAME: Win32_Cleanup.dat
#
# USED BY: Batch job Win32FileCleanup
#
# SYNOPSIS: Directives for cleaning up win32 drives. This file is
# identical to Unix_Cleanup.dat but for the --compress (Win32 only)
# argument which is missing and the required --host argument (Unix Only).
#
#
# FORMAT : HOSTNAME drive drive drive.
#
# DETAILS : The arguments are the arguments to purge_old_win32_files.pl
#
# --host=<host>
# --hours=<hours>
# --days=<days>
# --dir=<dir>
# [--del] - do you wish to delete the file?
# [--done] - only work on files with .done extension
#
# EXAMPLE FORMATS:
#
# Delete 14+ day old files on host h1
# --days=14 --dir=//adsrv012/c$/oracle/admin/SCPROD/udump --del
#
# Delete files with .done extension older than 2 days old
# Note: this will delete .done.gz files and .done files
# --days=2 --dir=//SRVR001/d$/logshipbuffer --del --done
#
#######################################################################
#
# START RECOMMENDED PURGES
# --days=7 --dir=C:/gem/data/logs -del
# --days=7 --dir=C:/gem/data/batchjob_errors -del
# --days=7 --dir=C:/gem/data/GEM_BATCHJOB_LOGS -del
# --days=7 --dir=C:/gem/data/CONSOLE_REPORTS -del
# --days=7 --dir=C:/gem/data/cronlogs -del
# --days=7 --dir=C:/gem/data/errors -del
# --days=7 --dir=C:/gem/data/backup_reports -del
# --days=7 --dir=C:/gem/data/html_output -del
# --days=30 --dir=C:/gem/data/BACKUP_LOGS\*\sessionlog -del
# --days=30 --dir=C:/gem/data/BACKUP_LOGS\*\errors -del
# END RECOMMENDED PURGES
The Unix_Cleanup.dat file will control unix file/directory cleanups. It is similar to the Windows cleanup file but for the addition of the --hostname argument.
Port Monitoring is a quick availability test - and a good way to find out if a database or system is down. The file <font color=red>conf/port_monitor.dat</font> contains a list of Hosts and Ports to Monitor.
This file is autogenerated by configure.pl but will probably require hand editing.
Simple format:
# hostname PORT SERVER DESCRIPTION SRVR001 1433 SRVR001 Microsoft SQL Server SRVR007 1433 SRVR007 Microsoft SQL Server brokersql 1433 BROKERSQL Microsoft SQL Server mail1 25 Sendmail on mail1
The File <font color=red>conf/Replication.dat</font> is used by the Sybase replication server monitoring scripts and will need to be popluated with your Rep Servers and Routes.
# FILENAME: replication.dat
#
# SYNOPSIS: Lists your sybase replication server information
#
# FORMAT:
#
# REPSERVER $NAME $LOGIN $PASSWORD $THRESHOLD_MB
# REPROUTE $SOURCE_SVR $SOURCE_DB $DEST_SVR $DEST_DB $REPSERVER
#
# EXAMPLES (350MB Thresholds):
#
# REPSERVER REP1DR_RS sa <pass> 350
# REPSERVER REP2_RS sa <pass> 350
# REPSERVER REP1_RS sa <pass> 350
#
# REPROUTE SYB1 my_db SYB2 my_db REP2_RS
# REPROUTE SYB1 my_db SYB1DR my_db REP2_RS
#
# ONCE YOU SET THIS FILE UP - PLEASE RUN
# ADMIN_SCRIPTS/monitoring/SybRepMonitor.pl --ACTION=INSTALL
# MONITORING IS HANDLED AUTOMATICALLY BY THE GEM BATCH SybRepMonAgent
##############################
There are two programs that check replication. The batch SybRepServerChecker will check all your rep servers to be sure that no routes are suspended, that disk space is reasonable, and that there are no other serious errors. There are also three batches, SybRepMonAgent, SybRepMonCheck, and SybRepMonReport that compute latencies of your replication, and will warn you when replication falls behind. These three batch programs call SybRepMonitor.pl. This program takes an argument --ACTION.
To initialize this system, you need to set up your configuration file and then run <font color=red>myperl SybRepMonitor.pl --ACTION=INSTALfont
After the system is INSTALLED, the above mentioned batches will perform your monitoring automatically. Below is the syntax for SybRepMonitor.pl:
Usage: SybRepMonitor.pl --ACTION=INSTALL|UNINSTALL|MONITOR|REPORT|CHECK
[--SERVER=xxx] [--DEBUG] [--OUTFILE=file]
if --ACTION=INSTALL will install
--REBUILD redoes table creates
if --ACTION=CHECK will check latency vs --ALARM_MINS
- mail to --ALARM_MAILTO as needed
if --ACTION=REPORT will create a report
- will print data for --REPORT_HOURS hours
- will only show rows > --REPORT_THRESH seconds
- will aggregate based on --BUCKETSIZE
- will print HTML output if --HTML is passed
if --ACTION=MONITOR will insert rows into all source databases
- will insert --ITERATIONS rows
- sleeps --FREQUENCY secs between each insert
You may not wish all your alarms to work at the same thresholds - preferring to set some thresholds individually. Generally speaking, this is most needed when some resourse is nearing ``full'' and generating alarms, but you can not fix the problem immediately - intending to fix the problem later. The <font color=red>conf/threshold_overrides.dat</font> file allows you that control.
# FILENAME: threshold_overrides.dat
#
# SYNOPSIS: Monitoring Thresholds
#
# DETAILS : This file permits you to turn off alarms and to control their
# severity. This is a Comma Separated File
#
# SAMPLE FORMAT:
# <system> <monitor_program> <level> <threshold> <optional subsystem>
#
# monitor_program=UnixDiskspace|PcDiskspace|SybSpaceMonitor
# |SpaceMonitorSqlsvr|ThresholdManager
# level=CRITICAL|ERROR|WARNING
#
# remember to reset higher priority thresholds so there is no overlap - ie if you
# set ERROR to 98%, then you should set critical to something larger (say 99%).
# The programs may act wierd if the CRITICAL threshold was less than the ERROR threshold.
#
# By default - SybSpaceMonitor & SpaceMonitorSqlsvr will use 90/92/95% as
# their default thresholds
#
# Replication thresholds are in minutes. For replication, you can put a
# blank target name in if you want but not a blank source name
#
# EXAMPLES
# eg. directfeed,UnixDiskspace,WARNING,80,Disk: /var
# eg. direct1,UnixDiskspace,WARNING,80,Disk: /var
# eg. direct2 ,UnixDiskspace,WARNING,80,Disk: /var
#
# eg. SYBSVR1,SybSpaceMonitor,WARNING,80,db1
# eg. SYB1,SybSpaceMonitor,ERROR,95
# eg. SYB1,SybSpaceMonitor,ERROR,95,clientdb
#
# eg. ADSRV057,PcDiskspace,ERROR,98,d$
# eg. ADSRV057,PcDiskspace,CRITICAL,99,d$
#
# eg. SERVERNM,ThresholdManager,CRITICAL,99,logsegment|default
#
# EXAMPLE REPLICATION STUFF
# eg. SOURCESVR,SybRepMonCheck,CRITICAL,30,TARGETNM
# eg. SOURCESVR,SybRepMonCheck,ERROR,15,TARGETNM
# eg. SYB1,SybRepMonCheck,CRITICAL,80,IMAGSYB2
# eg. SYB1,SybRepMonCheck,CRITICAL,80,
# eg. SYB1,SybRepMonCheck,ERROR,60
#
#########################################
Another separately installed <EM><STRONG>GEM</STRONG></EM> component is the Sybase threshold manager. This program (ADMIN_SCRIPTS/bin/threshold_manager.pl) installs customized sp_thresholdaction procedures into your databases and allows you to find peak usage (most important for log file sizes). This proactive utility will hopefully prevent you from running out of log space - because you will have been warned in advance in such a way that you can expand capacity before there is a production alert.
The sp_thresholdaction procedures we install are simple reporting versions that insert peak usage data into a small table in sybsystemprocs. You can run a report showing peak usage of data and log segments in your databases which is the Threshold Report you can view in the console.
This procedure needs to be installed manually by running
<font color=red>ADMIN_SCRIPTS/bin/threshold_manager.pl --ACTION=INSTALfont
after you have registered your servers. One warning. If you increase the *size* of any of your databases, you will need to rerun the above installation command. Sybase thresholds are based on free pages and the percentages are therefore set when you run the installation. So, if you double the size of a database, your 60% threshold will be firing at 30% and recording it as 60% full! Just reinstall and you will be ok!.
Other commands of interest:
<font color=red>ADMIN_SCRIPTS/bin/threshold_manager.pl --ACTION=CHECK</font>
<font color=red>ADMIN_SCRIPTS/bin/threshold_manager.pl --ACTION=REPORT</font>
This feature only works with Sybase
This information is stored in <font color=red>conf/pc_service.dat</font> and can be created from configure.pl using ``Snapshot Win32 Services'' on the Post install tasklist. The file can also be created by the unscheduled batch file PcServiceCheckerInit.ksh.
You will probably need to hand edit this file - the snapshot is at a point in time and some services may get turned off or just dont stay up. After you have created this file, simply remove lines which get flagged by the monitors yet are not important.
cd win32_batch_scripts/interactive PcServiceCheckerInit.ksh
The alarms tab will populate the directory cgi-bin with runnable web scripts. The button to actually distribute those scripts to your web server is currently unimplemented. You will need to copy the scripts by hand.
This is made easier by the fact that the scripts have been modified to run in your environments, but it is possible that the scripts do not execute.
a) copy the contents of cgi-bin to the web server scripts directory you identified for your use b) chmod 755 these scripts to make them executable (unix only) c) try to run mimi.pl when in that directory. Mimi is a cgi-bin script - so it will hang asking for parameters if it runs successfully. It might also give you a command not found or library not found message. d) you might need to change the hashbang (first line) to point to your perl e) try running printenv.pl from your web browser... this is a very simple script that should always work. f) try running mimi.pl from your web browser
It is beyond the scope of this document to discuss web server setup. All the directory requires is script access.
The top of these scripts should look like:
#!/usr/local/bin/perl-5.8.1
use lib qw(/apps/sybmon/dev/lib)
my($COMPANYNAME)="YOUR COMPANY";
BEGIN {
if( ! defined $ENV{SYBASE} or ! -d $ENV{SYBASE} ) {
if( 1==2 ) { print 'doh'; }
} elsif( -d "/export/home/sybase" ) {
$ENV{SYBASE} = "/export/home/sybase";
} elsif( -d "/apps/sybase" ) {
$ENV{SYBASE} = "/apps/sybase";
}
}
}
The paths, company name, sybase directories, and perl version have all been customized. Make sure they all look ok.
Test this script through your web browser.
<p><font color=RED>perl monitor.pl</FONT><br><font color=RED>perl eventviewer.pl</FONT></p>
Two modules are currently shipped with <EM><STRONG>GEM</STRONG></EM> - an event viewer and a monitor. Both are located in the root directory. To run, use <path to your perl>/monitor.pl. Under windows, you should be able to just double click on these programs to run them and can create desktop shortcuts as needed.
<p><font color=RED>perl ADMIN_SCRIPTS/bin/fix_db.pl -Ax</FONT></p>
This is a cute little database configuration checker. You might as well run it. It shows dboptions you might wish to correct and identifies common configuration problems. To correct these problems remove the -x (noexec) option.
You can add customized reports to the console easily using conf/console.dat. These reports can be pretty much any command or sql statement. Check out the file for the syntax.
It is recommended that you create object level backups of your server DDL once per week. GEM provides the excellent script dbschema.pl (written by Michael Peppler and David Owen) in the ADMIN_SCRIPTS/bin directory to facilitate this process. Because this script is written in SybPerl (not DBI::Sybase), we do not provide an integrated approach to creating these audits. I recommend you back up your ddl once per week on your production servers.
You can set up conf/dns.dat and then do dns monitoring with
0,20,40 * * * * /usr/bin/nice /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/monitoring/dnsmonitor.pl --OUTFILE=/apps/sybmon/dev/data/html_output/dnsmonitor.pl.txt
The gem configuration utility will schedule your jobs on win32 monitoring stations.
At 2 points in configure.pl, you can create a unix crontab file. This file is named crontab.txt and is placed in the <EM><STRONG>GEM</STRONG></EM> root directory. You should schedule this file using yoru crontab on unix. To do this run
export EDITOR=vi # presuming u like vi
crontab -e
<vi window>
:r crontab.txt
:wq
You should also place a .forward file in your home directory to redirect mails to your account.
Put the output of any of your routine scripts into data/html_output or data/cronlogs to have the reports show in the GEM console. This is a convenient spot to redirect output in any case. The filename itself is used to generate the console report.
# the following job compare a server to its dr copy
1 22 * * * /perl/bin/perl /apps/sybmon/dev/ADMIN_SCRIPTS/bin/datacompare.pl -Usa -SSYB1 -Pxxx -sSYB1DR -pxxx -uxxx -h > /apps/sybmon/dev/data/html_output/Rowcount_Compare_Prod_DR.html 2>&1
Schedule additional alarming reports. These need to be set up in the cgi-bin mimi.pl reports screen before they can be run. But you can snapshot anything you can see on those screens and mail the results as appropriate
0 10,12,14,16,18 * * 1-5 /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/monitoring/RunMimiReport.pl --SUBJECT="2 Hour Unix Status Report" --MAILTO=def@bac.com,abc@abc.com --REPORT=Unix2Hour --TIME="2 Hours" > /apps/sybmon/dev/data/GEM_BATCHJOB_LOGS/RunMimiReport_Unix2Hour 2> /apps/sybmon/dev/data/batchjob_errors/RunMimiReport_Unix2Hour
I sync logins between my replicated servers with
38 9 * * 0 /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/bin/copy_syslogins.pl -FSYB1 -Usa -Pxxx -tSYB2 -usa -pyyy > /apps/sybmon/dev/data/GEM_BATCHJOB_LOGS/copy_syslogins_SYB1.log 2>/apps/sybmon/dev/data/batchjob_errors/copy_syslogins_SYB1.log
I monitor cisco logs with
18,48 * * * * /usr/bin/nice /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/monitoring/cisco_logmon.pl --INFILE=/var/log/cisco.log --TODAYONLY >/apps/sybmon/dev/data/GEM_BATCHJOB_LOGS/cisco_logmon.log 2>/apps/sybmon/dev/data/batchjob_errors/cisco_logmon.log
I monitor centrally collected (via syslogsd) unix logs with
5,15,25,35,45,55 * * * * /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/monitoring/unix_monitor.pl --INFILE=/var/log/messages --TODAYONLY --EXCLUDEFILE=/apps/sybmon/dev/ADMIN_SCRIPTS/bin/solaris.excl --MAXLINES=100000000 --HTML --OUTFILE=/apps/sybmon/dev/data/html_output/unix_monitor.html >/apps/sybmon/dev/data/GEM_BATCHJOB_LOGS/unix_monitor.log 2>/apps/sybmon/dev/data/batchjob_errors/unix_monitor.log
<EM><STRONG>GEM</STRONG></EM> is an administrative tool, designed for system administrators. As such, the conf directory contains all your passwords. You need to protect this data via operating system permissions.
We strongly recommend that all other directories in the tree, with the exception of the data/CONSOLE_REPORTS tree, be also made secure. Technically, you can allow read/execute access to these files, but it would be easier to only allow owner/group access - much of the data contained may be sensitive.
How To Upgrade GEM
<em><strong>GEM</strong></em> is designed for simple upgrades. The steps are as follows
On your windows monitoring server, you may not be able to extract the code because the task scheduler is running a batch job that calls one of the perl scripts. Windows will not let you overwrite a program that is running. This does not happen under Unix. You can either go into the task scheduler, stop all running jobs before rerunning the extract, or kill the jobs and then stop the task scheduler process. I normally just stop the jobs, but sometimes (because end task does not work cleanly) i am forced to stop the scheduler process for the duration of the extract.
Run <font color=red><code>perl upgrade.pl</code></font> to finalize the installation. This step is MANDATORY and will perform a variety of steps like updating your configuration files and re-installing your alarm database. During the period you are upgrading - your <em><strong>GEM</strong></em> batch jobs will probably fail and send you alarm mail. Ignore these failures notifications.
The alarm database reinstall will save table data, drop all tables, reinstall tables, and finally copy back data. If there is a problem, it will leave a tables with the extension _BAK containing the original data. So, under the hood, the upgrade process for the table Event is ``select into Event_BAK from Event'', ``drop table Event'', ``create table Event'', ``insert Event select * from Event_BAK'', and finally ``drop table Event_BAK''. Upgrades can not start if any tables with the extension _BAK exists - you must remove them before the upgrade.
If there is a problem, you can probably just drop the BAK table - but it would be better to recover by following a process akin to ``truncate table Event'', ``insert Event select * from Event_BAK'', ``Drop table Event'' (you have to do these by hand). Got it? Any questions on a failed install please contact SQL Technologies tech support.
The final install step redoes your hashbangs, perl library paths, and removes all control M's from the code line. A log of the upgrade is saved in the logs directory.
<font color=red>ONCE YOU HAVE SUCCESSFULLY RUN upgrade.pl YOU ARE DON&/FONT;