The Complete GEM Documentation

GEM Documentation Navigation Page. Documentation is also available at The GEM Home Page. Documentation and Code are both Copyright © 1995-2006 by Sql Technologies. All Rights are Reserved.

The Complete GEM Documentation Slide Show: What Is GEM Download GEM
 
An Introduction To GEM How To Install Getting Started Guide
Batch Jobs Overview GEM Screenshots Server Backup and Maintenance
Utilities Guide The GEM Console New System Stored Procedures
FAQ and Change Log Common Perl Libraries Pricing & Right To Use

Introduction To GEM

GEM is an open source server management toolkit for Database Administrators. GEM is our solution to managing many Database Servers without a lot of stress.

GEM is a feature rich, low cost package that is easy to install, configure, and manage. GEM uses a small footprint, centralized, agent less architecture that has minimal server impact and will revolutionize the way you perform system administration.

The Generic Enterprise Manager or GEM automates the routine parts of server management. It monitors your systems and notifies you of problems before your users do. GEM configuration auditing can reveal serious issues before they become a problem. For example, a database backup checker reads system logs to ensure your backups succeed. GEM gives small and mid sized companies the tools that up until now only the very largest companies possessed.

GEM provides a simple, complete solution to help you manage your Database Servers and associated Windows, Unix, and Linux systems. While GEM provides comprehensive operating system level monitoring and alarming, the strength of GEM is database infrastructure management. GEM is designed for database administrators, by database administrators.

GEM is a non-invasive, agent less system that runs on a central monitoring station. Nothing runs on your remote systems. Even our backup solution runs centrally. You will need a Win32 monitoring station if you are using GEM with Win32 systems, and a Unix monitoring station if you are using GEM on Unix. GEM will install our excellent system stored procedure library on all your database servers.


What Is the Generic Enterprise Manager

The Generic Enterprise Manager (GEM) is a systems management package designed specifically for Database Administrators. GEM provides a simple, complete solution to help you manage your Database Servers and associated Windows, Unix, and Linux systems. While GEM provides comprehensive operating system level monitoring and alarming, the strength of GEM is its simple proven mechanisms to help you manage your database infrastructure. GEM is designed for database administrators, by database administrators.

The Generic Enterprise Manager is written in open source perl. It is broken into several component packages, each consisting of small, self-documenting, easy to read perl programs. These packages use common libraries, a common directory structure, and common configuration files.


The Benefits of GEM

Fundamentally, GEM benefits your users by reducing systems problems. GEM effectively automates cumbersome tasks like collecting and parsing database error logs and delivers that information to the administrators in a clear, concise manner. This dramatically increases your ability to spot problems - GEM helps you find issues before they become problems and allows you to proactively manage your servers.

By effectively delivering system health information, GEM decreases the time spent on routine tasks and increase quality of service provided. GEM will both reduce downtime and decreasing the time you spend on system maintenance. GEM dramatically increases the information about each server that administrators have at their fingertips.

GEM protects your servers by guaranteeing your server back ups and by providing the configuration audit information you need to restore in the event of an emergency.

GEM monitors important services in your database infrastructure and alarms operators if there is a problem. Our alarm routing mechanism allows you to receive only get the messages you care about.

GEM is most effective for small and mid sized organizations (3-100 database servers). GEM requires administrator access to your databases, so a DBA must be responsible for installing and running it.

GEM contains numerous features for both developers and administrators. GEM provides a Database Utilities Toolkit of command line tools and contains an excellent library of new system stored procedures.

GEM is a must for companies that perform remote server management or have servers in multiple branch offices.


The Power of GEM

The power of GEM is based on its open architecture. Fundamentally, GEM is an open source set of libraries and management programs that are used by numerous tiny programs (300 line is pretty typical) designed to monitor and analyze your environment. The underlying components, the libraries and standard configuration files are designed to be easily understood and well documented, but most importantly, hundreds of working example programs are readily available. Because all the scripts that acutally do work are simple and small, you can troubleshoot and customize GEM with minimal effort.

As an example, let us say YOU wish to add a new report to the console. All you would need is to create a perl script that creates either html or text output, modifying one line in a configuration file, and then scheduling your program. The report then appears integrated into the console. If this is too complex, the console provides a second, even easier, interface. Just put a query/command into a second config file and the report automatically appears.

The same applies to integrating your own custom alarming into our alarming system. Just take your existing monitoring script, add a few lines of perl (our alarm interface is open), and your alarms will automatically be routed and managed from GEM.

GEM is an open ARCHITECTURE that includes a complete set of open source programs. This allows you to make GEM a complete solution to both your infrastructure and your business monitoring and reporting needs.


Component Packages


Architecture Overview

GEM is a non-invasive agent less system that runs on a central monitoring station. Nothing runs on your remote systems. Even our backup solution is runs centrally. You will need a Windows monitoring station if you are using GEM with Win32 systems/databases, and a Unix monitoring station if you are using GEM for Unix systems/databases.

GEM will install our popular system stored procedure library on all servers.

GEM provides a database maintenance package which can optionally be used for server back ups and maintenance.

The GEM Configuration Utility creates a set of custom monitoring and maintenance scripts known as the GEM Batch Scripts, which are scheduled on your monitoring stations. Information is collected in the GEM Repository and the GEM Monitoring Database, from which it is reformatted and delivered.

If you have a hybrid environment (both Win32 and Unix Systems), GEM is normally installed on a samba shared drive. This permits the same code to be used on your windows, Linux, and Unix systems. The windows monitoring system will run scheduled tasks on your windows domain and the Unix/Linux server will use cron to schedule jobs in your Unix domain.

GEM is written in open source perl. Perl is regarded as the language of choice for systems administrators and your administrators will find it easy to understand GEM. With a normal level of perl experience, they can troubleshoot any problems that arise. GEM uses standard perl/DBI for database independent connectivity and perl/Tk and perl/CGI for user interfaces. Programs are self-documented using the standard perldoc format (i.e. See documentation via perldoc program.pl) and follow standard best practices. The GEM documentation is created from this same perldoc.

The core of GEM is a set of flexible perl library modules. Use of this library interfaces permits the majority of GEM programs to be short and simple. This has the benefit of simplified systems maintenance. The combination of an API and practical examples allows users great flexibility in terms of customization.

Every environment has numerous specific needs. GEM is specifically designed for easy expansion to meet your needs.


GEM Prerequisites

GEM is a server administration tool, and it is therefore expected that you have administrator access (sa, sa_role) to your databases. For your Unix systems/databases, you will need access to a Unix account with read permission on your Sybase error logs. To monitor win32 systems, you need to schedule batch jobs using an account with read access to the event logs and services (in practice this means administrator access).

GEM requires one or two monitoring stations. You need a Win32 monitoring station if you wish to monitor Win32 servers/databases and a Unix monitoring station if you wish to monitor Unix servers/databases. The ideal Unix monitoring system is an older 4 cpu system, possibly a recently decommissioned server. The ideal Win32 monitoring system is a 2-3 year old Win2K or Win2005 server with 2 GB RAM.

You will also need a small monitoring database. This monitoring database can be SQL Server you are running all Win32 or Sybase for other configurations. For cleanliness, it is recommended that this server be running on one of your monitoring stations.

The GEM monitoring stations require a recent version of perl. The DBI module and the appropriate DBD modules (DBD::ODBC or DBD::Sybase) must be installed.

Finally, you will need access to a Web Server that can run cgi scripts.


Usage License

All GEM software and documentation are Copyright (c) 1995-2006 by SQL Technologies, inc. All rights are reserved.

GEM is currently under strict distribution control. The GEM may be downloaded from www.edbarlow.com for evaluation and use, but the software may not be redistributed in any manner without express consent.

NO WARANTY OF ANY SORT IS PROVIDED WITH THIS SOFTWARE

The terms of this license will change once the software goes general availability.

GEM is distributed as open source code, and it is ok for you to inspect and modify the code line so long so long as you don't remove any of the copyright notices and do not redistribute the changes without the consent of SQL Technologies.

Pricing

GEM Project GEM is a major effort designed to help you manage business critical infrastructure at commercial organizations. It is expected that commercial users will support this project financially. We do, however, come from a free software background you can expect the package to be low cost and high functionality. Non profit and government organizations, companies with 1-4 database systems, and indviduals are welcome to use this software for free.

GEM uses licensed consultants and distributors for primary user support. These 'Certified Product Specialists' will be paid a sales and support fees above and beyond the prices listed below, in return for installation and production support. Support charges for these individuals will be negotiated by the customer on a case by case basis.

Training is available for individuals interested in being Certified. If you are a consultant or distributor and are interested in reselling GEM, please contact us. For now, support will be provided by SQL Technologies free of charge.

PACKAGE

PRICE LIST (as of Aug 1 2006)

GEM Standalone Backup Scripts

FREE

GEM Stored Procedure Library

FREE

Free GEM (1-4 Server License)

FREE

Commercial GEM

$500 / server initial license

$200 / server / year maintenance


Customization

GEM is open source and supports an open perl API. The software is designed for easy, customization. Most simple customization ( adding an alarm monitor or adding a report to the console) can be done trivially. Contact SQL Technologies for details.

Complex customization will be performed on a contract basis. Contact us for more information.

Getting Started Guide

for the GEM

This guide will give information on how to get started with the Generic Enterprise Manger (GEM). The scope of this project is large, but if you follow this guide, we can make this tool a success for you. This document describes the structure of the code line - necessary information to understand the application.

GEM provides a framework to manage multiple database systems. The system is built upon a set of perl libraries which provide the glue to bind the system together. Since its all open source perl, expansion and customization are easy. This guide endeavors to guide you through the process of installing and customizing GEM.

One thing to remember. GEM is designed for simplicity. The goal is to put code, data, and libraries in locations where a reasonable administrator would expect them to be found. After you extract GEM, please poke around and familiarize yourself with the product. Look for utilities in the ADMIN_SCRIPTS subdirectory and configuration files in the conf subdirectory. Knowledge of the layout is essential to maintaining and operating GEM as your production server managment interface.

This document provides an overview of GEM layout. To truly understand the layout, you will need to have an existing copy of the code. To get this, please read the Installation Guide.

Directory Structure

The first thing to understand is the basic directory structure of the installation. The command line programs that implement most of the functionality exist in the ADMIN_SCRIPTS subdirectory, which is further subdividied by package. Configuration files are stored in the conf subdirectory, and data and formatted output is stored in the data subdirectory. The data subdirectory consists of the GEM File Repository (where we keep collected files and system log files) and the GEM Console (in CONSOLE_REPORTS) where we keep report output. The main GEM programs (eventviewer.pl, monitor.pl, troubleshoot.pl, and configure.pl) are in the the root directory. The rest of the heirarchy is for support files.

ftfo.gif (126 bytes) GEM Base Directory This is where you extracted the software.
ftn.gif (70 bytes)ftfo.gif (126 bytes) ADMIN_SCRIPTS
Command line programs reside here.
ftn.gif (70 bytes)ftfo.gif (126 bytes) doc
Directory of GEM documentation
ftn.gif (70 bytes)ftfo.gif (126 bytes) Unix_batch_scripts
This is where batch scritps that work with Unix reside
ftn.gif (70 bytes)ftfo.gif (126 bytes) win32_batch_scripts
This is where batch scripts that work with Windows reside
ftn.gif (70 bytes)ftfo.gif (126 bytes) lib
Perl Module Library
ftn.gif (70 bytes)ftfo.gif (126 bytes) plugins
Location for GEM Manager plugins
ftn.gif (70 bytes)ftfo.gif (126 bytes) conf
Configuration Files (these are human editable and must be kept secure)
ftln.gif (64 bytes) ftfo.gif (126 bytes) data Sub directory for discovered and generated data - this directory can get big

GEM User Interfaces

ftfo.gif (126 bytes) GEM Base Directory This is where you extracted the software.
ftn.gif (70 bytes) gem.pl
This is the GEM enterprise manager - which rolls up all the plugins. It is not currently functional.
ftn.gif (70 bytes) configure.pl
This is the GEM setup utility
ftn.gif (70 bytes) troubleshoot.pl
This is the diagnostic utility. It is used to diagnose configuration and setup problems.
ftn.gif (70 bytes) monitor.pl
The systems monitor. It allows drill down server diagnostics (works on system tables)
ftln.gif (64 bytes) eventviewer.pl
The system event viewer

The ADMIN SCRIPTS Directory

The main command line programs of the GEM are stored in ADMIN_SCRIPTS and broken down by package

ftfo.gif (126 bytes) GEM Base Directory This is where you extracted the software.
ftln.gif (64 bytes)ftfo.gif (126 bytes) ADMIN_SCRIPTS
Command line programs reside here.
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)bin
Miscellaneous utilities
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)dbi_backup_scripts
Sybase and Sql Server Backup Scripts
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)MDA_procs
Sybase MDA Table monitoring procedures
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)monitoring
Programs designed for monitoring
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)oracle
Oracle Scripts
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)procs
Sybase and Sql Server Extended Stored Procedure Library
ftb.gif (135 bytes)ftln.gif (64 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)console
Scripts that manage the GEM Console

The Data Directory

The data directory is where information is stored about your systems. This information is broken down by package.
ftln.gif (64 bytes)ftfo.gif (126 bytes) data Sub directory for generated data - this directory can get big
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes)
ftb.gif (135 bytes)|
ftb.gif (135 bytes)|
ftb.gif (135 bytes)BACKUP_LOGS


This directory contains the output from your maintenance plan
scripts broken down by server. Within each server specific
subdirectory exists a directory for audits, dbcc, errors, and sessionlogs.
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)batchjob_errors
Standard Error from the maintenance scripts
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)GEM_BATCHJOB_LOGS
Standard Output from the maintenance scripts
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)CONSOLE_REPORTS
This is the console. You point your browser here to see it.
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)depends
Files required for the dependency analyzer
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)cronlogs
Logs of your scheduled jobs
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)html_output
Output of any html reports you want auto included in documentation
ftb.gif (135 bytes)ftn.gif (70 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)lockfiles Lock Files for programs that can be overscheduled
ftb.gif (135 bytes)ftln.gif (64 bytes)ftfo.gif (126 bytes) ftb.gif (135 bytes)system_information_data
Monitoring data and configuration information by server

The conf Directory

The conf directory contains system configuration files. The following are the main configuration files

Files are shiped in the distribution with .sample extensions - these are the templates with standard comments. No files are shipped with .dat, .xml, or .cfg extensions. This permits you to perform an upgrade by extracting the new tarball over the existing code line with no problems.

GEM Scheduled Tasks

One of the core features of GEM is its monitoring and maintenance scripts. These scripts perform the systems monitoring and alarming, perform the system administration operations you define in your maintenance plans, and provide useful routine maintenance activities. The scripts are customized to your environment by the GEM configuration manager. For version 1, some of these scripts may involve hand definition of a configuration file.

The configuration manager will create scheduled for some of these jobs on your Windows scheduler and will create a Unix crontab file for jobs you indicate you wish to run on Unix. These tasks fundamentally entail a call to one of the command line scripts located in the ADMIN_SCRIPTS directory, but the GEM hides this from you. GEM creates a set of .bat and .ksh driver scripts that have obvious names to be the scripts that you actually run or schedule.

These scripts are stored in the win32_batch_scripts and Unix_batch_scritpts directories. Within those directories are 4 subdirectories as follows:

ftfo.gif (126 bytes) GEM Base
ftn.gif (70 bytes) ftfo.gif (126 bytes) unix_batch_scripts This is where scripts that run under Unix exist. These directories are populated with customized scripts.
ftn.gif (70 bytes)ftfo.gif (126 bytes) interactive Output from these commands can bee seen. These are the scripts that you use
ftn.gif (70 bytes)ftfo.gif (126 bytes) batch These scripts are for use by your scheduler. They redirect output to specified log and error files. If you run them you will get no output.
ftn.gif (70 bytes)ftfo.gif (126 bytes) plan_interactive Scripts for the backup / maintenance plan system. These scripts are interactive as per above
ftln.gif (64 bytes)ftfo.gif (126 bytes) plan_batch Scripts for the backup / maintenance plan system These scripts are for use by the scheduler as per above

The above tree is duplicated in the win32_batch_scripts subdirectory.

Simply put, each script is created as a .ksh on Unix and as both a .bat and .ksh script on Windows. Copies of the script exist in an interactive and batch directory (or a plan_interactive/plan_batch directory). The batch scripts redirect output to locations in the data tree (so you cant see it) but are identical to the interactive scripts. When you run a scheduled task, use the batch version, and when you run interactively, use the interactive version (you want to see output if you are running interactively). The plan_* directories store pregenerated maintenance plan scritps. The plan directories were separated from the regular directories because the tree becomes hard to navigate when you get large numbers of servers (and a large number of maintenance plans).

Backup and Maintenance Plans

GEM offers a complete maintenance plan based system maintenace facility.  Plans are set up using the GEM Installer and stored in the configuration file configure.cfg.  All normal features related to systems maintenance are covered including Backups, DBCC, Update Statistics, and Log Shipping.  Additionally, backup compression for SQL Server, notification, and extensive reporting are provided.  The command line interface provides a level of flexibility not available with native SQL Server facilities. 

GEM Enterprise Manager

The GEM Enterprise Manager is the graphical user interface for GEM. It is a modular system that uses plugins for all functionality. Plugins use single console for all your systems. It uses perl DBI - a database independant api - and provides you with a seamless api. The trick here is the RosettaStone.pm perl module, which basically maps database independant requests (get me locks/logins/space) to database specific ddl. This gives you consistency. The GEM. is designed to use plugins, which are basically small perl modules that conform to an api. Most of your services are provided by the other GEM libraries, so these modules basically define what user interface elements they want (menus, how they want their explorer to look, right click functionality) and then manage a Tk frame however they want. The goal here is to have third parties develop plugins that add functionality. And the nice thing here, is that this can be business specific functionality (ie. a trade or order volume report). The GEM plugins are managed from a central store on the internet which is available to you to distribute your plugins in the same manner that CPAN does for perl modules.

Documentation and Diagnostics

All system documentation uses the standard perldoc format. Perldoc is a utility provided with perl to extract documentation from programs. To get full documentation on a program, simply run perldoc program. The majority of the GEM documentation is composed from the embedded perldoc. You can see a short version of the help by simply running perl program.pl -?, which will provide useful syntax help. 

Older database programs use the standard syntax of -Uuser -Ppassword -Sserver -Ddatabase for arguments. Newer programs use --USER=username --PASSWORD=password --SERVER=servername --DATABASE=database convention.

If you have problems with a programs, try running with -d or --DEBUG flags set. These will run the program in verbose/diagnostic mode and the program will print useful diagnostic information. This is the standard way of debugging problems.


INSTALLATION GUIDE

The following guide describes how to install GEM.

If you are planning to using GEM, please contact Sql Technologies by email. We are interested in making your GEM Installation a success, and will provide installation support. We are aiming for full customer satisfaction and will work with you to make this product a success.

Pre-install check list

  • PICK 1 or 2 SYSTEM(s) TO USE AS GEM MONITORING SERVER(s)

    GEM manages your systems from central monitoring systems. A windows systems is used to manage your windows databases. A unix/linux server is used for your unix servers. If you have both (a "Hybrid Environment") windows and unix based databases, you will need 2 monitoring systems.

    These systems run a lot of perl batch jobs. It is recommended that you use dedicated systems for your production GEM monitoring servers. For your initial trial setup, use a development system.

    On windows monitoring systems, we suggest you have access to a windows login OTHER than the login you normally use for login. On windows, scheduled GEM batch jobs might result in the pop up of dos cmd boxes (which is annoying). Scheduling the jobs using a 'batch' windows account prevents this.

    You will need appropriate SQL Server and/or Sybase/Oracle client software to be installed on your monitoring servers.

  • PICK A HOME FOR THE GEM CODE

    Identify the directory you are going to install GEM into. This is only tricky if you have a Hybrid Environment, in which case your admins need to create a samba share such that a single directory appear in both unix/linux and windows. When GEM is installed on a samba share, the same code works unchanged to monitor both your Unix and Windows environments seamlessly.

  • INSTALL & VERIFY perl 5.8.1 OR LATER

    Each monitoring station requires perl to be installed at version 5.8.1 or later. This can be verified by running the command perl -v (prints the version number).

    On windows, we recommend the latest Activestate Perl from http://www.activestate.com. Use the default installation options.

    On UNIX, binary releases can be downloaded from http://www.cpan.org. You can have your unix admins install/provide perl for you or can install your own version of perl in your own home directory. The second approach has the advantage that you have control, and will not need permission of the unix administrators to add/update modules.

  • ADD PERL MODULES

    Verify availability of the perl DBI module and associated DBD perl modules. DBI and DBD::ODBC are required on windows servers. DBI and Native DBD Drivers (DBD::Sybase, DBD::Oracle) are required for the perl on your unix monitoring servers.

    On Windows, install these modules via :

    [StartKey] -> [Run] and then type ppm install DBI
    and
    [StartKey] -> [Run] and then type ppm install DBD::ODem/em.

    If your company has a firewall/proxy set up such that the ppm commands fail we suggest:

    Bring up a browser and go to http://ppm.activestate.com/PPMPackages/zips/ - select your version and go to the Windows subdirectoy and download the latest zip file. The file will contain 3 files when extracted - a .ppd file, a readme, and a .tar.gz file. To install this ActiveState PPM package, run the following command in the current directory:

    ppm install DBD-ODBC.ppd

    You should download both the DBD::ODBC and DBI libraries. The latest versions (as of October 2007 are 1.13 and 1.58 respectively.

    On Unix, the DBI and DBD::Sybase modules must be installed by hand. You may also wish to install DBD::Oracle, although oracle support is currently limited with full support scheduled for 3Q08. Module installation on unix may require administrator help (the make install step requires write access to the perl library directories which are write protected unless you are using a local version of perl.

  • IDENTIFY SCRIPT CAPABLE DIRECTORY ON YOUR WEB SERVER

    While GEM does not "technically" require running scripts on an web server, several of the interfaces were designed primarly for web access. To run these scripts, you must create a 'script enabled' directory on your web server. GEM provides perl scripts which, when deployed to this directory will provide web based features. The primary interfaces provided at this time are a monitoring and reporting tool for your database servers. You should secure this web directory as you see fit for information of that type.

  • CREATE A MONITORING DATABASE

    GEM requires a small monitoring database. Place this database on Sybase if you have a Hybrid or Unix Only environment and on either Sybase or Microsoft SQL Server in windows only environments. This database will store enterprise monitoring information and must be accessible from all your monitoring servers. If you have Unix Systems yet wish (for some reason) to use a SQL Server as your repository, you can install FreeTds and the unix verison of DBD::ODBC.

    • Create A Database Named gemalarms with size=200MB
    • Turn on select into and truncate log on checkpoint database options
    • Create login named gemalarms with password gemalarms
    • Add login gemalarms to database gemalarms in group public (not a dbo account)

    use master
    go
    create database gemalarms on data_device_6=200
    go
    use master
    go
    sp_dboption gemalarms,"select",true
    go
    sp_dboption gemalarms,"trunc",true
    go
    use gemalarms
    go
    checkpoint
    go
    sp_addlogin "gemalarms","gemalarms","gemalarms"
    go
    sp_adduser gemalarms
    go

  • IDENTIFY THE DATABASE SERVERS YOU WANT TO MONITOR

    If you are trialing GEM, we recommend installing on 4-6 database servers on 4-6 separate sytems and that the trial be either 'all unix' or 'all windows' databases. You will require the following rights on these database servers:

    You need windows administrator access your windows database servers

    • needed to monitor disk space, server processes, and event logs
    You will require sa_role access for all your Sybase database servers
    • for installation of the stored procedure library
    • you can get away with non-sa access for monitoring
    You will require "no-password" access via ssh or rsh for your unix systems
    • for access to log files
    You will require sybase unix account access for your Sybase database servers.
    • this is used to read sybase event logs

  • SET UP YOUR ODBC DATA SOURCES OR interfaces FILE

    Ensure that all the database servers identified in the last step have span>ODBC entries (Windows Monitoring Box), and have appropriate Sybase interfaces file (and Oracle tnsnames.ora) entries. The configuration process (configure.pl) will automatically discover your servers based on these values.

  • SET UP SECURE INTRA SYSTEM COMMUNICATIONS (UNIX)

    All UNIX systems involved with GEM must be accessible from the unix monitoring system by ssh/scp (preferred method), rsh/rcp, or standard ftp. The account you will be using to schedule the batch jobs and to run gem must be able to get into these systems without passwords (except in the case of ftp where a password is ok).

    monitor ebarlow: ssh sybase@sybprod
    Last login: Wed Apr 2 11:37:29 2008 from monitor
    $

Suggestions on your first implementation

Database administrators can be, and should be, a cautious lot. We obviously roll out a tool like GEM to a single test box and install the client on our personal workstation. Lowest risk configuration and all that. An installation like that totally and completely buys you nothing. GEM will be of no value. That is totally and completely the wrong way to roll out GEM.

The correct way to roll out GEM is to implement it on between 4 and 6 test and dev servers at once using a 'server' class box as a monitoring station (it does NOT need to be a dedicated box but it runs a bunch of perl scripts). Obviously dont trial GEM on anything production (that includes your workstation). Your Trial needs this many systems to let you to see how the GEM software works. Plus, GEM is a LOT of effort to implement, so you might as well get some use out of your GEM trial.

Finally, ensure that you have defined ALL items on the GEM pre-install check list. For example, GEM loses half its value in a unix environment if you dont have 'no password' access via either ssh or rsh connectivity to your unix systems. Simarly, "ADMINSITRATOR GROUP" rights are required for direct access to your Windows servers log files. You need these. Additionally, it is highly strongly very suggested that you have a web server with a script capable directory into which you can place our cgi perl interface. While we do have a TK version of the interface, the web interface is the one you should use.

Presuming you have a reasonable network configuration, GEM has a pretty low footprint on everything but the monitoring host system. Network impact is mostly semi hourly fetches of your database error logs. GEM does install our system stored procedures on all your database servers. They are a nice package and you will like them. On the monitoring system, GEM runs a bunch of perl batch jobs - and frequently multiple jobs will run at the same time. We recommend multi processor systems as your monitoring systems, but most 2-4 year old server class systems will be fine (find some depricated system). Linux is a good choice for a monitoring system to monitor unix database.

As far as your database servers go, GEM has the experience to be a good citizen. GEM is designed by active dbas, not a lab by developers. Your developers and users should have not have any noticable degredation in performance as a result of GEM.

A good solid burn in environment for GEM should have between 4 and 6 database servers and 4 to 6 different host systems (of course nothing production should be touched). It would be slightly simpler to implement the trial if running all windows or all unix databases.

Download & initial setup

  • DOWNLOAD AND EXTRACT GEM

    You should now download and extract the GEM software from http://www.edbarlow.com/download.pl.

    You will need a passcode to download the software which can be recieved by mail from SQL Technolgies.

    Extract into the directory identified in the pre-install checklist. You should need to install into only one location regardless of whether you have a hybrid or non-hybrid environment.

    The GEM download file is a standard compressed tar file and can be extracted with winzip (windows), gnu tar (tar xvzf gem.tgz), or with gzip -d gem.tgz; tar xvf gem.tar. If you install GEM into a subdirectory, do not use a directory that includes space characters (ie. Program Files). White space in the directory name is not a tested option.

  • SET YOUR DISPLAY ENVIRONMENT VARIABLE IF UNIX

    Many of the GEM tools use perl/Tk for a user interface. On windows this works without support, but on unix this requires some type of X terminal manager (like exceed). On UNIX, you must set the environment variable DISPLAY correctly.

    EXPORT DISPLAY=your_ip_address:0.0.

    Unix users will also require an X terminal manager installed on their workstation. Your infrastructure support can help you install this if you are not allready set up.

  • VERIFY YOUR PERL INSTALLATION

    perl troubleshoot.pl

    Use the program troubleshoot.pl in the root of the installation tree to test for the presence of necessary perl libraries. troubleshoot.pl is the GEM installation trouble shooting program and it can be run to diagnose any GEM issues - a successful run indicates that your configuration is stable.

  • RUN configure.pl

    See the section on configure.pl for details on what this program does.


configure.pl

GEM Configuration Tool

configure.pl is the GEM configuration utility. This utility will build the configuration files (stored in the conf subdirectory) and runs some basic setup tasks for GEM. The GEM Configuration Tool is quite intelligent, and will survey your systems to give you a good baseline that should require minimal tuning to work within your environment.

Some of the things configure.pl does include:

  • Create Configuration Files

    One primary output of configure.pl is the configuration files stored in the conf directory. These files can be hand edited once GEM is set up.

  • Rebuild of all the code

    Changing the hashbangs (the first line of the file on unix is known as a hashbang and defines the command line interpreter to use), removing Control-M (dos newline) characters from the files, and modifying library paths.

  • Creation of the GEM Batch Jobs

    configure.pl also creates the GEM Batch Jobs. On a win32 monitoring station, GEM can schedule these jobs. On a unix environment, we create a crontab.txt file they can be schedule via cron.

  • Monitoring Setup

    The initial database installation is done via configure.pl

  • Initial Build

    This program also will generate an initial version of the GEM console and prepopulate the alarm system with alerts and reports relevant to your environment.

    Important notes

    • Configure.pl creates a file logs/configure.log which includes both console messages AND diagnostic messages. If you have a problem please look at this file. It contains all output including output that should be only produced with configure.pl --DEBUG.

    • Work through the tabs from left to right.

    • "Save Configuration Changes" writes your configuration files. While some operations (adding stored procedures to servers) are permanent once completed, the majority are not. It is important to save before you quit.

    • Leave the backups tab alone for your first pass

    How to run configure.pl

    perl configure.pl

    If you wish to see diagnostic messages, you may run

    perl configure.pl --DEBUG

    A log file of the run will be placed in the logs subdirectory.

    FOLLOW THE FOLLOWING STEP BY STEP INSTRUCTIONS TO COMPLETE YOUR INSTALLATION

    The welcome tab

    PURPOSE: License Details and Acceptance

    This tab gives licensing options for GEM. Select "I AGREE WITH THE LICENSE" to continue after you have read all the information provided when you click on the various buttons on this page.

    The GEM Help button will help guide you through your install.

    All GEM code is copyright (c) 1995-2007 by SQL Technologies=head1 The Register Tab

    PURPOSE: Basic configuration setup plus register your product.

    • Enter Administrator Info. This information is never shared so make it correct.

    • Select Environment, Product, and License. Running on a samba share is appropriate in Hybrid Windows/Unix environment. Select Free GEM for product trials. Contact barlowedward@hotmail.com for a product key for "Full Gem".

    • Select Preferred Connect Options to define the legal system connectivity options.

    • Click "REGISTER YOUR VERSION" to register your product.

    The paths tab

    [This tab is trivial if you have an All Windows or All Unix environment.]

    • Enter FULL Path(s) to your version of perl. eg. /usr/local/bin/perl or C:/perl/bin/perl.

    • Enter All legal paths to the GEM top level directory to define paths to required libraries. On a samba / hybrid installation where /apps/gem maps to c:/gem on win32 host win2, you would have '/apps/gem' (the Unix Code Location) and '//win2/c$/gem' as the NT Code Location, and Alternate #1 would be 'C:/gem'. Alternate #2 will be blank.

    • Enter Unix/Linux Web Browser Its probably /usr/local/bin/firefox. Skip if you dont know it.

    • Enter Sybase Home Directories in a list so GEM knows what paths to search while surveying remote systems.

    PURPOSE: Ensure Mail Connectivity

    This tab is only required on Windows servers. For these servers, you need to provide your SMTP mail server name. If "Test Mail Install" should send a test message to your GEM Administrator Mail Address (entered earlier).

    The servers tab

    PURPOSE: Register your Databases & Systems with GEM

    • Access To Databases must be via 'sa' or an account with similar roles/permissions. GEM does not support blank/empty administrator passwords.

    • Access To Unix Systems must be via ssh, rsh, or ftp using an account with reads permissions on the DBMS log files in $SYBASE/$ORA_HOME. The ssh or rsh connection should not require a password.

    • Access To Win32 Systems is via native authentication - the account must read Event Logs and Administrative File Shares (ie. requires administrator).

    • The Full GEM Add Server Wizard creates a best guess at your environment - prompting for logins and passwords appropriately. To save time, it guesses passwords by reusing previously entered ones. This wizard only registers NEW servers and is re-runnable.

    • The Free GEM Add Server Wizard For FREE Gem provides a simple data entry screen to enter up to 4 DATABASEs.

    • The Server Type Buttons allow you to define category (only PRODUCTION/CRITICAL systems are page-able), change passwords, and deregister servers

    • The Quick Add Button allows you to register a single Database or System.

    • The UnixComm & WinComm columns allow you to specify preferred communication methods from Unix/Windows Monitoring stations respectively.

    The files tab

    PURPOSE: Identify Database Home Directories & Important FIles

    This tab contains rows for each UNIX/Linux system, listing survey results (collected earlier) in blue and configuration directives in black. If survey results look incorrect, correct the configuration directives (with the add/delete buttons) and rerun the surveys.

    • Sybase ASE Server $SYBASE Directory is a valid sybase server directory and is searched for Error Logs, RUN_* files, and other config information.

    • Application Log File (unused in v1)

    • Ignore This RUN_* File directives that the Sybase server RUN_* file is unused and should be "ignored" to prevent confusion about what runs where.

    • Operating System Log File (unused in v1)

    • Sybase Open Client $SYBASE Directory are surveyed only for interface file consistency checks.

    • Oracle ORAHOME Directory (unused in v1)

    • SQL Server Maintenance Plan Logs (unused in v1)

    When completed, you consider clicking "SAVE CONFIGURATION CHANGES"=head1 The Connect Tab

    PURPOSE: Test connectivity and survey your systems.

    • Page-Top Buttons allow you to can see servers by type.

    • Connect To All to test connectivity. Windows GEM requires odbc settings for all databases and native administrator access for systems. Unix GEM requires normal database access (interfaces file for sybase) plus host access via ssh (preferred), rsh, or ftp. Fix up any servers you cant connect to.

    • Survey All to run a survey of all your servers. Normally, surveys are scheduled but we need some initial data. Click "Dont Skip" to not skip servers which have been recently surveyed.

    The backups tab

    When you create your servers, it will also create two backup plans for you: one that has the same name as the server and the other which has _WEEKEND appended to the server name. Data on your plans is stored in the configure.cfg file and is hand editable. Please see documentation on the GEM Maintenance And Backup System.

    • For initial setup, you can ignore this section

    The console tab

    PURPOSE: Define Console Special Behaviors

    • For an initial setup or trial, you can ignore this section

    • This page allows you to copy the console to another location. You may be interested in doing this for security reasons. By default, the CONSOLE Web Pages are created the data/CONSOLE_REPORTS directory but these reports can be published to another location (perhaps on your web server). Most users will NOT wish to do this.

      You may copy the files via unix copies (ssh, rsh, ftp - select the Copy using FTP button) or to a local directory (like //NEWSERVER/c$/web/gem). Some users may want to do thsi copy for security reasons - to allow non administrators to use the GEM console. This keeps those same users away from the GEM directory tree which stores other highly sensitive data.

    The alarms tab

    PURPOSE: Set Up Alarms & Monitoring Systems

    • Fill in the Alarm Database Server Info with information about the 200MB (approximate size) alarm database you set up before running configure.pl (see pre-requesites). The email address is arbitrary and all alarms will be mailed using that as a reply-to address. It is reasonable to set this to an INVALID mail address (alarms@yourcompany.com or Gem@yourcompany.com) because you wont ever reply to the messages.

    • Install/Upgrade Alarm Database will create alarm database tables & procedures

    • Validate Alarm Database checks the database for sanity and version.

    • Fill in Web Server information including the map of hostname/filename to alarming url. Once this is done click Populate Local cgi-bin to copy formatted scripts to .../cgi-bin and Copy to Web Server to copy those files to the web server.

    The install software tab

    Now would be a good time to Save Configuration Changes!

    Click the "Install Software" button to run the numbered steps listed below that button.

    • Reformat Code Removes Control-M's and Sets Hashbangs

    • Create GEM Batch Scripts creates your batch scripts. In a hybrid environment run this on UNIX last - so none of the files have control-M characters embedded

    • Backup Manager sets up default directories for the backup system

    • Modify Console Scripts

    The post install tasks tab

    • Create a crontab file for your unix installation and schedule it.

    • Snapshot your windows servers services to create conf/pc_service.dat. This basically creates a list of all the running services on all your windows boxes for future monitoring. You will be notified if any services go down. Assuming your environement is currently stable, a current snapshot will give you a pretty good listing of what needs to be monitored.

    The scheduler tab

    PURPOSE: Define Scheduling and Actually Schedule Batch Jobs

    • Look over the jobs as defined to ensure that they are correct - you can use the default Frequencies by default

    • Schedule on this server (Win32 Monitoring Only) provides a native interface to the windows task scheduler. Provide an account & password which has access to windows event logs, windows disk administrative shares, and windows services (probably requires adminstrator).

    • Create Unix Crontab will create a crontab.txt file in the gem home directory that is appropriate for unix cron. To install into cron:

         export EDITOR=vi
      

      crontab -e

      <vi window pops up>

      :r crontab.txt

      Be sure that the account you schedule this on has a .forward file so any errors will not be routed! A .forward file is a file in your HOME directory that contains mail addresses.


    POST INSTALLATION STEPS

    Other things to do

    Once you are done with your GEM installation, there are a variety of post-installation tasks . This checklist is designed to help you with these tasks.

    CLEAN UP UNIX SCRIPTS DIRECTORY

    There are outstanding issues with chmod() and dos newline removal that impact Hybrid environments. Specifically, you must run configure.pl on unix AFTER you run configure.pl on windows. This is because there is no way to get windows to use just a standard newline instead of CR/newline (so all files will have a ^M in them if you run on windows last).

    You should also run the Batch File creation on UNIX last. If you ran your Win32 install after your Unix install, go to your unix server and run perl upgrade.pl to fix these issues.

    Modify your Password Files

    The password files are located in conf/*_password.dat and are hand editable.

       sybase_passwords.dat
       oracle_password.dat
       pc_password.dat
       unix_passwords.dat
       sqlsvr_password.dat
    

    All configuration files are self documented. An archive of your old configuration files is kept in conf/saved_files

    If you add new databases or systems, recreate & reschedule your Batch Scripts. Most scripts work against all servers in the configuration files, but some scripts work on a specific database (specifically the ones with the Database name in the file name).

    Windows Directory Cleaner

    One of the most horrific tasks of the windows DBA is disk space cleanup. Something is *always* running out of space... GEM automates this task for you.

    The file conf/Win32_Cleanup.dat contains a list of disk space Cleanup instructions for the windows cleanup batch job. It is suggested that you customize this file to include your tran/archive log directories, your application log directories, and any other directories that fill up on your systems. A best practice is to add rows to this file over time, every time the windows disk space monitor spots a directory that is filling up. The lines in this file conform to the arguments of ADMIN_SCRIPTS/bin/purge_old_files.pl, and include --hours and --days to identify the keep times, --dir to identify the directory to purge, and either --compress= or --del (to identify what to do with the old files). See the syntax of purge_old_files.pl for other options.

       # FILENAME: Win32_Cleanup.dat
       #
       # USED BY:  Batch job Win32FileCleanup
       #
       # SYNOPSIS: Directives for cleaning up win32 drives.  This file is
       #    identical to Unix_Cleanup.dat but for the --compress (Win32 only)
       #    argument which is missing and the required --host argument (Unix Only).
       #
       #
       # FORMAT  : HOSTNAME drive drive drive.
       #
       # DETAILS : The arguments are the arguments to purge_old_win32_files.pl
       #
       #  --host=<host>
       #  --hours=<hours>
       #  --days=<days>
       #  --dir=<dir>
       #  [--del]                - do you wish to delete the file?
       #  [--done]             - only work on files with .done extension
       #
       # EXAMPLE FORMATS:
       #
       # Delete 14+ day old files on host h1
       # --days=14 --dir=//adsrv012/c$/oracle/admin/SCPROD/udump --del
       #
       # Delete files with .done extension older than 2 days old
       # Note: this will delete .done.gz files and .done files
       # --days=2 --dir=//SRVR001/d$/logshipbuffer --del --done
       #
       #######################################################################
       #
       # START RECOMMENDED PURGES
       # --days=7 --dir=C:/gem/data/logs -del
       # --days=7 --dir=C:/gem/data/batchjob_errors -del
       # --days=7 --dir=C:/gem/data/GEM_BATCHJOB_LOGS -del
       # --days=7 --dir=C:/gem/data/CONSOLE_REPORTS -del
       # --days=7 --dir=C:/gem/data/cronlogs -del
       # --days=7 --dir=C:/gem/data/errors -del
       # --days=7 --dir=C:/gem/data/backup_reports -del
       # --days=7 --dir=C:/gem/data/html_output -del
       # --days=30 --dir=C:/gem/data/BACKUP_LOGS\*\sessionlog -del
       # --days=30 --dir=C:/gem/data/BACKUP_LOGS\*\errors -del
       # END RECOMMENDED PURGES
    

    Unix Directory Cleaner

    The Unix_Cleanup.dat file will control unix file/directory cleanups. It is similar to the Windows cleanup file but for the addition of the --hostname argument.

    port_monitor.dat

    Port Monitoring is a quick availability test - and a good way to find out if a database or system is down. The file conf/port_monitor.dat contains a list of Hosts and Ports to Monitor.

    This file is autogenerated by configure.pl but will probably require hand editing.

    Simple format:

      # hostname                      PORT SERVER   DESCRIPTION
      SRVR001                        1433 SRVR001   Microsoft SQL Server
      SRVR007                        1433 SRVR007   Microsoft SQL Server
      brokersql                      1433 BROKERSQL Microsoft SQL Server
      mail1                            25   Sendmail on mail1
    

    Sybase Replication Monitoring

    The File conf/Replication.dat is used by the Sybase replication server monitoring scripts and will need to be popluated with your Rep Servers and Routes.

       # FILENAME: replication.dat
       #
       # SYNOPSIS: Lists your sybase replication server information
       #
       # FORMAT:
       #
       # REPSERVER $NAME $LOGIN $PASSWORD $THRESHOLD_MB
       # REPROUTE  $SOURCE_SVR $SOURCE_DB $DEST_SVR $DEST_DB $REPSERVER
       #
       # EXAMPLES (350MB Thresholds):
       #
       #  REPSERVER REP1DR_RS sa <pass> 350
       #  REPSERVER REP2_RS   sa <pass> 350
       #  REPSERVER REP1_RS   sa <pass> 350
       #
       #  REPROUTE  SYB1 my_db SYB2   my_db REP2_RS
       #  REPROUTE  SYB1 my_db SYB1DR my_db REP2_RS
       #
       # ONCE YOU SET THIS FILE UP - PLEASE RUN
       #      ADMIN_SCRIPTS/monitoring/SybRepMonitor.pl --ACTION=INSTALL
       # MONITORING IS HANDLED AUTOMATICALLY BY THE GEM BATCH SybRepMonAgent
       ##############################
    

    There are two programs that check replication. The batch SybRepServerChecker will check all your rep servers to be sure that no routes are suspended, that disk space is reasonable, and that there are no other serious errors. There are also three batches, SybRepMonAgent, SybRepMonCheck, and SybRepMonReport that compute latencies of your replication, and will warn you when replication falls behind. These three batch programs call SybRepMonitor.pl. This program takes an argument --ACTION.

    To initialize this system, you need to set up your configuration file and then run myperl SybRepMonitor.pl --ACTION=INSTALL

    After the system is INSTALLED, the above mentioned batches will perform your monitoring automatically. Below is the syntax for SybRepMonitor.pl:

       Usage: SybRepMonitor.pl --ACTION=INSTALL|UNINSTALL|MONITOR|REPORT|CHECK
            [--SERVER=xxx] [--DEBUG] [--OUTFILE=file]
    

            if --ACTION=INSTALL will install
                    --REBUILD redoes table creates
            if --ACTION=CHECK will check latency vs --ALARM_MINS
                    - mail to --ALARM_MAILTO as needed
            if --ACTION=REPORT will create a report
                    - will print data for --REPORT_HOURS hours
                    - will only show rows > --REPORT_THRESH seconds
                    - will aggregate based on --BUCKETSIZE
                    - will print HTML output if --HTML is passed
            if --ACTION=MONITOR will insert rows into all source databases
                    - will insert --ITERATIONS rows
                    - sleeps --FREQUENCY secs between each insert
    

    Adjust Alarming Thresholds

    You may not wish all your alarms to work at the same thresholds - preferring to set some thresholds individually. Generally speaking, this is most needed when some resourse is nearing "full" and generating alarms, but you can not fix the problem immediately - intending to fix the problem later. The conf/threshold_overrides.dat file allows you that control.

       # FILENAME: threshold_overrides.dat
       #
       # SYNOPSIS: Monitoring Thresholds
       #
       # DETAILS : This file permits you to turn off alarms and to control their
       # severity.  This is a Comma Separated File
       #
       # SAMPLE FORMAT:
       # <system> <monitor_program> <level> <threshold> <optional subsystem>
       #
       # monitor_program=UnixDiskspace|PcDiskspace|SybSpaceMonitor
       #                         |SpaceMonitorSqlsvr|ThresholdManager
       # level=CRITICAL|ERROR|WARNING
       #
       # remember to reset higher priority thresholds so there is no overlap - ie if you
       # set ERROR to 98%, then you should set critical to something larger (say 99%).
       # The programs may act wierd if the CRITICAL threshold was less than the ERROR threshold.
       #
       # By default - SybSpaceMonitor & SpaceMonitorSqlsvr will use 90/92/95% as
       # their default thresholds
       #
       # Replication thresholds are in minutes.  For replication,  you can put a
       # blank target name in if you want but not a blank source name
       #
       # EXAMPLES
       # eg.  directfeed,UnixDiskspace,WARNING,80,Disk: /var
       # eg.  direct1,UnixDiskspace,WARNING,80,Disk: /var
       # eg.  direct2 ,UnixDiskspace,WARNING,80,Disk: /var
       #
       # eg.  SYBSVR1,SybSpaceMonitor,WARNING,80,db1
       # eg.  SYB1,SybSpaceMonitor,ERROR,95
       # eg.  SYB1,SybSpaceMonitor,ERROR,95,clientdb
       #
       # eg.  ADSRV057,PcDiskspace,ERROR,98,d$
       # eg.  ADSRV057,PcDiskspace,CRITICAL,99,d$
       #
       # eg.  SERVERNM,ThresholdManager,CRITICAL,99,logsegment|default
       #
       # EXAMPLE REPLICATION STUFF
       # eg.  SOURCESVR,SybRepMonCheck,CRITICAL,30,TARGETNM
       # eg.  SOURCESVR,SybRepMonCheck,ERROR,15,TARGETNM
       # eg.  SYB1,SybRepMonCheck,CRITICAL,80,IMAGSYB2
       # eg.  SYB1,SybRepMonCheck,CRITICAL,80,
       # eg.  SYB1,SybRepMonCheck,ERROR,60
       #
       #########################################
    

    The sybase threshold manager

    Another separately installed GEM component is the Sybase threshold manager. This program (ADMIN_SCRIPTS/bin/threshold_manager.pl) installs customized sp_thresholdaction procedures into your databases and allows you to find peak usage (most important for log file sizes). This proactive utility will hopefully prevent you from running out of log space - because you will have been warned in advance in such a way that you can expand capacity before there is a production alert.

    The sp_thresholdaction procedures we install are simple reporting versions that insert peak usage data into a small table in sybsystemprocs. You can run a report showing peak usage of data and log segments in your databases which is the Threshold Report you can view in the console.

    This procedure needs to be installed manually by running

    ADMIN_SCRIPTS/bin/threshold_manager.pl --ACTION=INSTALL

    after you have registered your servers. One warning. If you increase the *size* of any of your databases, you will need to rerun the above installation command. Sybase thresholds are based on free pages and the percentages are therefore set when you run the installation. So, if you double the size of a database, your 60% threshold will be firing at 30% and recording it as 60% full! Just reinstall and you will be ok!.

    Other commands of interest:

    ADMIN_SCRIPTS/bin/threshold_manager.pl --ACTION=CHECK

    ADMIN_SCRIPTS/bin/threshold_manager.pl --ACTION=REPORT

    This feature only works with Sybase

    Snapshot Win32 Services

    This information is stored in conf/pc_service.dat and can be created from configure.pl using "Snapshot Win32 Services" on the Post install tasklist. The file can also be created by the unscheduled batch file PcServiceCheckerInit.ksh.

    You will probably need to hand edit this file - the snapshot is at a point in time and some services may get turned off or just dont stay up. After you have created this file, simply remove lines which get flagged by the monitors yet are not important.

      cd win32_batch_scripts/interactive
      PcServiceCheckerInit.ksh
    

    Install web server scripts

    The alarms tab will populate the directory cgi-bin with runnable web scripts. The button to actually distribute those scripts to your web server is currently unimplemented. You will need to copy the scripts by hand.

    This is made easier by the fact that the scripts have been modified to run in your environments, but it is possible that the scripts do not execute.

      a) copy the contents of cgi-bin to the web server scripts directory you identified for your use
      b) chmod 755 these scripts to make them executable (unix only)
      c) try to run mimi.pl when in that directory.  Mimi is a cgi-bin script - so it will hang asking for parameters if it runs successfully.  It might also give you a command not found or library not found message.
      d) you might need to change the hashbang (first line) to point to your perl
      e) try running printenv.pl from your web browser... this is a very simple script that should always work.
      f) try running mimi.pl from your web browser
    

    It is beyond the scope of this document to discuss web server setup. All the directory requires is script access.

    The top of these scripts should look like:

       #!/usr/local/bin/perl-5.8.1
    

       use lib qw(/apps/sybmon/dev/lib)
       my($COMPANYNAME)="YOUR COMPANY";
       BEGIN {
          if( ! defined $ENV{SYBASE} or ! -d $ENV{SYBASE} ) {
             if( 1==2 ) { print 'doh'; }
             } elsif( -d "/export/home/sybase" ) {
                $ENV{SYBASE} = "/export/home/sybase";
             } elsif( -d "/apps/sybase" ) {
                $ENV{SYBASE} = "/apps/sybase";
             }
          }
       }
    

    The paths, company name, sybase directories, and perl version have all been customized. Make sure they all look ok.

    Test this script through your web browser.

    RUN monitor.pl and eventviewer.pl

    perl monitor.pl
    perl eventviewer.pl

    Two modules are currently shipped with GEM - an event viewer and a monitor. Both are located in the root directory. To run, use /monitor.pl. Under windows, you should be able to just double click on these programs to run them and can create desktop shortcuts as needed.

    Run fix_db.pl to for initial db config errors

    perl ADMIN_SCRIPTS/bin/fix_db.pl -Ax

    This is a cute little database configuration checker. You might as well run it. It shows dboptions you might wish to correct and identifies common configuration problems. To correct these problems remove the -x (noexec) option.

    ADDITIONAL CONSOLE REPORTS

    You can add customized reports to the console easily using conf/console.dat. These reports can be pretty much any command or sql statement. Check out the file for the syntax.

    SCHEMA GENERATION

    It is recommended that you create object level backups of your server DDL once per week. GEM provides the excellent script dbschema.pl (written by Michael Peppler and David Owen) in the ADMIN_SCRIPTS/bin directory to facilitate this process. Because this script is written in SybPerl (not DBI::Sybase), we do not provide an integrated approach to creating these audits. I recommend you back up your ddl once per week on your production servers.

    DNS Montitoring

    You can set up conf/dns.dat and then do dns monitoring with

    0,20,40 * * * * /usr/bin/nice /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/monitoring/dnsmonitor.pl --OUTFILE=/apps/sybmon/dev/data/html_output/dnsmonitor.pl.txt

    Scheduling the batch jobs

    Win32 Scheduling

    The gem configuration utility will schedule your jobs on win32 monitoring stations.

    Unix Scheduling

    At 2 points in configure.pl, you can create a unix crontab file. This file is named crontab.txt and is placed in the GEM root directory. You should schedule this file using yoru crontab on unix. To do this run

       export EDITOR=vi   # presuming u like vi
       crontab -e
       <vi window>
       :r crontab.txt
       :wq
    

    You should also place a .forward file in your home directory to redirect mails to your account.

    Your own scheduled tasks

    Put the output of any of your routine scripts into data/html_output or data/cronlogs to have the reports show in the GEM console. This is a convenient spot to redirect output in any case. The filename itself is used to generate the console report.

       # the following job compare a server to its dr copy
       1 22 * * * /perl/bin/perl /apps/sybmon/dev/ADMIN_SCRIPTS/bin/datacompare.pl -Usa -SSYB1 -Pxxx -sSYB1DR -pxxx -uxxx -h > /apps/sybmon/dev/data/html_output/Rowcount_Compare_Prod_DR.html 2>&1
    

    Schedule additional alarming reports. These need to be set up in the cgi-bin mimi.pl reports screen before they can be run. But you can snapshot anything you can see on those screens and mail the results as appropriate

       0 10,12,14,16,18 * * 1-5 /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/monitoring/RunMimiReport.pl --SUBJECT="2 Hour Unix Status Report" --MAILTO=def@bac.com,abc@abc.com --REPORT=Unix2Hour --TIME="2 Hours" > /apps/sybmon/dev/data/GEM_BATCHJOB_LOGS/RunMimiReport_Unix2Hour 2> /apps/sybmon/dev/data/batchjob_errors/RunMimiReport_Unix2Hour
    

    I sync logins between my replicated servers with

       38 9 * * 0 /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/bin/copy_syslogins.pl -FSYB1 -Usa -Pxxx -tSYB2 -usa -pyyy > /apps/sybmon/dev/data/GEM_BATCHJOB_LOGS/copy_syslogins_SYB1.log 2>/apps/sybmon/dev/data/batchjob_errors/copy_syslogins_SYB1.log
    

    I monitor cisco logs with

       18,48 * * * * /usr/bin/nice /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/monitoring/cisco_logmon.pl --INFILE=/var/log/cisco.log --TODAYONLY >/apps/sybmon/dev/data/GEM_BATCHJOB_LOGS/cisco_logmon.log 2>/apps/sybmon/dev/data/batchjob_errors/cisco_logmon.log
    

    I monitor centrally collected (via syslogsd) unix logs with

       5,15,25,35,45,55 * * * * /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/monitoring/unix_monitor.pl --INFILE=/var/log/messages --TODAYONLY --EXCLUDEFILE=/apps/sybmon/dev/ADMIN_SCRIPTS/bin/solaris.excl --MAXLINES=100000000 --HTML --OUTFILE=/apps/sybmon/dev/data/html_output/unix_monitor.html >/apps/sybmon/dev/data/GEM_BATCHJOB_LOGS/unix_monitor.log 2>/apps/sybmon/dev/data/batchjob_errors/unix_monitor.log
    

    Directory security

    GEM is an administrative tool, designed for system administrators. As such, the conf directory contains all your passwords. You need to protect this data via operating system permissions.

    We strongly recommend that all other directories in the tree, with the exception of the data/CONSOLE_REPORTS tree, be also made secure. Technically, you can allow read/execute access to these files, but it would be easier to only allow owner/group access - much of the data contained may be sensitive.


    How To Upgrade GEM

    GEM is designed for simple upgrades. The steps are as follows

    • Download the latest codeline

    • Extract on top of your existing distribution!!!

      No configuration files are shipped with the installation so there should be no problems. All configuration files are shipped with a .sample extension.

      On your windows monitoring server, you may not be able to extract the code because the task scheduler is running a batch job that calls one of the perl scripts. Windows will not let you overwrite a program that is running. This does not happen under Unix. You can either go into the task scheduler, stop all running jobs before rerunning the extract, or kill the jobs and then stop the task scheduler process. I normally just stop the jobs, but sometimes (because end task does not work cleanly) i am forced to stop the scheduler process for the duration of the extract.

    • Run perl upgrade.pl

      After you download, your GEM scripts will have *OUR* hashbangs, library paths, and sybase environment. This is probably not what you want.

      Run perl upgrade.pl to finalize the installation. This step is MANDATORY and will perform a variety of steps like updating your configuration files and re-installing your alarm database. During the period you are upgrading - your GEM batch jobs will probably fail and send you alarm mail. Ignore these failures notifications.

      The alarm database reinstall will save table data, drop all tables, reinstall tables, and finally copy back data. If there is a problem, it will leave a tables with the extension _BAK containing the original data. So, under the hood, the upgrade process for the table Event is "select into Event_BAK from Event", "drop table Event", "create table Event", "insert Event select * from Event_BAK", and finally "drop table Event_BAK". Upgrades can not start if any tables with the extension _BAK exists - you must remove them before the upgrade.

      If there is a problem, you can probably just drop the BAK table - but it would be better to recover by following a process akin to "truncate table Event", "insert Event select * from Event_BAK", "Drop table Event" (you have to do these by hand). Got it? Any questions on a failed install please contact SQL Technologies tech support.

      The final install step redoes your hashbangs, perl library paths, and removes all control M's from the code line. A log of the upgrade is saved in the logs directory.

      ONCE YOU HAVE SUCCESSFULLY RUN upgrade.pl YOU ARE DONE


    GUIDE TO GEM BATCH JOBS

    DESCRIPTION

    Batch jobs are created by the GEM configuration program and are used to monitor your servers and to create the GEM Console. These scripts are organized into 8 directories that contain duplicate scripts. The batch jobs name is the "monitor program" shown by the monitoring system.

    The directory structure is

       win32_batch_scripts or
           batch
           interactive
           plan_batch
           plan_interactive
       unix_batch_scripts
           batch
           interactive
           plan_batch
           plan_interactive
    

    Files are created in these directories as needed by configure.pl. For example, if you tell configure.pl that you have only windows based systems, the subdirectories under unix_batch_scripts will not be populated. Similarly, if you are in an all-sybase environment, the oracle and sql server scripts will not be created. Batch jobs are created with a .ksh extension. Additionally, the win32_batch_scripts tree will contain duplicates with a .bat extension. This simplfies scheduling.

    The major difference between the "*_batch" and "*_interactive" directories is the redirection of standard output. Scheduled jobs use the "batch" directories which differ from the "interactive" directories only in that they redirect standard output and standard error into the GEM_HOME/data/GEM_BATCHJOB_LOGS directory. These files can be viewed through the Console.

    You are responsible for scheduling these jobs (sort of). If you are running on windows, configure.pl (the GEM Configuration utility) can auto schedule your jobs in the windows task scheduler. If unix, configure.pl can create a customized GEM_HOME/crontab.txt file for your inclusion in your unix crontab (via crontab -e; set the variable 'EDITOR=vi' if 'crontab -e' leaves you in 'ed' mode. On Win32, GEM provides both .ksh and .bat versions of all scripts to support those who use a unix emulator like cygwin or MKS toolkit.

    TO RUN THE JOBS INTERACTIVELY

    When running batch jobs by hand (for example to test), always use the files in the 'interactive' or 'plan_interactive' subdirectory trees. If you use the scripts in batch or plan_batch, you wont be able to view the output because it is redirected to files!!!


    SUMMARY OF BATCH JOBS

    The following table is a summary of the jobs that can be created by GEM's configuration utility.

    Scripts In The 'plan' Subdirectories
    Job Name Win32 or
    Unix
    Description Recommended
    Frequency
    Backup_<PLANNAME> Win32 Main Nightly PLANNAME Backup Run (and possibly transfer & load them)daily
    FixLogship_<PLANNAME> Win32 Guarantees Transfer of logshiped files for plan PLANNAME never
    LoadTranlogs_<PLANNAME> Win32 Loads Transfered but Unapplied Transaction Logs never
    Trandump_<PLANNAME> Win32 Run a Transaction Log Dump (and possibly transfer & load them) Hourly

    Scripts In The 'interactive' Subdirectories
    Job Name Win32 or
    Unix
    Description Recommended
    Frequency
    AlarmCleanupDB Both Internal cleanup of the gemalarms database daily
    AlarmRoutingAgent Both Read Alerts & Route Them Frequently
    CheckServer_<SERVER> Win32 Is SERVER ok? Hourly
    CleanupUnixGEM UnixCleans up unix files on GEM Daily
    ConsoleArchiver.ksh BothOnce a month make a datestamped copy of the console Monthly
    ConsoleBuildAndFtp either Assembles the Console. Optionally publish the Console to external web server Hourly
    ConsoleUnixHourly Unix Builds frequently changing pages in the Console - scheduled 1ce/hour Hourly
    ConsoleUnixNightly Unix Build 1ce/day pages in the Console daily
    ConsoleUnixWeekly Unix Build 1ce/week pages in the Console Weekly
    ConsoleWin32Hourly Win32 Builds frequently changing pages in the Console - scheduled 1ce/hour Hourly
    ConsoleWin32Nightly Win32 Build 1ce/day pages in the Console Daily
    ConsoleWin32Weekly Win32 Build 1ce/week pages in the Console Weekly
    DnsMonitor Both Monitors your DNS Servers Hourly
    GemRpt_AgentReport Both Create Backup Reports Hourly
    GemRpt_BackupReport Both Create Backup Reports Hourly
    GemRpt_Backups Both Create Backup Reports Hourly
    GemRpt_BlackoutReport Both Creates the Blackout Report Daily
    GemRpt_PCDiskspace Both Win32 Disk Space Monitor Hourly
    GemRpt_PCEventlog Both Win32 Event Log Monitor Hourly
    GemRpt_ProductionErrors Both Create Production Overnight Report Hourly
    GemRpt_ProductionList.ksh Both Creates the Production List Report Daily
    GemRpt_ProductionWarnings Both Create Full List Of All Alerts Hourly
    GemRpt_Sybase_ASE Both Create Sybase Event Log Reports Hourly
    GemRpt_Sybase_BackupServer Both Create Sybase Backup Reports Hourly
    GemRpt_Sybase_RepServer Both Create Sybase Replication Report hourly
    GemRpt_Unix_Errors.ksh Unix Create a report with current unix errors Daily
    GemRpt_Unix_Warnings.ksh UnixCreate a report with current unix warnings Daily
    GetUnixDiskSpace Unix Collect Unix Disk Space Alerts Frequently
    LargeLogFileArchive.ksh Archive database error logs that have grown too large Weekly
    OptdiagAnalyze.ksh Analyze your servers based on optdiag output Weekly
    OraSurvey Both Survey all oracle servers daily
    PortMonitor Both Test Services (sybase servers, win32 services...) Frequently
    SybAnalyzeDepends Both Create Dependency Analysis Files Of All Your Databases Weekly
    SybAuditor Both Reverse Engineer an Audit of your Sybase Servers Weekly
    SybBkpSvrConfig Both Create Configuration Report on Backup File Fetch Daily
    SybBkpSvrLogFetch Both Fetch All Sybase Backup Files Hourly
    SybBkpSvrLogRpt Both Process Sybase Backup Files Hourly
    SybCheck_Development Both Check availability of your Development Servers Hourly
    SybDailyCheck Both Daily Sybase Server Check Daily
    SybErrLogAllRpt Both Create complete Sybase Error Log REport Hourly
    SybErrLogSaveAlarms Both Parse Sybase Error Logs & Save Any Alerts Hourly
    SybErrLogConfig Both Create Sybase Error Log Configuration Report daily
    SybErrLogFetch Both Fetch Sybase Error Logs Hourly
    SybErrLogRpt Both Analyze & Report on Sybase Error Logs Hourly
    SybHistSvrConfig Both Report Historical Server Configuration daily
    SybHistSvrLogFetch Both Fetch Historical Server Log Files Hourly
    SybHistSvrLogRpt Both Analyze & Create Historical Server Report Hourly
    SybMonSvrConfig Both Report Monitor Server Configuration daily
    SybMonSvrLogFetch Both Fetch Historical Server Log Files Hourly
    SybMonSvrLogRpt Both Analyze & Create Monitor Server Report Hourly
    SybPing Both Monitor Your Sybase Servers via ping Frequently
    SybSurvey Both survey daily
    SybRepCompare Compare Rowcounts between servers that are replicated Daily
    SybRepServerChecker Both Check Replication Hourly
    SybRepSvrConfig Both Report On Your Replication Configuration daily
    SybRepSvrLogFetch Both Fetch Replication Server Log Files Frequently
    SybRepSvrLogRpt Both Report on Repliation Server Logs Hourly
    SybRepSvrSaveAlarms Both Parse & Save Replication Server Errors Frequently
    SybaseBackupFileReport Both Report on Backups daily
    SybRepMonCheck Checks that your replication monitoring is working Daily
    SybRepMonReport Both Create Report of Replication Latencieis Frequently
    SybSpaceMonitor Both Sybase Space Monitor Hourly
    SybStoredProcUpdater Ensure that all stored procedure libraries are at the latest revision Weekly
    SybThresholdManager Both Run Threshold Manager Daily
    SybThresholdManagerReinstall Both Reinstall Threshold Manager for databases that have changed in size Daily
    SybTruncateMaster Both dump tran master with nolog Daily
    SybTruncateSybsystemdb Both dump tran sybsystemdb with nolog Weekly
    SybTruncateSybsystemprocs Both dump tran sybsystemprocs with nolog Weekly
    MssqlAnalyzeDepends Win32 Optionally analyze dependencies on Sql Server Daily
    MssqlAuditor Win32 Audit your Mssql configurations Daily
    MssqlBackupCrosscheck_AM Win32 Morning batch to check Mssql Backups Daily
    MssqlBackupCrosscheck_PM Win32 Evening batch to check Mssql Backups Hourly
    MssqlBackupHistory Win32 Check your backup history Daily
    MssqlCheck_Development Win32 Check your development servers Hourly
    GetBackupStateMssql Win32 Have All Your SQL Server Databases Been Backed Up? Hourly
    GetBackupStateSybase Both Have All Your Sybase Server Databases Been Backed Up? Hourly
    MssqlCycleEventlogs Win32 Cycle Sql Server Event Logs daily
    MssqlDailyCheck Win32 Daily Check of your Mssql Servers
    MssqlJobStatusRpt Win32 Check Sql Server Scheduled Jobs daily
    MssqlLogMaintPlan Win32 Check Sql Server Transaction Logs Plans Hourly
    MssqlPing Win32 Monitor Your Sql Servers via ping Frequently
    MssqlScheduledJobRpt Win32 Check SQL Server Scheduled Jobs daily
    MssqlShrinkData