GEM Documentation Navigation Page. Documentation is also available at The GEM Home Page. Documentation and Code are both Copyright © 1995-2006 by Sql Technologies. All Rights are Reserved.
GEM is an open source server management toolkit for Database Administrators. GEM is our solution to managing many Database Servers without a lot of stress.
- Server Monitoring And Alarming
- Maintenance Plan based Server Backup
- Configuration Reporting and Auditing
- Extended System Stored Procedure Library
- Swiss army knife set of command line utilities
- Central Monitoring Station Architecture
- Easy Setup
- Database Support: Sybase & SQL Server
- Host Support: Win32, Linux, and Unix systems
- Based on 15+ years of free software
- Open source perl
GEM is a feature rich,
low cost package that is easy to install, configure, and manage. GEM
uses a small footprint, centralized, agent less
architecture that has minimal server impact and will revolutionize the way you
perform system administration.
The Generic Enterprise Manager or GEM automates the
routine parts of server management. It monitors your systems and notifies you
of problems before your users do. GEM configuration auditing can reveal serious
issues before they become a problem. For example, a database backup checker
reads system logs to ensure your backups succeed. GEM gives small and mid sized
companies the tools that up until now only the very largest companies
possessed.
GEM provides a simple, complete solution to help you
manage your Database Servers and associated Windows, Unix, and Linux
systems. While GEM provides comprehensive operating system level
monitoring and alarming, the strength of GEM is database infrastructure
management. GEM is designed for database administrators, by database
administrators.
GEM is a non-invasive, agent less system that runs on a central monitoring station.
Nothing runs on your remote systems. Even our backup solution runs
centrally. You will need a Win32 monitoring station if you are using GEM
with Win32 systems, and a Unix monitoring station if you are using GEM on Unix.
GEM will install our excellent system stored procedure library on all your
database servers.
What Is the Generic Enterprise Manager
|
The Generic Enterprise Manager (GEM) is a systems management package designed specifically for Database
Administrators. GEM provides a simple, complete solution to help you manage your Database Servers and
associated Windows, Unix, and Linux systems. While GEM provides comprehensive operating system level
monitoring and alarming, the strength of GEM is its simple proven mechanisms to help you manage your
database infrastructure. GEM is designed for database administrators, by database administrators.
The Generic Enterprise Manager is written in open source perl. It is broken into several component packages, each consisting of small, self-documenting, easy to read perl programs. These packages use common libraries, a common directory structure, and common configuration files.
Fundamentally, GEM benefits your users by reducing systems problems. GEM effectively automates
cumbersome tasks like collecting and parsing database error logs and delivers that information to the administrators in a clear,
concise manner. This dramatically increases your ability to spot problems - GEM helps you find
issues before they become problems and allows you to proactively manage your servers.
By effectively delivering system health information, GEM decreases the time spent on routine tasks and
increase quality of service provided. GEM will both reduce downtime and decreasing the time you spend on system maintenance.
GEM dramatically increases the information about each server that administrators have at their fingertips.
GEM protects your servers by guaranteeing your server back ups and by providing the configuration audit
information you need to restore in the event of an emergency.
GEM monitors important services in your database infrastructure and alarms operators if there is a problem. Our alarm routing mechanism allows you to receive only get the messages you care about.
GEM is most effective for small and mid sized organizations (3-100 database servers). GEM requires administrator access to your databases, so a DBA must be responsible for installing and running it.
GEM contains numerous features for both developers and administrators. GEM provides a Database Utilities Toolkit of command line tools and contains an excellent library of new system stored procedures.
GEM is a must for companies that perform remote server management or have servers in multiple branch offices.
The power of GEM is based on its open architecture. Fundamentally, GEM is an open source
set of libraries and management programs that are used by numerous tiny programs (300 line is pretty typical)
designed to monitor and analyze your environment. The underlying components, the libraries and standard configuration
files are designed to be
easily understood and well documented, but most importantly, hundreds of working example programs are readily
available. Because all the scripts that acutally do work are simple and small, you can troubleshoot and customize
GEM with minimal effort.
As an example, let us say YOU wish to add a new report to the console. All you would need is to create a perl script
that creates either html or text output, modifying one line in a configuration file, and then scheduling your program. The
report then appears integrated into the console. If this is too complex, the console provides a second, even easier, interface.
Just put a query/command into a second config file and the report automatically appears.
The same applies to integrating your own custom alarming into our alarming system. Just take your existing monitoring script,
add a few lines of perl (our alarm interface is open), and your alarms will automatically be routed and managed from GEM.
GEM is an open ARCHITECTURE that includes a complete set of open source programs. This allows you to make
GEM a complete solution to both your infrastructure and your business monitoring and reporting needs.
- The GEM Configuration Utility is a graphical utility that sets up and customizes your configuration.
- The GEM Console is a set of web pages that provide cross enterprise reports.
- The GEM Batch Scripts provide monitoring, data collection, and analytics on your systems.
- The GEM Enterprise Manager is a set of user interface programs
- Our Maintenance Scripts are a complete solution for scheduled database administration.
- The GEM Stored Procedure Library provides numerous new system stored procedures.
- The GEM Database Utilities consist of numerous well-documented command line utilities.
- The GEM Web Interface provides a secure web interface
- The GEM File Repository stores collected system information in an easily read directory structure.
- The GEM Monitoring Database stores monitoring and alarming information.
GEM is a non-invasive agent less system that runs on a central monitoring station. Nothing runs on your remote systems. Even our backup solution is runs centrally. You will need a Windows monitoring station if you are using GEM with Win32 systems/databases, and a Unix monitoring station if you are using GEM for Unix systems/databases.
GEM will install our popular system stored procedure library on all servers.
GEM provides a
database maintenance package which can optionally
be used for server back ups and maintenance.
The GEM Configuration Utility creates a set of custom monitoring and maintenance scripts known as the GEM Batch Scripts, which are scheduled on your monitoring stations. Information is collected in the GEM Repository and the GEM Monitoring Database, from which it is reformatted and delivered.
If you have a hybrid environment (both Win32 and Unix Systems), GEM is normally installed on a samba shared drive. This permits the same code to be used on your windows, Linux, and Unix systems. The windows monitoring system will run scheduled tasks on your windows domain and the Unix/Linux server will use cron to schedule jobs in your Unix domain.
GEM is written in open source perl. Perl is regarded as the language of choice for systems administrators and your administrators will find it easy to understand GEM. With a normal level of perl experience, they can troubleshoot any problems that arise. GEM uses standard perl/DBI for database independent connectivity and perl/Tk and perl/CGI for user interfaces. Programs are self-documented using the standard perldoc format (i.e. See documentation via perldoc program.pl) and follow standard best practices. The GEM documentation is created from this same perldoc.
The core of GEM is a set of flexible perl library modules. Use of this library interfaces permits the majority of GEM programs to be short and simple. This has the benefit of simplified systems maintenance. The combination of an API and practical examples allows users great flexibility in terms of customization.
Every environment has numerous specific needs. GEM is specifically designed for easy expansion to meet your needs.
GEM is a server administration tool, and it is therefore expected that you have administrator access (sa, sa_role) to your databases. For your Unix systems/databases, you will need access to a Unix account with read permission on your Sybase error logs. To monitor win32 systems, you need to schedule batch jobs using an account with read access to the event logs and services (in practice this means administrator access).
GEM requires one or two monitoring stations. You need a Win32 monitoring station if you wish to monitor Win32 servers/databases and a Unix monitoring station if you wish to monitor Unix servers/databases. The ideal Unix monitoring system is an older 4 cpu system, possibly a recently decommissioned server. The ideal Win32 monitoring system is a 2-3 year old Win2K or Win2005 server with 2 GB RAM.
You will also need a small monitoring database. This monitoring database can be SQL Server you are running all Win32 or Sybase for other configurations. For cleanliness, it is recommended that this server be running on one of your monitoring stations.
The GEM monitoring stations require a recent version of perl. The DBI module and the appropriate DBD modules (DBD::ODBC or DBD::Sybase) must be installed.
Finally, you will need access to a Web Server that can run cgi scripts.
All GEM software and documentation are Copyright (c) 1995-2006 by
SQL Technologies, inc. All rights are reserved.
GEM is currently under strict distribution control. The GEM may be
downloaded from www.edbarlow.com for evaluation and use, but the software may not be
redistributed in any manner without express consent.
NO WARANTY OF ANY SORT IS PROVIDED WITH THIS SOFTWARE
The terms of this license will change once the software goes general
availability.
GEM is distributed as open source code, and it is ok for you to inspect and modify the code line so long so long as you don't remove any of the copyright notices and do not redistribute the changes without the consent of SQL Technologies.
GEM
Project GEM is a major effort designed to help you manage business
critical infrastructure at commercial organizations. It is expected
that commercial users will support this project financially.
We do, however, come from a free software background you can expect the
package to be low cost and high functionality.
Non profit and government organizations, companies with 1-4 database systems,
and indviduals are welcome to use this software for free.
GEM uses licensed consultants and distributors for primary user support.
These 'Certified Product Specialists' will be paid a sales and support fees above and beyond
the prices listed below, in return for installation and production support. Support charges
for these individuals will be negotiated by the customer on a case by case basis.
Training is available for individuals interested in being Certified. If you are a consultant
or distributor and are interested in
reselling GEM, please contact us. For now, support will be provided by SQL Technologies free
of charge.
|
PACKAGE
|
PRICE LIST
|
|
GEM Standalone Backup Scripts
|
FREE
|
|
GEM Stored Procedure Library
|
FREE
|
|
Free GEM (1-4 Server License)
|
FREE
|
|
Commercial GEM
|
$500 /
server initial license
$200 / server / year maintenance
|
GEM is open source and supports an open perl API. The software is designed for easy,
customization. Most simple customization ( adding an alarm monitor or adding a report
to the console) can be done trivially. Contact SQL Technologies for details.
Complex customization will be performed on a contract basis. Contact us for more information.
Getting Started Guide
for the GEM
|
This guide will give information on how to get started with the Generic
Enterprise Manger (GEM).
The scope of this project is large, but if you follow this guide, we can
make this tool a success for you. This document describes the structure of the
code line - necessary information to understand the application.
GEM provides a framework to manage multiple database systems. The system is built upon a set of perl
libraries which provide the glue to bind the system together. Since its
all open source perl, expansion and customization are easy. This guide endeavors
to guide you through the process of installing and customizing GEM.
One thing to remember. GEM is designed for simplicity. The goal is
to put code, data, and libraries in locations where a reasonable administrator would expect them to be found.
After you extract GEM, please poke around and familiarize yourself with the product.
Look for utilities in the ADMIN_SCRIPTS subdirectory and configuration files in the conf subdirectory.
Knowledge of the layout is essential to maintaining and operating
GEM as your production server managment interface.
This document provides an overview of GEM layout. To truly understand the layout, you will need to have
an existing copy of the code. To get this, please read the Installation Guide.
The first thing to understand is the basic
directory structure of the installation. The command line programs that implement
most of the functionality exist in the ADMIN_SCRIPTS subdirectory, which
is further subdividied by package. Configuration files are stored in the conf
subdirectory, and data and formatted output is stored in the data subdirectory. The data subdirectory
consists of the GEM File Repository (where we keep collected files and system log files) and the GEM
Console (in CONSOLE_REPORTS) where we keep report output. The
main
GEM programs (eventviewer.pl,
monitor.pl,
troubleshoot.pl, and
configure.pl)
are in the the root directory. The rest of the heirarchy is for support files.
 |
GEM Base Directory |
This is where you extracted
the software. |
  |
ADMIN_SCRIPTS |
Command line programs reside here. |
  |
doc |
Directory of GEM documentation |
  |
Unix_batch_scripts |
This is where batch scritps that work with Unix reside
|
  |
win32_batch_scripts |
This is where batch scripts that work with Windows
reside |
  |
lib |
Perl Module Library |
  |
plugins |
Location for GEM Manager plugins |
  |
conf |
Configuration Files (these are human editable and must
be kept secure) |
|
data |
Sub directory for discovered
and generated data - this directory can get big |
 |
GEM Base Directory |
This is where you extracted
the software. |
 |
gem.pl |
This is the GEM enterprise
manager - which rolls up all the plugins. It is not currently functional.
|
 |
configure.pl |
This is the GEM setup utility
|
 |
troubleshoot.pl |
This is the diagnostic utility. It is used to diagnose
configuration and setup problems. |
 |
monitor.pl
|
The systems monitor.
It allows drill down server diagnostics (works on system tables)
|
 |
eventviewer.pl
|
The system event viewer
|
The ADMIN SCRIPTS Directory |
The main command line programs of the GEM are stored in ADMIN_SCRIPTS and broken
down by package
 |
GEM Base Directory |
This is where you extracted
the software. |
  |
ADMIN_SCRIPTS |
Command line programs reside here. |
   |
bin |
Miscellaneous utilities |
   |
dbi_backup_scripts
|
Sybase and Sql Server Backup Scripts |
   |
MDA_procs
|
Sybase MDA Table monitoring procedures |
   |
monitoring
|
Programs designed for monitoring |
   |
oracle |
Oracle Scripts |
   |
procs |
Sybase and Sql Server Extended Stored Procedure Library
|
   |
console
|
Scripts that manage the GEM Console |
The data directory is where information is stored about your systems. This information
is broken down by package.
  |
data |
Sub directory for generated
data - this directory can get big |
       |
BACKUP_LOGS
|
This directory contains the
output from your maintenance plan
scripts broken down by server. Within each server specific
subdirectory exists a directory for audits, dbcc, errors, and sessionlogs.
|
   |
batchjob_errors
|
Standard Error from the maintenance scripts |
   |
GEM_BATCHJOB_LOGS
|
Standard Output from the maintenance scripts |
   |
CONSOLE_REPORTS
|
This is the console. You point your browser here to see it. |
   |
depends |
Files required for the dependency analyzer |
   |
cronlogs
|
Logs of your scheduled jobs
|
   |
html_output
|
Output of any html reports
you want auto included in documentation |
   |
lockfiles |
Lock Files for programs that
can be overscheduled |
   |
system_information_data
|
Monitoring data and configuration information by server |
The conf directory contains system configuration files. The following are the
main configuration files
| FILE NAME |
FILE DESCRIPTION |
| gem.xml
|
This file is managed by the survey scripts and used
by the enterprise manager. It contains discovered configuration
information. |
| configure.cfg
|
this file is the main configuration file for the backup
scripts |
| documenter.dat |
this file determines extra reports and programs to
include in the GEM Console |
| oracle_passwords.dat |
Oracle Password File |
| pc_password.dat |
Password file for Windows servers; |
| pc_service.dat |
File listing services on Windows that must be up |
| port_monitor.dat |
File listing systems/ports to monitor |
| sqlsvr_password.dat |
Sql Server Password file |
| sybase_passwords.dat |
Sybase Password file |
| threshold_overrides.dat |
|
| Unix_passwords.dat |
Unix password file |
| Win32_Cleanup.dat
|
Directories under Windows for routine purge/cleanup |
Files are shiped in the distribution with .sample extensions
- these are the templates with standard comments. No files are shipped with
.dat, .xml, or .cfg extensions. This permits you to perform an upgrade by extracting
the new tarball over the existing code line with no problems.
One of the core features of GEM is its monitoring
and maintenance scripts. These scripts perform the systems monitoring and
alarming, perform the system administration operations you define in your
maintenance plans, and provide useful routine maintenance activities. The
scripts are customized to your environment by the GEM configuration manager.
For version 1, some of these scripts may involve hand definition of a configuration
file.
The configuration manager will create scheduled for some of these jobs on
your Windows scheduler and will create a Unix crontab file for jobs you indicate
you wish to run on Unix. These tasks fundamentally entail a call to one of
the command line scripts located in the ADMIN_SCRIPTS directory, but the GEM
hides this from you. GEM creates a set of .bat and .ksh driver scripts that
have obvious names to be the scripts that you actually run or schedule.
These scripts are stored in the win32_batch_scripts and Unix_batch_scritpts directories.
Within those directories are 4 subdirectories as follows:
|
GEM Base |
|
| unix_batch_scripts |
This is where scripts that run under Unix exist. These directories are populated with customized scripts. |

|
interactive |
Output from these commands can bee seen.
These are the scripts that you use |

|
batch |
These scripts are for use by your scheduler.
They redirect output to specified log and error files.
If you run them you will get no output. |

|
plan_interactive |
Scripts for the backup / maintenance plan system.
These scripts are interactive as per above |
  |
plan_batch |
Scripts for the backup / maintenance plan system
These scripts are for use by the scheduler as per above |
The above tree is duplicated in the win32_batch_scripts subdirectory.
Simply put, each script is created as a .ksh
on Unix and as both a .bat and .ksh script on Windows. Copies of the script
exist in an interactive and batch directory (or a plan_interactive/plan_batch
directory). The batch scripts redirect output to locations in the data tree
(so you cant see it) but are identical to the interactive scripts. When you
run a scheduled task, use the batch version, and when you run interactively,
use the interactive version (you want to see output if you are running interactively).
The plan_* directories store pregenerated maintenance plan scritps. The plan
directories were separated from the regular directories because the tree becomes
hard to navigate when you get large numbers of servers (and a large number
of maintenance plans).
Backup and Maintenance Plans |
GEM offers a complete maintenance plan based system maintenace facility.
Plans are set up using the GEM Installer and stored in the configuration
file configure.cfg. All normal features related to systems maintenance
are covered including Backups, DBCC, Update Statistics, and Log Shipping.
Additionally, backup compression for SQL Server, notification, and extensive
reporting are provided. The command line interface provides a level of
flexibility not available with native SQL Server facilities.
The
GEM Enterprise Manager is the graphical
user interface for GEM. It is a modular system that uses plugins for all functionality.
Plugins use single console for all your systems. It uses perl DBI - a database
independant api - and provides you with a seamless api. The trick here is
the
RosettaStone.pm perl module, which basically maps database independant
requests (get me locks/logins/space) to database specific ddl. This gives
you consistency. The GEM. is designed to use plugins, which are basically
small
perl modules that conform to
an api. Most of your services are provided by the other
GEM libraries, so
these modules basically define what user interface elements they want (menus,
how they want their explorer to look, right click functionality) and then
manage a Tk frame however they want. The goal here is to have third parties
develop plugins that add functionality. And the nice thing here, is that this
can be business specific functionality (ie. a trade or order volume report).
The
GEM plugins are managed from a central store on the internet which is
available to you to distribute your plugins in the same manner that CPAN does
for perl modules.
Documentation and Diagnostics |
All system documentation uses the standard perldoc format. Perldoc is a utility
provided with perl to extract documentation from programs. To get full documentation
on a program, simply run perldoc program.
The majority of the GEM documentation is composed from the embedded perldoc.
You can see a short version of the help by simply running perl
program.pl -?, which will provide useful syntax help.
Older database programs use the standard syntax of -Uuser -Ppassword -Sserver -Ddatabase for
arguments. Newer programs use --USER=username
--PASSWORD=password --SERVER=servername --DATABASE=database convention.
If you have problems with a programs, try running with -d or --DEBUG flags set. These will run the program
in verbose/diagnostic mode and the program will print useful diagnostic information.
This is the standard way of debugging problems.
The following guide describes how to install GEM.
If you are planning to using GEM, please contact Sql Technologies by email. We are interested in making your GEM Installation a success, and will provide installation support. We are aiming for full customer satisfaction and will work with you to make this product a success.
Pre-install check list
- PICK 1 or 2 SYSTEM(s) TO USE AS GEM MONITORING SERVER(s)
GEM manages your systems from central monitoring systems. A windows systems is used to manage your windows databases. A unix/linux server is used for your unix servers. If you have both (a "Hybrid Environment") windows and unix based databases, you will need 2 monitoring systems.
These systems run a lot of perl batch jobs. It is recommended that you use dedicated systems for your production GEM monitoring servers. For your initial trial setup, use a development system.
On windows monitoring systems, we suggest you have access to a windows login OTHER than the login you normally use for login. On windows, scheduled GEM batch jobs might result in the pop up of dos cmd boxes (which is annoying). Scheduling the jobs using a 'batch' windows account prevents this.
You will need appropriate SQL Server and/or Sybase/Oracle client software to be installed on your monitoring servers.
- PICK A HOME FOR THE GEM CODE
Identify the directory you are going to install GEM into. This is only tricky if you have a Hybrid Environment, in which case your admins need to create a samba share such that a single directory appear in both unix/linux and windows. When GEM is installed on a samba share, the same code works unchanged to monitor both your Unix and Windows environments seamlessly.
- INSTALL & VERIFY perl 5.8.1 OR LATER
Each monitoring station requires perl to be installed at version 5.8.1 or later. This can be verified by running the command perl -v (prints the version number).
On windows, we recommend the latest Activestate Perl from http://www.activestate.com. Use the default installation options.
On UNIX, binary releases can be downloaded from http://www.cpan.org. You can have your unix admins install/provide perl for you or can install your own version of perl in your own home directory. The second approach has the advantage that you have control, and will not need permission of the unix administrators to add/update modules.
- ADD PERL MODULES
Verify availability of the perl DBI module and associated DBD perl modules. DBI and DBD::ODBC are required on windows servers. DBI and Native DBD Drivers (DBD::Sybase, DBD::Oracle) are required for the perl on your unix monitoring servers.
On Windows, install these modules via :
[StartKey] -> [Run] and then type ppm install DBI
and
[StartKey] -> [Run] and then type ppm install DBD::ODem/em.
If your company has a firewall/proxy set up such that the ppm commands fail we suggest:
Bring up a browser and go to http://ppm.activestate.com/PPMPackages/zips/ - select your version and go to the Windows subdirectoy and download the latest zip file. The file will contain 3 files when extracted - a .ppd file, a readme, and a .tar.gz file. To install this ActiveState PPM package, run the following command in the current directory:
ppm install DBD-ODBC.ppd
You should download both the DBD::ODBC and DBI libraries. The latest versions (as of October 2007 are 1.13 and 1.58 respectively.
On Unix, the DBI and DBD::Sybase modules must be installed by hand. You may also wish to install DBD::Oracle, although oracle support is currently limited with full support scheduled for 3Q08. Module installation on unix may require administrator help (the make install step requires write access to the perl library directories which are write protected unless you are using a local version of perl.
- IDENTIFY SCRIPT CAPABLE DIRECTORY ON YOUR WEB SERVER
While GEM does not "technically" require running scripts on an web server, several of the interfaces were designed primarly for web access. To run these scripts, you must create a 'script enabled' directory on your web server. GEM provides perl scripts which, when deployed to this directory will provide web based features. The primary interfaces provided at this time are a monitoring and reporting tool for your database servers. You should secure this web directory as you see fit for information of that type.
- CREATE A MONITORING DATABASE
GEM requires a small monitoring database. Place this database on Sybase if you have a Hybrid or Unix Only environment and on either Sybase or Microsoft SQL Server in windows only environments. This database will store enterprise monitoring information and must be accessible from all your monitoring servers. If you have Unix Systems yet wish (for some reason) to use a SQL Server as your repository, you can install FreeTds and the unix verison of DBD::ODBC.
- Create A Database Named gemalarms with size=200MB
- Turn on select into and truncate log on checkpoint database options
- Create login named gemalarms with password gemalarms
- Add login gemalarms to database gemalarms in group public (not a dbo account)
use master
go
create database gemalarms on data_device_6=200
go
use master
go
sp_dboption gemalarms,"select",true
go
sp_dboption gemalarms,"trunc",true
go
use gemalarms
go
checkpoint
go
sp_addlogin "gemalarms","gemalarms","gemalarms"
go
sp_adduser gemalarms
go
- IDENTIFY THE DATABASE SERVERS YOU WANT TO MONITOR
If you are trialing GEM, we recommend installing on 4-6 database servers on 4-6 separate sytems and that the trial be either 'all unix' or 'all windows' databases. You will require the following rights on these database servers:
You need windows administrator access your windows database servers
- needed to monitor disk space, server processes, and event logs
You will require sa_role access for all your Sybase database servers - for installation of the stored procedure library
- you can get away with non-sa access for monitoring
You will require "no-password" access via ssh or rsh for your unix systems You will require sybase unix account access for your Sybase database servers. - this is used to read sybase event logs
- SET UP YOUR ODBC DATA SOURCES OR interfaces FILE
Ensure that all the database servers identified in the last step have span>ODBC entries (Windows Monitoring Box), and have appropriate Sybase interfaces file (and Oracle tnsnames.ora) entries. The configuration process (configure.pl) will automatically discover your servers based on these values.
- SET UP SECURE INTRA SYSTEM COMMUNICATIONS (UNIX)
All UNIX systems involved with GEM must be accessible from the unix monitoring system by ssh/scp (preferred method), rsh/rcp, or standard ftp. The account you will be using to schedule the batch jobs and to run gem must be able to get into these systems without passwords (except in the case of ftp where a password is ok).
monitor ebarlow: ssh sybase@sybprod
Last login: Wed Apr 2 11:37:29 2008 from monitor
$
Suggestions on your first implementation
Database administrators can be, and should be, a cautious lot. We obviously roll out a tool like GEM to a single test box and install the client on our personal workstation. Lowest risk configuration and all that. An installation like that totally and completely buys you nothing. GEM will be of no value. That is totally and completely the wrong way to roll out GEM.
The correct way to roll out GEM is to implement it on between 4 and 6 test and dev servers at once using a 'server' class box as a monitoring station (it does NOT need to be a dedicated box but it runs a bunch of perl scripts). Obviously dont trial GEM on anything production (that includes your workstation). Your Trial needs this many systems to let you to see how the GEM software works. Plus, GEM is a LOT of effort to implement, so you might as well get some use out of your GEM trial.
Finally, ensure that you have defined ALL items on the GEM pre-install check list. For example, GEM loses half its value in a unix environment if you dont have 'no password' access via either ssh or rsh connectivity to your unix systems. Simarly, "ADMINSITRATOR GROUP" rights are required for direct access to your Windows servers log files. You need these. Additionally, it is highly strongly very suggested that you have a web server with a script capable directory into which you can place our cgi perl interface. While we do have a TK version of the interface, the web interface is the one you should use.
Presuming you have a reasonable network configuration, GEM has a pretty low footprint on everything but the monitoring host system. Network impact is mostly semi hourly fetches of your database error logs. GEM does install our system stored procedures on all your database servers. They are a nice package and you will like them. On the monitoring system, GEM runs a bunch of perl batch jobs - and frequently multiple jobs will run at the same time. We recommend multi processor systems as your monitoring systems, but most 2-4 year old server class systems will be fine (find some depricated system). Linux is a good choice for a monitoring system to monitor unix database.
As far as your database servers go, GEM has the experience to be a good citizen. GEM is designed by active dbas, not a lab by developers. Your developers and users should have not have any noticable degredation in performance as a result of GEM.
A good solid burn in environment for GEM should have between 4 and 6 database servers and 4 to 6 different host systems (of course nothing production should be touched). It would be slightly simpler to implement the trial if running all windows or all unix databases.
Download & initial setup
- DOWNLOAD AND EXTRACT GEM
You should now download and extract the GEM software from http://www.edbarlow.com/download.pl.
You will need a passcode to download the software which can be recieved by mail from SQL Technolgies.
Extract into the directory identified in the pre-install checklist. You should need to install into only one location regardless of whether you have a hybrid or non-hybrid environment.
The GEM download file is a standard compressed tar file and can be extracted with winzip (windows), gnu tar (tar xvzf gem.tgz), or with gzip -d gem.tgz; tar xvf gem.tar. If you install GEM into a subdirectory, do not use a directory that includes space characters (ie. Program Files). White space in the directory name is not a tested option.
- SET YOUR DISPLAY ENVIRONMENT VARIABLE IF UNIX
Many of the GEM tools use perl/Tk for a user interface. On windows this works without support, but on unix this requires some type of X terminal manager (like exceed). On UNIX, you must set the environment variable DISPLAY correctly.
EXPORT DISPLAY=your_ip_address:0.0.
Unix users will also require an X terminal manager installed on their workstation. Your infrastructure support can help you install this if you are not allready set up.
- VERIFY YOUR PERL INSTALLATION
perl troubleshoot.pl
Use the program troubleshoot.pl in the root of the installation tree to test for the presence of necessary perl libraries. troubleshoot.pl is the GEM installation trouble shooting program and it can be run to diagnose any GEM issues - a successful run indicates that your configuration is stable.
- RUN configure.pl
See the section on configure.pl for details on what this program does.
configure.pl
GEM Configuration Tool
|
configure.pl is the GEM configuration utility.
This utility will build the configuration files (stored in the conf subdirectory) and runs
some basic setup tasks for GEM. The GEM Configuration Tool is quite intelligent, and will
survey your systems to give you a good baseline that should require minimal tuning to work
within your environment.
Some of the things configure.pl does include:
- Create Configuration Files
One primary output of configure.pl is the configuration files stored in the conf directory. These files can be hand edited once GEM is set up.
- Rebuild of all the code
Changing the hashbangs (the first line of the file on unix is known as a hashbang and defines the command line
interpreter to use), removing Control-M (dos newline) characters from the files, and modifying library paths.
- Creation of the GEM Batch Jobs
configure.pl also creates the GEM Batch Jobs. On a win32 monitoring station, GEM can
schedule these jobs. On a unix environment, we create a crontab.txt file they can be schedule via cron.
- Monitoring Setup
The initial database installation is done via configure.pl
- Initial Build
This program also will generate an initial version of the GEM console and prepopulate
the alarm system with alerts and reports relevant to your environment.
Important notes
- Configure.pl creates a file logs/configure.log
which includes both console messages AND diagnostic messages. If you
have a problem please look at this file. It contains all output including
output that should be only produced with configure.pl --DEBUG.
- Work through the tabs from left to right.
- "Save Configuration Changes"
writes your configuration files. While some
operations (adding stored procedures to servers)
are permanent once completed, the majority are not. It is important to save before
you quit.
- Leave the backups tab alone for your first pass
perl configure.pl
If you wish to see diagnostic messages, you may run
perl configure.pl --DEBUG
A log file of the run will be placed in the logs subdirectory.
FOLLOW THE FOLLOWING STEP BY STEP INSTRUCTIONS TO COMPLETE YOUR INSTALLATION
The welcome tab
PURPOSE: License Details and Acceptance
This tab gives licensing options for GEM. Select "I AGREE WITH THE LICENSE" to continue after you have read all the information provided when you click on the various buttons on this page.
The GEM Help button will help guide you through your install.
All GEM code is copyright (c) 1995-2007 by SQL Technologies=head1 The Register Tab
PURPOSE: Basic configuration setup plus register your product.
- Enter Administrator Info.
This information is never shared so make it correct.
- Select Environment, Product, and License.
Running on a samba share is appropriate in Hybrid Windows/Unix environment. Select Free GEM for product trials. Contact barlowedward@hotmail.com for a product key for "Full Gem".
- Select Preferred Connect Options
to define the legal system connectivity options.
- Click "REGISTER YOUR VERSION"
to register your product.
The paths tab
[This tab is trivial if you have an All Windows or All Unix environment.]
- Enter FULL Path(s) to your version of perl.
eg. /usr/local/bin/perl or C:/perl/bin/perl.
- Enter All legal paths to the GEM top level directory
to define paths to required libraries. On a samba / hybrid installation where /apps/gem maps to c:/gem on win32 host win2, you would have '/apps/gem' (the Unix Code Location) and '//win2/c$/gem' as the NT Code Location, and Alternate #1 would be 'C:/gem'. Alternate #2 will be blank.
- Enter Unix/Linux Web Browser
Its probably /usr/local/bin/firefox. Skip if you dont know it.
- Enter Sybase Home Directories
in a list so GEM knows what paths to search while surveying remote systems.
PURPOSE: Ensure Mail Connectivity
This tab is only required on Windows servers. For these servers, you need to provide your SMTP mail server name. If "Test Mail Install" should send a test message to your GEM Administrator Mail Address (entered earlier).
The servers tab
PURPOSE: Register your Databases & Systems with GEM
- Access To Databases
must be via 'sa' or an account with similar roles/permissions. GEM does not support blank/empty administrator passwords.
- Access To Unix Systems
must be via ssh, rsh, or ftp using an account with reads permissions on the DBMS log files in $SYBASE/$ORA_HOME. The ssh or rsh connection should not require a password.
- Access To Win32 Systems
is via native authentication - the account must read Event Logs and Administrative File Shares (ie. requires administrator).
- The Full GEM Add Server Wizard
creates a best guess at your environment - prompting for logins and passwords appropriately. To save time, it guesses passwords by reusing previously entered ones. This wizard only registers NEW servers and is re-runnable.
- The Free GEM Add Server Wizard
For FREE Gem provides a simple data entry screen to enter up to 4 DATABASEs.
- The Server Type Buttons
allow you to define category (only PRODUCTION/CRITICAL systems are page-able), change passwords, and deregister servers
- The Quick Add Button
allows you to register a single Database or System.
- The UnixComm & WinComm columns
allow you to specify preferred communication methods from Unix/Windows Monitoring stations respectively.
The files tab
PURPOSE: Identify Database Home Directories & Important FIles
This tab contains rows for each UNIX/Linux system, listing survey results (collected earlier) in blue and configuration directives in black. If survey results look incorrect, correct the configuration directives (with the add/delete buttons) and rerun the surveys.
- Sybase ASE Server $SYBASE Directory
is a valid sybase server directory and is searched for Error Logs, RUN_* files, and other config information.
- Application Log File (unused in v1)
- Ignore This RUN_* File
directives that the Sybase server RUN_* file is unused and should be "ignored" to prevent confusion about what runs where.
- Operating System Log File (unused in v1)
- Sybase Open Client $SYBASE Directory
are surveyed only for interface file consistency checks.
- Oracle ORAHOME Directory (unused in v1)
- SQL Server Maintenance Plan Logs (unused in v1)
When completed, you consider clicking "SAVE CONFIGURATION CHANGES"=head1 The Connect Tab
PURPOSE: Test connectivity and survey your systems.
- Page-Top Buttons
allow you to can see servers by type.
- Connect To All
to test connectivity. Windows GEM requires odbc settings for all databases and native administrator access for systems. Unix GEM requires normal database access (interfaces file for sybase) plus host access via ssh (preferred), rsh, or ftp. Fix up any servers you cant connect to.
- Survey All
to run a survey of all your servers. Normally, surveys are scheduled but we need some initial data. Click "Dont Skip" to not skip servers which have been recently surveyed.
The backups tab
When you create your servers, it will also create two backup plans for you: one that has the same name as the server and the other which has _WEEKEND appended to the server name. Data on your plans is stored in the configure.cfg file and is hand editable. Please see documentation on the GEM Maintenance And Backup System.
- For initial setup, you can ignore this section
The console tab
PURPOSE: Define Console Special Behaviors
- For an initial setup or trial, you can ignore this section
- This page allows you to copy the console to another location.
You may be interested in doing this for security reasons. By default, the CONSOLE Web Pages are created the data/CONSOLE_REPORTS directory but these reports can be published to another location (perhaps on your web server). Most users will NOT wish to do this.
You may copy the files via unix copies (ssh, rsh, ftp - select the Copy using FTP button) or to a local directory (like //NEWSERVER/c$/web/gem). Some users may want to do thsi copy for security reasons - to allow non administrators to use the GEM console. This keeps those same users away from the GEM directory tree which stores other highly sensitive data.
The alarms tab
PURPOSE: Set Up Alarms & Monitoring Systems
- Fill in the Alarm Database Server Info
with information about the 200MB (approximate size) alarm database you set up before running configure.pl (see pre-requesites). The email address is arbitrary and all alarms will be mailed using that as a reply-to address. It is reasonable to set this to an INVALID mail address (alarms@yourcompany.com or Gem@yourcompany.com) because you wont ever reply to the messages.
- Install/Upgrade Alarm Database
will create alarm database tables & procedures
- Validate Alarm Database
checks the database for sanity and version.
- Fill in Web Server information
including the map of hostname/filename to alarming url. Once this is done click Populate Local cgi-bin to copy formatted scripts to .../cgi-bin and Copy to Web Server to copy those files to the web server.
The install software tab
Now would be a good time to Save Configuration Changes!
Click the "Install Software" button to run the numbered steps listed below that button.
- Reformat Code
Removes Control-M's and Sets Hashbangs
- Create GEM Batch Scripts
creates your batch scripts. In a hybrid environment run this on UNIX last - so none of the files have control-M characters embedded
- Backup Manager
sets up default directories for the backup system
- Modify Console Scripts
The post install tasks tab
- Create a crontab file for your unix installation and schedule it.
- Snapshot your windows servers services
to create conf/pc_service.dat. This basically creates a list of all the running services on all your windows boxes for future monitoring. You will be notified if any services go down. Assuming your environement is currently stable, a current snapshot will give you a pretty good listing of what needs to be monitored.
The scheduler tab
PURPOSE: Define Scheduling and Actually Schedule Batch Jobs
Other things to do
Once you are done with your GEM installation, there are a variety of post-installation tasks . This checklist is designed to help you with these tasks.
CLEAN UP UNIX SCRIPTS DIRECTORY
There are outstanding issues with chmod() and dos newline removal that impact Hybrid environments. Specifically, you must run configure.pl on unix AFTER you run configure.pl on windows. This is because there is no way to get windows to use just a standard newline instead of CR/newline (so all files will have a ^M in them if you run on windows last).
You should also run the Batch File creation on UNIX last. If you ran your Win32 install after your Unix install, go to your unix server and run perl upgrade.pl to fix these issues.
Modify your Password Files
The password files are located in conf/*_password.dat and are hand editable.
sybase_passwords.dat
oracle_password.dat
pc_password.dat
unix_passwords.dat
sqlsvr_password.dat
All configuration files are self documented. An archive of your old configuration files is kept in conf/saved_files
If you add new databases or systems, recreate & reschedule your Batch Scripts. Most scripts work against all servers in the configuration files, but some scripts work on a specific database (specifically the ones with the Database name in the file name).
Windows Directory Cleaner
One of the most horrific tasks of the windows DBA is disk space cleanup. Something is *always* running out of space... GEM automates this task for you.
The file conf/Win32_Cleanup.dat contains a list of disk space Cleanup instructions for the windows cleanup batch job. It is suggested that you customize this file to include your tran/archive log directories, your application log directories, and any other directories that fill up on your systems. A best practice is to add rows to this file over time, every time the windows disk space monitor spots a directory that is filling up. The lines in this file conform to the arguments of ADMIN_SCRIPTS/bin/purge_old_files.pl, and include --hours and --days to identify the keep times, --dir to identify the directory to purge, and either --compress= or --del (to identify what to do with the old files). See the syntax of purge_old_files.pl for other options.
# FILENAME: Win32_Cleanup.dat
#
# USED BY: Batch job Win32FileCleanup
#
# SYNOPSIS: Directives for cleaning up win32 drives. This file is
# identical to Unix_Cleanup.dat but for the --compress (Win32 only)
# argument which is missing and the required --host argument (Unix Only).
#
#
# FORMAT : HOSTNAME drive drive drive.
#
# DETAILS : The arguments are the arguments to purge_old_win32_files.pl
#
# --host=<host>
# --hours=<hours>
# --days=<days>
# --dir=<dir>
# [--del] - do you wish to delete the file?
# [--done] - only work on files with .done extension
#
# EXAMPLE FORMATS:
#
# Delete 14+ day old files on host h1
# --days=14 --dir=//adsrv012/c$/oracle/admin/SCPROD/udump --del
#
# Delete files with .done extension older than 2 days old
# Note: this will delete .done.gz files and .done files
# --days=2 --dir=//SRVR001/d$/logshipbuffer --del --done
#
#######################################################################
#
# START RECOMMENDED PURGES
# --days=7 --dir=C:/gem/data/logs -del
# --days=7 --dir=C:/gem/data/batchjob_errors -del
# --days=7 --dir=C:/gem/data/GEM_BATCHJOB_LOGS -del
# --days=7 --dir=C:/gem/data/CONSOLE_REPORTS -del
# --days=7 --dir=C:/gem/data/cronlogs -del
# --days=7 --dir=C:/gem/data/errors -del
# --days=7 --dir=C:/gem/data/backup_reports -del
# --days=7 --dir=C:/gem/data/html_output -del
# --days=30 --dir=C:/gem/data/BACKUP_LOGS\*\sessionlog -del
# --days=30 --dir=C:/gem/data/BACKUP_LOGS\*\errors -del
# END RECOMMENDED PURGES
Unix Directory Cleaner
The Unix_Cleanup.dat file will control unix file/directory cleanups. It is similar to the Windows cleanup file but for the addition of the --hostname argument.
port_monitor.dat
Port Monitoring is a quick availability test - and a good way to find out if a database or system is down. The file conf/port_monitor.dat contains a list of Hosts and Ports to Monitor.
This file is autogenerated by configure.pl but will probably require hand editing.
Simple format:
# hostname PORT SERVER DESCRIPTION
SRVR001 1433 SRVR001 Microsoft SQL Server
SRVR007 1433 SRVR007 Microsoft SQL Server
brokersql 1433 BROKERSQL Microsoft SQL Server
mail1 25 Sendmail on mail1
Sybase Replication Monitoring
The File conf/Replication.dat is used by the Sybase replication server monitoring scripts and will need to be popluated with your Rep Servers and Routes.
# FILENAME: replication.dat
#
# SYNOPSIS: Lists your sybase replication server information
#
# FORMAT:
#
# REPSERVER $NAME $LOGIN $PASSWORD $THRESHOLD_MB
# REPROUTE $SOURCE_SVR $SOURCE_DB $DEST_SVR $DEST_DB $REPSERVER
#
# EXAMPLES (350MB Thresholds):
#
# REPSERVER REP1DR_RS sa <pass> 350
# REPSERVER REP2_RS sa <pass> 350
# REPSERVER REP1_RS sa <pass> 350
#
# REPROUTE SYB1 my_db SYB2 my_db REP2_RS
# REPROUTE SYB1 my_db SYB1DR my_db REP2_RS
#
# ONCE YOU SET THIS FILE UP - PLEASE RUN
# ADMIN_SCRIPTS/monitoring/SybRepMonitor.pl --ACTION=INSTALL
# MONITORING IS HANDLED AUTOMATICALLY BY THE GEM BATCH SybRepMonAgent
##############################
There are two programs that check replication. The batch SybRepServerChecker will check all your rep servers to be sure that no routes are suspended, that disk space is reasonable, and that there are no other serious errors. There are also three batches, SybRepMonAgent, SybRepMonCheck, and SybRepMonReport that compute latencies of your replication, and will warn you when replication falls behind. These three batch programs call SybRepMonitor.pl. This program takes an argument --ACTION.
To initialize this system, you need to set up your configuration file and then run myperl SybRepMonitor.pl --ACTION=INSTALL
After the system is INSTALLED, the above mentioned batches will perform your monitoring automatically. Below is the syntax for SybRepMonitor.pl:
Usage: SybRepMonitor.pl --ACTION=INSTALL|UNINSTALL|MONITOR|REPORT|CHECK
[--SERVER=xxx] [--DEBUG] [--OUTFILE=file]
if --ACTION=INSTALL will install
--REBUILD redoes table creates
if --ACTION=CHECK will check latency vs --ALARM_MINS
- mail to --ALARM_MAILTO as needed
if --ACTION=REPORT will create a report
- will print data for --REPORT_HOURS hours
- will only show rows > --REPORT_THRESH seconds
- will aggregate based on --BUCKETSIZE
- will print HTML output if --HTML is passed
if --ACTION=MONITOR will insert rows into all source databases
- will insert --ITERATIONS rows
- sleeps --FREQUENCY secs between each insert
Adjust Alarming Thresholds
You may not wish all your alarms to work at the same thresholds - preferring to set some thresholds individually. Generally speaking, this is most needed when some resourse is nearing "full" and generating alarms, but you can not fix the problem immediately - intending to fix the problem later. The conf/threshold_overrides.dat file allows you that control.
# FILENAME: threshold_overrides.dat
#
# SYNOPSIS: Monitoring Thresholds
#
# DETAILS : This file permits you to turn off alarms and to control their
# severity. This is a Comma Separated File
#
# SAMPLE FORMAT:
# <system> <monitor_program> <level> <threshold> <optional subsystem>
#
# monitor_program=UnixDiskspace|PcDiskspace|SybSpaceMonitor
# |SpaceMonitorSqlsvr|ThresholdManager
# level=CRITICAL|ERROR|WARNING
#
# remember to reset higher priority thresholds so there is no overlap - ie if you
# set ERROR to 98%, then you should set critical to something larger (say 99%).
# The programs may act wierd if the CRITICAL threshold was less than the ERROR threshold.
#
# By default - SybSpaceMonitor & SpaceMonitorSqlsvr will use 90/92/95% as
# their default thresholds
#
# Replication thresholds are in minutes. For replication, you can put a
# blank target name in if you want but not a blank source name
#
# EXAMPLES
# eg. directfeed,UnixDiskspace,WARNING,80,Disk: /var
# eg. direct1,UnixDiskspace,WARNING,80,Disk: /var
# eg. direct2 ,UnixDiskspace,WARNING,80,Disk: /var
#
# eg. SYBSVR1,SybSpaceMonitor,WARNING,80,db1
# eg. SYB1,SybSpaceMonitor,ERROR,95
# eg. SYB1,SybSpaceMonitor,ERROR,95,clientdb
#
# eg. ADSRV057,PcDiskspace,ERROR,98,d$
# eg. ADSRV057,PcDiskspace,CRITICAL,99,d$
#
# eg. SERVERNM,ThresholdManager,CRITICAL,99,logsegment|default
#
# EXAMPLE REPLICATION STUFF
# eg. SOURCESVR,SybRepMonCheck,CRITICAL,30,TARGETNM
# eg. SOURCESVR,SybRepMonCheck,ERROR,15,TARGETNM
# eg. SYB1,SybRepMonCheck,CRITICAL,80,IMAGSYB2
# eg. SYB1,SybRepMonCheck,CRITICAL,80,
# eg. SYB1,SybRepMonCheck,ERROR,60
#
#########################################
The sybase threshold manager
Another separately installed GEM component is the Sybase threshold manager. This program (ADMIN_SCRIPTS/bin/threshold_manager.pl) installs customized sp_thresholdaction procedures into your databases and allows you to find peak usage (most important for log file sizes). This proactive utility will hopefully prevent you from running out of log space - because you will have been warned in advance in such a way that you can expand capacity before there is a production alert.
The sp_thresholdaction procedures we install are simple reporting versions that insert peak usage data into a small table in sybsystemprocs. You can run a report showing peak usage of data and log segments in your databases which is the Threshold Report you can view in the console.
This procedure needs to be installed manually by running
ADMIN_SCRIPTS/bin/threshold_manager.pl --ACTION=INSTALL
after you have registered your servers. One warning. If you increase the *size* of any of your databases, you will need to rerun the above installation command. Sybase thresholds are based on free pages and the percentages are therefore set when you run the installation. So, if you double the size of a database, your 60% threshold will be firing at 30% and recording it as 60% full! Just reinstall and you will be ok!.
Other commands of interest:
ADMIN_SCRIPTS/bin/threshold_manager.pl --ACTION=CHECK
ADMIN_SCRIPTS/bin/threshold_manager.pl --ACTION=REPORT
This feature only works with Sybase
Snapshot Win32 Services
This information is stored in conf/pc_service.dat and can be created from configure.pl using "Snapshot Win32 Services" on the Post install tasklist. The file can also be created by the unscheduled batch file PcServiceCheckerInit.ksh.
You will probably need to hand edit this file - the snapshot is at a point in time and some services may get turned off or just dont stay up. After you have created this file, simply remove lines which get flagged by the monitors yet are not important.
cd win32_batch_scripts/interactive
PcServiceCheckerInit.ksh
Install web server scripts
The alarms tab will populate the directory cgi-bin with runnable web scripts. The button to actually distribute those scripts to your web server is currently unimplemented. You will need to copy the scripts by hand.
This is made easier by the fact that the scripts have been modified to run in your environments, but it is possible that the scripts do not execute.
a) copy the contents of cgi-bin to the web server scripts directory you identified for your use
b) chmod 755 these scripts to make them executable (unix only)
c) try to run mimi.pl when in that directory. Mimi is a cgi-bin script - so it will hang asking for parameters if it runs successfully. It might also give you a command not found or library not found message.
d) you might need to change the hashbang (first line) to point to your perl
e) try running printenv.pl from your web browser... this is a very simple script that should always work.
f) try running mimi.pl from your web browser
It is beyond the scope of this document to discuss web server setup. All the directory requires is script access.
The top of these scripts should look like:
#!/usr/local/bin/perl-5.8.1
use lib qw(/apps/sybmon/dev/lib)
my($COMPANYNAME)="YOUR COMPANY";
BEGIN {
if( ! defined $ENV{SYBASE} or ! -d $ENV{SYBASE} ) {
if( 1==2 ) { print 'doh'; }
} elsif( -d "/export/home/sybase" ) {
$ENV{SYBASE} = "/export/home/sybase";
} elsif( -d "/apps/sybase" ) {
$ENV{SYBASE} = "/apps/sybase";
}
}
}
The paths, company name, sybase directories, and perl version have all been customized. Make sure they all look ok.
Test this script through your web browser.
perl monitor.pl
perl eventviewer.pl
Two modules are currently shipped with GEM - an event viewer and a monitor. Both are located in the root directory. To run, use /monitor.pl. Under windows, you should be able to just double click on these programs to run them and can create desktop shortcuts as needed.
Run fix_db.pl to for initial db config errors
perl ADMIN_SCRIPTS/bin/fix_db.pl -Ax
This is a cute little database configuration checker. You might as well run it. It shows dboptions you might wish to correct and identifies common configuration problems. To correct these problems remove the -x (noexec) option.
ADDITIONAL CONSOLE REPORTS
You can add customized reports to the console easily using conf/console.dat. These reports can be pretty much any command or sql statement. Check out the file for the syntax.
SCHEMA GENERATION
It is recommended that you create object level backups of your server DDL once per week. GEM provides the excellent script dbschema.pl (written by Michael Peppler and David Owen) in the ADMIN_SCRIPTS/bin directory to facilitate this process. Because this script is written in SybPerl (not DBI::Sybase), we do not provide an integrated approach to creating these audits. I recommend you back up your ddl once per week on your production servers.
DNS Montitoring
You can set up conf/dns.dat and then do dns monitoring with
0,20,40 * * * * /usr/bin/nice /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/monitoring/dnsmonitor.pl --OUTFILE=/apps/sybmon/dev/data/html_output/dnsmonitor.pl.txt
Scheduling the batch jobs
Win32 Scheduling
The gem configuration utility will schedule your jobs on win32 monitoring stations.
Unix Scheduling
At 2 points in configure.pl, you can create a unix crontab file. This file is named crontab.txt and is placed in the GEM root directory. You should schedule this file using yoru crontab on unix. To do this run
export EDITOR=vi # presuming u like vi
crontab -e
<vi window>
:r crontab.txt
:wq
You should also place a .forward file in your home directory to redirect mails to your account.
Your own scheduled tasks
Put the output of any of your routine scripts into data/html_output or data/cronlogs to have the reports show in the GEM console. This is a convenient spot to redirect output in any case. The filename itself is used to generate the console report.
# the following job compare a server to its dr copy
1 22 * * * /perl/bin/perl /apps/sybmon/dev/ADMIN_SCRIPTS/bin/datacompare.pl -Usa -SSYB1 -Pxxx -sSYB1DR -pxxx -uxxx -h > /apps/sybmon/dev/data/html_output/Rowcount_Compare_Prod_DR.html 2>&1
Schedule additional alarming reports. These need to be set up in the cgi-bin mimi.pl reports screen before they can be run. But you can snapshot anything you can see on those screens and mail the results as appropriate
0 10,12,14,16,18 * * 1-5 /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/monitoring/RunMimiReport.pl --SUBJECT="2 Hour Unix Status Report" --MAILTO=def@bac.com,abc@abc.com --REPORT=Unix2Hour --TIME="2 Hours" > /apps/sybmon/dev/data/GEM_BATCHJOB_LOGS/RunMimiReport_Unix2Hour 2> /apps/sybmon/dev/data/batchjob_errors/RunMimiReport_Unix2Hour
I sync logins between my replicated servers with
38 9 * * 0 /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/bin/copy_syslogins.pl -FSYB1 -Usa -Pxxx -tSYB2 -usa -pyyy > /apps/sybmon/dev/data/GEM_BATCHJOB_LOGS/copy_syslogins_SYB1.log 2>/apps/sybmon/dev/data/batchjob_errors/copy_syslogins_SYB1.log
I monitor cisco logs with
18,48 * * * * /usr/bin/nice /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/monitoring/cisco_logmon.pl --INFILE=/var/log/cisco.log --TODAYONLY >/apps/sybmon/dev/data/GEM_BATCHJOB_LOGS/cisco_logmon.log 2>/apps/sybmon/dev/data/batchjob_errors/cisco_logmon.log
I monitor centrally collected (via syslogsd) unix logs with
5,15,25,35,45,55 * * * * /usr/local/bin/perl-5.8.1 /apps/sybmon/dev/ADMIN_SCRIPTS/monitoring/unix_monitor.pl --INFILE=/var/log/messages --TODAYONLY --EXCLUDEFILE=/apps/sybmon/dev/ADMIN_SCRIPTS/bin/solaris.excl --MAXLINES=100000000 --HTML --OUTFILE=/apps/sybmon/dev/data/html_output/unix_monitor.html >/apps/sybmon/dev/data/GEM_BATCHJOB_LOGS/unix_monitor.log 2>/apps/sybmon/dev/data/batchjob_errors/unix_monitor.log
Directory security
GEM is an administrative tool, designed for system administrators. As such, the conf directory contains all your passwords. You need to protect this data via operating system permissions.
We strongly recommend that all other directories in the tree, with the exception of the data/CONSOLE_REPORTS tree, be also made secure. Technically, you can allow read/execute access to these files, but it would be easier to only allow owner/group access - much of the data contained may be sensitive.
GEM is designed for simple upgrades. The steps are as follows
- Download the latest codeline
- Extract on top of your existing distribution!!!
No configuration files are shipped with the installation so there should be no problems. All configuration files are shipped with a .sample extension.
On your windows monitoring server, you may not be able to extract the code because the task scheduler is running a batch job that calls one of the perl scripts. Windows will not let you overwrite a program that is running. This does not happen under Unix. You can either go into the task scheduler, stop all running jobs before rerunning the extract, or kill the jobs and then stop the task scheduler process. I normally just stop the jobs, but sometimes (because end task does not work cleanly) i am forced to stop the scheduler process for the duration of the extract.
- Run
perl upgrade.pl
After you download, your GEM scripts will have *OUR* hashbangs, library paths, and sybase environment. This is probably not what you want.
Run perl upgrade.pl to finalize the installation. This step is MANDATORY and will perform a variety of steps like updating your configuration files and re-installing your alarm database. During the period you are upgrading - your GEM batch jobs will probably fail and send you alarm mail. Ignore these failures notifications.
The alarm database reinstall will save table data, drop all tables, reinstall tables, and finally copy back data. If there is a problem, it will leave a tables with the extension _BAK containing the original data. So, under the hood, the upgrade process for the table Event is "select into Event_BAK from Event", "drop table Event", "create table Event", "insert Event select * from Event_BAK", and finally "drop table Event_BAK". Upgrades can not start if any tables with the extension _BAK exists - you must remove them before the upgrade.
If there is a problem, you can probably just drop the BAK table - but it would be better to recover by following a process akin to "truncate table Event", "insert Event select * from Event_BAK", "Drop table Event" (you have to do these by hand). Got it? Any questions on a failed install please contact SQL Technologies tech support.
The final install step redoes your hashbangs, perl library paths, and removes all control M's from the code line. A log of the upgrade is saved in the logs directory.
ONCE YOU HAVE SUCCESSFULLY RUN upgrade.pl YOU ARE DONE
DESCRIPTION
Batch jobs are created by the GEM configuration program and are used to monitor your servers and to create the GEM Console. These scripts are organized into 8 directories that contain duplicate scripts. The batch jobs name is the "monitor program" shown by the monitoring system.
The directory structure is
win32_batch_scripts or
batch
interactive
plan_batch
plan_interactive
unix_batch_scripts
batch
interactive
plan_batch
plan_interactive
Files are created in these directories as needed by configure.pl. For example, if you tell configure.pl that you have only windows based systems, the subdirectories under unix_batch_scripts will not be populated. Similarly, if you are in an all-sybase environment, the oracle and sql server scripts will not be created. Batch jobs are created with a .ksh extension. Additionally, the win32_batch_scripts tree will contain duplicates with a .bat extension. This simplfies scheduling.
The major difference between the "*_batch" and "*_interactive" directories is the redirection of standard output. Scheduled jobs use the "batch" directories which differ from the "interactive" directories only in that they redirect standard output and standard error into the GEM_HOME/data/GEM_BATCHJOB_LOGS directory. These files can be viewed through the Console.
You are responsible for scheduling these jobs (sort of). If you are running on windows, configure.pl (the GEM Configuration utility) can auto schedule your jobs in the windows task scheduler. If unix, configure.pl can create a customized GEM_HOME/crontab.txt file for your inclusion in your unix crontab (via crontab -e; set the variable 'EDITOR=vi' if 'crontab -e' leaves you in 'ed' mode. On Win32, GEM provides both .ksh and .bat versions of all scripts to support those who use a unix emulator like cygwin or MKS toolkit.
TO RUN THE JOBS INTERACTIVELY
When running batch jobs by hand (for example to test), always use the files in the 'interactive' or 'plan_interactive' subdirectory trees. If you use the scripts in batch or plan_batch, you wont be able to view the output because it is redirected to files!!!
The following table is a summary of the jobs that can be created by GEM's configuration utility.
| Scripts In The 'plan' Subdirectories |
| Job Name | Win32 or Unix | Description | Recommended Frequency |
| Backup_<PLANNAME> | Win32 | Main Nightly PLANNAME Backup Run (and possibly transfer & load them) | daily |
| FixLogship_<PLANNAME> | Win32 | Guarantees Transfer of logshiped files for plan PLANNAME | never |
| LoadTranlogs_<PLANNAME> | Win32 | Loads Transfered but Unapplied Transaction Logs | never |
| Trandump_<PLANNAME> | Win32 | Run a Transaction Log Dump (and possibly transfer & load them) | Hourly |
| Scripts In The 'interactive' Subdirectories |
| Job Name | Win32 or Unix | Description | Recommended Frequency |
| AlarmCleanupDB | Both | Internal cleanup of the gemalarms database | daily |
| AlarmRoutingAgent | Both | Read Alerts & Route Them | Frequently |
| CheckServer_<SERVER> | Win32 | Is SERVER ok? | Hourly |
| CleanupUnixGEM | Unix | Cleans up unix files on GEM | Daily |
| ConsoleArchiver.ksh | Both | Once a month make a datestamped copy of the console | Monthly |
| ConsoleBuildAndFtp | either | Assembles the Console. Optionally publish the Console to external web server | Hourly |
| ConsoleUnixHourly | Unix | Builds frequently changing pages in the Console - scheduled 1ce/hour | Hourly |
| ConsoleUnixNightly | Unix | Build 1ce/day pages in the Console | daily |
| ConsoleUnixWeekly | Unix | Build 1ce/week pages in the Console | Weekly |
| ConsoleWin32Hourly | Win32 | Builds frequently changing pages in the Console - scheduled 1ce/hour | Hourly |
| ConsoleWin32Nightly | Win32 | Build 1ce/day pages in the Console | Daily |
| ConsoleWin32Weekly | Win32 | Build 1ce/week pages in the Console | Weekly |
| DnsMonitor | Both | Monitors your DNS Servers | Hourly |
| GemRpt_AgentReport | Both | Create Backup Reports | Hourly |
| GemRpt_BackupReport | Both | Create Backup Reports | Hourly |
| GemRpt_Backups | Both | Create Backup Reports | Hourly |
| GemRpt_BlackoutReport | Both | Creates the Blackout Report | Daily |
| GemRpt_PCDiskspace | Both | Win32 Disk Space Monitor | Hourly |
| GemRpt_PCEventlog | Both | Win32 Event Log Monitor | Hourly |
| GemRpt_ProductionErrors | Both | Create Production Overnight Report | Hourly |
| GemRpt_ProductionList.ksh | Both | Creates the Production List Report | Daily |
| GemRpt_ProductionWarnings | Both | Create Full List Of All Alerts | Hourly |
| GemRpt_Sybase_ASE | Both | Create Sybase Event Log Reports | Hourly |
| GemRpt_Sybase_BackupServer | Both | Create Sybase Backup Reports | Hourly |
| GemRpt_Sybase_RepServer | Both | Create Sybase Replication Report | hourly | |
| GemRpt_Unix_Errors.ksh | Unix | Create a report with current unix errors | Daily |
| GemRpt_Unix_Warnings.ksh | Unix | Create a report with current unix warnings | Daily |
| GetUnixDiskSpace | Unix | Collect Unix Disk Space Alerts | Frequently |
| LargeLogFileArchive.ksh | | Archive database error logs that have grown too large | Weekly |
| OptdiagAnalyze.ksh | | Analyze your servers based on optdiag output | Weekly |
| OraSurvey | Both | Survey all oracle servers | daily |
| PortMonitor | Both | Test Services (sybase servers, win32 services...) | Frequently |
| SybAnalyzeDepends | Both | Create Dependency Analysis Files Of All Your Databases | Weekly |
| SybAuditor | Both | Reverse Engineer an Audit of your Sybase Servers | Weekly |
| SybBkpSvrConfig | Both | Create Configuration Report on Backup File Fetch | Daily |
| SybBkpSvrLogFetch | Both | Fetch All Sybase Backup Files | Hourly |
| SybBkpSvrLogRpt | Both | Process Sybase Backup Files | Hourly |
| SybCheck_Development | Both | Check availability of your Development Servers | Hourly |
| SybDailyCheck | Both | Daily Sybase Server Check | Daily |
| SybErrLogAllRpt | Both | Create complete Sybase Error Log REport | Hourly |
| SybErrLogSaveAlarms | Both | Parse Sybase Error Logs & Save Any Alerts | Hourly |
| SybErrLogConfig | Both | Create Sybase Error Log Configuration Report | daily |
| SybErrLogFetch | Both | Fetch Sybase Error Logs | Hourly |
| SybErrLogRpt | Both | Analyze & Report on Sybase Error Logs | Hourly |
| SybHistSvrConfig | Both | Report Historical Server Configuration | daily |
| SybHistSvrLogFetch | Both | Fetch Historical Server Log Files | Hourly |
| SybHistSvrLogRpt | Both | Analyze & Create Historical Server Report | Hourly |
| SybMonSvrConfig | Both | Report Monitor Server Configuration | daily |
| SybMonSvrLogFetch | Both | Fetch Historical Server Log Files | Hourly |
| SybMonSvrLogRpt | Both | Analyze & Create Monitor Server Report | Hourly |
| SybPing | Both | Monitor Your Sybase Servers via ping | Frequently |
| SybSurvey | Both | survey | daily |
| SybRepCompare | | Compare Rowcounts between servers that are replicated | Daily |
| SybRepServerChecker | Both | Check Replication | Hourly |
| SybRepSvrConfig | Both | Report On Your Replication Configuration | daily |
| SybRepSvrLogFetch | Both | Fetch Replication Server Log Files | Frequently |
| SybRepSvrLogRpt | Both | Report on Repliation Server Logs | Hourly |
| SybRepSvrSaveAlarms | Both | Parse & Save Replication Server Errors | Frequently |
| SybaseBackupFileReport | Both | Report on Backups | daily |
| SybRepMonCheck | | Checks that your replication monitoring is working | Daily |
| SybRepMonReport | Both | Create Report of Replication Latencieis | Frequently |
| SybSpaceMonitor | Both | Sybase Space Monitor | Hourly |
| SybStoredProcUpdater | | Ensure that all stored procedure libraries are at the latest revision | Weekly |
| SybThresholdManager | Both | Run Threshold Manager | Daily |
| SybThresholdManagerReinstall | Both | Reinstall Threshold Manager for databases that have changed in size | Daily |
| SybTruncateMaster | Both | dump tran master with nolog | Daily |
| SybTruncateSybsystemdb | Both | dump tran sybsystemdb with nolog | Weekly |
| SybTruncateSybsystemprocs | Both | dump tran sybsystemprocs with nolog | Weekly |
| MssqlAnalyzeDepends | Win32 | Optionally analyze dependencies on Sql Server | Daily |
| MssqlAuditor | Win32 | Audit your Mssql configurations | Daily |
| MssqlBackupCrosscheck_AM | Win32 | Morning batch to check Mssql Backups | Daily |
| MssqlBackupCrosscheck_PM | Win32 | Evening batch to check Mssql Backups | Hourly |
| MssqlBackupHistory | Win32 | Check your backup history | Daily |
| MssqlCheck_Development | Win32 | Check your development servers | Hourly |
| GetBackupStateMssql | Win32 | Have All Your SQL Server Databases Been Backed Up? | Hourly |
| GetBackupStateSybase | Both | Have All Your Sybase Server Databases Been Backed Up? | Hourly |
| MssqlCycleEventlogs | Win32 | Cycle Sql Server Event Logs | daily |
| MssqlDailyCheck | Win32 | Daily Check of your Mssql Servers | |
| MssqlJobStatusRpt | Win32 | Check Sql Server Scheduled Jobs | daily |
| MssqlLogMaintPlan | Win32 | Check Sql Server Transaction Logs Plans | Hourly |
| MssqlPing | Win32 | Monitor Your Sql Servers via ping | Frequently |
| MssqlScheduledJobRpt | Win32 | Check SQL Server Scheduled Jobs | daily |
| MssqlShrinkData |