A Guide To GEM Batch Jobs

The Complete GEM Documentation Slide Show: What Is GEM Download GEM

An Introduction To GEM How To Install Getting Started Guide

Batch Jobs Overview GEM Screenshots Server Backup and Maintenance

Utilities Guide The GEM Console New System Stored Procedures

FAQ and Change Log Common Perl Libraries Pricing & Right To Use

GUIDE TO GEM BATCH JOBS

DESCRIPTION

Batch jobs are created by the GEM configuration program and are used to monitor your servers and to create the GEM Console. These scripts are organized into 8 directories that contain duplicate scripts. The batch jobs name is the "monitor program" shown by the monitoring system.

The directory structure is

   win32_batch_scripts or
       batch
       interactive
       plan_batch
       plan_interactive
   unix_batch_scripts
       batch
       interactive
       plan_batch
       plan_interactive

Files are created in these directories as needed by configure.pl. For example, if you tell configure.pl that you have only windows based systems, the subdirectories under unix_batch_scripts will not be populated. Similarly, if you are in an all-sybase environment, the oracle and sql server scripts will not be created. Batch jobs are created with a .ksh extension. Additionally, the win32_batch_scripts tree will contain duplicates with a .bat extension. This simplfies scheduling.

The major difference between the "*_batch" and "*_interactive" directories is the redirection of standard output. Scheduled jobs use the "batch" directories which differ from the "interactive" directories only in that they redirect standard output and standard error into the GEM_HOME/data/GEM_BATCHJOB_LOGS directory. These files can be viewed through the Console.

You are responsible for scheduling these jobs (sort of). If you are running on windows, configure.pl (the GEM Configuration utility) can auto schedule your jobs in the windows task scheduler. If unix, configure.pl can create a customized GEM_HOME/crontab.txt file for your inclusion in your unix crontab (via crontab -e; set the variable 'EDITOR=vi' if 'crontab -e' leaves you in 'ed' mode. On Win32, GEM provides both .ksh and .bat versions of all scripts to support those who use a unix emulator like cygwin or MKS toolkit.

TO RUN THE JOBS INTERACTIVELY

When running batch jobs by hand (for example to test), always use the files in the 'interactive' or 'plan_interactive' subdirectory trees. If you use the scripts in batch or plan_batch, you wont be able to view the output because it is redirected to files!!!

SUMMARY OF BATCH JOBS

The following table is a summary of the jobs that can be created by GEM's configuration utility.

Scripts In The 'plan' Subdirectories

Job Name Win32 or
Unix Description Recommended
Frequency

Backup_<PLANNAME> Win32 Main Nightly PLANNAME Backup Run (and possibly transfer & load them) daily

FixLogship_<PLANNAME> Win32 Guarantees Transfer of logshiped files for plan PLANNAME never

LoadTranlogs_<PLANNAME> Win32 Loads Transfered but Unapplied Transaction Logs never

Trandump_<PLANNAME> Win32 Run a Transaction Log Dump (and possibly transfer & load them) Hourly

Scripts In The 'interactive' Subdirectories

Job Name Win32 or
Unix Description Recommended
Frequency

AlarmCleanupDB Both Internal cleanup of the gemalarms database daily

AlarmRoutingAgent Both Read Alerts & Route Them Frequently

CheckServer_<SERVER> Win32 Is SERVER ok? Hourly

CleanupUnixGEM Unix Cleans up unix files on GEM Daily

ConsoleArchiver.ksh Both Once a month make a datestamped copy of the console Monthly

ConsoleBuildAndFtp either Assembles the Console. Optionally publish the Console to external web server Hourly

ConsoleUnixHourly Unix Builds frequently changing pages in the Console - scheduled 1ce/hour Hourly

ConsoleUnixNightly Unix Build 1ce/day pages in the Console daily

ConsoleUnixWeekly Unix Build 1ce/week pages in the Console Weekly

ConsoleWin32Hourly Win32 Builds frequently changing pages in the Console - scheduled 1ce/hour Hourly

ConsoleWin32Nightly Win32 Build 1ce/day pages in the Console Daily

ConsoleWin32Weekly Win32 Build 1ce/week pages in the Console Weekly

DnsMonitor Both Monitors your DNS Servers Hourly

GemRpt_AgentReport Both Create Backup Reports Hourly

GemRpt_BackupReport Both Create Backup Reports Hourly

GemRpt_Backups Both Create Backup Reports Hourly

GemRpt_BlackoutReport Both Creates the Blackout Report Daily

GemRpt_PCDiskspace Both Win32 Disk Space Monitor Hourly

GemRpt_PCEventlog Both Win32 Event Log Monitor Hourly

GemRpt_ProductionErrors Both Create Production Overnight Report Hourly

GemRpt_ProductionList.ksh Both Creates the Production List Report Daily

GemRpt_ProductionWarnings Both Create Full List Of All Alerts Hourly

GemRpt_Sybase_ASE Both Create Sybase Event Log Reports Hourly

GemRpt_Sybase_BackupServer Both Create Sybase Backup Reports Hourly

GemRpt_Sybase_RepServer Both Create Sybase Replication Report hourly

GemRpt_Unix_Errors.ksh Unix Create a report with current unix errors Daily

GemRpt_Unix_Warnings.ksh Unix Create a report with current unix warnings Daily

GetUnixDiskSpace Unix Collect Unix Disk Space Alerts Frequently
LargeLogFileArchive.ksh Archive database error logs that have grown too large Weekly

OptdiagAnalyze.ksh Analyze your servers based on optdiag output Weekly

OraSurvey Both Survey all oracle servers daily

PortMonitor Both Test Services (sybase servers, win32 services...) Frequently

SybAnalyzeDepends Both Create Dependency Analysis Files Of All Your Databases Weekly

SybAuditor Both Reverse Engineer an Audit of your Sybase Servers Weekly

SybBkpSvrConfig Both Create Configuration Report on Backup File Fetch Daily

SybBkpSvrLogFetch Both Fetch All Sybase Backup Files Hourly

SybBkpSvrLogRpt Both Process Sybase Backup Files Hourly

SybCheck_Development Both Check availability of your Development Servers Hourly

SybDailyCheck Both Daily Sybase Server Check Daily
SybErrLogAllRpt Both Create complete Sybase Error Log REport Hourly
SybErrLogSaveAlarms Both Parse Sybase Error Logs & Save Any Alerts Hourly

SybErrLogConfig Both Create Sybase Error Log Configuration Report daily

SybErrLogFetch Both Fetch Sybase Error Logs Hourly

SybErrLogRpt Both Analyze & Report on Sybase Error Logs Hourly
SybHistSvrConfig Both Report Historical Server Configuration daily

SybHistSvrLogFetch Both Fetch Historical Server Log Files Hourly

SybHistSvrLogRpt Both Analyze & Create Historical Server Report Hourly
SybMonSvrConfig Both Report Monitor Server Configuration daily

SybMonSvrLogFetch Both Fetch Historical Server Log Files Hourly

SybMonSvrLogRpt Both Analyze & Create Monitor Server Report Hourly
SybPing Both Monitor Your Sybase Servers via ping Frequently

SybSurvey Both survey daily

SybRepCompare Compare Rowcounts between servers that are replicated Daily

SybRepServerChecker Both Check Replication Hourly

SybRepSvrConfig Both Report On Your Replication Configuration daily

SybRepSvrLogFetch Both Fetch Replication Server Log Files Frequently

SybRepSvrLogRpt Both Report on Repliation Server Logs Hourly

SybRepSvrSaveAlarms Both Parse & Save Replication Server Errors Frequently
SybaseBackupFileReport Both Report on Backups daily

SybRepMonCheck Checks that your replication monitoring is working Daily

SybRepMonReport Both Create Report of Replication Latencieis Frequently

SybSpaceMonitor Both Sybase Space Monitor Hourly

SybStoredProcUpdater Ensure that all stored procedure libraries are at the latest revision Weekly

SybThresholdManager Both Run Threshold Manager Daily

SybThresholdManagerReinstall Both Reinstall Threshold Manager for databases that have changed in size Daily

SybTruncateMaster Both dump tran master with nolog Daily

SybTruncateSybsystemdb Both dump tran sybsystemdb with nolog Weekly

SybTruncateSybsystemprocs Both dump tran sybsystemprocs with nolog Weekly

MssqlAnalyzeDepends Win32 Optionally analyze dependencies on Sql Server Daily

MssqlAuditor Win32 Audit your Mssql configurations Daily

MssqlBackupCrosscheck_AM Win32 Morning batch to check Mssql Backups Daily

MssqlBackupCrosscheck_PM Win32 Evening batch to check Mssql Backups Hourly

MssqlBackupHistory Win32 Check your backup history Daily

MssqlCheck_Development Win32 Check your development servers Hourly
GetBackupStateMssql Win32 Have All Your SQL Server Databases Been Backed Up? Hourly

GetBackupStateSybase Both Have All Your Sybase Server Databases Been Backed Up? Hourly

MssqlCycleEventlogs Win32 Cycle Sql Server Event Logs daily

MssqlDailyCheck Win32 Daily Check of your Mssql Servers

MssqlJobStatusRpt Win32 Check Sql Server Scheduled Jobs daily

MssqlLogMaintPlan Win32 Check Sql Server Transaction Logs Plans Hourly

MssqlPing Win32 Monitor Your Sql Servers via ping Frequently

MssqlScheduledJobRpt Win32 Check SQL Server Scheduled Jobs daily

MssqlShrinkDatabase Win32 Shrink Database Weekly

MssqlShrinklogs Win32 Shrink Transaction Logs Hourly

MssqlStoredProcUpdater Win32 Ensure latest revision of Stored Procedure Library

SpaceMonitorMssql Win32 Monitor Space on Mssql Servers Hourly

MssqlSurvey Win32 survey your Win32 Servers daily

MssqlTruncateMaster Win32 Truncate Master database log

Win32FileCleanup Win32 Cleanup Windows Directories daily

Win32Diskspace Win32 Monitors Windows Diskspace Frequently

Win32Eventlog Win32 Monitors Windows Event Logs Frequently

Win32GetHosts Win32 Fetch Win32 Host Information daily

Win32Ping Win32 Pings your Win32 Hosts Frequently

Win32HostsRpt Win32 Create Win32 Hosts Report daily

Win32ServiceChecker Win32 Monitors Windows Services Frequently

Win32ServiceCheckerUpdate Win32 Append to Services List for any new servers never

Win32Survey Win32 Survey Your Win32 Servers daily

Win32TaskScheduler Win32 Monitors Windows Task Scheduler Hourly

TestDfltConnectivity Both Tests Connectivity to your Hosts Hourly

TraceRoute Both Run & Analyze Trace Route Info To Your Servers Hourly

UnixPing Both Monitor Your unix systems via ping Frequently

UnixSurvey Unix Survey Your Unix Servers Daily

UnixBackupStateStaticInfo Both Sanity Check Backup Configuration Static Info (lists of databases...) Daily

UnixFileCleanup Unix Cleanup Unix Directories Daily

Survey Scripts

The survey scripts discover information about your servers and store the results in the gem.xml data file. The scritps are broken up by type for scheduling convenience. The Sql Server and Windows surveys must be run from a Win32 scheduler, but the Oracle and Sybase jobs can be run on either unix or windows. I personally prefer to run these jobs on unix.

OraSurvey

  Package:  Gem Console
  Synopsis: Surveys your oracle servers - collects configuration information
  Executes: <CODEDIR>/server_documenter/discover.pl -Toracle

SurveySqlServer

  Package:  Gem Console
  Synopsis: Surveys your sql servers - collects configuration information
  Executes: <CODEDIR>/server_documenter/discover.pl -Tsqlsvr

SybSurvey

  Package:  Gem Console
  Synopsis: Surveys your sybase servers - collects configuration information
  Executes: <CODEDIR>/server_documenter/discover.pl -Tsybase

Win32Survey

  Package:  Gem Console
  Synopsis: Surveys your windows servers - collects configuration information
  Executes: <CODEDIR>/server_documenter/discover.pl -Tpcdisks

MAINTENANCE PLAN JOBS

These jobs, stored in plan_batch and plan_interactive are used for your nightly maintenance. Each plan you created will have its own associated set of scripts. There will either 2 or 4 batches created for each plan, depending on whether log shipping is enabled.

Backup_{PLANNAME}

  Package:  Gem Backup Jobs
  Synopsis: Backs up your databases
  Executes: <CODEDIR>/dbi_backup_scripts/backup.pl -J<JOB>

FixLogship_{PLANNAME}

  Package:  Gem Backup Jobs
  Synopsis: Copy transaction logs for log shipping
  Executes: <CODEDIR>/dbi_backup_scripts/fix_logship.pl -J<JOB>

LoadTranlogs_{PLANNAME}

  Package:  Gem Backup Jobs
  Synopsis: Cleans up old events and heartbeats from your GEM Alarms Database
  Executes: <CODEDIR>/dbi_backup_scripts/load_all_tranlogs.pl -JOB=<JOB>

Trandump_{PLANNAME}

  Package:  Gem Backup Jobs
  Synopsis: Cleans up old events and heartbeats from your GEM Alarms Database
  Executes: <CODEDIR>/dbi_backup_scripts/backup.pl -t -J<JOB>

CONSOLE BATCH JOBS

These jobs create and manage the Console

ConsoleUnixHourly, ConsoleWin32Hourly

  Package:  GEM Console
  Synopsis: builds the Console documentation tree with recently changed stuff
  Executes: <CODEDIR>/server_documenter/consoler_hourly.pl

ConsoleUnixNightly, ConsoleWin32Nightly

  Package:  GEM Console
  Synopsis: builds the Console documentation tree - overnight rebuild
  Executes: <CODEDIR>/server_documenter/console_nightly.pl

ConsoleUnixWeekly, ConsoleWin32Weekly

  Package:  GEM Console
  Synopsis: builds the Console documentation tree - complete rebuild (slow)
  Executes: <CODEDIR>/server_documenter/console_weekly.pl

CONSOLE BATCH REPORTS

These reports use the alarm viewer to create pre-designed reports about your servers. These reports are stored in /html_output. When any of the Console build utilities run, they search for these reports and copy them in to the GEM menu heirarchy. These reports are, in other words, batched. As you will probably be rebuilding the Console hourly, they may be up to 1 hour delayed (it depends on how often you schedule the job). Note that the same command executable (RunMimiReport.pl) takes other arguments including email addresses for direct mailing of reports to specific email addresses.

GemRpt_Backups

  Package:  GEM Console
  Synopsis: Creates Custom Report GemRpt_Backups.html
  Executes: <CODEDIR>/monitoring/RunMimiReport.pl --REPORT=GemRpt_Backups
     --OUTFILE=<DATADIR>/html_output/GemRpt_Backups.html

GemRpt_PCDiskspace

  Package:  GEM Console
  Synopsis: Creates Custom Report GemRpt_PCDiskspace.html
  Executes: <CODEDIR>/monitoring/RunMimiReport.pl --REPORT=GemRpt_PCDiskspace
     --OUTFILE=<DATADIR>/html_output/GemRpt_PCDiskspace.html

GemRpt_PCEventlog

  Package:  GEM Console
  Synopsis: Creates Custom Report GemRpt_PCEventlog.html
  Executes: <CODEDIR>/monitoring/RunMimiReport.pl --REPORT=GemRpt_PCEventlog
     --OUTFILE=<DATADIR>/html_output/GemRpt_PCEventlog.html

GemRpt_ProductionOvernight

   Package:  GEM Console
   Synopsis: Creates Custom Report GemRpt_ProductionOvernight.html
   Executes: <CODEDIR>/monitoring/RunMimiReport.pl --BATCHID=GemRpt_ProductionOvernight
      --REPORT=GemRpt_ProductionOvernight
      --OUTFILE=<DATADIR>/html_output/GemRpt_ProductionOvernight.html

GemRpt_ProductionWarnings

   Package:  GEM Console
   Synopsis: Creates Custom Report GemRpt_ProductionWarnings.html
   Executes: <CODEDIR>/monitoring/RunMimiReport.pl --BATCHID=GemRpt_ProductionWarnings
      --REPORT=GemRpt_ProductionWarnings
      --OUTFILE=<DATADIR>/html_output/GemRpt_ProductionWarnings.html

GemRpt_SybaseASE

   Package:  GEM Console
   Synopsis: Creates Custom Report GemRpt_SybaseASE.html
   Executes: <CODEDIR>/monitoring/RunMimiReport.pl --BATCHID=GemRpt_SybaseASE
      --REPORT=GemRpt_SybaseASE
      --OUTFILE=<DATADIR>/html_output/GemRpt_SybaseASE.html

GemRpt_SybaseBackupServer

   Package:  GEM Console
   Synopsis: Creates Custom Report GemRpt_SybaseBackupServer.html
   Executes: <CODEDIR>/monitoring/RunMimiReport.pl --BATCHID=GemRpt_SybaseBackupServer
      --REPORT=GemRpt_SybaseBackupServer
      --OUTFILE=<DATADIR>/html_output/GemRpt_SybaseBackupServer.html

GemRpt_SybaseRepServer

   Package:  GEM Console
   Synopsis: Creates Custom Report GemRpt_SybaseRepServer.html
   Executes: <CODEDIR>/monitoring/RunMimiReport.pl --BATCHID=GemRpt_SybaseRepServer
      --REPORT=GemRpt_SybaseRepServer
      --OUTFILE=<DATADIR>/html_output/GemRpt_SybaseRepServer.html

NORMAL JOBS

These jobs, stored in the batch and interactive subdirectories are used for normal maintenance.

AlarmCleanup

  Package:  Gem Maintenance
  Synopsis: Cleans up old events and heartbeats from your GEM Alarms Database
  Executes: <CODEDIR>/monitoring/MlpAlarmCleanupDB.pl --DAYS=7

AlarmRoutingAgent

  Package:  Gem Maintenance
  Synopsis: Reads GEM Alarms and routes them via pager and email as appropriate
  Executes: <CODEDIR>/monitoring/MlpAlarmRoutingAgent.pl --FROM=MonitorAdm@hotmail.com --MAXHOURS=24

AnalyzeDepends

  Package:  Gem Console / Utilities
  Synopsis: Builds a real ddl dependency tree for your servers
  Executes: <CODEDIR>/bin/depends_analyze.pl

CheckServer_{SERVER}

  Package:  GEM Console / Monitoring
  Monitor:  CheckForBlocks
  Synopsis: Checks a single server for blocks.  Will have --MAILTO if it is a production server.
  Optional Arguments: --TIME=secs and --MAILTO=a@b,c@d  can be used for higher granualarity monitoring.k
  Executes: <CODEDIR>/monitoring/CheckServer.pl -TYPE=sqlsvr -TIME=300 --SERVER={SERVER}
     -MICROSOFT
  Executes: <CODEDIR>/monitoring/CheckServer.pl -TYPE=sqlsvr -TIME=60 --SERVER={SERVER}
     -MICROSOFT --MAILTO="barlowedward@hotmail.com"

Win32GetHosts

  Package:  GEM Console
  Synopsis: Collects hosts files from your windows boxes
  Executes: <CODEDIR>/bin/win32_get_file.pl --BATCH_ID=Win32GetHosts
     --INDIR=/winnt/system32/drivers/etc,/windows/system32/drivers/etc,/windows,/winnt
     --INFILE=hosts -OUTDIR=<DATADIR>/system_information_data/hosts

Win32TaskScheduler

  Package:  GEM Console
  Synopsis: Collects windows task scheduler log files from your windows boxes
  Executes: <CODEDIR>/bin/win32_get_file.pl --BATCH_ID=Win32TaskScheduler --INDIR=/winnt,/windows
     --INFILE=Schedlgu.txt --NEWLABEL=Win32_Job_Scheduler_Log
     -OUTDIR=<DATADIR>/system_information_data/schedlgu

MssqlJobStatusRpt

  Executes: <CODEDIR>/bin/mssql_jobstatus_rpt.pl --HTML --OUTFILE=<DATADIR>/html_output/mssql_jobstatus_rpt.html

This creates the file html_output/mssql_jobstatus_rpt.html which lists failed sql server jobs that are enabled. To remove rows from this report you should disable the failed job.

MssqlLogMaintPlan

  Monitor: MssqlLogMaintPlan
  Executes: <CODEDIR>/bin/mssql_logmaintplan.pl -CONFIGFILE=<DATADIR>/logmaintplan.dat

This saves maintenance plan information as heartbeats with the monitor_key=LogMaintPlan. This filters only system maintenance plans for SQL Server.

Win32FileCleanup

This is the most important of the maintenance scripts. The only issue is that you need to set it up by hand. Every directory you want purged should have lines added to it. Maintain this program and you wont ever need to run out of disk space again.

  Package: Normal Maintenance
  Synopsis: Cleans up directories on windows.  You should edit your conf/Win32_Cleanup.dat file
  Monitor: Win32FileCleanup
  Executes: <CODEDIR>/bin/purge_old_files.pl --BATCH_ID=Win32FileCleanup --ctlfile=//samba/sybmon/dev/conf/Win32_Cleanup.dat

Win32Diskspace

  Package:  Gem Console
  Monitor:  Win32Diskspace
  Synopsis: Monitor and Report on Windows Diskspace
  Executes: <CODEDIR>/bin/win32_diskspace.pl --ALARMTHRESH=95 --WARNTHRESH=90
     --LOGALARM --OUTFILE=<DATADIR>/html_output/win32_diskspace.txt

win32_diskspace.bat will save space usage information for your servers with embedded alarm thresholds of 90 for warning and 95% for errors. The output is placed in html_output/win32_diskspace.txt to be picked up later.

PcEventlog

  Package:  Gem Console
  Monitor:  PcEventlog
  Synopsis: Monitor and Report on Windows System Event Logs
  Executes: <CODEDIR>/bin/win32_eventlog.pl --HOURS=2 --LOGEVENTS --CONFIGFILE=<DATADIR>/win32_eventlog.dat --SHOWINFOMSGS

win32_eventlog.bat will save slightly reduced list of event log messages into the alarm manager. Several specific services are filtered out - the type can be adjusted by editing the win32_eventlog.pl script (which gives directions).

win32_eventlog.bat uses the file data/win32_eventlog.dat for its base configuration information. It should run frequently (every 1-2 hours). Messages are saved in the alarm system with monitor_program = PcEventlog

Win32ScheduledJobRpt

  Executes: <CODEDIR>/bin/win32_scheduled_job_rpt.pl --HTMLFILE=<DATADIR>/html_output/win32_scheduled_job.html

Job scheduler reports will be generated. The mechanism for this is win32_scheduled_job_rpt.*. and sqlsvr_backupreport.ksh. These call the primitive routines win32_backupreport.pl and win32_scheduled_job_rpt.pl and result in html_output of win32_scheduled_job.html.

The primitive for this is win32_scheduled_job_rpt.pl and win32_backupreprot.pl

Win32ServiceChecker

  Package:  Gem Console
  Synopsis: Monitor and Report on Windows Services
  Monitor:  Win32ServiceChecker
  Executes: <CODEDIR>/bin/win32_service.pl --ACTION=VALIDATE

Checks service availability on all your windows servers and compares results to a snapshot of services that should be running. The results are saved in the alarms database. This snapshot is not created by default (yet). The snapshot is stored in the conf/win32_service.dat file. The first thing to do is to run the following which creates a snapshot file of all running services on all your defined Win32 servers

  Win32_service_checker_init.bat

The above command should probably not be scheduled, or scheduled only once per week. The init routine writes the win32_service.dat configuration file and will clear your alarm database. If you wish to reset the services for a particular system (perhaps it was rebuilt) you should add --SYSTEMS=server. In other words you should run either

   Win32_service_checker_init --SYSTEMS=winhost1

which is the same as running the primitive

   win32_service.pl --SYSTEMS=winhost1 --ACTION=SNAPSHOT

win32_service_checker.bat should probably be run fairly frequently. The primitive for this is win32_service.pl

Win32ServiceCheckerInit

  Package:  Gem Console
  Synopsis: One off batch - takes a snapshot of services on all your windows boxes for use by Win32ServiceChecker
  Executes: <CODEDIR>/bin/win32_service.pl --ACTION=SNAPSHOT

Win32ShrinkLogs

  Package:  Maintenance
  Synopsis: Shrink the transactionlogs on all your windows servers
  Executes: <CODEDIR>/bin/mssql_shrinklogs.pl --DOSHRINK --SYSTEMS=ALL

The win32_shrinklogs.bat script will shrink all logs in all your sql servers using the mssql_shrinklogs.pl primitive.

PingSqlsvr

  Monitor: PingSqlsvr
  Executes: <CODEDIR>/bin/ping_server.pl -TYPE=SQLSVR

The Ping* scripts ping your database servers and save the results in the alarms database

PingSybase

  Monitor: PingSybase
  Executes: <CODEDIR>/bin/ping_server.pl -TYPE=SYBASE

The Ping* scripts ping your database servers and save the results in the alarms database

PingSystems

  Monitor: PingSystems
  Executes: <CODEDIR>/bin/ping_systems.pl -TYPE=SYBASE

The Ping* scripts ping your database servers and save the results in the alarms database

PortMonitor

  Monitor: PortMonitor
  Executes: <CODEDIR>/monitoring/port_monitor.pl --LOGALARM

The port monitor uses the file conf/port_monitor.dat for data. It does a "ping" on ports of your servers for a response. To initialize this file from known port data, run

   PortMonitor.ksh --GENERATE

Additionally, the first time you run PortMonitor.ksh, it will generate the file based on your sybase interfaces file and GEM configuration data. Since your interface file may contain junk servers, it will comment out servers that are not up at the time of the run by placing a # in front of the line.

You should add other servers/services to the config file. The initialization only monitors databases, but the pgogram can monitor everything that is based on sockets. Note windows NT services are allready handled.

To remove a service that generates a warning or error, simply comment it out or remove it from the configuraiton file.

PcHostsRpt

  Monitor: PcHostsRpt
  Executes: <CODEDIR>/bin/parse_hosts.pl -BPcHostsRpt -h -O<DATADIR>/html_output/Win32_Hosts_Report.html
     -D<DATADIR>/system_information_data/hosts

SpaceMonitorMssql

  Monitor: SpaceMonitorMssql
  Executes: <CODEDIR>/server_documenter/space_monitor.pl -AR -Tsqlsvr

SybSpaceMonitor

  Monitor: SpaceMonitorMssql
  Executes: <CODEDIR>/server_documenter/space_monitor.pl -AR -Tsybase

TraceRoute

  Monitor: TraceRoute
  Executes: <CODEDIR>/ADMIN_SCRIPTS\monitoring\tracert.pl -BATCH_ID=TraceRoute

SybRepMonAgent

   Executes: <CODEDIR>/ADMIN_SCRIPTS/monitoring/SybaseRepMonitor.pl --ACTION=MONITOR
     --ITERATIONS=5 --FREQUENCY=30
     --OUTFILE=<DATADIR>/html_output/SybRepMonAgent.html $*

SybRepMonReport

   Executes: <CODEDIR>/ADMIN_SCRIPTS/monitoring/SybaseRepMonitor.pl --ACTION=REPORT
     --HTML --OUTFILE=<DATADIR>/html_output/SybRepMonReport.html $*

MssqlCycleEventLogs

   Cycles the sql server event logs

MssqlShrinklogs

   Shrinks the sql server log files on all your servers

Log File Monitors

These files will collect error logs for your servers. There are multiple reports for each log type. These reports are a Configuration Report (what it should do), a Fetch Report (how the file fetch went), a Report (usually giving 48 hours of log messages), and a SaveAlarms report, which uses a patern file and saves recent alarms into the alarm database. The SaveAlarms report will ensure that duplicates are not stored.

SybErrLogAllRpt

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybErrLogAllRpt -TASE_SERVER -hrtF
     -O<DATADIR>/html_output/GemRpt_SybErrLogAllRpt.html $*

SybErrLogConfig

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybErrLogConfig -TASE_SERVER -hm
     -O<DATADIR>/html_output/GemRpt_SybErrLogConfig.html $*

SybErrLogFetch

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybErrLogFetch -TASE_SERVER -hf
     -O<DATADIR>/html_output/GemRpt_SybErrLogFetch.html $*

SybErrLogRpt

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybErrLogRpt -TASE_SERVER -hrt
     -O<DATADIR>/html_output/GemRpt_SybErrLogRpt.html $*

SybErrLogSaveAlarms

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybErrLogSaveAlarms -TASE_SERVER -hrpA
     -O<DATADIR>/html_output/GemRpt_SybErrLogSaveAlarms.html $*

SybHistSvrConfig

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybHistSvrConfig -THISTORICAL_SERVER -hm
     -O<DATADIR>/html_output/GemRpt_SybHistSvrConfig.html $*

SybHistSvrLogFetch

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybHistSvrLogFetch -THISTORICAL_SERVER -hf
     -O<DATADIR>/html_output/GemRpt_SybHistSvrLogFetch.html $*

SybHistSvrLogRpt

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybHistSvrLogRpt -THISTORICAL_SERVER -htr
     -O<DATADIR>/html_output/GemRpt_SybHistSvrLogRpt.html $*

SybMonSvrConfig

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybMonSvrConfig -TMONITOR_SERVER -hm
     -O<DATADIR>/html_output/GemRpt_SybMonSvrConfig.html $*

SybMonSvrLogFetch

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybMonSvrLogFetch -TMONITOR_SERVER -hf
     -O<DATADIR>/html_output/GemRpt_SybMonSvrLogFetch.html $*

SybMonSvrLogRpt

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybMonSvrLogRpt -TMONITOR_SERVER -htr
     -O<DATADIR>/html_output/GemRpt_SybMonSvrLogRpt.html $*

SybRepSvrConfig

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybRepSvrConfig -TREP_SERVER -hm
     -O<DATADIR>/html_output/GemRpt_SybRepSvrConfig.html $*

SybRepSvrLogFetch

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybRepSvrLogFetch -TREP_SERVER -hf
     -O<DATADIR>/html_output/GemRpt_SybRepSvrLogFetch.html $*

SybRepSvrLogRpt

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybRepSvrLogRpt -TREP_SERVER -htr
     -O<DATADIR>/html_output/GemRpt_SybRepSvrLogRpt.html $*

SybRepSvrSaveAlarms

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybRepSvrSaveAlarms -TREP_SERVER -hrpA
     -O<DATADIR>/html_output/GemRpt_SybRepSvrAlarm.html $*

SybaseBackupFileReport

   Executes: <CODEDIR>/ADMIN_SCRIPTS/console/gem_file_manager.pl -BSybaseBackupFileReport -TBACKUP_SERVER -hm
     -O<DATADIR>/html_output/SybaseBackupFileReport.html $*

OTHER

  check_sybase_repserver.pl     SybRepServerChecker
  documenter report 14          UnixDiskspace

THE WIN32 BACKUP CROSSCHECK PROCESS

Process

The win32 backup crosscheck process is an extensive monitor within GEM that determines whether your SQL Server databases are appropriately backed up. This process consists of the following batch processes

Batch Jobs Involved

MssqlBackupCrosscheck MssqlBackupCrosscheckWeekly UpdateBackupStateStaticInfo

What Happens

Event log records are read on all your sql servers and stored in a table in the gemalarms database named BackupState. The MssqlBackupCrosscheckWeekly batch reads 1 full week of records while the MssqlBackupCrosscheck just reads a few hours worth of records. This table contains the last time any backup events happened - the last time the db was dumped, loaded, tran dumped etc... The table can, of course be non-representative of what your actual environment is. For example, you could drop a database named AA. Your event logs would show a backup of AA from last week - which would trigger an alarm because the db has not been backed up in a week! The UpdateBackupStateStaticInfo should run nightly to remove (actually mark deleted) information that is not current.

The final part of this process is the crosscheck reporting. The --REPORT arg (normally set in the batches) will produce reports and alarm based on the info collected in the table. The reporting process uses crosscheck.dat config file to determine thresholds for what and how to alarm.

Diagnostics Procedure

MssqlBackupCrosscheck.ksh --DEBUG or --DIAGNOSTICS select * from BackupState perl Backup_Crosscheck_Report.pl --SQLSVRONLY --TRANHOURS=2 --NOALARM # use noalarm to avoid a flood of emails

CONSOLE ERROR GUIDE

What to do whith console errors

PcServiceChecker: Service XXX state=SERVICE STOPPED
This implies that a service on windows is down. You should either correct the problem or ignore it. To ignore the problem, edit your conf/pc_service.dat file and remove the line associated with the service. The format is simple. It is SERVER:SERVICE_DESCRIPTION:INTERNAL_KEY. Just find the line and delete it. That service will no longer be monitored... You will then need to use the user interface and delete the offending error message so you do not continue to see it.
MAIL SUBJECT: E.BROKERSQL1 Surveyor Service Surveyor state=SERVICE STOPPED
DETAILS: system=BROKERSQL1 subsystem=Surveyor state=ERROR Service Surveyor state=SERVICE STOPPED
FIX: Restart Service or edit pc_service.dat and delete the message from the error handler
SybThresholdManager: Thresholds show peak usage=95 at Sep 11 2007 10:21AM
Threshold tables are in sybsystemprocs - table gem_threshold_hist4
just delete the appropriate rows (truncating the table should work)
Data Space at 97.72% (override using threshold_overrides.dat)
The config file threshold_overrides.dat
XXX ProcLib Outdated Stored Procedure Library
cd .../gem/ADMIN_SCRIPTS/procs configure.pl -USER=sa -SERVER=xxx -PASSWORD=pass

This output is documentation for the SQL Technologies A Guide To GEM Batch Jobs.

The Complete GEM Documentation	Slide Show: What Is GEM	Download GEM

An Introduction To GEM	How To Install	Getting Started Guide
Batch Jobs Overview	GEM Screenshots	Server Backup and Maintenance
Utilities Guide	The GEM Console	New System Stored Procedures
FAQ and Change Log	Common Perl Libraries	Pricing & Right To Use

Scripts In The 'plan' Subdirectories
Job Name	Win32 or Unix	Description	Recommended Frequency
Backup_<PLANNAME>	Win32	Main Nightly PLANNAME Backup Run (and possibly transfer & load them)	daily
FixLogship_<PLANNAME>	Win32	Guarantees Transfer of logshiped files for plan PLANNAME	never
LoadTranlogs_<PLANNAME>	Win32	Loads Transfered but Unapplied Transaction Logs	never
Trandump_<PLANNAME>	Win32	Run a Transaction Log Dump (and possibly transfer & load them)	Hourly