[ Back | Print ]
Technical Information Document
Obituary Troubleshooting in eDirectory - TID10064117 (last modified 20SEP2004)
printer friendly tell a friend
Click here if this does not solve your problem
10064117 10064117 10064117
goal

Obituary Troubleshooting in eDirectory

fact

Novell Directory Services 6

Novell Directory Services 7

Novell Directory Services 8

Novell eDirectory 8.5 (85.xx)

Novell eDirectory 8.6 for All Platforms

symptom

Obituaries are created when an eDirectory object is DELETED, MOVED, RENAMED or RESTORED. The obituary itself is an attribute of the object on which the change took place. The reason an obituary attribute is created is to maintain database consistency between real copies of the object and external references of the changed object. An external reference is a lightweight/scaled down copy of the real object, used as a referencing placeholder. In order to achieve this various attributes relating to the obituary process are maintained, these are BACKLINK, USED_BY, NOTIFY and REFERENCE. These attributes are maintained using the Backlinker, Janitor and Distributed Reference Links (DRL).

Obituaries are reported in eDirectory by various utilities and tools. Sometimes they are only noticed when they prevent other tasks from being performed, such as partition operations. Although there may be many tools available and many to come in future, the process and information knowledge required for troubleshooting them is basically the same for all eDirectory and NDS versions.

This document provides the steps, guidelines and information you need to troubleshoot obituaries in NDS.

**  Reports Required **

   iMONITOR: Obituary Listing Report from several servers within the tree
   DSREPAIR: Check External Reference reports as possible from servers in the tree
   DSREPAIR: Time Synchronization report from a server holding a replica of the [ROOT] partition
   DSDIAG: List Replica Rings report from one server
   DSDIAG: Compare Replica Rings report from one server

** Troubleshooting Process **

    - Check the DS module version.
    - Check that Time is synchronized.
    - Is replica synchronization synchronizing without errors?
    - Check that servers in the backlink list are up and working.
    - Are there backlink values without Primary Obituaries?
    - Check for known bugs.
    - What is the Purge Vector's time set to?
    - Are Move & Inhibit_move obituaries consistent? Does each Move have an Inhibit_move?
    - Are log files missing for some servers?
    - Are there obituaries stuck that don't exist on the Master replica?
    - Are obituaries stuck that are on the Master?
     - Are there obituaries on External Reference objects?

fix

** Possible Fix for Issues **

* DS Module Version
There are known issues with some version of DS. The DS versions involved provide an idea of the types of problems that could be expected.

* Backlinked Servers
Ideally there should be no servers reporting communication errors. If there are, resolve these errors.

Check that servers have not been removed from the network, which still have server objects appearing in the NDS. Resolve issues with removed servers as the first step. Typically you would see errors: -625, -626 Communication or -630 Different Tree.

* Time Synchronization
Newer versions of DS have become more sensitive to time synchronization issues. Obituaries can not be processed when the time is out of sync,   because DS will not synchronize. When DS does not synchronize all replicas will not receive the obituary attribute updates. This could be a possible   cause of replicas reporting different obituary flag values. Resolve the time sync errors as the second step.

* Replica Synchronization
Sync errors will prevent obituary attribute changes from synchronizing as well as preventing the obituary process advancing the flag status. Resolve the sync errors as the third step.

* Backlink values without Primary Obituaries
This can be resolved by running DSREPAIR: Local Database Repair with the default options.

* Known bugs and Issues
When objects have two Primary obituaries, they don't get processed. Be aware of other issues, by looking for TIDs.

* Purge Vector or Obituary Vector (8.6.0 or newer versions of eDirectory)
This is the most overlooked aspect of obituary processing. Obituaries will only be purged per partition, if the MTS (Modification Time Stamp) of the PURGEABLE obituary is OLDER then the timestamp value recorded for the Purge Vector attribute, for each server replica number used in the TV (Transitive Vector). The Purge Vector will only be updated if the partition has a successful sync cycle. The purpose of this attribute is to prevent obituaries from being purged before all replicas have been notified about it. The values in the attribute are recorded per replica number, therefore per server in the replica ring. DSREPAIR checks and fixes the Purge Vector values.

NOTE: While the Purge Vector is actually stored as an attribute on the partition root object the Obituary Vector is not.  It is stored in memory and is recalculated each time that the Janitor Process runs.  To see what the Obituary Vector actually is you can turn on the Janitor filter in DSTrace.

* Move & Inhibit_move
Does each Move have an Inhibit_move? The Move has a reference to the Inhibit_move, but the Inhibit_move does not point back to the Move.

In order to check for consistency, verify that each Move obituary points to a valid Inhibit_move obituary. The Move obituary is where the source object existed and the Inhibit_move is where the object was moved to.

When all Move obituaries have been checked, then the left over Inhibit_move obituaries could be inconsistent. They may not be as they could exist on Externally Referenced objects.

* Missing log files
This impacts the results for apparently inconsistent Move/Inhibit_move obituaries. If you don't have all the logs, how do you know that an Inhibit_move does not have a Move?

NOTE: The Move has a reference to the Inhibit_move, but the Inhibit_move does not have one back to the Move.

* Obituaries DON'T exist on the Master
Either timestamp these obituaries or make this server the Master of the partition.Caution!! Never make a subordinate Reference the Master. As the subordinate reference does not contain REAL objects, You will LOSE all the objects in that partition. This can be achieved by using a version of DSREPAIR that supports the -OT switch. To timestamp the obituaries, load DSREPAIR -OT and Repair Local DS Database with the default options.  Make sure the version of DSREPAIR you intend to use does not introduce problems. Check the readme file for newer DSREPAIR versions or field test readme files.  For Linux or Unix use iMonitor, go to the repair option (example https://10.0.0.11:8009/nds/dsrepair) click on the advanced button and place the -OT switch there.  Then run the repair.

* Obituaries DO exist on the Master
Are the servers in the backlink list available and communicating properly? If not, resolve this. If they are, then either timestamp these obituaries or make this server the Master of the partition. This can be achieved by using a version of DSREPAIR that supports the -OT switch. To timestamp the obituaries, load DSREPAIR -OT and Repair Local DS Database with the default options.  It is possible for obituaries on the server holding the Master of the partition not to process after all this.

If every thing else has been checked and is correct, try unloading DS.NLM or running DSREPAIR: Local Database Repair with the default options.

* Obituaries stuck on External Reference objects
Objects in the External Reference partition reporting obituaries can only be cleaned up by:

   Using DSREPAIR -xk3
OR
   Place a replica of those objects on the server and removing it

Either of these step should only be followed once all the other steps have been performed and it has been concluded that these obituaries are stuck to the point that they will never process or are inconsistent. By inconsistent it is meant that there is no obituary attribute for this obituary type, on any real copy of the object. A real copy of an object is an object which is located in a Master, R/W or R/O replica on the server.

Sometimes the use of the -xk3 switch can cause the file system trustees to be lost. This does not often occur, but as a precaution it is best to take a trustee backup using TBACKUP. If the trustees are lost, printing would also be affected.

NOTE: Before making any major changes to eDirectory, like using DSREPAIR ·XK switches, ensure you have a DIB backup of the DS and file system trustee backup on that server. It sounds like a lot of work, but recovery is a lot harder.

** eDirectory Report Collection **

* iMONITOR: Obituary Listing Report
To collect all obituaries on a specific server you can run this.  This is the easiest way to troubleshoot obituaries, as once you have the list of entry records that have obituaries on them you can then go look at the obituary values.  From there you can determine which server is holding up the processing of the obituaries and troubleshoot the health of the agent without ever leaving your browser.

* DSREPAIR: Check External Reference
To collect all the obituaries on a server use DSREPAIR -A. Without the -A switch the Move/Inhibit_move obituaries will not be reported. In DS 7, 8 & 85.xx the process actually attempts to contact other servers to check the external references. This is a lot slower than with DS 6.

The collection can be automated using ONSITE ADMIN and Stuffkey. There are a number of scripts available for this. The tool can be downloaded from http://support.novell.com

* DSDIAG: List Replica Rings
This report is run from only one server in the tree. Use the default DSDIAG settings.

This report provides information about all the partitions in the tree as well as which servers are in the replica ring for each partition.

* DSDIAG: Compare Replica Rings
This report is run from only one server in the tree. Use the .default DSDIAG settings.

This report will indicate which partitions could be experiencing synchronization issues. Communications issues in synchronization are a guaranteed stopper of obituary processing.  

* DSDIAG: Background Process Check
This report is run from only one server in the tree. Use the default DSDIAG settings.

To generate this report, DSDIAG must authenticate to the eDirectory tree. This is done by using the Manage Identities option under the Preferences menu. If DSDIAG is not authenticated, errors such as -672 will be reported.
This report collects the same data as SET DSTRACE=*ST. Status information is displayed for, LIMBER, SCHEMA, BACKLINKER and OBITUARIES.

When this DSDIAG report is created, the above status information is collected.

Not all obituaries are shown in this report. Only obituaries that are located on a server that holds the Master partition and which DS is attempting to process. All other obituaries can only be reported using DSREPAIR or iMonitor.

* DSREPAIR: Time Synchronization
Collect this report from then main menu in DSREPAIR. Run the report on a server holding the ROOT partition.

This report provides information about all servers in the tree, DS version, Time Sync status and communications issues.

* DSREPAIR: Display Replica Information
There are scripts that collect the DSREPAIR: Display Replica Information report. Although there are some tools that use this report, this menu option is not available in DS 8 & 85.xx.

* DSBROWSE: Object search
Using the Object Search menu option in DSBROWSE it is possible to quickly report on all obituaries. But, this information is only available on the screen. There is no option to write the results to a file using DSBROWSE. Using DSBROWSE is a lot quicker than other tools, but it is server centric.

++ Note when using DSDIAG:
* Increase the IPX sockets if this server uses IPX, 4000-6000, to avoid communications errors in large trees:
       LOAD SPXCONFG Q=1 I=4000

* When generating reports on NetWare 5 servers, the search context should be changed to [ROOT], to avoid the -708 error. If the report is being used for    DSDESIGNER, then remove the [ROOT] from the report after it has been created.

* When scripting reports on NetWare 5 using Stuffkey, on some menu options the second option from the top is highlighted by default, instead of the top one..

Document Title: Obituary Troubleshooting in eDirectory
Document ID: 10064117
Solution ID: NOVL55452
Creation Date: 10AUG2001
Modified Date: 20SEP2004
Novell Product Class: Management Products
NetWare
Novell eDirectory

Disclaimer

The Origin of this information may be internal or external to Novell. Novell makes all reasonable efforts to verify this information. However, the information provided in this document is for your information only. Novell makes no explicit or implied claims to the validity of this information.

Any trademarks referenced in this document are the property of their respective owners. Consult your product manuals for complete trademark information.


[ Back | Print ]