MasterIndex Developer Documentation¶

MasterIndex Developer Documentation
Overview
Internal Helper Components
Core Workflow
Constructor
API Reference
Usage Examples
Integration with Database Class
Error Types
Best Practices

Overview¶

The MasterIndex class manages cross-instance coordination for GAS DB using ScriptProperties. It provides virtual locking, conflict detection, and collection metadata management. Following the Section 4 refactoring, MasterIndex serves as the primary source of truth for collection metadata, with the Database class delegating all collection management operations to it.

Key Responsibilities:

Cross-instance coordination via ScriptProperties
Virtual locking for collection access
Conflict detection using modification tokens
Collection metadata management (primary source of truth)
Integration with Database class for collection lifecycle management

Storage: ScriptProperties with key GASDB_MASTER_INDEX

Integration with Database Class: The Database class delegates collection operations to MasterIndex:

Collection creation, access, and deletion
Collection listing and metadata retrieval
Backup synchronisation to Drive-based index files

Internal Helper Components¶

MasterIndexMetadataNormaliser¶

Location: src/04_core/MasterIndex/01_MasterIndexMetadataNormaliser.js

Encapsulates the transformation of incoming metadata into CollectionMetadata instances. The normaliser clamps timestamps, clones lock status payloads, and ensures modification tokens are generated when missing. This keeps _addCollectionInternal and bulk operations lean while guaranteeing consistent metadata regardless of input type (plain object or existing CollectionMetadata).

MasterIndexLockManager Helper Methods ⭐ NEW in v0.0.5¶

Location: src/04_core/MasterIndex/02_MasterIndexLockManager.js

Added in: MI2 refactoring

`_setAndPersistLockStatus(collectionName, collection, lockStatus)`¶

Centralized helper for setting and persisting lock status with guaranteed ordering.

Parameters:
collectionName (String): Name of collection to update
collection (CollectionMetadata): Collection metadata object
lockStatus (Object|null): Lock status to apply
Behaviour:
Sets lock status on collection metadata
Persists to MasterIndex via _updateCollectionMetadataInternal()
Usage: Used by acquireCollectionLock(), renewCollectionLock(), releaseCollectionLock(), cleanupExpiredLocks()
Benefits: Single source of truth for lock persistence, guaranteed update ordering

MasterIndexConflictResolver Helper Methods ⭐ NEW in v0.0.5¶

Location: src/04_core/MasterIndex/04_MasterIndexConflictResolver.js

Added in: MI1 refactoring

`_applyMetadataUpdates(collectionMetadata, updates)`¶

Centralized helper for applying metadata field updates during conflict resolution.

Parameters:
collectionMetadata (CollectionMetadata): Metadata object to update
updates (Object): Map of field names to values
Behaviour:
Iterates through update keys
Applies known fields (documentCount, lockStatus) via setter methods
Ignores unknown fields
Usage: Used by _applyLastWriteWins() for conflict resolution
Benefits: Single source of truth for metadata updates, consistent update semantics, easier to extend

Core Workflow¶

Collection Access Protocol¶

Database class updates to an existing collection follow this protocol when delegating to MasterIndex:

// 1. Database retrieves the existing collection metadata from MasterIndex
// 2. Acquire virtual lock for thread safety
const acquired = masterIndex.acquireCollectionLock('users', operationId);
if (!acquired) {
  throw new Error('Collection is already locked');
}

// 3. Check for conflicts (if updating existing collection)
const hasConflict = masterIndex.hasConflict('users', expectedToken);

// 4. Perform operations (Database coordinates with Drive operations)
// 5. Renew the lease if the write is close to expiry
masterIndex.renewCollectionLock('users', operationId, 45000);

// 6. Apply the metadata updates produced by the write
masterIndex.updateCollectionMetadata('users', updates);

// 7. Release lock
masterIndex.releaseCollectionLock('users', operationId);

Virtual Locking¶

Prevents concurrent modifications across script instances:

Locks expire automatically (default: 30 seconds)
Operation ID required for lock acquisition/release
Active locks may be renewed by the owning operation before final metadata persistence
Expired locks may be explicitly removed by calling cleanupExpiredLocks()
All ScriptLock-protected mutation paths reload the latest ScriptProperties snapshot after acquiring the lock, so read-modify-write operations do not act on stale in-memory metadata

Lock-held snapshot refresh¶

MasterIndex._withScriptLock() now reloads the latest ScriptProperties payload while the ScriptLock is held before any mutation logic reads collection metadata. This means separate MasterIndex instances pointing at the same masterIndexKey observe the newest collection and lock state before they:

add or remove collections
update collection metadata
acquire or release collection locks
clean up expired locks
resolve conflicts that persist metadata back to the index

This closes the stale-instance window where one execution could acquire the ScriptLock yet still overwrite a newer lock or collection update using outdated in-memory state.

Data Structure¶

{
  version: Number,
  lastUpdated: String,
  collections: {
    [collectionName]: {
      name: String,
      fileId: String | null,
      created: String,
      lastUpdated: String,
      documentCount: Number,
      modificationToken: String,
      lockStatus: null | {
        isLocked: Boolean,
        lockedBy: String | null,
        lockedAt: Number | null,
        lockTimeout: Number | null
      }
    }
  }
}

lockStatus may be null, an active lock object, or a persisted unlocked object with isLocked: false and null-valued lock fields immediately after an explicit release.

Constructor¶

class MasterIndex {
  constructor(config = {}) {
    // ...
  }
}

Parameters:

config.masterIndexKey (String): ScriptProperties key (default: 'GASDB_MASTER_INDEX')
config.lockTimeout (Number): Lock timeout in ms (default: 30000)
config.version (Number): Master index version (default: 1)

Behaviour: Creates configuration, initialises data structure, loads from ScriptProperties if available.

API Reference¶

Core Methods¶

`addCollection(name, metadata)`¶

Adds collection to master index. Called by Database.createCollection() during collection creation.

name (String): Collection name
metadata (Object): Collection metadata (fileId, documentCount, etc.)
Returns: Collection data object
Throws: CONFIGURATION_ERROR for invalid name
Database Integration: Primary method used by Database class to register new collections

`getCollection(name)` / `getCollections()`¶

Retrieves collection metadata. Used by Database.getCollection() and Database.listCollections() to access collection information.

Returns: Collection object or collections map

`acquireCollectionLock(collectionName, operationId, timeout)` / `renewCollectionLock(collectionName, operationId, timeout)` / `releaseCollectionLock(collectionName, operationId)`¶

Coordinates collection-level virtual locks.

acquireCollectionLock(...): claims a free or expired lock for an operation
renewCollectionLock(...): refreshes an active lock owned by the same operation before it expires
releaseCollectionLock(...): clears the lock for the owning operation
Database Integration: Called by Database class methods when coordinating writes to collections that are already registered in the master index

`updateCollectionMetadata(name, updates)`¶

Updates collection metadata with new modification token.

updates (Object): Metadata changes
Returns: Updated collection data
Concurrency behaviour: Reloads the latest ScriptProperties snapshot while holding ScriptLock before applying updates, so stale instances preserve newer lock state and metadata written by other executions

`removeCollection(name)`¶

Removes a collection from the master index. Called by Database.dropCollection() during collection deletion.

name (String): Collection name to remove
Returns: Boolean - true if the collection was removed, false otherwise
Database Integration: Used by Database class to remove collections from the primary metadata store
Concurrency behaviour: Refreshes from ScriptProperties under ScriptLock first, so an older instance can still remove a collection added by a newer instance instead of missing it in stale memory

`getCollections()`¶

Retrieves all collections in the master index.

Returns: Object - map of collection metadata

Locking Methods¶

`acquireCollectionLock(collectionName, operationId, timeout)`¶

Acquires virtual lock for collection. Used by Database class before performing collection operations.

Parameters: timeout is optional and defaults to the configured lock lease duration
Returns: true if successful, false if already locked
Throws: COLLECTION_NOT_FOUND if the collection is not yet registered in the master index
Database Integration: Called by Database methods during coordinated updates and deletion of already-registered collections
Concurrency behaviour: Evaluates lock ownership from a freshly reloaded snapshot while ScriptLock is held, preventing stale instances from stealing a newer lock

`renewCollectionLock(collectionName, operationId, timeout)`¶

Renews an active virtual lock for a collection owned by the same operation.

Parameters: timeout is optional and defaults to the configured lock lease duration
Returns: true if the active lock was renewed, false if it was missing, expired, or owned by another operation
Database Integration: Used by CollectionCoordinator to extend a near-expiry lease before final metadata persistence
Concurrency behaviour: Reloads the latest lock state under ScriptLock before renewing, so stale instances cannot extend a newer lock they do not own

`releaseCollectionLock(collectionName, operationId)`¶

Releases virtual lock (must match operation ID).

Returns: true when the lock is released, when no lock is held, or when the collection no longer exists; false if another operation owns the active lock
Concurrency behaviour: Reloads the latest lock state under ScriptLock first, allowing a stale instance to release the current lock correctly when it still owns it

`isCollectionLocked(collectionName)`¶

Checks if collection is currently locked.

Remarks: Reads the current in-memory snapshot; call load() first when the caller needs the latest persisted lock state

`cleanupExpiredLocks()`¶

Removes expired locks.

Concurrency behaviour: Reloads current collection metadata under ScriptLock before scanning, so cleanup removes genuinely expired locks without clearing newer active locks written by another instance

Conflict Management¶

`hasConflict(collectionName, expectedToken)`¶

Checks if collection was modified since token was generated.

`resolveConflict(collectionName, newData, strategy)`¶

Resolves conflicts using specified strategy ('LAST_WRITE_WINS').

`generateModificationToken()`¶

Creates unique modification token.

`validateModificationToken(token)`¶

Validates token format (timestamp-randomstring).

Usage Examples¶

Basic Operations¶

// Typically called via Database class, not directly
const masterIndex = new MasterIndex();

// Database.createCollection() triggers this workflow:
const collection = masterIndex.addCollection('users', {
  fileId: 'abc123',
  documentCount: 0
});

// Database.getCollection() and Database.listCollections() use:
const users = masterIndex.getCollection('users');

// Collection operations trigger metadata updates:
masterIndex.updateCollectionMetadata('users', {
  documentCount: 5
});

Locking Pattern¶

const operationId = 'op_' + Date.now();

if (masterIndex.acquireCollectionLock('users', operationId)) {
  try {
    // Perform operations
    masterIndex.updateCollectionMetadata('users', updates);
  } finally {
    masterIndex.releaseCollectionLock('users', operationId);
  }
}

Conflict Resolution¶

const expectedToken = 'previously-read-token';

if (masterIndex.hasConflict('users', expectedToken)) {
  const resolution = masterIndex.resolveConflict('users', newData, 'LAST_WRITE_WINS');
} else {
  masterIndex.updateCollectionMetadata('users', newData);
}

Integration with Database Class¶

The MasterIndex serves as the primary source of truth for collection metadata, working closely with the Database class in the following ways:

Collection Lifecycle Integration¶

Creation Flow:

Database.createCollection() validates collection name
Database creates Drive file for collection data
Database calls MasterIndex.addCollection() to register metadata
Database updates Drive-based index file as backup
Database caches collection object in memory

Access Flow:

Database.getCollection() checks in-memory cache first
If not cached, Database calls MasterIndex.getCollection()
If not in MasterIndex, Database falls back to Drive index file
Auto-creation triggers if enabled and collection doesn't exist

Deletion Flow:

Database.dropCollection() removes from memory cache
Database calls MasterIndex.removeCollection() to update metadata
Database deletes Drive file and updates index file

Data Synchronisation¶

The Database class maintains consistency between MasterIndex and Drive-based storage:

Primary Operations: MasterIndex handles all metadata operations
Backup Operations: Database synchronises to Drive index file via backupIndexToDrive()
Recovery Operations: Database can restore from Drive index to MasterIndex if needed

Locking Coordination¶

Database class uses MasterIndex locking for thread safety:

// Example from a coordinated update to an existing collection
const operationId = 'update_' + Date.now();
if (this._masterIndex.acquireCollectionLock(name, operationId)) {
  try {
    // Perform Drive operations for the existing collection
    this._fileService.updateFile(fileId, updatedData);
    // Update MasterIndex metadata for the existing collection
    this._masterIndex.updateCollectionMetadata(name, {
      documentCount: updatedDocumentCount,
      modificationToken: nextModificationToken
    });
  } finally {
    this._masterIndex.releaseCollectionLock(name, operationId);
  }
}

Error Types¶

CONFIGURATION_ERROR: Invalid parameters
COLLECTION_NOT_FOUND: Collection doesn't exist
LOCK_TIMEOUT: Failed to acquire ScriptLock
MASTER_INDEX_ERROR: General operation failures

Best Practices¶

Always release locks in finally blocks
Use appropriate timeouts for lock operations
Handle lock acquisition failures with retry logic
Validate modification tokens before updates
Clean up expired locks periodically