Overview
Each connector runs as an independent service (a Docker container in self-hosted deployments, or an ECS task on AWS) and syncs data independently. The connector-manager service orchestrates all connector operations, including scheduling syncs, tracking progress, and managing concurrent operations.
Connector Manager
The connector-manager service is the central orchestrator for all connector operations. It provides:
| Capability | Description |
|---|
| Sync Orchestration | Triggers syncs on-demand or on schedule |
| Progress Tracking | Real-time sync progress via Server-Sent Events |
| Concurrency Control | Limits concurrent syncs (default: 10 global, 3 per type) |
| Stale Sync Detection | Marks hung syncs as failed after 10 minutes |
| SDK Endpoints | API for custom connectors built with Python/TypeScript SDK |
Configuration
Key connector-manager settings (via environment variables):
| Variable | Default | Description |
|---|
MAX_CONCURRENT_SYNCS | 10 | Maximum concurrent syncs across all sources |
MAX_CONCURRENT_SYNCS_PER_TYPE | 3 | Maximum concurrent syncs per connector type |
SCHEDULER_INTERVAL_SECONDS | 30 | How often scheduler checks for due syncs |
STALE_SYNC_TIMEOUT_MINUTES | 10 | Timeout to mark a sync as stale/failed |
Viewing Connector Status
Navigate to Settings → Integrations to see all configured connectors.
For each connector, you can see:
- Enabled/Disabled status
- Last sync timestamp
- Sync progress when a sync is running (including document count)
Enabling and Disabling Connectors
Each connector can be enabled or disabled independently. A disabled connector:
- Stops syncing new content
- Retains previously indexed documents
- Can be re-enabled at any time
To toggle a connector’s status, click Configure and change the enabled setting.
Sync Operations
Automatic Sync
Connectors automatically sync on a schedule. After the initial full sync, incremental syncs run periodically to capture new and updated content.
Manual Sync
To trigger a manual sync:
- Navigate to Settings → Integrations
- Select the connector
- Click Sync Now
You can only trigger a manual sync if no sync is currently running. If a sync is in progress, the button will be disabled.
Removing a Connector
To disconnect a data source:
- Navigate to Settings → Integrations
- Select the connector
- Click Remove
- Confirm the removal
When you remove a connector:
- The connector service is stopped
- All indexed documents from that source are deleted
- Users will no longer see results from that source
Troubleshooting
Connector shows “Error” status
Check the connector logs. For Docker Compose deployments:
# Check specific connector logs
docker logs omni-<connector-name>-connector
# Check connector-manager logs for orchestration issues
docker logs omni-connector-manager
- Expired or revoked credentials
- API rate limiting
- Network connectivity problems
- Connector-manager unable to reach connector service
Sync seems stuck
If a sync has been running for an unusually long time:
- Check the connector-manager logs for heartbeat failures:
docker logs omni-connector-manager | grep -i "stale\|timeout"
- Check the connector logs for errors
- Syncs are marked as failed after 10 minutes of inactivity (configurable via
STALE_SYNC_TIMEOUT_MINUTES)
- Restart the connector service if needed
Connector not receiving sync requests
Verify connector-manager can reach the connector:
- Check connector URL configuration:
# Ensure CONNECTOR_<TYPE>_URL is set correctly
echo $CONNECTOR_GOOGLE_URL
- Test connectivity from connector-manager:
docker exec omni-connector-manager curl http://<connector>:<port>/health
- Check connector-manager logs for connection errors
Too many concurrent syncs
If syncs are queuing or timing out:
- Check current sync count in connector-manager logs
- Adjust concurrency limits:
MAX_CONCURRENT_SYNCS (default: 10)MAX_CONCURRENT_SYNCS_PER_TYPE (default: 3)
- Consider staggering sync schedules for different sources
