> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getomni.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Filesystem

> Index local files and directories for unified search across your file system

The Filesystem connector enables Omni to index and search content from local directories, making your organization's file repositories searchable alongside other data sources.

## Overview

### What Gets Indexed

| Content Type   | Description                                                                |
| -------------- | -------------------------------------------------------------------------- |
| **Text files** | Full content extraction for text-based files (`.txt`, `.md`, `.csv`, etc.) |
| **Documents**  | Office documents (`.docx`), PDFs, and other structured text                |
| **Metadata**   | File name, path, size, timestamps for all files                            |

<Note>
  Source code files (`.py`, `.js`, `.ts`, `.rs`, `.go`, `.java`, etc.) and config files (`.json`, `.yaml`, `.toml`, `.xml`, `.html`, `.css`, etc.) are **excluded by default**. The filesystem connector is designed for indexing documents and knowledge content, not code repositories.
</Note>

### Supported File Formats

**Full content indexing** (text extracted and searchable):

* Plain text files (`.txt`, `.md`, `.rst`, `.csv`)
* Office documents (`.docx`)
* Other text-based formats

**Excluded by default:**

* Source code files
* Config and markup files

**Metadata-only indexing** (searchable by name, path, date):

* Binary files (images, etc.)
* Files larger than 10MB

### How It Works

1. The connector scans configured directories recursively
2. Text files are read and content is extracted for full-text search
3. File metadata (name, path, size, timestamps) is indexed for all files
4. A file watcher detects changes in real-time between full scans

<Note>
  The connector uses read-only access. Omni cannot modify or delete any files in your filesystem.
</Note>

## Prerequisites

Before setting up the Filesystem connector, ensure you have:

* **Omni deployment** with the Filesystem connector service running
* **Directory access** to the files you want to index
* **Docker volume mounting** configured (for containerized deployments)

## Setup

### Step 1: Configure Docker Volume Mounts

For Docker Compose deployments, add volume mounts to the Filesystem connector service in your `docker-compose.override.yml`:

```yaml theme={null}
services:
  filesystem-connector:
    volumes:
      - /path/to/your/documents:/data/documents:ro
      - /path/to/another/folder:/data/shared:ro
```

<Warning>
  Always use read-only mounts (`:ro`) for security. The connector only needs read access to index files.
</Warning>

### Step 2: Add Environment Variables

Add the following to your `.env` file:

```bash theme={null}
FILESYSTEM_CONNECTOR_PORT=4013
```

### Step 3: Connect in Omni

1. Navigate to **Settings** → **Integrations** in Omni
2. Find **Filesystem** and click **Connect**
3. Configure the source settings:

| Setting                 | Required | Default    | Description                                                        |
| ----------------------- | -------- | ---------- | ------------------------------------------------------------------ |
| `base_path`             | Yes      | -          | Root directory to scan (inside container, e.g., `/data/documents`) |
| `scan_interval_seconds` | No       | `300`      | Full scan interval (5 minutes)                                     |
| `file_extensions`       | No       | -          | Whitelist of extensions (e.g., `["txt", "md", "json"]`)            |
| `exclude_patterns`      | No       | -          | Patterns to exclude (e.g., `[".git", "node_modules"]`)             |
| `max_file_size_bytes`   | No       | `10485760` | Max file size for content extraction (10MB)                        |

### Step 4: Start Initial Sync

1. Click **Save Configuration**
2. Click **Sync Now** to start the initial scan
3. Monitor progress in the admin panel

<Check>
  Your Filesystem connector is now configured. The connector will scan and index files from the configured directory.
</Check>

## Example Configurations

### Documentation Repository

Index a documentation folder with Markdown and text files:

```json theme={null}
{
  "base_path": "/data/documents",
  "scan_interval_seconds": 600,
  "file_extensions": ["md", "txt", "rst"],
  "exclude_patterns": [".git", "_build", "node_modules"]
}
```

### Shared File Server

Index a shared network drive with all text content:

```json theme={null}
{
  "base_path": "/data/shared",
  "scan_interval_seconds": 1800,
  "exclude_patterns": ["Thumbs.db", ".DS_Store", "~$*"],
  "max_file_size_bytes": 52428800
}
```

## Managing the Integration

### Viewing Sync Status

Navigate to **Settings** → **Integrations** to view the sync status for each source directly on the list, including last sync time, number of indexed items, and any errors. Click **Configure** on a source for more details.

### Sync Behavior

The Filesystem connector uses two synchronization mechanisms:

| Mechanism        | Frequency             | Description                        |
| ---------------- | --------------------- | ---------------------------------- |
| **Full Scan**    | Every 5 min (default) | Walks entire directory tree        |
| **File Watcher** | Real-time             | Detects file changes between scans |

The file watcher polls for changes every 2 seconds and batches events with a 30-second idle timeout.

### Adding More Directories

To index additional directories:

1. Add another volume mount in Docker Compose
2. Create a new Filesystem source in Omni with the new `base_path`

### Removing the Integration

1. Navigate to **Settings** → **Integrations**
2. Click **Configure** against the Filesystem source
3. Click **Delete Permanently**

## Troubleshooting

<AccordionGroup>
  <Accordion title="Files not appearing in search">
    Common causes:

    * Volume not mounted correctly in Docker
    * File extension not in whitelist
    * File matches an exclude pattern
    * File larger than max size (content-only issue)

    **Solution**:

    1. Verify the volume mount: `docker exec omni-filesystem-connector ls /data/documents`
    2. Check configuration for file\_extensions and exclude\_patterns
    3. Trigger a manual sync
  </Accordion>

  <Accordion title="Permission denied errors">
    The container doesn't have read access to the files.

    **Solution**:

    * Ensure files are readable by the container user
    * Check that SELinux/AppArmor isn't blocking access
    * Try mounting with `:ro,z` (for SELinux) or `:ro,Z` (for private mount)
  </Accordion>

  <Accordion title="Large files not indexed">
    Files larger than 10MB (default) are indexed for metadata only.

    **Solution**: Increase `max_file_size_bytes` in the source configuration. Be aware this increases memory usage.
  </Accordion>

  <Accordion title="Scan taking too long">
    Large directories with many files will take longer to scan.

    **Factors affecting scan time**:

    * Number of files in the directory tree
    * Average file size
    * Disk I/O speed

    **Optimization tips**:

    * Use `file_extensions` to limit which files are processed
    * Use `exclude_patterns` to skip unnecessary directories
    * Increase `scan_interval_seconds` for stable directories
  </Accordion>

  <Accordion title="Changes not detected">
    The file watcher may not detect all changes.

    **Solution**: The full scan (every 5 minutes by default) will catch any missed changes. You can also trigger a manual sync.
  </Accordion>

  <Accordion title="Symlinks not followed">
    By default, symlinks are not followed to prevent infinite loops.

    **Solution**: Currently, symlink following is disabled. Place actual files in the indexed directory.
  </Accordion>
</AccordionGroup>

## Security Considerations

* **Read-only mounts**: Always use `:ro` flag for volume mounts
* **File permissions**: The connector respects filesystem permissions
* **No network access**: The connector only reads local files
* **Container isolation**: Files are accessed through Docker volumes

<Warning>
  Ensure sensitive files (credentials, private keys, etc.) are excluded using `exclude_patterns` or not mounted at all.
</Warning>

## What's Next

<CardGroup cols={3}>
  <Card title="Search Your Data" icon="magnifying-glass" href="/user-guide/search">
    Learn how to search across your files
  </Card>

  <Card title="AI Assistant" icon="robot" href="/user-guide/ai-assistant">
    Ask questions about your documents
  </Card>

  <Card title="Add More Connectors" icon="plug" href="/connectors/overview">
    Connect additional data sources
  </Card>
</CardGroup>