---
title: Performance & Scaling
slug: performance-and-scaling
createdAt: 2025-10-29T21:30:02.175Z
---

Connector performance depends on the size of the source dataset, schema complexity, and workflow configuration. Optimising each stage ensures efficient imports and timely change processing.

# Key Performance Factors

| Area              | Description                                           | Optimisation Tip                                                    |
| ----------------- | ----------------------------------------------------- | ------------------------------------------------------------------- |
| **Import**        | Reading from source (file, SQL, API).                 | Use indexes and minimal column sets. Avoid SELECT \*.               |
| **ProcessImport** | Detecting differences between import and data tables. | Ensure both tables have primary keys and statistics are up-to-date. |
| **Process**       | Executing change handlers.                            | Adjust batch sizes and parallelise connectors across instances.     |
| **History Table** | Logging all change events.                            | Archive or purge older history rows regularly.                      |

# Import Optimisation

- Use **stored procedures** for SQL providers — they can pre-filter and aggregate data. 
- For large CSVs, split into smaller batches and use the batchSize parameter. 
- Run imports from local disks rather than network shares for better throughput. 
- Validate file encoding (UTF‑8) to prevent parsing overhead. 

# ProcessImport Optimisation

- Index the key column on both **import** and **data** tables. 
- Keep non-key columns lightweight; large text fields slow comparisons. 
- Set maxChanges thresholds relative to expected delta sizes. 
- Use **incremental imports** (WHERE LastModified > @LastRun) when possible. 

# Process (Handler) Optimisation

- Tune batchSize in the process options. Typical range: 100–500. 
- If handlers run scripts or submit jobs, verify downstream systems can absorb the load. 
- Use separate connectors for unrelated domains instead of one large workflow. 
- Schedule high-volume connectors during low-traffic windows. 

# Scaling Out

- Multiple connectors can run concurrently on the same instance if resources permit. 
- Use multiple Activate servers to distribute connector workloads. 
- Monitor **CPU** and **SQL I/O** during heavy runs to identify bottlenecks. 
- Review Windows Task Scheduler or Orchestrator timing to avoid overlap. 

# Long-term Maintenance

- Archive history tables by date or row count. 
- Periodically review indexes for fragmentation. 
- Measure average import duration and compare against baselines. 
- Reassess connector scheduling after major data source changes.