Performance & Scaling
5 min
Connector performance depends on the size of the source dataset, schema complexity, and workflow configuration. Optimising each stage ensures efficient imports and timely change processing.
Key Performance Factors
Area | Description | Optimisation Tip |
|---|---|---|
Import | Reading from source (file, SQL, API). | Use indexes and minimal column sets. Avoid SELECT *. |
ProcessImport | Detecting differences between import and data tables. | Ensure both tables have primary keys and statistics are up-to-date. |
Process | Executing change handlers. | Adjust batch sizes and parallelise connectors across instances. |
History Table | Logging all change events. | Archive or purge older history rows regularly. |
Import Optimisation
- Use stored procedures for SQL providers — they can pre-filter and aggregate data.
- For large CSVs, split into smaller batches and use the batchSize parameter.
- Run imports from local disks rather than network shares for better throughput.
- Validate file encoding (UTF‑8) to prevent parsing overhead.
ProcessImport Optimisation
- Index the key column on both import and data tables.
- Keep non-key columns lightweight; large text fields slow comparisons.
- Set maxChanges thresholds relative to expected delta sizes.
- Use incremental imports (WHERE LastModified > @LastRun) when possible.
Process (Handler) Optimisation
- Tune batchSize in the process options. Typical range: 100–500.
- If handlers run scripts or submit jobs, verify downstream systems can absorb the load.
- Use separate connectors for unrelated domains instead of one large workflow.
- Schedule high-volume connectors during low-traffic windows.
Scaling Out
- Multiple connectors can run concurrently on the same instance if resources permit.
- Use multiple Activate servers to distribute connector workloads.
- Monitor CPU and SQL I/O during heavy runs to identify bottlenecks.
- Review Windows Task Scheduler or Orchestrator timing to avoid overlap.
Long-term Maintenance
- Archive history tables by date or row count.
- Periodically review indexes for fragmentation.
- Measure average import duration and compare against baselines.
- Reassess connector scheduling after major data source changes.