Inside Activate
Connectors
Performance & Scaling
5 min
connector performance depends on the size of the source dataset, schema complexity, and workflow configuration optimising each stage ensures efficient imports and timely change processing key performance factors area description optimisation tip import reading from source (file, sql, api) use indexes and minimal column sets avoid select processimport detecting differences between import and data tables ensure both tables have primary keys and statistics are up to date process executing change handlers adjust batch sizes and parallelise connectors across instances history table logging all change events archive or purge older history rows regularly import optimisation use stored procedures for sql providers — they can pre filter and aggregate data for large csvs, split into smaller batches and use the batchsize parameter run imports from local disks rather than network shares for better throughput validate file encoding (utf‑8) to prevent parsing overhead processimport optimisation index the key column on both import and data tables keep non key columns lightweight; large text fields slow comparisons set maxchanges thresholds relative to expected delta sizes use incremental imports (where lastmodified > @lastrun) when possible process (handler) optimisation tune batchsize in the process options typical range 100–500 if handlers run scripts or submit jobs, verify downstream systems can absorb the load use separate connectors for unrelated domains instead of one large workflow schedule high volume connectors during low traffic windows scaling out multiple connectors can run concurrently on the same instance if resources permit use multiple activate servers to distribute connector workloads monitor cpu and sql i/o during heavy runs to identify bottlenecks review windows task scheduler or orchestrator timing to avoid overlap long term maintenance archive history tables by date or row count periodically review indexes for fragmentation measure average import duration and compare against baselines reassess connector scheduling after major data source changes