Skip to main content

Schedule sync runs

Infrahub Sync runs as a single CLI command. Schedule it with whatever tooling already runs scheduled jobs in your environment — cron, CI, Prefect, Dagster, or a homegrown runner.

Common orchestration options​

Any tool that can run a CLI command on a schedule and capture its output works. Pick whichever fits your existing operational model.

Cron​

The most direct option. A cron entry runs infrahub-sync sync on a defined schedule. Output and exit codes are captured by your normal cron logging.

When this works well:

  • You already operate a cron host for scheduled jobs.
  • The sync is straightforward — one project, one schedule, no dependencies on other jobs.
  • Failure handling can be reactive (logs are reviewed when something looks off).

Trade-offs: No native retry on failure, no visibility beyond logs, no built-in alerting. For mission-critical syncs, wrap the cron entry in a script that handles retries and alerts.

CI jobs (GitHub Actions, GitLab CI, etc.)​

Define the sync as a scheduled workflow in your CI system. CI provides run history, log retention, secret management, and notification on failure.

When this works well:

  • You treat infrastructure data movement as part of your software delivery pipeline.
  • You want run history and structured failure notifications from tooling you already operate.
  • Sync project configurations live in a git repository alongside other infrastructure code.

Trade-offs: CI systems are designed for short-lived jobs. For very large or long-running syncs, watch out for job timeout limits.

Prefect​

Infrahub Sync provides two Prefect-backed execution surfaces:

  • Install the prefect extra to serve one direct deployment for plan and confirmed sync requests through Prefect's API. Sync output is forwarded into the flow-run log. See the Prefect remote run reference and examples/prefect_remote_run/README.md in the repository.
  • Install the managed extra when an automation client needs a Sync-owned authenticated HTTP API, durable records and artifacts, reviewed-plan apply, idempotency, cancellation, and worker execution. See the managed Sync HTTP API reference.

When this works well:

  • You already use Prefect for other infrastructure or data workflows.
  • A sync needs to be triggered on demand from another system, over an API rather than a shell.
  • You want centralized observability across many sync projects.

Trade-offs: Prefect itself has to be operated. If you do not already run it, this adds operational surface area. The direct deployment runs and observes one plan or confirmed sync; it does not add scheduling, retry, queue, or overlap policy.

Use the managed API for the supported worker-backed run lifecycle. For schedules, retry policies, triggers, or composing several syncs into a larger workflow, configure Prefect around either packaged surface or wrap the CLI in a flow you own.

Dagster, Airflow, and other workflow engines​

Any workflow engine that can run a CLI command works the same way as Prefect. Pick based on what you already use and what other workloads share the orchestration platform.

Event-driven execution​

Some sync use cases are better triggered by an event than by a schedule. For example: "when a device is created in Infrahub, sync its details from the inventory system." For event-driven patterns, Infrahub's trigger event system can call infrahub-sync directly instead of running on a fixed cadence.

Event-driven sync is the exception, not the rule. Most data movement is well-served by a recurring schedule. Use event-driven patterns when the data volume is small, the freshness requirement is high, or the source system is itself event-driven.

Choosing a cadence​

Match the cadence to the rate at which source data changes and your tolerance for staleness:

  • Every few minutes — for high-change-rate sources that need near-real-time freshness. Verify the sync run completes faster than the cadence to avoid overlap.
  • Hourly — a common default for active infrastructure data.
  • Daily — appropriate for slower-moving data or for syncs where the next run can absorb a missed one.
  • On demand — for migrations, audits, or one-off seeding rather than steady-state operation.

Each sync run calculates a fresh diff and applies only deltas, so running more often does not multiply work. The constraint is the time the sync takes, not the work it does.

Observability and failure handling​

Whatever scheduling tool you use, plan for these operational concerns:

  • Logging. Infrahub Sync emits lifecycle and adapter records through Python logging. Forward them to the logging system used by the scheduler. Public Python API lifecycle records include structured run, operation, stage, and outcome attributes.
  • Failure detection. The CLI returns a non-zero exit code on failure. Wire the scheduling tool to catch this and alert the appropriate channel.
  • Failure recovery. Inspect the run record and destination after a failed write, then calculate a fresh plan. A failed operation can have written part of its change. Safe re-application is not established for destination kinds whose identity crosses a relationship; follow the run recovery guidance.
  • Run isolation. Avoid overlapping runs of the same sync project. Set the cadence longer than the run time, or have the scheduler skip a run if the previous one is still active.
  • Sync project versioning. Store sync project directories in version control. Tag or release configuration changes the same way you handle infrastructure code.

Running Infrahub Sync in a container​

To run Infrahub Sync from any orchestrator while keeping a single Python environment under your control, package it as a container. Build the image with infrahub-sync installed and the sync project directory included, set the sync command as the entrypoint, and pass credentials via environment variables.

Infrahub Sync does not ship a reference container image — examples in the OpsMill GitHub repository show common patterns to adapt.

What's not in Infrahub Sync (and what to use instead)​

  • Built-in scheduler. Use one of the options above.
  • Sync history dashboard. The scheduling tool typically provides this — CI run history, Prefect UI, or aggregated logs.
  • Built-in retry policies. Configure retries in the scheduling tool. Treat write retries according to the documented recovery and convergence boundaries.
  • Alerting on failure. Configure alerts in the scheduling tool or log aggregator.