Skip to content

Master Sync Orchestration

Critical Workflow

This is the PRIMARY automation workflow running every 5 minutes. If this fails, all automated syncs stop.

Overview

Trigger: Cron job */5 * * * *
Entry Point: sync:manager console command
Purpose: Master orchestrator for all provider syncs
Frequency: Every 5 minutes
Impact: Critical - orchestrates all automation

Complete Flow Diagram

flowchart TD
    Start[Cron: Every 5 minutes] --> Manager[sync:manager command]
    Manager --> Service[SyncManagementService::run]

    Service --> Query[Query active SyncProfiles]
    Query --> Check{For each profile}

    Check --> Create[Create SyncTask]
    Create --> Dispatch[Dispatch SyncProfileCommand]

    Dispatch --> Router[SyncProfileHandler<br/>Master Router]

    Router --> TypeCheck{Check SyncType}

    TypeCheck -->|KPN_SP16 or GRIP| KPN[KpnSp16SyncCommand]
    TypeCheck -->|MY_VODAFONE| VOD[VodafoneSyncCommand]
    TypeCheck -->|T_MOBILE| TMO[TMobileCalviSyncCommand]
    TypeCheck -->|KPN_EEN| EEN[KpnEenSyncCommand]
    TypeCheck -->|YIELDER| YIELD[YielderSyncCommand]

    KPN --> KPNHandler[KpnSp16SyncHandler]
    VOD --> VODHandler[VodafoneSyncHandler]
    TMO --> TMOHandler[TMobileCalviSyncHandler]
    EEN --> EENHandler[KpnEenSyncHandler]
    YIELD --> YIELDHandler[YielderSyncHandler]

    KPNHandler --> Totals1[Dispatch Totals Chain]
    VODHandler --> Totals2[Dispatch Totals Chain]
    TMOHandler --> Totals3[Dispatch Totals Chain]
    EENHandler --> Totals4[Dispatch Totals Chain]
    YIELDHandler --> Totals5[Dispatch Totals Chain]

    Totals1 --> Complete[Update SyncTask: COMPLETED]
    Totals2 --> Complete
    Totals3 --> Complete
    Totals4 --> Complete
    Totals5 --> Complete

    Complete --> Check

    style Start fill:#ffebee
    style Router fill:#fff3e0
    style Complete fill:#e8f5e9

Detailed Step-by-Step

Step 1: Cron Triggers sync:manager

# Every 5 minutes
*/5 * * * * php /var/www/expensis/bin/console sync:manager

What happens: - Cron daemon executes the command - Console command initializes - Calls SyncManagementService::run()

Step 2: Query Active Sync Profiles

// SyncManagementService.php
public function run() {
    $syncProfiles = $this->syncProfileRepository->findActiveProfiles();

    foreach ($syncProfiles as $syncProfile) {
        if ($this->shouldSync($syncProfile)) {
            $this->executeSyncProfile($syncProfile);
        }
    }
}

Database Query: 

SELECT * FROM sync_profile 
WHERE active = 1 
AND next_sync_date <= NOW()
ORDER BY priority DESC

Step 3: Create SyncTask for Tracking

$syncTask = new SyncTask();
$syncTask->setSyncProfile($syncProfile);
$syncTask->setCycleStartDate($billingCycle);
$syncTask->setStatus(SyncTask::STATUS_PENDING);
$this->entityManager->persist($syncTask);
$this->entityManager->flush();

Database Write: - Table: sync_task - Fields: sync_profile_id, cycle_start_date, status, created_at

Step 4: Dispatch SyncProfileCommand

$command = new SyncProfileCommand(
    $syncProfile,
    $billingCycle,
    $syncTask,
    $chainCommands // Optional chain
);

$this->commandBus->dispatch($command);

Location: src/Service/SyncManagementService.php:378

Step 5: SyncProfileHandler Routes by Type

// SyncProfileHandler.php:153
public function __invoke(SyncProfileCommand $command) {
    $syncType = $command->getSyncProfile()->getSyncType();

    $syncCommand = match($syncType) {
        SyncType::KPN_SP16, SyncType::GRIP 
            => new KpnSp16SyncCommand(...),
        SyncType::MY_VODAFONE 
            => new VodafoneSyncCommand(...),
        SyncType::T_MOBILE 
            => new TMobileCalviSyncCommand(...),
        SyncType::KPN_EEN 
            => new KpnEenSyncCommand(...),
        SyncType::YIELDER 
            => new YielderSyncCommand(...),
    };

    if ($chainCommands) {
        $syncCommand->withChain($chainCommands);
    }

    $this->commandBus->dispatch($syncCommand);
}

This is the MASTER ROUTER - all provider syncs flow through here.

Step 6: Provider-Specific Handler Executes

Each provider handler follows the same pattern:

sequenceDiagram
    participant Handler
    participant ImportService
    participant Provider
    participant DB
    participant MessageBus

    Handler->>ImportService: Import data
    ImportService->>Provider: Scrape/API call
    Provider-->>ImportService: Return data
    ImportService->>DB: Store CDRs, subscriptions
    Handler->>MessageBus: DispatchTotalsCommand
    MessageBus->>Handler: Execute chain

Common Operations: 1. Call import service 2. Scrape provider portal or call API 3. Store CDRs and subscription data 4. Dispatch totals calculation 5. Update sync task status

Step 7: Totals Calculation Chain

After sync completes, totals are calculated:

$totalsCommand = new DispatchTotalsCommand($customerId, $billingCycle);
$totalsCommand->withChain([
    new DispatchTotalsBackupCommand($customerId, $billingCycle)
]);
$this->commandBus->dispatch($totalsCommand);

Chain execution: 1. DispatchTotalsCommand → Routes to calculation 2. CalculateTotalsCommand → Calculates billing totals 3. DispatchTotalsBackupCommand → Backs up totals

Handled by: MessageConsumedHandler (event-driven)

Database Operations

Tables Read:

  • sync_profile - Active sync configurations
  • sync_task - Previous sync status
  • customer - Customer details
  • billing_cycle - Cycle dates

Tables Written:

  • sync_task - New task creation, status updates
  • call_detail_record - CDR imports
  • subscription - Subscription updates
  • device - Device information
  • totals - Billing calculations
  • totals_backup - Historical totals

Typical Query Volume per Run:

  • SELECT: ~50-100 queries
  • INSERT: ~1000-10000 CDRs (per provider)
  • UPDATE: ~10-50 sync tasks
  • Total time: 2-10 minutes (all providers)

Error Handling

Authentication Failures

try {
    $importService->import($customer, $billingCycle);
} catch (SyncAuthenticationFailedException $e) {
    $this->syncManagementService->setStatus(
        $syncProfile, 
        $billingCycle, 
        SyncTask::STATUS_INVALID_CREDENTIALS,
        $e->getMessage()
    );
    return; // Stop processing
}

Result: SyncTask marked as INVALID_CREDENTIALS, email notification sent

Portal Unavailable

catch (SyncInvoiceNotAvailableException $e) {
    $this->syncManagementService->setStatus(
        $syncProfile,
        $billingCycle,
        SyncTask::STATUS_INVOICE_NOT_YET_AVAILABLE,
        $e->getMessage()
    );
    return; // Will retry on next cron run
}

Result: Task marked for retry, will attempt again in 5 minutes

Unknown Errors

catch (\Exception $e) {
    $this->logger->error("Sync failed: {$e->getMessage()}");
    $this->syncManagementService->setStatus(
        $syncProfile,
        $billingCycle,
        SyncTask::STATUS_UNKNOWN_ERROR,
        $e->getMessage()
    );

    // Calculate totals after 5 failed attempts
    if ($syncTask->getRetries() >= 5) {
        $this->dispatchTotalsAnyway($customer, $billingCycle);
    }
}

Result: After 5 retries, calculates totals with existing data

Success Flow Example

Time: 10:00:00 - Cron triggers

Time Action Status
10:00:00 Cron executes sync:manager Running
10:00:01 Query 15 active profiles Found 15
10:00:02 Create 15 SyncTask records Created
10:00:03 Dispatch 15 SyncProfileCommands Dispatched
10:00:04-10:05:00 Handlers execute in parallel Processing
10:05:01 KPN SP16 completes Success
10:05:03 Vodafone completes Success
10:05:05 T-Mobile completes Success
10:05:10 All totals calculated Complete

Total Duration: ~5 minutes
Success Rate: Typically 95%+ of profiles

Monitoring Points

Critical Metrics

-- Failed syncs in last hour
SELECT COUNT(*) FROM sync_task
WHERE status = 'failed'
AND created_at > NOW() - INTERVAL 1 HOUR;

-- Average sync duration
SELECT AVG(TIMESTAMPDIFF(SECOND, created_at, updated_at))
FROM sync_task
WHERE status = 'completed'
AND created_at > NOW() - INTERVAL 1 DAY;

-- Sync success rate
SELECT 
    status,
    COUNT(*) as count,
    COUNT(*) * 100.0 / SUM(COUNT(*)) OVER () as percentage
FROM sync_task
WHERE created_at > NOW() - INTERVAL 1 DAY
GROUP BY status;

Alert Thresholds

Metric Threshold Action
Cron missed 2 consecutive Page on-call
Fail rate > 20% Investigate
Duration > 10 minutes Check performance
Auth failures > 5 Check credentials

Manual Intervention

Force Sync for Customer

# Bypass schedule, force immediate sync
php bin/console sync:profile --customer=123 --force

Retry Failed Sync

-- Reset failed sync task for retry
UPDATE sync_task 
SET status = 'pending', retries = 0
WHERE id = 12345;

Debug Sync

# Run with verbose output
php bin/console sync:manager -vvv

Performance Characteristics

Normal Operation

  • Profiles per run: 10-20
  • Duration: 3-7 minutes
  • CDRs imported: 5,000-50,000
  • Database queries: 100-500
  • Memory usage: 50-200 MB

Peak Load

  • Profiles per run: 30-40
  • Duration: 8-12 minutes
  • CDRs imported: 100,000+
  • Database queries: 1000+
  • Memory usage: 300-500 MB

Key Takeaways

Understanding the Master Sync

  1. Runs every 5 minutes (critical timing)
  2. SyncProfileHandler is the master router
  3. Executes multiple provider syncs in parallel
  4. Each sync follows: Import → Calculate → Backup
  5. MessageConsumedHandler enables all chaining
  6. Failures retry automatically on next cron run

Single Point of Failure

If sync:manager cron stops, NO automated syncs occur. Monitor this closely!