Critical Paths

The 4 most critical dependency paths in the Expensis automation system. If any component in these paths fails, major functionality breaks.

---

Path 1: Master Sync Orchestration

Impact: ALL automated provider syncs stop

graph TB
    CRON[Cron Daemon<br/>System Service] --> CMD[sync:manager<br/>Console Command]
    CMD --> SVC[SyncManagementService<br/>Business Logic]
    SVC --> MB1[MessageBus<br/>Dispatch Layer]
    MB1 --> SPH[SyncProfileHandler<br/>CRITICAL ROUTER]
    SPH --> MB2[MessageBus<br/>Forward to providers]
    MB2 --> KPN[KpnSp16SyncHandler]
    MB2 --> VOD[VodafoneSyncHandler]
    MB2 --> TMO[TMobileCalviSyncHandler]
    MB2 --> YIELD[YielderSyncHandler]
    MB2 --> EEN[KpnEenSyncHandler]

    style CRON fill:#ff6b6b
    style CMD fill:#ff6b6b
    style SPH fill:#ff6b6b
    style MB1 fill:#ffd93d
    style MB2 fill:#ffd93d

Dependency Chain

1. Cron Daemon (system service)
2. Status: systemctl status cron

3. If stopped: NO automation runs

4. sync:manager Command (file)

5. Location: src/Command/SyncManagementCommand.php

6. If missing/broken: Cron fails

7. SyncManagementService (service class)

8. Location: src/Service/SyncManagementService.php:378

9. If broken: No sync logic executes

10. MessageBus (Symfony component)

11. Config: config/packages/messenger.yaml

12. If misconfigured: Commands not dispatched

13. SyncProfileHandler CRITICAL ROUTER

14. Location: src/MessageBus/AsynchronousHandler/SyncProfileHandler.php:153

15. If broken: ALL provider routing stops

16. Provider Handlers (5 handlers)

17. If one breaks: That provider stops (others continue)

Monitoring

-- Check if syncs are running
SELECT
    COUNT(*) as sync_count,
    MAX(created_at) as last_sync
FROM sync_task
WHERE created_at > NOW() - INTERVAL 10 MINUTE;

-- Should return rows every 5 minutes
-- If no results: sync:manager not running

Recovery

# 1. Check cron status
sudo systemctl status cron

# 2. Verify cron entry
crontab -l | grep sync:manager

# 3. Test manual execution
cd /var/www/expensis
php bin/console sync:manager -vvv

# 4. Check logs
tail -100 /var/log/syslog | grep sync:manager

---

Path 2: Command Chain Execution

Impact: ALL workflows with chains break (99% of workflows)

graph LR
    CMD[Any Command<br/>with chain] --> HDL[Handler<br/>executes]
    HDL --> EVT[WorkerMessageHandledEvent<br/>fires]
    EVT --> MCH[MessageConsumedHandler<br/>CRITICAL]
    MCH --> CHECK{Has chain?}
    CHECK -->|Yes| DISPATCH[Dispatch next command]
    CHECK -->|No| DONE[Done]
    DISPATCH --> MB[MessageBus]
    MB --> NEXT[Next Handler]

    style MCH fill:#ff6b6b
    style EVT fill:#ffd93d

Dependency Chain

1. Handler Execution
2. Any MessageBus handler

3. Fires WorkerMessageHandledEvent on completion

4. WorkerMessageHandledEvent (Symfony event)

5. Part of Symfony Messenger

6. If event system broken: Chains don't work

7. MessageConsumedHandler CRITICAL

8. Location: src/MessageBus/MessageConsumedHandler.php
9. Subscribes to WorkerMessageHandledEvent

10. If broken: ZERO command chains work

11. MessageBus (for chained dispatch)

12. Must be properly configured
13. If broken: Chained commands not dispatched

Test Chain Execution

# Dispatch command with chain
php bin/console totals:new --customer=123 --cycle=2025-01-01

# Check if backup also created (part of chain)

-- Verify chain executed
SELECT
    t.id as totals_id,
    tb.id as backup_id,
    t.customer_id,
    t.cycle_start_date
FROM total_usage_new_table t
LEFT JOIN total_usage_new_backup_table tb ON
    t.customer_id = tb.customer_id
    AND t.cycle_start_date = tb.cycle_start_date
WHERE t.customer_id = 123
AND t.cycle_start_date = '2025-01-01';

-- If backup_id is NULL: Chain didn't execute

Recovery

# 1. Verify MessageConsumedHandler exists
ls -la src/MessageBus/MessageConsumedHandler.php

# 2. Check event subscriber registration
php bin/console debug:event-dispatcher kernel.message.consumed

# 3. Clear cache
php bin/console cache:clear

# 4. Restart messenger workers
php bin/console messenger:stop-workers

---

Path 3: Totals Calculation

Impact: ALL billing calculations fail

graph TB
    SYNC[Provider Sync<br/>completes] --> DTC[DispatchTotalsCommand<br/>ROUTER]
    DTC --> DTH[DispatchTotalsHandler<br/>determines type]
    DTH --> CALC{Customer<br/>has rate plan?}
    CALC -->|Yes| CRP[CalculateTotalsRateplanHandler]
    CALC -->|No| CT[CalculateTotalsHandler<br/>CRITICAL]
    CT --> DB[(Query CDRs<br/>from database)]
    CRP --> DB
    DB --> AGG[Aggregate totals]
    AGG --> SAVE[Update totals table]
    SAVE --> BI[BillingInsightHandler<br/>chain]

    style DTC fill:#ffd93d
    style DTH fill:#ffd93d
    style CT fill:#ff6b6b
    style DB fill:#95e1d3

Dependency Chain

1. DispatchTotalsCommand (router command)
2. Dispatched by all provider syncs

3. If broken: No routing to calculation

4. DispatchTotalsHandler (router)

5. Location: src/MessageBus/AsynchronousHandler/DispatchTotalsHandler.php
6. Determines: rate plan vs standard calculation

7. If broken: No totals calculated

8. CalculateTotalsHandler CRITICAL

9. Location: src/MessageBus/AsynchronousHandler/CalculateTotalsHandler.php
10. Performs actual calculation

11. If broken: NO billing totals

12. Database (call_detail_record table)

13. Must have CDRs
14. Must have proper indexes

15. If slow: Calculations timeout

16. Totals Table (target)

17. Must be writable
18. If locked: Calculations fail

Monitoring

-- Check if calculations are happening
SELECT
    customer_id,
    cycle_start_date,
    total_cost,
    updated_at
FROM total_usage_new_table
WHERE updated_at > NOW() - INTERVAL 1 HOUR
ORDER BY updated_at DESC
LIMIT 20;

-- Should see recent updates

Recovery

# 1. Test calculation manually
php bin/console totals:new --customer=123 --cycle=2025-01-01 -vvv

# 2. Check for database locks
# MySQL:
SHOW PROCESSLIST;

# 3. Check CDR data exists

SELECT COUNT(*) FROM call_detail_record
WHERE customer_id = 123
AND call_date >= '2025-01-01'
AND call_date < '2025-02-01';

---

Path 4: CDR Import (Routit Example)

Impact: Mobile CDR imports fail (affects Routit billing)

graph TB
    REQ[RoutitRequestCdrsHandler<br/>request from API] --> CHECK{CDRs ready?}
    CHECK -->|No| WAIT[Exit, retry later]
    CHECK -->|Yes| DL[RoutitDownloadCdrsHandler<br/>download ZIP]
    DL --> FS[File System<br/>extract CSV]
    FS --> IMP[RoutitImportCdrsHandler<br/>CRITICAL BULK INSERT]
    IMP --> BATCH[Batch processing<br/>1000 at a time]
    BATCH --> DB[(Database<br/>call_detail_record)]
    DB --> INV[RoutitInvoiceHandler<br/>generate invoice]
    INV --> CALC[CalculateTotalsHandler<br/>calculate billing]

    style IMP fill:#ff6b6b
    style FS fill:#ffd93d
    style DB fill:#95e1d3

Dependency Chain

1. RoutitRequestCdrsHandler
2. Requests CDRs from Routit API

3. If broken: No CDR data requested

4. Routit API (external)

5. Must be accessible
6. Must have data ready

7. If down: Can't download CDRs

8. RoutitDownloadCdrsHandler

9. Downloads ZIP files

10. If broken: Files not retrieved

11. File System (/var/www/expensis/storage/)

12. Must be writable
13. Must have space

14. If full: Downloads fail

15. RoutitImportCdrsHandler CRITICAL

16. Location: src/MessageBus/AsynchronousHandler/RoutitImportCdrsHandler.php
17. Performs bulk insert (10k-50k CDRs)

18. If broken: NO CDRs imported

19. Database (write performance)

20. Must handle bulk inserts
21. If slow: Imports timeout

Monitoring

-- Check import progress
SELECT
    id,
    type,
    locked,
    complete,
    created_at,
    updated_at
FROM import_queue
WHERE created_at > NOW() - INTERVAL 24 HOUR
AND type = 'routit_cdr'
ORDER BY created_at DESC;

-- Check CDR volume
SELECT
    DATE(call_date) as date,
    COUNT(*) as cdr_count
FROM call_detail_record
WHERE call_date > NOW() - INTERVAL 7 DAY
GROUP BY DATE(call_date)
ORDER BY date DESC;

Recovery

# 1. Check disk space
df -h /var/www/expensis/storage

# 2. Check import queue

SELECT * FROM import_queue
WHERE type = 'routit_cdr'
AND locked = 1
AND complete = 0
ORDER BY created_at DESC;

# 3. Reset stuck import

UPDATE import_queue
SET locked = 0, message = 'Reset manually'
WHERE id = 123 AND type = 'routit_cdr';

# 4. Retry manually
php bin/console routit:cdr:sync --customer=123

---

Critical Path Summary

Path	SPOF Count	Most Critical Component	Impact Scope
Master Sync	3	SyncProfileHandler	ALL syncs (100%)
Chain Execution	1	MessageConsumedHandler	ALL chains (99% workflows)
Totals Calculation	2	CalculateTotalsHandler	ALL billing (100%)
CDR Import	2	RoutitImportCdrsHandler	Routit billing only

---

Health Check Script

#!/bin/bash
# check_critical_paths.sh

echo "=== Critical Path Health Check ==="

# Path 1: Master Sync
echo ""
echo "1. Master Sync Path:"
systemctl is-active cron && echo " Cron running" || echo " Cron stopped"
[ -f src/Command/SyncManagementCommand.php ] && echo " sync:manager exists" || echo " sync:manager missing"
[ -f src/MessageBus/AsynchronousHandler/SyncProfileHandler.php ] && echo " SyncProfileHandler exists" || echo " SyncProfileHandler missing"

# Path 2: Chain Execution
echo ""
echo "2. Chain Execution Path:"
[ -f src/MessageBus/MessageConsumedHandler.php ] && echo " MessageConsumedHandler exists" || echo " MessageConsumedHandler missing"

# Path 3: Totals Calculation
echo ""
echo "3. Totals Calculation Path:"
[ -f src/MessageBus/AsynchronousHandler/DispatchTotalsHandler.php ] && echo " DispatchTotalsHandler exists" || echo " DispatchTotalsHandler missing"
[ -f src/MessageBus/AsynchronousHandler/CalculateTotalsHandler.php ] && echo " CalculateTotalsHandler exists" || echo " CalculateTotalsHandler missing"

# Path 4: CDR Import
echo ""
echo "4. CDR Import Path:"
[ -f src/MessageBus/AsynchronousHandler/RoutitImportCdrsHandler.php ] && echo " RoutitImportCdrsHandler exists" || echo " RoutitImportCdrsHandler missing"
[ -d /var/www/expensis/storage ] && echo " Storage directory exists" || echo " Storage directory missing"

---

Related Documentation

• Single Points of Failure - Detailed SPOF analysis
• Dependency Graph - Complete system dependencies
• Common Issues - Troubleshooting paths
• Workflows - See paths in action