Skip to content

Common Issues

Solutions to frequently encountered problems in the Expensis automation system.

Critical Issues

Syncs Not Running at All

Symptom: No syncs executing for any provider

Root Cause: sync:manager cron job stopped

Check: 

# Check cron status
sudo systemctl status cron

# Check if sync:manager is in crontab
crontab -l | grep sync:manager

# Check recent cron execution
grep sync:manager /var/log/syslog | tail -20

Solution: 

# Verify cron entry exists
*/5 * * * * php /var/www/expensis/bin/console sync:manager

# Restart cron if needed
sudo systemctl restart cron

# Test manual execution
cd /var/www/expensis
php bin/console sync:manager -vvv

Prevention: Monitor cron execution with alerting

Command Chains Not Executing

Symptom: First command completes but chained commands never run

Root Cause: MessageConsumedHandler broken or not subscribed

Check: 

# Verify handler exists
ls -la src/MessageBus/MessageConsumedHandler.php

# Check Symfony event subscribers
php bin/console debug:event-dispatcher kernel.message.consumed

Diagnosis Query: 

-- Check if totals backup is happening after calculations
SELECT
    st.id,
    st.status,
    t.id as totals_id,
    tb.id as backup_id
FROM sync_task st
LEFT JOIN total_usage_new_table t ON st.customer_id = t.customer_id
LEFT JOIN total_usage_new_backup_table tb ON t.customer_id = tb.customer_id
WHERE st.created_at > NOW() - INTERVAL 1 HOUR
AND st.status = 'completed'
AND tb.id IS NULL;  -- If results found, backup chain not working

Solution: 1. Verify MessageConsumedHandler class exists and is properly configured 2. Check event subscriber registration in config/services.yaml 3. Clear Symfony cache: php bin/console cache:clear 4. Restart message consumers

Sync Issues

Sync Fails with "Invalid Credentials"

Symptom: sync_task.status = 'invalid_credentials'

Root Cause: Provider portal credentials expired or incorrect

Check Database: 

SELECT
    sp.id,
    sp.sync_type,
    c.name as customer_name,
    st.status,
    st.error_message
FROM sync_task st
JOIN sync_profile sp ON st.sync_profile_id = sp.id
JOIN customer c ON sp.customer_id = c.id
WHERE st.status = 'invalid_credentials'
AND st.created_at > NOW() - INTERVAL 24 HOUR;

Solutions: 1. Update Credentials: - Go to customer sync profile settings in UI - Update username/password - Save and test connection

  1. For Calvi (T-Mobile): 

    # Encrypt new password
php bin/console calvi-password:encrypt-passwords

  2. Manual Test: 

    # Test specific provider sync
php bin/console sync:manager --customer=123 --force -vvv

Prevention: Set up credential expiration alerts

Sync Fails with "Invoice Not Yet Available"

Symptom: sync_task.status = 'invoice_not_yet_available'

Root Cause: Provider hasn't published invoice data yet (too early in billing cycle)

Behavior: This is EXPECTED - sync will automatically retry on next cron run (5 minutes)

Check: 

SELECT
    customer_id,
    cycle_start_date,
    status,
    created_at,
    retries
FROM sync_task
WHERE status = 'invoice_not_yet_available'
ORDER BY created_at DESC;

Solution: Wait for provider to publish data (usually available 3-5 days into billing cycle)

No Action Needed: System will retry automatically

KPN SP16 Sync Hangs on Import Queue

Symptom: KPN SP16 sync creates import queue record but never processes it

Root Cause: Import queue stuck in 'processing' status

Check: 

SELECT * FROM kpn_import_queue
WHERE status = 'processing'
AND created_at < NOW() - INTERVAL 1 HOUR;

Solution: 

-- Reset stuck queue items
UPDATE kpn_import_queue
SET status = 'pending', retries = 0
WHERE status = 'processing'
AND created_at < NOW() - INTERVAL 1 HOUR;

Then trigger manually: 

php bin/console kpn:sp16:sync --customer=123

Routit CDR Import Fails Halfway

Symptom: CDR import starts but fails partway through, leaving partial data

Root Cause: Memory exhaustion or database timeout during bulk insert

Check Logs: 

tail -100 /var/www/expensis/var/log/prod.log | grep -i "memory\|timeout"

Solutions:

  1. Increase PHP Memory: 

    # php.ini
memory_limit = 512M  # Increase if needed

  2. Increase MySQL Timeout: 

    SET GLOBAL max_execution_time = 300;
SET GLOBAL wait_timeout = 300;

  3. Reduce Batch Size: 

    // In RoutitImportCdrsHandler.php
$batchSize = 500;  // Reduce from 1000

  4. Manual Cleanup and Retry: 

    -- Delete partial import
DELETE FROM call_detail_record
WHERE customer_id = 123
AND call_date >= '2025-01-01'
AND import_id = 'partial_import_id';

-- Reset import request
UPDATE routit_cdr_request
SET status = 'pending'
WHERE id = 456;

Totals Calculation Issues

Totals Are Zero Despite CDRs Existing

Symptom: Totals calculation completes but shows zero costs

Root Cause: CDRs not linked to devices/subscriptions or rate plan missing

Diagnosis: 

-- Check for unlinked CDRs
SELECT COUNT(*) as unlinked_cdrs
FROM call_detail_record cdr
WHERE cdr.device_id IS NULL
AND cdr.call_date >= '2025-01-01';

-- Check for CDRs without rate plans
SELECT
    cdr.id,
    cdr.call_type,
    d.subscription_id,
    s.rate_plan_id
FROM call_detail_record cdr
LEFT JOIN device d ON cdr.device_id = d.id
LEFT JOIN subscription s ON d.subscription_id = s.id
WHERE cdr.call_date >= '2025-01-01'
AND s.rate_plan_id IS NULL
LIMIT 10;

Solutions:

  1. Link CDRs to Devices: 

    php bin/console link:cdrs:to:devices --customer=123

  2. Assign Rate Plans:

  3. Go to subscription management UI
  4. Assign appropriate rate plan to subscriptions
  5. Recalculate totals: 
    php bin/console totals:new --customer=123 --cycle=2025-01-01

Totals Differ from Provider Invoice

Symptom: Calculated totals don't match provider's invoice

Root Causes: 1. Missing CDRs 2. Wrong rate plan applied 3. Provider charges not in CDRs (fees, taxes)

Diagnosis: 

-- Compare CDR count with provider
SELECT
    COUNT(*) as expensis_cdr_count,
    SUM(cost) as expensis_total_cost
FROM call_detail_record
WHERE customer_id = 123
AND call_date >= '2025-01-01'
AND call_date < '2025-02-01';

-- Check totals breakdown
SELECT * FROM total_usage_new_table
WHERE customer_id = 123
AND cycle_start_date = '2025-01-01';

Solutions:

  1. Re-import CDRs: 

    # Force fresh import
php bin/console [provider]:sync --customer=123 --cycle=2025-01-01 --force

  2. Verify Rate Plan: 

    SELECT s.id, s.name, s.rate_plan_id, rp.name as rate_plan_name
FROM subscription s
LEFT JOIN rate_plan rp ON s.rate_plan_id = rp.id
WHERE s.customer_id = 123;

  3. Manual Adjustment (if needed): 

    -- Add manual adjustment for fees not in CDRs
UPDATE totals
SET additional_fees = 50.00,
    total_cost = total_cost + 50.00
WHERE customer_id = 123
AND cycle_start_date = '2025-01-01';

API & Network Issues

Provider API Timeout

Symptom: Sync fails with timeout error

Check Logs: 

grep -i "timeout\|connection" /var/www/expensis/var/log/prod.log | tail -50

Solutions:

  1. Increase Timeout: 

    // In HTTP client configuration
'timeout' => 60,  // Increase from 30

  2. Check Provider Status:

  3. Verify provider portal is accessible
  4. Check for maintenance windows

  5. Test API manually with curl

  6. Retry: 

    # Manual retry
php bin/console [provider]:sync --customer=123 --retry

Rate Limit Exceeded (Nina API)

Symptom: Addon activation fails with "Too many requests"

Root Cause: Exceeded 10 requests/minute limit for Routit Nina API

Check: 

SELECT
    customer_id,
    COUNT(*) as request_count,
    MAX(created_at) as last_request
FROM nina_request_limit
WHERE created_at > NOW() - INTERVAL 1 MINUTE
GROUP BY customer_id
HAVING COUNT(*) >= 10;

Solution: Wait 1 minute and retry

Prevention: Implement request throttling in UI

Database Issues

Deadlock During Totals Calculation

Symptom: Totals calculation fails with deadlock error

Check: 

SHOW ENGINE INNODB STATUS\G
-- Look for LATEST DETECTED DEADLOCK section

Solution: 1. Retry calculation (usually succeeds on retry) 2. If persistent, check for concurrent totals calculations: 

SELECT * FROM sync_task
WHERE status = 'processing'
AND updated_at > NOW() - INTERVAL 5 MINUTE;

Prevention: Ensure syncs for same customer don't run concurrently

Table Full (Partition Issue)

Symptom: CDR insert fails with "table is full"

Root Cause: CDR table partition full or not created for current month

Check: 

-- Check partition status
SELECT
    PARTITION_NAME,
    TABLE_ROWS,
    DATA_LENGTH / 1024 / 1024 as SIZE_MB
FROM INFORMATION_SCHEMA.PARTITIONS
WHERE TABLE_NAME = 'call_detail_record'
ORDER BY PARTITION_NAME DESC;

Solution: 

-- Create new partition for next month
ALTER TABLE call_detail_record
ADD PARTITION (
    PARTITION p202501 VALUES LESS THAN (UNIX_TIMESTAMP('2025-02-01'))
);

User-Triggered Issues

Manual Totals Recalculation Hangs

Symptom: User clicks "Recalculate" but page never refreshes

Root Cause: Command dispatched but handler failed silently

Check: 

# Check recent handler errors
tail -100 /var/www/expensis/var/log/prod.log | grep CalculateTotalsHandler

User Solution: 1. Wait 60 seconds 2. Refresh page manually 3. Check if totals were updated

Admin Solution: 

# Check task status
php bin/console sync:status --customer=123

# If stuck, trigger manually
php bin/console totals:new --customer=123 --cycle=2025-01-01

Debugging Commands

Check Sync Status

php bin/console sync:status --customer=123
php bin/console sync:history --customer=123 --limit=10

View Recent Errors

tail -100 /var/www/expensis/var/log/prod.log | grep ERROR

Test Specific Sync

php bin/console sync:manager --customer=123 --force -vvv

Verify Message Bus

php bin/console debug:messenger

Check Event Subscribers

php bin/console debug:event-dispatcher