Error Handling

The consumer application captures data-related errors and provides mechanisms for tracking and resolving failed messages.

Error Log Table

The consumer automatically creates and manages an error_log table in your database. Any data-related errors (e.g., invalid data) are captured and stored in this table.

Key Points

• The table is created automatically during consumer startup
• Failed messages are logged with details for troubleshooting
• Messages in this table are not automatically cleaned up
• Customers should monitor this table and handle cleanup as needed

Monitoring Errors

Query Recent Errors

SELECT TOP 100 *
FROM error_log
ORDER BY created_at DESC;

Query Errors by Topic

SELECT topic_name, COUNT(*) as error_count
FROM error_log
GROUP BY topic_name
ORDER BY error_count DESC;

Query Error Details

SELECT 
  topic_name,
  error_message,
  created_at
FROM error_log
WHERE created_at > DATEADD(day, -1, GETDATE())
ORDER BY created_at DESC;

Common Error Types

Error Type	Description	Resolution
Data validation	Invalid data format	Review source data
Schema mismatch	Table schema doesn't match message	Consumer will auto-adjust
Constraint violation	Duplicate key or FK violation	Check data integrity
Connection timeout	Database connection lost	Check database connectivity

Error Handling Behavior

When the consumer encounters an error:

1. Error is logged to the error_log table
2. Message offset is committed (error won't be retried automatically)
3. Processing continues with the next message
4. Improved logging (v2.1.0+) provides better error messaging

Version 2.1.0 Improvements

• Improved error logging with better messaging for database, network, or business exceptions
• Updated error handling to clean up messages left behind in the MDC via other threads

Handling Failed Messages

Option 1: Manual Reprocessing

1. Identify the failed message from error_log
2. Fix the underlying data issue
3. Manually insert/update the record in the target table

Option 2: Data Refresh

For widespread issues, consider a data refresh to reprocess all messages.

Option 3: Contact Support

For persistent or unexplained errors, contact LimePoint Support.

Cleanup Recommendations

Since error logs are not automatically cleaned, implement a cleanup strategy:

-- Delete errors older than 90 days
DELETE FROM error_log
WHERE created_at < DATEADD(day, -90, GETDATE());

Or archive before deletion:

-- Archive to separate table
INSERT INTO error_log_archive
SELECT * FROM error_log
WHERE created_at < DATEADD(day, -90, GETDATE());

-- Then delete
DELETE FROM error_log
WHERE created_at < DATEADD(day, -90, GETDATE());

Logging Configuration

Configure logging levels in application.yml:

logging:
  level:
    com.limepoint.solifi: INFO  # Use DEBUG for more detail

DEBUG Mode

When set to DEBUG, the consumer will log records in clear text. These records may contain sensitive data. Use DEBUG mode only sparingly.

Log Output Examples

Info Level (Normal Operation)

INFO  - Processed 1000 records from topic cs_master_nf

Error Level (When Issues Occur)

ERROR - Failed to process record: Invalid date format in column 'effective_date'

Debug Level (Detailed Troubleshooting)

DEBUG - Received ConsumerRecord: key=12345, partition=0, offset=5678
DEBUG - Processing record: {id: 12345, name: "Example"}

Best Practices

1. Monitor regularly - Check error_log table daily
2. Set up alerts - Create database alerts for new errors
3. Document resolutions - Track how errors are resolved
4. Clean up periodically - Implement retention policies
5. Use appropriate logging - Don't leave DEBUG on in production

Next Steps

• Configure health monitoring
• Understand data refresh
• Review the sizing guide