Sizing Guide
The consumer service is lightweight and does not require significant system resources under steady-state conditions. This guide helps you plan resource allocation for different scenarios.
Consumer Resource Requirements
Normal Operation
When the consumer is processing messages in real time and not significantly behind the upstream broker:
| Resource | Minimum | Recommended |
|---|---|---|
| vCPUs | 2 | 2-4 |
| Memory | 2 GB | 4 GB |
This configuration is sufficient for ongoing consumption workloads.
Initial Load or Backlog Processing
When the consumer needs to process a large backlog (for example, over 20 million messages):
| Resource | Minimum | Recommended |
|---|---|---|
| vCPUs | 4 | Up to 16 |
| Memory | 8 GB | Up to 32 GB |
This allows multiple consumers to run in parallel for efficient catch-up.
Once the backlog has been processed and the consumer has caught up to the upstream, resources can be reduced to the normal operating configuration.
Database Considerations
Database as Bottleneck
While the consumer can be scaled horizontally to process messages in parallel, the database is typically the largest bottleneck in the ingestion pipeline.
The database must persist all consumed records, and its performance is limited by:
- Transaction management
- Indexing
- Checkpointing
- Disk I/O
Key Insight
Adding more consumers beyond a certain point does not improve throughput and may instead reduce efficiency. The maximum sustainable throughput is determined by the database's ability to perform inserts in parallel, not necessarily by the number of consumers.
Database Resource Recommendations
| Scenario | vCPUs | Memory | Storage |
|---|---|---|---|
| Small (< 100 topics) | 4 | 8 GB | SSD |
| Medium (100-500 topics) | 8 | 16 GB | SSD |
| Large (> 500 topics) | 16+ | 32 GB+ | SSD |
Storage Planning
Main Data Tables
Estimate storage based on:
- Number of topics
- Average message size
- Data retention period
Audit Tables
If auditing is enabled:
- Plan for 2x or more the storage of main data tables
- Audit tables grow continuously
- Implement retention policies
Formula
Estimated Storage =
(Number of Topics × Avg Records × Avg Record Size) +
(Audit Factor × Main Data Size) +
20% Buffer
Network Requirements
| Requirement | Specification |
|---|---|
| Bandwidth | 100+ Mbps recommended |
| Latency | < 100ms to Kafka brokers |
| Firewall | Allow outbound to Kafka and Schema Registry |
Sample Sizing Scenarios
Scenario 1: Small Deployment
- Topics: 50
- Messages/day: 100,000
- Audit: Enabled
Recommended:
- Consumer: 2 vCPU, 4 GB RAM
- Database: 4 vCPU, 8 GB RAM, 100 GB SSD
Scenario 2: Medium Deployment
- Topics: 200
- Messages/day: 1,000,000
- Audit: Enabled
Recommended:
- Consumer: 4 vCPU, 8 GB RAM
- Database: 8 vCPU, 16 GB RAM, 500 GB SSD
Scenario 3: Large Deployment
- Topics: 500+
- Messages/day: 10,000,000+
- Audit: Enabled
Recommended:
- Consumers: Multiple instances, each 4 vCPU, 8 GB RAM
- Database: 16 vCPU, 32 GB RAM, 1+ TB SSD
Initial Load Sizing
For initial load scenarios with large data volumes:
Sample Workload (Internal Testing)
| Parameter | Value |
|---|---|
| Total topics | 144 |
| Average partitions per topic | 6 |
| Largest partition | 1 million messages |
| Total messages | 46 million |
Configurations Used
- Consumer instances: 8 (each with 4 GB memory and 4 CPU)
- Database instance: 8 CPU, 16 GB memory
- Network: Local, no firewalls or packet inspection
Observed Duration
| Workload | Duration |
|---|---|
| Total messages (46M) | 2 hours |
| With Audit (92M) | 3 hours 47 mins |
| With Full Audit (95M) | 4 hours 17 mins |
JVM Tuning
For production deployments, consider JVM options:
java \
-Xms2g \
-Xmx4g \
-XX:MaxDirectMemorySize=64m \
-XX:+UseG1GC \
-jar solifi-consumer-<version>.jar
JVM Settings Reference
| Setting | Purpose | Recommendation |
|---|---|---|
-Xms | Initial heap | 50% of container memory |
-Xmx | Maximum heap | 75% of container memory |
-XX:MaxDirectMemorySize | Direct memory | 64m-128m |
-XX:+UseG1GC | Garbage collector | Recommended for large heaps |
Monitoring for Sizing
Key metrics to monitor for right-sizing:
| Metric | Healthy Range | Action if Exceeded |
|---|---|---|
| Heap usage | < 80% | Increase -Xmx |
| CPU usage | < 70% | Increase vCPUs |
| Consumer lag | Decreasing | Add consumers |
| DB connections | < pool max | Increase pool size |
Next Steps
- Configure health monitoring
- Review the changelog
- Learn about scaling