Scaling Consumer Application

Learn how to scale the Solifi Consumer application for optimal performance.

Understanding Kafka Scaling

Scaling is highly dependent on the number of partitions in a Kafka topic. According to Kafka concepts, consumers can be defined within a consumer group to listen to a topic in parallel.

Key Insight

If a topic has 5 partitions, then starting 5 consumers under the same consumer group ID will enable parallel processing by different consumers.

Scaling with Concurrency

Rather than starting separate consumer instances individually, the Solifi Consumer application supports scaling by adjusting the number of concurrent threads/listeners using the solifi.concurrency configuration.

solifi:
  concurrency: 5  # Equivalent to having 5 consumers listening to the topics

Concurrency Guidelines

Scenario	Recommended Concurrency
Default	Server CPU count (usually 8-10)
Optimal	Equal to topic partition count
High-volume topics	Match to partition count

Scaling with Multiple Instances

For topics with higher TPS (transactions per second), you can:

1. Start a separate consumer instance dedicated to high-volume topic(s)
2. Set the thread count (solifi.concurrency) equal to the number of partitions

Example: High-Volume Topic

# consumer-high-volume.yml
solifi:
  topics:
    - high_volume_topic
  concurrency: 6  # Matches the 6 partitions of high_volume_topic

Resource Sizing Guide

The consumer service is lightweight and doesn't require significant resources under steady-state conditions.

Normal Operation

When the consumer is processing messages in real time and not significantly behind:

Resource	Minimum
vCPUs	2
Memory	2 GB

Initial Load / Backlog Processing

When processing a large backlog (e.g., over 20 million messages):

Resource	Recommended
vCPUs	Up to 16
Memory	Up to 32 GB

This allows multiple consumers to run in parallel for efficient catch-up.

Resource Reduction

Once the backlog is processed and the consumer has caught up, resources can be reduced to normal operating configuration.

Database Bottleneck

While the consumer can be scaled horizontally, the database is typically the largest bottleneck in the ingestion pipeline.

Why Databases Become Bottlenecks

The database must persist all consumed records, and its performance is limited by:

• Transaction management
• Indexing
• Checkpointing
• Disk I/O

Key Insight

Adding more consumers beyond a certain point does not improve throughput and may instead reduce efficiency. The maximum sustainable throughput is determined by the database's ability to perform inserts in parallel, not necessarily by the number of consumers.

Database Optimization Tips

1. Use appropriate indexes - Don't over-index
2. Configure connection pooling - Match pool size to concurrency
3. Use SSD storage - Faster I/O for writes
4. Monitor lock contention - Especially during high-volume writes
5. Consider partitioned tables - For very large datasets

Kubernetes Scaling

Manual Scaling

kubectl scale deployment solifi-consumer --replicas=3

Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: solifi-consumer-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: solifi-consumer
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

Handling Date & Time

As recommended, all date and time should be stored in UTC. This solves many problems when dealing with date/time, especially in regions with Daylight Saving Time (DST).

solifi:
  timezone:  # Leave empty or omit to store in UTC (recommended)

Best Practice

Keep all values in UTC and don't specify the timezone property. Handle timezone conversion in your reporting/application layer.

Scaling Best Practices

1. Monitor before scaling - Use Prometheus/Grafana to identify actual bottlenecks
2. Scale incrementally - Add resources gradually and measure impact
3. Watch consumer lag - Key metric for determining scaling needs
4. Consider database capacity - Often the limiting factor
5. Use dedicated consumers - For high-volume topics
6. Match concurrency to partitions - Optimal resource utilization

Next Steps

• Configure initial load mode for large data migrations
• Set up health monitoring
• Understand auditing