NO.40 Kafka Connect is running on a two node cluster in distributed mode. The connector is a source connector that pulls data from Postgres tables (users/payment/orders), writes to topics with two partitions, and with replication factor two. The development team notices that the data is lagging behind.
What should be done to reduce the data lag*?
The Connector definition is listed below:
{
“name”: “confluent-postgresql-source”,
“connector class”: “PostgresSource”,
“topic.prefix”: “postgresql_”,
& nbsp;& nbsp;& nbsp;…
“db.name”: “postgres”,
“table.whitelist”: “users.payment.orders”,
“timestamp.column.name”: “created_at”,
“output.data format”: “JSON”,
“db.timezone”: “UTC”,
“tasks.max”: “1”
}
The connector is currently configured with “tasks.max”: “1”, which means only one task is handling all tables (users, payment, orders). This can create a bottleneck and lead to lag. Increasing tasks.max allows Kafka Connect to parallelize work across multiple tasks, which can pull data from different tables concurrently and reduce lag.