Horizontally scaling and vertically scaling. — Key Concepts in Kafka Apache Kafka is a publish-subscribe messaging system which lets you send messages between processes, applications, and servers. Applications which need to read data from Kafka use a KafkaConsumer to subscribe to Kafka topics and receive messages from these topics. Reading data from Kafka is a bit different than reading data…

A common problem in streaming applications Kafka is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system which is widely adopted by enterprises. Kafka has the nature of zero-copy, which does not load data into memory and makes Kafka operate fast. When Kafka gets data from a producer, it takes bytes as input and publishes them…

Get started with Apache Spark Structure Streaming for your realtime applications. Why Spark Spark offers a compute engine which easily scales big data applications. …

Diagnose and decrease AWS Lambda iterator age for stream-based invocations. AWS Lambda can use stream based services as invocation sources, essentially making your Lambda function a consumer of streams such as Kafka stream, Kinesis or DynamoDB streams. If you are dealing with high volume data, you probably have to fine-tune…