io.druid.indexing.kafka.supervisor

Class KafkaSupervisor

java.lang.Object

io.druid.indexing.kafka.supervisor.KafkaSupervisor

All Implemented Interfaces:

Supervisor

public class KafkaSupervisor
extends Object
implements Supervisor

Supervisor responsible for managing the KafkaIndexTasks for a single dataSource. At a high level, the class accepts a KafkaSupervisorSpec which includes the Kafka topic and configuration as well as an ingestion spec which will be used to generate the indexing tasks. The run loop periodically refreshes its view of the Kafka topic's partitions and the list of running indexing tasks and ensures that all partitions are being read from and that there are enough tasks to satisfy the desired number of replicas. As tasks complete, new tasks are queued to process the next range of Kafka offsets.

Field Summary

Fields

Modifier and Type	Field and Description
static String	IS_INCREMENTAL_HANDOFF_SUPPORTED

Constructor Summary

Constructors

Constructor and Description
KafkaSupervisor(TaskStorage taskStorage, TaskMaster taskMaster, IndexerMetadataStorageCoordinator indexerMetadataStorageCoordinator, KafkaIndexTaskClientFactory taskClientFactory, com.fasterxml.jackson.databind.ObjectMapper mapper, KafkaSupervisorSpec spec)

Method Summary

Modifier and Type	Method and Description
void	checkpoint(String sequenceName, DataSourceMetadata previousCheckpoint, DataSourceMetadata currentCheckpoint) The definition of checkpoint is not very strict as currently it does not affect data or control path.
SupervisorReport	getStatus()
protected int	getTaskGroupIdForPartition(int partition)
void	possiblyRegisterListener()
void	reset(DataSourceMetadata dataSourceMetadata)
void	start()
void	stop(boolean stopGracefully)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

IS_INCREMENTAL_HANDOFF_SUPPORTED

public static final String IS_INCREMENTAL_HANDOFF_SUPPORTED

See Also:

Constant Field Values

Constructor Detail

KafkaSupervisor

public KafkaSupervisor(TaskStorage taskStorage,
                       TaskMaster taskMaster,
                       IndexerMetadataStorageCoordinator indexerMetadataStorageCoordinator,
                       KafkaIndexTaskClientFactory taskClientFactory,
                       com.fasterxml.jackson.databind.ObjectMapper mapper,
                       KafkaSupervisorSpec spec)

Method Detail

start

public void start()

Specified by:

start in interface Supervisor

stop

public void stop(boolean stopGracefully)

Specified by:

stop in interface Supervisor

Parameters:

stopGracefully - If true, supervisor will cleanly shutdown managed tasks if possible (for example signalling them to publish their segments and exit). The implementation may block until the tasks have either acknowledged or completed. If false, supervisor will stop immediately and leave any running tasks as they are.

getStatus

public SupervisorReport getStatus()

Specified by:

getStatus in interface Supervisor

reset

public void reset(DataSourceMetadata dataSourceMetadata)

Specified by:

reset in interface Supervisor

checkpoint

public void checkpoint(String sequenceName,
                       DataSourceMetadata previousCheckpoint,
                       DataSourceMetadata currentCheckpoint)

Description copied from interface: Supervisor

The definition of checkpoint is not very strict as currently it does not affect data or control path. On this call Supervisor can potentially checkpoint data processed so far to some durable storage for example - Kafka Supervisor uses this to merge and handoff segments containing at least the data represented by DataSourceMetadata

Specified by:

checkpoint in interface Supervisor

Parameters:

sequenceName - unique Identifier to figure out for which sequence to do checkpointing

previousCheckpoint - DataSourceMetadata checkpointed in previous call

currentCheckpoint - current DataSourceMetadata to be checkpointed

possiblyRegisterListener

public void possiblyRegisterListener()

getTaskGroupIdForPartition

protected int getTaskGroupIdForPartition(int partition)