public class KafkaSupervisor extends Object implements Supervisor
KafkaSupervisorSpec
which includes the Kafka topic and configuration as well as an ingestion spec which will
be used to generate the indexing tasks. The run loop periodically refreshes its view of the Kafka topic's partitions
and the list of running indexing tasks and ensures that all partitions are being read from and that there are enough
tasks to satisfy the desired number of replicas. As tasks complete, new tasks are queued to process the next range of
Kafka offsets.Modifier and Type | Field and Description |
---|---|
static String |
IS_INCREMENTAL_HANDOFF_SUPPORTED |
Constructor and Description |
---|
KafkaSupervisor(TaskStorage taskStorage,
TaskMaster taskMaster,
IndexerMetadataStorageCoordinator indexerMetadataStorageCoordinator,
KafkaIndexTaskClientFactory taskClientFactory,
com.fasterxml.jackson.databind.ObjectMapper mapper,
KafkaSupervisorSpec spec) |
Modifier and Type | Method and Description |
---|---|
void |
checkpoint(String sequenceName,
DataSourceMetadata previousCheckpoint,
DataSourceMetadata currentCheckpoint)
The definition of checkpoint is not very strict as currently it does not affect data or control path.
|
SupervisorReport |
getStatus() |
protected int |
getTaskGroupIdForPartition(int partition) |
void |
possiblyRegisterListener() |
void |
reset(DataSourceMetadata dataSourceMetadata) |
void |
start() |
void |
stop(boolean stopGracefully) |
public static final String IS_INCREMENTAL_HANDOFF_SUPPORTED
public KafkaSupervisor(TaskStorage taskStorage, TaskMaster taskMaster, IndexerMetadataStorageCoordinator indexerMetadataStorageCoordinator, KafkaIndexTaskClientFactory taskClientFactory, com.fasterxml.jackson.databind.ObjectMapper mapper, KafkaSupervisorSpec spec)
public void start()
start
in interface Supervisor
public void stop(boolean stopGracefully)
stop
in interface Supervisor
stopGracefully
- If true, supervisor will cleanly shutdown managed tasks if possible (for example signalling
them to publish their segments and exit). The implementation may block until the tasks have
either acknowledged or completed. If false, supervisor will stop immediately and leave any
running tasks as they are.public SupervisorReport getStatus()
getStatus
in interface Supervisor
public void reset(DataSourceMetadata dataSourceMetadata)
reset
in interface Supervisor
public void checkpoint(String sequenceName, DataSourceMetadata previousCheckpoint, DataSourceMetadata currentCheckpoint)
Supervisor
checkpoint
in interface Supervisor
sequenceName
- unique Identifier to figure out for which sequence to do checkpointingpreviousCheckpoint
- DataSourceMetadata checkpointed in previous callcurrentCheckpoint
- current DataSourceMetadata to be checkpointedpublic void possiblyRegisterListener()
protected int getTaskGroupIdForPartition(int partition)
Copyright © 2011–2018. All rights reserved.