io.druid.indexing.overlord

Class RemoteTaskRunner

java.lang.Object

io.druid.indexing.overlord.RemoteTaskRunner

All Implemented Interfaces:

TaskRunner, io.druid.tasklogs.TaskLogStreamer

public class RemoteTaskRunner
extends Object
implements TaskRunner, io.druid.tasklogs.TaskLogStreamer

The RemoteTaskRunner's primary responsibility is to assign tasks to worker nodes. The RemoteTaskRunner uses Zookeeper to keep track of which workers are running which tasks. Tasks are assigned by creating ephemeral nodes in ZK that workers must remove. Workers announce the statuses of the tasks they are running. Once a task completes, it is up to the RTR to remove the task status and run any necessary cleanup. The RemoteTaskRunner is event driven and updates state according to ephemeral node changes in ZK.

The RemoteTaskRunner will assign tasks to a node until the node hits capacity. At that point, task assignment will fail. The RemoteTaskRunner depends on another component to create additional worker resources.

If a worker node becomes inexplicably disconnected from Zk, the RemoteTaskRunner will fail any tasks associated with the worker after waiting for RemoteTaskRunnerConfig.taskCleanupTimeout for the worker to show up.

The RemoteTaskRunner uses ZK for job management and assignment and http for IPC messages.

Constructor Summary

Constructors

Constructor and Description

RemoteTaskRunner(com.fasterxml.jackson.databind.ObjectMapper jsonMapper, RemoteTaskRunnerConfig config, IndexerZkConfig indexerZkConfig, org.apache.curator.framework.CuratorFramework cf, PathChildrenCacheFactory pathChildrenCacheFactory, com.metamx.http.client.HttpClient httpClient, com.google.common.base.Supplier<WorkerBehaviorConfig> workerConfigRef, ScheduledExecutorService cleanupExec, ResourceManagementStrategy<RemoteTaskRunner> resourceManagement)

Method Summary

Modifier and Type	Method and Description
ZkWorker	findWorkerRunningTask(String taskId)
Collection<RemoteTaskRunnerWorkItem>	getKnownTasks()
List<ZkWorker>	getLazyWorkers()
Collection<RemoteTaskRunnerWorkItem>	getPendingTasks()
Collection<RemoteTaskRunnerWorkItem>	getRunningTasks()
com.google.common.base.Optional<ScalingStats>	getScalingStats() Some runners are able to scale up and down their capacity in a dynamic manner.
Collection<ZkWorker>	getWorkers()
boolean	isWorkerRunningTask(Worker worker, String taskId)
List<ZkWorker>	markWorkersLazy(com.google.common.base.Predicate<ZkWorker> isLazyWorker, int maxWorkers)
List<com.metamx.common.Pair<Task,com.google.common.util.concurrent.ListenableFuture<TaskStatus>>>	restore() Some task runners can restart previously-running tasks after being bounced.
com.google.common.util.concurrent.ListenableFuture<TaskStatus>	run(Task task) A task will be run only if there is no current knowledge in the RemoteTaskRunner of the task.
void	shutdown(String taskId) Finds the worker running the task and forwards the shutdown signal to the worker.
void	start()
void	stop() Stop this task runner.
com.google.common.base.Optional<com.google.common.io.ByteSource>	streamTaskLog(String taskId, long offset)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RemoteTaskRunner

public RemoteTaskRunner(com.fasterxml.jackson.databind.ObjectMapper jsonMapper,
                        RemoteTaskRunnerConfig config,
                        IndexerZkConfig indexerZkConfig,
                        org.apache.curator.framework.CuratorFramework cf,
                        PathChildrenCacheFactory pathChildrenCacheFactory,
                        com.metamx.http.client.HttpClient httpClient,
                        com.google.common.base.Supplier<WorkerBehaviorConfig> workerConfigRef,
                        ScheduledExecutorService cleanupExec,
                        ResourceManagementStrategy<RemoteTaskRunner> resourceManagement)

Method Detail

start

public void start()

stop

public void stop()

Description copied from interface: TaskRunner

Stop this task runner. This may block until currently-running tasks can be gracefully stopped. After calling stopping, "run" will not accept further tasks.

Specified by:

stop in interface TaskRunner

restore

public List<com.metamx.common.Pair<Task,com.google.common.util.concurrent.ListenableFuture<TaskStatus>>> restore()

Description copied from interface: TaskRunner

Some task runners can restart previously-running tasks after being bounced. This method does that, and returns the list of tasks (and status futures).

Specified by:

restore in interface TaskRunner

getWorkers

public Collection<ZkWorker> getWorkers()

getRunningTasks

public Collection<RemoteTaskRunnerWorkItem> getRunningTasks()

Specified by:

getRunningTasks in interface TaskRunner

getPendingTasks

public Collection<RemoteTaskRunnerWorkItem> getPendingTasks()

Specified by:

getPendingTasks in interface TaskRunner

getKnownTasks

public Collection<RemoteTaskRunnerWorkItem> getKnownTasks()

Specified by:

getKnownTasks in interface TaskRunner

getScalingStats

public com.google.common.base.Optional<ScalingStats> getScalingStats()

Description copied from interface: TaskRunner

Some runners are able to scale up and down their capacity in a dynamic manner. This returns stats on those activities

Specified by:

getScalingStats in interface TaskRunner

Returns:

ScalingStats if the runner has an underlying resource which can scale, Optional.absent() otherwise

findWorkerRunningTask

public ZkWorker findWorkerRunningTask(String taskId)

isWorkerRunningTask

public boolean isWorkerRunningTask(Worker worker,
                                   String taskId)

run

public com.google.common.util.concurrent.ListenableFuture<TaskStatus> run(Task task)

A task will be run only if there is no current knowledge in the RemoteTaskRunner of the task.

Specified by:

run in interface TaskRunner

Parameters:

task - task to run

Returns:

task status, eventually

shutdown

public void shutdown(String taskId)

Finds the worker running the task and forwards the shutdown signal to the worker.

Specified by:

shutdown in interface TaskRunner

Parameters:

taskId - - task id to shutdown

streamTaskLog

public com.google.common.base.Optional<com.google.common.io.ByteSource> streamTaskLog(String taskId,
                                                                                      long offset)

Specified by:

streamTaskLog in interface io.druid.tasklogs.TaskLogStreamer

markWorkersLazy

public List<ZkWorker> markWorkersLazy(com.google.common.base.Predicate<ZkWorker> isLazyWorker,
                                      int maxWorkers)

getLazyWorkers

public List<ZkWorker> getLazyWorkers()