io.druid.indexing.overlord

Class RemoteTaskRunner

java.lang.Object

io.druid.indexing.overlord.RemoteTaskRunner

All Implemented Interfaces:

TaskRunner, io.druid.tasklogs.TaskLogStreamer

public class RemoteTaskRunner
extends Object
implements TaskRunner, io.druid.tasklogs.TaskLogStreamer

The RemoteTaskRunner's primary responsibility is to assign tasks to worker nodes. The RemoteTaskRunner uses Zookeeper to keep track of which workers are running which tasks. Tasks are assigned by creating ephemeral nodes in ZK that workers must remove. Workers announce the statuses of the tasks they are running. Once a task completes, it is up to the RTR to remove the task status and run any necessary cleanup. The RemoteTaskRunner is event driven and updates state according to ephemeral node changes in ZK.

The RemoteTaskRunner will assign tasks to a node until the node hits capacity. At that point, task assignment will fail. The RemoteTaskRunner depends on another component to create additional worker resources. For example, ResourceManagementScheduler can take care of these duties.

If a worker node becomes inexplicably disconnected from Zk, the RemoteTaskRunner will fail any tasks associated with the worker after waiting for RemoteTaskRunnerConfig.taskCleanupTimeout for the worker to show up.

The RemoteTaskRunner uses ZK for job management and assignment and http for IPC messages.

Constructor Summary

Constructors

Constructor and Description
RemoteTaskRunner(com.fasterxml.jackson.databind.ObjectMapper jsonMapper, RemoteTaskRunnerConfig config, IndexerZkConfig indexerZkConfig, org.apache.curator.framework.CuratorFramework cf, PathChildrenCacheFactory pathChildrenCacheFactory, com.metamx.http.client.HttpClient httpClient, com.google.common.base.Supplier<WorkerBehaviorConfig> workerConfigRef, ScheduledExecutorService cleanupExec)

Method Summary

Methods

Modifier and Type	Method and Description
ZkWorker	findWorkerRunningTask(String taskId)
Collection<RemoteTaskRunnerWorkItem>	getKnownTasks()
List<ZkWorker>	getLazyWorkers()
Collection<RemoteTaskRunnerWorkItem>	getPendingTasks()
Collection<RemoteTaskRunnerWorkItem>	getRunningTasks()
Collection<ZkWorker>	getWorkers()
boolean	isWorkerRunningTask(Worker worker, String taskId)
List<ZkWorker>	markWorkersLazy(com.google.common.base.Predicate<ZkWorker> isLazyWorker, int maxWorkers)
com.google.common.util.concurrent.ListenableFuture<TaskStatus>	run(Task task) A task will be run only if there is no current knowledge in the RemoteTaskRunner of the task.
void	shutdown(String taskId) Finds the worker running the task and forwards the shutdown signal to the worker.
void	start()
void	stop()
com.google.common.base.Optional<com.google.common.io.ByteSource>	streamTaskLog(String taskId, long offset)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RemoteTaskRunner

public RemoteTaskRunner(com.fasterxml.jackson.databind.ObjectMapper jsonMapper,
                RemoteTaskRunnerConfig config,
                IndexerZkConfig indexerZkConfig,
                org.apache.curator.framework.CuratorFramework cf,
                PathChildrenCacheFactory pathChildrenCacheFactory,
                com.metamx.http.client.HttpClient httpClient,
                com.google.common.base.Supplier<WorkerBehaviorConfig> workerConfigRef,
                ScheduledExecutorService cleanupExec)

Method Detail

start

public void start()

stop

public void stop()

getWorkers

public Collection<ZkWorker> getWorkers()

Specified by:

getWorkers in interface TaskRunner

getRunningTasks

public Collection<RemoteTaskRunnerWorkItem> getRunningTasks()

Specified by:

getRunningTasks in interface TaskRunner

getPendingTasks

public Collection<RemoteTaskRunnerWorkItem> getPendingTasks()

Specified by:

getPendingTasks in interface TaskRunner

getKnownTasks

public Collection<RemoteTaskRunnerWorkItem> getKnownTasks()

Specified by:

getKnownTasks in interface TaskRunner

findWorkerRunningTask

public ZkWorker findWorkerRunningTask(String taskId)

isWorkerRunningTask

public boolean isWorkerRunningTask(Worker worker,
                          String taskId)

run

public com.google.common.util.concurrent.ListenableFuture<TaskStatus> run(Task task)

A task will be run only if there is no current knowledge in the RemoteTaskRunner of the task.

Specified by:

run in interface TaskRunner

Parameters:

task - task to run

Returns:

task status, eventually

shutdown

public void shutdown(String taskId)

Finds the worker running the task and forwards the shutdown signal to the worker.

Specified by:

shutdown in interface TaskRunner

Parameters:

taskId - - task id to shutdown

streamTaskLog

public com.google.common.base.Optional<com.google.common.io.ByteSource> streamTaskLog(String taskId,
                                                                             long offset)

Specified by:

streamTaskLog in interface io.druid.tasklogs.TaskLogStreamer

markWorkersLazy

public List<ZkWorker> markWorkersLazy(com.google.common.base.Predicate<ZkWorker> isLazyWorker,
                             int maxWorkers)

getLazyWorkers

public List<ZkWorker> getLazyWorkers()