io.druid.indexing.kafka

Class KafkaIndexTask

java.lang.Object

io.druid.indexing.common.task.AbstractTask

io.druid.indexing.kafka.KafkaIndexTask

All Implemented Interfaces:

Task, ChatHandler

public class KafkaIndexTask
extends AbstractTask
implements ChatHandler

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	KafkaIndexTask.Status

Field Summary

Fields

Modifier and Type	Field and Description
static long	PAUSE_FOREVER

Constructor Summary

Constructors

Constructor and Description
KafkaIndexTask(String id, TaskResource taskResource, DataSchema dataSchema, KafkaTuningConfig tuningConfig, KafkaIOConfig ioConfig, Map<String,Object> context, ChatHandlerProvider chatHandlerProvider, AuthorizerMapper authorizerMapper)

Method Summary

Modifier and Type	Method and Description
boolean	canRestore() Returns whether or not this task can restore its progress from its on-disk working directory.
Map<Integer,Map<Integer,Long>>	getCheckpoints()
Map<Integer,Map<Integer,Long>>	getCheckpointsHTTP(javax.servlet.http.HttpServletRequest req)
Map<Integer,Long>	getCurrentOffsets()
Map<Integer,Long>	getCurrentOffsets(javax.servlet.http.HttpServletRequest req)
DataSchema	getDataSchema()
Map<Integer,Long>	getEndOffsets()
Map<Integer,Long>	getEndOffsetsHTTP(javax.servlet.http.HttpServletRequest req)
KafkaIOConfig	getIOConfig()
int	getPriority() Returns task priority.
<T> QueryRunner<T>	getQueryRunner(Query<T> query) Returns query runners for this task.
org.joda.time.DateTime	getStartTime(javax.servlet.http.HttpServletRequest req)
KafkaIndexTask.Status	getStatus()
KafkaIndexTask.Status	getStatusHTTP(javax.servlet.http.HttpServletRequest req)
KafkaTuningConfig	getTuningConfig()
String	getType() Returns a descriptive label for this task type.
boolean	isReady(TaskActionClient taskActionClient) Execute preflight actions for a task.
javax.ws.rs.core.Response	pause(long timeout)
javax.ws.rs.core.Response	pauseHTTP(long timeout, javax.servlet.http.HttpServletRequest req) Signals the ingestion loop to pause.
void	resume()
javax.ws.rs.core.Response	resumeHTTP(javax.servlet.http.HttpServletRequest req)
TaskStatus	run(TaskToolbox toolbox) Execute a task.
javax.ws.rs.core.Response	setEndOffsets(Map<Integer,Long> offsets, boolean resume, boolean finish)
javax.ws.rs.core.Response	setEndOffsetsHTTP(Map<Integer,Long> offsets, boolean resume, boolean finish, javax.servlet.http.HttpServletRequest req)
javax.ws.rs.core.Response	stop(javax.servlet.http.HttpServletRequest req)
void	stopGracefully() Asks a task to arrange for its "run" method to exit promptly.

Methods inherited from class io.druid.indexing.common.task.AbstractTask

equals, getClasspathPrefix, getContext, getDataSource, getGroupId, getId, getNodeType, getTaskResource, hashCode, success, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface io.druid.indexing.common.task.Task

getContextValue, getContextValue

Field Detail

PAUSE_FOREVER

public static final long PAUSE_FOREVER

See Also:

Constant Field Values

Constructor Detail

KafkaIndexTask

public KafkaIndexTask(String id,
                      TaskResource taskResource,
                      DataSchema dataSchema,
                      KafkaTuningConfig tuningConfig,
                      KafkaIOConfig ioConfig,
                      Map<String,Object> context,
                      ChatHandlerProvider chatHandlerProvider,
                      AuthorizerMapper authorizerMapper)

Method Detail

getPriority

public int getPriority()

Description copied from interface: Task

Returns task priority. The task priority is currently used only for prioritized locking, but, in the future, it can be used for task scheduling, cluster resource management, etc.

Specified by:

getPriority in interface Task

Returns:

task priority

See Also:

for default task priorities

getType

public String getType()

Description copied from interface: Task

Returns a descriptive label for this task type. Used for metrics emission and logging.

Specified by:

getType in interface Task

Returns:

task type label

isReady

public boolean isReady(TaskActionClient taskActionClient)
                throws Exception

Description copied from interface: Task

Execute preflight actions for a task. This can be used to acquire locks, check preconditions, and so on. The actions must be idempotent, since this method may be executed multiple times. This typically runs on the coordinator. If this method throws an exception, the task should be considered a failure.

This method must be idempotent, as it may be run multiple times per task.

Specified by:

isReady in interface Task

Parameters:

taskActionClient - action client for this task (not the full toolbox)

Returns:

true if ready, false if not ready yet

Throws:

Exception - if the task should be considered a failure

getDataSchema

public DataSchema getDataSchema()

getTuningConfig

public KafkaTuningConfig getTuningConfig()

getIOConfig

public KafkaIOConfig getIOConfig()

run

public TaskStatus run(TaskToolbox toolbox)
               throws Exception

Description copied from interface: Task

Execute a task. This typically runs on a worker as determined by a TaskRunner, and will be run while holding a lock on our dataSource and implicit lock interval (if any). If this method throws an exception, the task should be considered a failure.

Specified by:

run in interface Task

Parameters:

toolbox - Toolbox for this task

Returns:

Some kind of finished status (isRunnable must be false).

Throws:

Exception - if this task failed

canRestore

public boolean canRestore()

Description copied from interface: Task

Returns whether or not this task can restore its progress from its on-disk working directory. Restorable tasks may be started with a non-empty working directory. Tasks that exit uncleanly may still have a chance to attempt restores, meaning that restorable tasks should be able to deal with potentially partially written on-disk state.

Specified by:

canRestore in interface Task

Overrides:

canRestore in class AbstractTask

stopGracefully

public void stopGracefully()

Description copied from interface: Task

Asks a task to arrange for its "run" method to exit promptly. This method will only be called if Task.canRestore() returns true. Tasks that take too long to stop gracefully will be terminated with extreme prejudice.

Specified by:

stopGracefully in interface Task

Overrides:

stopGracefully in class AbstractTask

getQueryRunner

public <T> QueryRunner<T> getQueryRunner(Query<T> query)

Description copied from interface: Task

Returns query runners for this task. If this task is not meant to answer queries over its datasource, this method should return null.

Specified by:

getQueryRunner in interface Task

Overrides:

getQueryRunner in class AbstractTask

Type Parameters:

T - query result type

Returns:

query runners for this task

stop

public javax.ws.rs.core.Response stop(@Context
                                      javax.servlet.http.HttpServletRequest req)

getStatusHTTP

public KafkaIndexTask.Status getStatusHTTP(@Context
                                           javax.servlet.http.HttpServletRequest req)

getStatus

public KafkaIndexTask.Status getStatus()

getCurrentOffsets

public Map<Integer,Long> getCurrentOffsets(@Context
                                           javax.servlet.http.HttpServletRequest req)

getCurrentOffsets

public Map<Integer,Long> getCurrentOffsets()

getEndOffsetsHTTP

public Map<Integer,Long> getEndOffsetsHTTP(@Context
                                           javax.servlet.http.HttpServletRequest req)

getEndOffsets

public Map<Integer,Long> getEndOffsets()

setEndOffsetsHTTP

public javax.ws.rs.core.Response setEndOffsetsHTTP(Map<Integer,Long> offsets,
                                                   boolean resume,
                                                   boolean finish,
                                                   @Context
                                                   javax.servlet.http.HttpServletRequest req)
                                            throws InterruptedException

Throws:

InterruptedException

setEndOffsets

public javax.ws.rs.core.Response setEndOffsets(Map<Integer,Long> offsets,
                                               boolean resume,
                                               boolean finish)
                                        throws InterruptedException

Throws:

InterruptedException

getCheckpointsHTTP

public Map<Integer,Map<Integer,Long>> getCheckpointsHTTP(@Context
                                                         javax.servlet.http.HttpServletRequest req)

getCheckpoints

public Map<Integer,Map<Integer,Long>> getCheckpoints()

pauseHTTP

public javax.ws.rs.core.Response pauseHTTP(long timeout,
                                           @Context
                                           javax.servlet.http.HttpServletRequest req)
                                    throws InterruptedException

Signals the ingestion loop to pause.

Parameters:

timeout - how long to pause for before resuming in milliseconds, <= 0 means indefinitely

Returns:

one of the following Responses: 400 Bad Request if the task has started publishing; 202 Accepted if the method has timed out and returned before the task has paused; 200 OK with a map of the current partition offsets in the response body if the task successfully paused

Throws:

InterruptedException

pause

public javax.ws.rs.core.Response pause(long timeout)
                                throws InterruptedException

Throws:

InterruptedException

resumeHTTP

public javax.ws.rs.core.Response resumeHTTP(@Context
                                            javax.servlet.http.HttpServletRequest req)
                                     throws InterruptedException

Throws:

InterruptedException

resume

public void resume()
            throws InterruptedException

Throws:

InterruptedException

getStartTime

public org.joda.time.DateTime getStartTime(@Context
                                           javax.servlet.http.HttpServletRequest req)