io.druid.segment.realtime.appenderator

Class AppenderatorDriver

java.lang.Object

io.druid.segment.realtime.appenderator.AppenderatorDriver

All Implemented Interfaces:

Closeable, AutoCloseable

public class AppenderatorDriver
extends Object
implements Closeable

A AppenderatorDriver drives an Appenderator to index a finite stream of data. This class does not help you index unbounded streams. All handoff is done at the end of indexing. This class helps with doing things that Appenderators don't, including deciding which segments to use (with a SegmentAllocator), publishing segments to the metadata store (with a SegmentPublisher), and monitoring handoff (with a SegmentHandoffNotifier). Note that the commit metadata stored by this class via the underlying Appenderator is not the same metadata as you pass in. It's wrapped in some extra metadata needed by the driver.

Constructor Summary

Constructors

Constructor and Description
AppenderatorDriver(Appenderator appenderator, SegmentAllocator segmentAllocator, SegmentHandoffNotifierFactory handoffNotifierFactory, UsedSegmentChecker usedSegmentChecker, com.fasterxml.jackson.databind.ObjectMapper objectMapper, FireDepartmentMetrics metrics) Create a driver.

Method Summary

Modifier and Type	Method and Description
AppenderatorDriverAddResult	add(InputRow row, String sequenceName, com.google.common.base.Supplier<Committer> committerSupplier) Add a row.
void	clear() Clears out all our state and also calls Appenderator.clear() on the underlying Appenderator.
void	close() Closes this driver.
void	moveSegmentOut(String sequenceName, List<SegmentIdentifier> identifiers) Move a set of identifiers out from "active", making way for newer segments.
Object	persist(Committer committer) Persist all data indexed through this driver so far.
com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata>	publish(TransactionalSegmentPublisher publisher, Committer committer, Collection<String> sequenceNames) Execute a task in background to publish all segments corresponding to the given sequence names.
com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata>	publishAndRegisterHandoff(TransactionalSegmentPublisher publisher, Committer committer, Collection<String> sequenceNames)
com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata>	registerHandoff(SegmentsAndMetadata segmentsAndMetadata) Register the segments in the given SegmentsAndMetadata to be handed off and execute a background task which waits until the hand off completes.
Object	startJob() Perform any initial setup and return currently persisted commit metadata.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AppenderatorDriver

public AppenderatorDriver(Appenderator appenderator,
                          SegmentAllocator segmentAllocator,
                          SegmentHandoffNotifierFactory handoffNotifierFactory,
                          UsedSegmentChecker usedSegmentChecker,
                          com.fasterxml.jackson.databind.ObjectMapper objectMapper,
                          FireDepartmentMetrics metrics)

Create a driver.

Parameters:

appenderator - appenderator

segmentAllocator - segment allocator

handoffNotifierFactory - handoff notifier factory

usedSegmentChecker - used segment checker

objectMapper - object mapper, used for serde of commit metadata

metrics - Firedepartment metrics

Method Detail

startJob

public Object startJob()

Perform any initial setup and return currently persisted commit metadata. Note that this method returns the same metadata you've passed in with your Committers, even though this class stores extra metadata on disk.

Returns:

currently persisted commit metadata

clear

public void clear()
           throws InterruptedException

Clears out all our state and also calls Appenderator.clear() on the underlying Appenderator.

Throws:

InterruptedException

add

public AppenderatorDriverAddResult add(InputRow row,
                                       String sequenceName,
                                       com.google.common.base.Supplier<Committer> committerSupplier)
                                throws IOException

Add a row. Must not be called concurrently from multiple threads.

Parameters:

row - the row to add

sequenceName - sequenceName for this row's segment

committerSupplier - supplier of a committer associated with all data that has been added, including this row

Returns:

segment to which this row was added, or null if segment allocator returned null for this row

Throws:

IOException - if there is an I/O error while allocating or writing to a segment

persist

public Object persist(Committer committer)
               throws InterruptedException

Persist all data indexed through this driver so far. Blocks until complete. Should be called after all data has been added through add(InputRow, String, Supplier).

Parameters:

committer - committer representing all data that has been added so far

Returns:

commitMetadata persisted

Throws:

InterruptedException

registerHandoff

public com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> registerHandoff(SegmentsAndMetadata segmentsAndMetadata)

Register the segments in the given SegmentsAndMetadata to be handed off and execute a background task which waits until the hand off completes.

Parameters:

segmentsAndMetadata - the result segments and metadata of publish(TransactionalSegmentPublisher, Committer, Collection)

Returns:

null if the input segmentsAndMetadata is null. Otherwise, a ListenableFuture for the submitted task which returns SegmentsAndMetadata containing the segments successfully handed off and the metadata of the caller of FiniteAppenderatorDriverMetadata

close

public void close()

Closes this driver. Does not close the underlying Appenderator; you should do that yourself.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

moveSegmentOut

public void moveSegmentOut(String sequenceName,
                           List<SegmentIdentifier> identifiers)

Move a set of identifiers out from "active", making way for newer segments.

publish

public com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> publish(TransactionalSegmentPublisher publisher,
                                                                                       Committer committer,
                                                                                       Collection<String> sequenceNames)

Execute a task in background to publish all segments corresponding to the given sequence names. The task internally pushes the segments to the deep storage first, and then publishes the metadata to the metadata storage.

Parameters:

publisher - segment publisher

committer - committer

sequenceNames - a collection of sequence names to be published

Returns:

a ListenableFuture for the submitted task which removes published sequenceNames from activeSegments and publishPendingSegments.

publishAndRegisterHandoff

public com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> publishAndRegisterHandoff(TransactionalSegmentPublisher publisher,
                                                                                                         Committer committer,
                                                                                                         Collection<String> sequenceNames)