io.druid.segment.realtime.appenderator

Class FiniteAppenderatorDriver

java.lang.Object

io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver

All Implemented Interfaces:

Closeable, AutoCloseable

public class FiniteAppenderatorDriver
extends Object
implements Closeable

A FiniteAppenderatorDriver drives an Appenderator to index a finite stream of data. This class does not help you index unbounded streams. All handoff is done at the end of indexing. This class helps with doing things that Appenderators don't, including deciding which segments to use (with a SegmentAllocator), publishing segments to the metadata store (with a SegmentPublisher), and monitoring handoff (with a SegmentHandoffNotifier). Note that the commit metadata stored by this class via the underlying Appenderator is not the same metadata as you pass in. It's wrapped in some extra metadata needed by the driver.

Constructor Summary

Constructors

Constructor and Description
FiniteAppenderatorDriver(Appenderator appenderator, SegmentAllocator segmentAllocator, SegmentHandoffNotifierFactory handoffNotifierFactory, UsedSegmentChecker usedSegmentChecker, com.fasterxml.jackson.databind.ObjectMapper objectMapper, int maxRowsPerSegment, long handoffConditionTimeout, FireDepartmentMetrics metrics) Create a driver.

Method Summary

Modifier and Type	Method and Description
SegmentIdentifier	add(InputRow row, String sequenceName, com.google.common.base.Supplier<Committer> committerSupplier) Add a row.
void	clear() Clears out all our state and also calls Appenderator.clear() on the underlying Appenderator.
void	close() Closes this driver.
SegmentsAndMetadata	finish(TransactionalSegmentPublisher publisher, Committer committer) Publish all data indexed through this driver so far, and waits for it to be handed off.
Object	persist(Committer committer) Persist all data indexed through this driver so far.
Object	startJob() Perform any initial setup and return currently persisted commit metadata.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FiniteAppenderatorDriver

public FiniteAppenderatorDriver(Appenderator appenderator,
                                SegmentAllocator segmentAllocator,
                                SegmentHandoffNotifierFactory handoffNotifierFactory,
                                UsedSegmentChecker usedSegmentChecker,
                                com.fasterxml.jackson.databind.ObjectMapper objectMapper,
                                int maxRowsPerSegment,
                                long handoffConditionTimeout,
                                FireDepartmentMetrics metrics)

Create a driver.

Parameters:

appenderator - appenderator

segmentAllocator - segment allocator

handoffNotifierFactory - handoff notifier factory

usedSegmentChecker - used segment checker

objectMapper - object mapper, used for serde of commit metadata

maxRowsPerSegment - maximum number of rows allowed in an entire segment (not a single persist)

handoffConditionTimeout - maximum number of millis allowed for handoff (not counting push/publish), zero means wait forever.

metrics - Firedepartment metrics

Method Detail

startJob

public Object startJob()

Perform any initial setup and return currently persisted commit metadata. Note that this method returns the same metadata you've passed in with your Committers, even though this class stores extra metadata on disk.

Returns:

currently persisted commit metadata

clear

public void clear()
           throws InterruptedException

Clears out all our state and also calls Appenderator.clear() on the underlying Appenderator.

Throws:

InterruptedException

add

public SegmentIdentifier add(InputRow row,
                             String sequenceName,
                             com.google.common.base.Supplier<Committer> committerSupplier)
                      throws IOException

Add a row. Must not be called concurrently from multiple threads.

Parameters:

row - the row to add

sequenceName - sequenceName for this row's segment

committerSupplier - supplier of a committer associated with all data that has been added, including this row

Returns:

segment to which this row was added, or null if segment allocator returned null for this row

Throws:

IOException - if there is an I/O error while allocating or writing to a segment

persist

public Object persist(Committer committer)
               throws InterruptedException

Persist all data indexed through this driver so far. Blocks until complete. Should be called after all data has been added through add(InputRow, String, Supplier).

Parameters:

committer - committer representing all data that has been added so far

Returns:

commitMetadata persisted

Throws:

InterruptedException

finish

public SegmentsAndMetadata finish(TransactionalSegmentPublisher publisher,
                                  Committer committer)
                           throws InterruptedException

Publish all data indexed through this driver so far, and waits for it to be handed off. Blocks until complete. Retries forever on transient failures, but may exit early on permanent failures. Should be called after all data has been added and persisted through add(InputRow, String, Supplier) and persist(Committer).

Parameters:

publisher - publisher to use for this set of segments

committer - committer representing all data that has been added so far

Returns:

segments and metadata published if successful, or null if segments could not be handed off due to transaction failure with commit metadata.

Throws:

InterruptedException

close

public void close()

Closes this driver. Does not close the underlying Appenderator; you should do that yourself.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable