io.druid.segment.realtime.appenderator

Interface Appenderator

All Superinterfaces:

AutoCloseable, Closeable, QuerySegmentWalker

All Known Implementing Classes:

AppenderatorImpl

public interface Appenderator
extends QuerySegmentWalker, Closeable

An Appenderator indexes data. It has some in-memory data and some persisted-on-disk data. It can serve queries on both of those. It can also push data to deep storage. But, it does not decide which segments data should go into. It also doesn't publish segments to the metadata store or monitor handoff; you have to do that yourself!

Any time you call one of the methods that adds, persists, or pushes data, you must provide a Committer, or a Supplier of one, that represents all data you have given to the Appenderator so far. The Committer will be used when that data has been persisted to disk.

Method Summary

Modifier and Type	Method and Description
int	add(SegmentIdentifier identifier, InputRow row, com.google.common.base.Supplier<Committer> committerSupplier) Add a row.
void	clear() Drop all in-memory and on-disk data, and forget any previously-remembered commit metadata.
void	close() Stop any currently-running processing and clean up after ourselves.
com.google.common.util.concurrent.ListenableFuture<?>	drop(SegmentIdentifier identifier) Drop all data associated with a particular pending segment.
String	getDataSource() Return the name of the dataSource associated with this Appenderator.
int	getRowCount(SegmentIdentifier identifier) Returns the number of rows in a particular pending segment.
List<SegmentIdentifier>	getSegments() Returns a list of all currently active segments.
com.google.common.util.concurrent.ListenableFuture<Object>	persistAll(Committer committer) Persist any in-memory indexed data to durable storage.
com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata>	push(List<SegmentIdentifier> identifiers, Committer committer) Merge and push particular segments to deep storage.
Object	startJob() Perform any initial setup.

Methods inherited from interface io.druid.query.QuerySegmentWalker

getQueryRunnerForIntervals, getQueryRunnerForSegments

Method Detail

getDataSource

String getDataSource()

Return the name of the dataSource associated with this Appenderator.

startJob

Object startJob()

Perform any initial setup. Should be called before using any other methods.

Returns:

currently persisted commit metadata

add

int add(SegmentIdentifier identifier,
        InputRow row,
        com.google.common.base.Supplier<Committer> committerSupplier)
 throws IndexSizeExceededException,
        SegmentNotWritableException

Add a row. Must not be called concurrently from multiple threads.

If no pending segment exists for the provided identifier, a new one will be created.

This method may trigger a persistAll(Committer) using the supplied Committer. If it does this, the Committer is guaranteed to be *created* synchronously with the call to add, but will actually be used asynchronously.

The add, clear, persistAll, and push methods should all be called from the same thread.

Parameters:

identifier - the segment into which this row should be added

row - the row to add

committerSupplier - supplier of a committer associated with all data that has been added, including this row

Returns:

positive number indicating how many summarized rows exist in this segment so far

Throws:

IndexSizeExceededException - if this row cannot be added because it is too large

SegmentNotWritableException - if the requested segment is known, but has been closed

getSegments

List<SegmentIdentifier> getSegments()

Returns a list of all currently active segments.

getRowCount

int getRowCount(SegmentIdentifier identifier)

Returns the number of rows in a particular pending segment.

Parameters:

identifier - segment to examine

Returns:

row count

Throws:

IllegalStateException - if the segment is unknown

clear

void clear()
    throws InterruptedException

Drop all in-memory and on-disk data, and forget any previously-remembered commit metadata. This could be useful if, for some reason, rows have been added that we do not actually want to hand off. Blocks until all data has been cleared. This may take some time, since all pending persists must finish first. The add, clear, persistAll, and push methods should all be called from the same thread.

Throws:

InterruptedException

drop

com.google.common.util.concurrent.ListenableFuture<?> drop(SegmentIdentifier identifier)

Drop all data associated with a particular pending segment. Unlike clear()), any on-disk commit metadata will remain unchanged. If there is no pending segment with this identifier, then this method will do nothing.

You should not write to the dropped segment after calling "drop". If you need to drop all your data and re-write it, consider clear() instead.

Parameters:

identifier - the pending segment to drop

Returns:

future that resolves when data is dropped

persistAll

com.google.common.util.concurrent.ListenableFuture<Object> persistAll(Committer committer)

Persist any in-memory indexed data to durable storage. This may be only somewhat durable, e.g. the machine's local disk. The Committer will be made synchronously will the call to persistAll, but will actually be used asynchronously. Any metadata returned by the committer will be associated with the data persisted to disk.

The add, clear, persistAll, and push methods should all be called from the same thread.

Parameters:

committer - a committer associated with all data that has been added so far

Returns:

future that resolves when all pending data has been persisted, contains commit metadata for this persist

push

com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> push(List<SegmentIdentifier> identifiers,
                                                                             Committer committer)

Merge and push particular segments to deep storage. This will trigger an implicit persistAll(Committer) using the provided Committer.

After this method is called, you cannot add new data to any segments that were previously under construction.

The add, clear, persistAll, and push methods should all be called from the same thread.

Parameters:

identifiers - list of segments to push

committer - a committer associated with all data that has been added so far

Returns:

future that resolves when all segments have been pushed. The segment list will be the list of segments that have been pushed and the commit metadata from the Committer.

close

void close()

Stop any currently-running processing and clean up after ourselves. This will not remove any on-disk persisted data, but it will drop any data that has not yet been persisted.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable