io.druid.segment.realtime.appenderator

Class AppenderatorImpl

java.lang.Object

io.druid.segment.realtime.appenderator.AppenderatorImpl

All Implemented Interfaces:

QuerySegmentWalker, Appenderator, Closeable, AutoCloseable

public class AppenderatorImpl
extends Object
implements Appenderator

Nested Class Summary

Nested classes/interfaces inherited from interface io.druid.segment.realtime.appenderator.Appenderator

Appenderator.AppenderatorAddResult

Constructor Summary

Constructors

Constructor and Description

AppenderatorImpl(DataSchema schema, AppenderatorConfig tuningConfig, FireDepartmentMetrics metrics, DataSegmentPusher dataSegmentPusher, com.fasterxml.jackson.databind.ObjectMapper objectMapper, QueryRunnerFactoryConglomerate conglomerate, DataSegmentAnnouncer segmentAnnouncer, ServiceEmitter emitter, ExecutorService queryExecutorService, IndexIO indexIO, IndexMerger indexMerger, Cache cache, CacheConfig cacheConfig)

Method Summary

Modifier and Type	Method and Description
Appenderator.AppenderatorAddResult	add(SegmentIdentifier identifier, InputRow row, com.google.common.base.Supplier<Committer> committerSupplier, boolean allowIncrementalPersists) Add a row.
void	clear() Drop all in-memory and on-disk data, and forget any previously-remembered commit metadata.
void	close() Stop any currently-running processing and clean up after ourselves.
void	closeNow() Unannounce the segments and wait for outstanding persists to finish.
com.google.common.util.concurrent.ListenableFuture<?>	drop(SegmentIdentifier identifier) Drop all data associated with a particular pending segment.
String	getDataSource() Return the name of the dataSource associated with this Appenderator.
<T> QueryRunner<T>	getQueryRunnerForIntervals(Query<T> query, Iterable<org.joda.time.Interval> intervals) Gets the Queryable for a given interval, the Queryable returned can be any version(s) or partitionNumber(s) such that it represents the interval.
<T> QueryRunner<T>	getQueryRunnerForSegments(Query<T> query, Iterable<SegmentDescriptor> specs) Gets the Queryable for a given list of SegmentSpecs.
int	getRowCount(SegmentIdentifier identifier) Returns the number of rows in a particular pending segment.
List<SegmentIdentifier>	getSegments() Returns a list of all currently active segments.
int	getTotalRowCount() Returns the number of total rows in this appenderator.
com.google.common.util.concurrent.ListenableFuture<Object>	persist(Collection<SegmentIdentifier> identifiers, Committer committer) Persist any in-memory indexed data for segments of the given identifiers to durable storage.
com.google.common.util.concurrent.ListenableFuture<Object>	persistAll(Committer committer) Persist any in-memory indexed data to durable storage.
com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata>	push(Collection<SegmentIdentifier> identifiers, Committer committer) Merge and push particular segments to deep storage.
Object	startJob() Perform any initial setup.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.druid.segment.realtime.appenderator.Appenderator

add

Constructor Detail

AppenderatorImpl

public AppenderatorImpl(DataSchema schema,
                        AppenderatorConfig tuningConfig,
                        FireDepartmentMetrics metrics,
                        DataSegmentPusher dataSegmentPusher,
                        com.fasterxml.jackson.databind.ObjectMapper objectMapper,
                        QueryRunnerFactoryConglomerate conglomerate,
                        DataSegmentAnnouncer segmentAnnouncer,
                        ServiceEmitter emitter,
                        ExecutorService queryExecutorService,
                        IndexIO indexIO,
                        IndexMerger indexMerger,
                        Cache cache,
                        CacheConfig cacheConfig)

Method Detail

getDataSource

public String getDataSource()

Description copied from interface: Appenderator

Return the name of the dataSource associated with this Appenderator.

Specified by:

getDataSource in interface Appenderator

startJob

public Object startJob()

Description copied from interface: Appenderator

Perform any initial setup. Should be called before using any other methods.

Specified by:

startJob in interface Appenderator

Returns:

currently persisted commit metadata

add

public Appenderator.AppenderatorAddResult add(SegmentIdentifier identifier,
                                              InputRow row,
                                              @Nullable
                                              com.google.common.base.Supplier<Committer> committerSupplier,
                                              boolean allowIncrementalPersists)
                                       throws IndexSizeExceededException,
                                              SegmentNotWritableException

Description copied from interface: Appenderator

Add a row. Must not be called concurrently from multiple threads.

If no pending segment exists for the provided identifier, a new one will be created.

This method may trigger a Appenderator.persistAll(Committer) using the supplied Committer. If it does this, the Committer is guaranteed to be *created* synchronously with the call to add, but will actually be used asynchronously.

If committer is not provided, no metadata is persisted. If it's provided, the add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.

Specified by:

add in interface Appenderator

Parameters:

identifier - the segment into which this row should be added

row - the row to add

committerSupplier - supplier of a committer associated with all data that has been added, including this row if is set to false then this will not be used as no persist will be done automatically

allowIncrementalPersists - indicate whether automatic persist should be performed or not if required. If this flag is set to false then the return value should have Appenderator.AppenderatorAddResult.isPersistRequired set to true if persist was skipped because of this flag and it is assumed that the responsibility of calling Appenderator.persistAll(Committer) is on the caller.

Returns:

Appenderator.AppenderatorAddResult

Throws:

IndexSizeExceededException - if this row cannot be added because it is too large

SegmentNotWritableException - if the requested segment is known, but has been closed

getSegments

public List<SegmentIdentifier> getSegments()

Description copied from interface: Appenderator

Returns a list of all currently active segments.

Specified by:

getSegments in interface Appenderator

getRowCount

public int getRowCount(SegmentIdentifier identifier)

Description copied from interface: Appenderator

Returns the number of rows in a particular pending segment.

Specified by:

getRowCount in interface Appenderator

Parameters:

identifier - segment to examine

Returns:

row count

getTotalRowCount

public int getTotalRowCount()

Description copied from interface: Appenderator

Returns the number of total rows in this appenderator.

Specified by:

getTotalRowCount in interface Appenderator

Returns:

total number of rows

getQueryRunnerForIntervals

public <T> QueryRunner<T> getQueryRunnerForIntervals(Query<T> query,
                                                     Iterable<org.joda.time.Interval> intervals)

Description copied from interface: QuerySegmentWalker

Gets the Queryable for a given interval, the Queryable returned can be any version(s) or partitionNumber(s) such that it represents the interval.

Specified by:

getQueryRunnerForIntervals in interface QuerySegmentWalker

Type Parameters:

T - query result type

Parameters:

query - the query to find a Queryable for

intervals - the intervals to find a Queryable for

Returns:

a Queryable object that represents the interval

getQueryRunnerForSegments

public <T> QueryRunner<T> getQueryRunnerForSegments(Query<T> query,
                                                    Iterable<SegmentDescriptor> specs)

Description copied from interface: QuerySegmentWalker

Gets the Queryable for a given list of SegmentSpecs.

Specified by:

getQueryRunnerForSegments in interface QuerySegmentWalker

Type Parameters:

T - the query result type

Parameters:

query - the query to return a Queryable for

specs - the list of SegmentSpecs to find a Queryable for

Returns:

the Queryable object with the given SegmentSpecs

clear

public void clear()
           throws InterruptedException

Description copied from interface: Appenderator

Drop all in-memory and on-disk data, and forget any previously-remembered commit metadata. This could be useful if, for some reason, rows have been added that we do not actually want to hand off. Blocks until all data has been cleared. This may take some time, since all pending persists must finish first.

The add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.

Specified by:

clear in interface Appenderator

Throws:

InterruptedException

drop

public com.google.common.util.concurrent.ListenableFuture<?> drop(SegmentIdentifier identifier)

Description copied from interface: Appenderator

Drop all data associated with a particular pending segment. Unlike Appenderator.clear()), any on-disk commit metadata will remain unchanged. If there is no pending segment with this identifier, then this method will do nothing.

You should not write to the dropped segment after calling "drop". If you need to drop all your data and re-write it, consider Appenderator.clear() instead.

Specified by:

drop in interface Appenderator

Parameters:

identifier - the pending segment to drop

Returns:

future that resolves when data is dropped

persist

public com.google.common.util.concurrent.ListenableFuture<Object> persist(Collection<SegmentIdentifier> identifiers,
                                                                          @Nullable
                                                                          Committer committer)

Description copied from interface: Appenderator

Persist any in-memory indexed data for segments of the given identifiers to durable storage. This may be only somewhat durable, e.g. the machine's local disk. The Committer will be made synchronously with the call to persist, but will actually be used asynchronously. Any metadata returned by the committer will be associated with the data persisted to disk.

If committer is not provided, no metadata is persisted. If it's provided, the add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.

Specified by:

persist in interface Appenderator

Parameters:

identifiers - segment identifiers to be persisted

committer - a committer associated with all data that has been added to segments of the given identifiers so far

Returns:

future that resolves when all pending data to segments of the identifiers has been persisted, contains commit metadata for this persist

persistAll

public com.google.common.util.concurrent.ListenableFuture<Object> persistAll(@Nullable
                                                                             Committer committer)

Description copied from interface: Appenderator

Persist any in-memory indexed data to durable storage. This may be only somewhat durable, e.g. the machine's local disk. The Committer will be made synchronously with the call to persistAll, but will actually be used asynchronously. Any metadata returned by the committer will be associated with the data persisted to disk.

If committer is not provided, no metadata is persisted. If it's provided, the add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.

Specified by:

persistAll in interface Appenderator

Parameters:

committer - a committer associated with all data that has been added so far

Returns:

future that resolves when all pending data has been persisted, contains commit metadata for this persist

push

public com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> push(Collection<SegmentIdentifier> identifiers,
                                                                                    @Nullable
                                                                                    Committer committer)

Description copied from interface: Appenderator

Merge and push particular segments to deep storage. This will trigger an implicit Appenderator.persist(Collection, Committer) using the provided Committer.

After this method is called, you cannot add new data to any segments that were previously under construction.

If committer is not provided, no metadata is persisted. If it's provided, the add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.

Specified by:

push in interface Appenderator

Parameters:

identifiers - list of segments to push

committer - a committer associated with all data that has been added so far

Returns:

future that resolves when all segments have been pushed. The segment list will be the list of segments that have been pushed and the commit metadata from the Committer.

close

public void close()

Description copied from interface: Appenderator

Stop any currently-running processing and clean up after ourselves. This allows currently running persists and pushes to finish. This will not remove any on-disk persisted data, but it will drop any data that has not yet been persisted.

Specified by:

close in interface Appenderator

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

closeNow

public void closeNow()

Unannounce the segments and wait for outstanding persists to finish. Do not unlock base persist dir as we are not waiting for push executor to shut down relying on current JVM to shutdown to not cause any locking problem if the task is restored. In case when task is restored and current task is still active because of push executor (which it shouldn't be since push executor starts daemon threads) then the locking should fail and new task should fail to start. This also means that this method should only be called when task is shutting down.

Specified by:

closeNow in interface Appenderator