public interface Appenderator extends QuerySegmentWalker, Closeable
You can provide a Committer
or a Supplier of one when you call one of the methods that adds, persists, or
pushes data. The Committer should represent all data you have given to the Appenderator so far. This Committer will
be used when that data has been persisted to disk.
Modifier and Type | Interface and Description |
---|---|
static class |
Appenderator.AppenderatorAddResult
Result of
add(SegmentIdentifier, InputRow, Supplier, boolean) containing following information
- SegmentIdentifier - identifier of segment to which rows are being added
- int - positive number indicating how many summarized rows exist in this segment so far and
- boolean - true if is set to false and persist is required; false otherwise |
Modifier and Type | Method and Description |
---|---|
default Appenderator.AppenderatorAddResult |
add(SegmentIdentifier identifier,
InputRow row,
com.google.common.base.Supplier<Committer> committerSupplier)
Same as
add(SegmentIdentifier, InputRow, Supplier, boolean) , with allowIncrementalPersists set to true |
Appenderator.AppenderatorAddResult |
add(SegmentIdentifier identifier,
InputRow row,
com.google.common.base.Supplier<Committer> committerSupplier,
boolean allowIncrementalPersists)
Add a row.
|
void |
clear()
Drop all in-memory and on-disk data, and forget any previously-remembered commit metadata.
|
void |
close()
Stop any currently-running processing and clean up after ourselves.
|
void |
closeNow()
Stop all processing, abandoning current pushes, currently running persist may be allowed to finish if they persist
critical metadata otherwise shutdown immediately.
|
com.google.common.util.concurrent.ListenableFuture<?> |
drop(SegmentIdentifier identifier)
Drop all data associated with a particular pending segment.
|
String |
getDataSource()
Return the name of the dataSource associated with this Appenderator.
|
int |
getRowCount(SegmentIdentifier identifier)
Returns the number of rows in a particular pending segment.
|
List<SegmentIdentifier> |
getSegments()
Returns a list of all currently active segments.
|
int |
getTotalRowCount()
Returns the number of total rows in this appenderator.
|
com.google.common.util.concurrent.ListenableFuture<Object> |
persist(Collection<SegmentIdentifier> identifiers,
Committer committer)
Persist any in-memory indexed data for segments of the given identifiers to durable storage.
|
default com.google.common.util.concurrent.ListenableFuture<Object> |
persistAll(Committer committer)
Persist any in-memory indexed data to durable storage.
|
com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> |
push(Collection<SegmentIdentifier> identifiers,
Committer committer)
Merge and push particular segments to deep storage.
|
Object |
startJob()
Perform any initial setup.
|
getQueryRunnerForIntervals, getQueryRunnerForSegments
String getDataSource()
Object startJob()
default Appenderator.AppenderatorAddResult add(SegmentIdentifier identifier, InputRow row, com.google.common.base.Supplier<Committer> committerSupplier) throws IndexSizeExceededException, SegmentNotWritableException
add(SegmentIdentifier, InputRow, Supplier, boolean)
, with allowIncrementalPersists set to trueAppenderator.AppenderatorAddResult add(SegmentIdentifier identifier, InputRow row, @Nullable com.google.common.base.Supplier<Committer> committerSupplier, boolean allowIncrementalPersists) throws IndexSizeExceededException, SegmentNotWritableException
If no pending segment exists for the provided identifier, a new one will be created.
This method may trigger a persistAll(Committer)
using the supplied Committer. If it does this, the
Committer is guaranteed to be *created* synchronously with the call to add, but will actually be used
asynchronously.
If committer is not provided, no metadata is persisted. If it's provided, the add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.
identifier
- the segment into which this row should be addedrow
- the row to addcommitterSupplier
- supplier of a committer associated with all data that has been added, including this row
if is set to false then this will not be used as no
persist will be done automaticallyallowIncrementalPersists
- indicate whether automatic persist should be performed or not if required.
If this flag is set to false then the return value should have
Appenderator.AppenderatorAddResult.isPersistRequired
set to true if persist was skipped
because of this flag and it is assumed that the responsibility of calling
persistAll(Committer)
is on the caller.Appenderator.AppenderatorAddResult
IndexSizeExceededException
- if this row cannot be added because it is too largeSegmentNotWritableException
- if the requested segment is known, but has been closedList<SegmentIdentifier> getSegments()
int getRowCount(SegmentIdentifier identifier)
identifier
- segment to examineIllegalStateException
- if the segment is unknownint getTotalRowCount()
void clear() throws InterruptedException
The add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.
InterruptedException
com.google.common.util.concurrent.ListenableFuture<?> drop(SegmentIdentifier identifier)
clear()
), any on-disk commit
metadata will remain unchanged. If there is no pending segment with this identifier, then this method will
do nothing.
You should not write to the dropped segment after calling "drop". If you need to drop all your data and
re-write it, consider clear()
instead.
identifier
- the pending segment to dropcom.google.common.util.concurrent.ListenableFuture<Object> persist(Collection<SegmentIdentifier> identifiers, @Nullable Committer committer)
If committer is not provided, no metadata is persisted. If it's provided, the add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.
identifiers
- segment identifiers to be persistedcommitter
- a committer associated with all data that has been added to segments of the given identifiers so
fardefault com.google.common.util.concurrent.ListenableFuture<Object> persistAll(@Nullable Committer committer)
If committer is not provided, no metadata is persisted. If it's provided, the add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.
committer
- a committer associated with all data that has been added so farcom.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> push(Collection<SegmentIdentifier> identifiers, @Nullable Committer committer)
persist(Collection, Committer)
using the provided Committer.
After this method is called, you cannot add new data to any segments that were previously under construction.
If committer is not provided, no metadata is persisted. If it's provided, the add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.
identifiers
- list of segments to pushcommitter
- a committer associated with all data that has been added so farvoid close()
close
in interface AutoCloseable
close
in interface Closeable
void closeNow()
Copyright © 2011–2018. All rights reserved.