public class AppenderatorImpl extends Object implements Appenderator
Appenderator.AppenderatorAddResult
Constructor and Description |
---|
AppenderatorImpl(DataSchema schema,
AppenderatorConfig tuningConfig,
FireDepartmentMetrics metrics,
DataSegmentPusher dataSegmentPusher,
com.fasterxml.jackson.databind.ObjectMapper objectMapper,
QueryRunnerFactoryConglomerate conglomerate,
DataSegmentAnnouncer segmentAnnouncer,
ServiceEmitter emitter,
ExecutorService queryExecutorService,
IndexIO indexIO,
IndexMerger indexMerger,
Cache cache,
CacheConfig cacheConfig) |
Modifier and Type | Method and Description |
---|---|
Appenderator.AppenderatorAddResult |
add(SegmentIdentifier identifier,
InputRow row,
com.google.common.base.Supplier<Committer> committerSupplier,
boolean allowIncrementalPersists)
Add a row.
|
void |
clear()
Drop all in-memory and on-disk data, and forget any previously-remembered commit metadata.
|
void |
close()
Stop any currently-running processing and clean up after ourselves.
|
void |
closeNow()
Unannounce the segments and wait for outstanding persists to finish.
|
com.google.common.util.concurrent.ListenableFuture<?> |
drop(SegmentIdentifier identifier)
Drop all data associated with a particular pending segment.
|
String |
getDataSource()
Return the name of the dataSource associated with this Appenderator.
|
<T> QueryRunner<T> |
getQueryRunnerForIntervals(Query<T> query,
Iterable<org.joda.time.Interval> intervals)
Gets the Queryable for a given interval, the Queryable returned can be any version(s) or partitionNumber(s)
such that it represents the interval.
|
<T> QueryRunner<T> |
getQueryRunnerForSegments(Query<T> query,
Iterable<SegmentDescriptor> specs)
Gets the Queryable for a given list of SegmentSpecs.
|
int |
getRowCount(SegmentIdentifier identifier)
Returns the number of rows in a particular pending segment.
|
List<SegmentIdentifier> |
getSegments()
Returns a list of all currently active segments.
|
int |
getTotalRowCount()
Returns the number of total rows in this appenderator.
|
com.google.common.util.concurrent.ListenableFuture<Object> |
persist(Collection<SegmentIdentifier> identifiers,
Committer committer)
Persist any in-memory indexed data for segments of the given identifiers to durable storage.
|
com.google.common.util.concurrent.ListenableFuture<Object> |
persistAll(Committer committer)
Persist any in-memory indexed data to durable storage.
|
com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> |
push(Collection<SegmentIdentifier> identifiers,
Committer committer)
Merge and push particular segments to deep storage.
|
Object |
startJob()
Perform any initial setup.
|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
add
public AppenderatorImpl(DataSchema schema, AppenderatorConfig tuningConfig, FireDepartmentMetrics metrics, DataSegmentPusher dataSegmentPusher, com.fasterxml.jackson.databind.ObjectMapper objectMapper, QueryRunnerFactoryConglomerate conglomerate, DataSegmentAnnouncer segmentAnnouncer, ServiceEmitter emitter, ExecutorService queryExecutorService, IndexIO indexIO, IndexMerger indexMerger, Cache cache, CacheConfig cacheConfig)
public String getDataSource()
Appenderator
getDataSource
in interface Appenderator
public Object startJob()
Appenderator
startJob
in interface Appenderator
public Appenderator.AppenderatorAddResult add(SegmentIdentifier identifier, InputRow row, @Nullable com.google.common.base.Supplier<Committer> committerSupplier, boolean allowIncrementalPersists) throws IndexSizeExceededException, SegmentNotWritableException
Appenderator
If no pending segment exists for the provided identifier, a new one will be created.
This method may trigger a Appenderator.persistAll(Committer)
using the supplied Committer. If it does this, the
Committer is guaranteed to be *created* synchronously with the call to add, but will actually be used
asynchronously.
If committer is not provided, no metadata is persisted. If it's provided, the add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.
add
in interface Appenderator
identifier
- the segment into which this row should be addedrow
- the row to addcommitterSupplier
- supplier of a committer associated with all data that has been added, including this row
if is set to false then this will not be used as no
persist will be done automaticallyallowIncrementalPersists
- indicate whether automatic persist should be performed or not if required.
If this flag is set to false then the return value should have
Appenderator.AppenderatorAddResult.isPersistRequired
set to true if persist was skipped
because of this flag and it is assumed that the responsibility of calling
Appenderator.persistAll(Committer)
is on the caller.Appenderator.AppenderatorAddResult
IndexSizeExceededException
- if this row cannot be added because it is too largeSegmentNotWritableException
- if the requested segment is known, but has been closedpublic List<SegmentIdentifier> getSegments()
Appenderator
getSegments
in interface Appenderator
public int getRowCount(SegmentIdentifier identifier)
Appenderator
getRowCount
in interface Appenderator
identifier
- segment to examinepublic int getTotalRowCount()
Appenderator
getTotalRowCount
in interface Appenderator
public <T> QueryRunner<T> getQueryRunnerForIntervals(Query<T> query, Iterable<org.joda.time.Interval> intervals)
QuerySegmentWalker
getQueryRunnerForIntervals
in interface QuerySegmentWalker
T
- query result typequery
- the query to find a Queryable forintervals
- the intervals to find a Queryable forpublic <T> QueryRunner<T> getQueryRunnerForSegments(Query<T> query, Iterable<SegmentDescriptor> specs)
QuerySegmentWalker
getQueryRunnerForSegments
in interface QuerySegmentWalker
T
- the query result typequery
- the query to return a Queryable forspecs
- the list of SegmentSpecs to find a Queryable forpublic void clear() throws InterruptedException
Appenderator
The add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.
clear
in interface Appenderator
InterruptedException
public com.google.common.util.concurrent.ListenableFuture<?> drop(SegmentIdentifier identifier)
Appenderator
Appenderator.clear()
), any on-disk commit
metadata will remain unchanged. If there is no pending segment with this identifier, then this method will
do nothing.
You should not write to the dropped segment after calling "drop". If you need to drop all your data and
re-write it, consider Appenderator.clear()
instead.
drop
in interface Appenderator
identifier
- the pending segment to droppublic com.google.common.util.concurrent.ListenableFuture<Object> persist(Collection<SegmentIdentifier> identifiers, @Nullable Committer committer)
Appenderator
If committer is not provided, no metadata is persisted. If it's provided, the add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.
persist
in interface Appenderator
identifiers
- segment identifiers to be persistedcommitter
- a committer associated with all data that has been added to segments of the given identifiers so
farpublic com.google.common.util.concurrent.ListenableFuture<Object> persistAll(@Nullable Committer committer)
Appenderator
If committer is not provided, no metadata is persisted. If it's provided, the add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.
persistAll
in interface Appenderator
committer
- a committer associated with all data that has been added so farpublic com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> push(Collection<SegmentIdentifier> identifiers, @Nullable Committer committer)
Appenderator
Appenderator.persist(Collection, Committer)
using the provided Committer.
After this method is called, you cannot add new data to any segments that were previously under construction.
If committer is not provided, no metadata is persisted. If it's provided, the add, clear, persist, persistAll, and push methods should all be called from the same thread to keep the metadata committed by Committer in sync.
push
in interface Appenderator
identifiers
- list of segments to pushcommitter
- a committer associated with all data that has been added so farpublic void close()
Appenderator
close
in interface Appenderator
close
in interface Closeable
close
in interface AutoCloseable
public void closeNow()
closeNow
in interface Appenderator
Copyright © 2011–2018. All rights reserved.