public class StreamAppenderatorDriver extends BaseAppenderatorDriver
APPENDING -> APPEND_FINISHED -> PUBLISHED
appenderator, executor, segments
Constructor and Description |
---|
StreamAppenderatorDriver(Appenderator appenderator,
SegmentAllocator segmentAllocator,
SegmentHandoffNotifierFactory handoffNotifierFactory,
UsedSegmentChecker usedSegmentChecker,
com.fasterxml.jackson.databind.ObjectMapper objectMapper,
FireDepartmentMetrics metrics)
Create a driver.
|
Modifier and Type | Method and Description |
---|---|
AppenderatorDriverAddResult |
add(InputRow row,
String sequenceName,
com.google.common.base.Supplier<Committer> committerSupplier,
boolean skipSegmentLineageCheck,
boolean allowIncrementalPersists)
Add a row.
|
void |
close()
Closes this driver.
|
void |
moveSegmentOut(String sequenceName,
List<SegmentIdentifier> identifiers)
Move a set of identifiers out from "active", making way for newer segments.
|
Object |
persist(Committer committer)
Persist all data indexed through this driver so far.
|
com.google.common.util.concurrent.ListenableFuture<Object> |
persistAsync(Committer committer)
Persist all data indexed through this driver so far.
|
com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> |
publish(TransactionalSegmentPublisher publisher,
Committer committer,
Collection<String> sequenceNames)
Execute a task in background to publish all segments corresponding to the given sequence names.
|
com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> |
publishAndRegisterHandoff(TransactionalSegmentPublisher publisher,
Committer committer,
Collection<String> sequenceNames) |
com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> |
registerHandoff(SegmentsAndMetadata segmentsAndMetadata)
Register the segments in the given
SegmentsAndMetadata to be handed off and execute a background task which
waits until the hand off completes. |
Object |
startJob()
Perform any initial setup and return currently persisted commit metadata.
|
append, clear
public StreamAppenderatorDriver(Appenderator appenderator, SegmentAllocator segmentAllocator, SegmentHandoffNotifierFactory handoffNotifierFactory, UsedSegmentChecker usedSegmentChecker, com.fasterxml.jackson.databind.ObjectMapper objectMapper, FireDepartmentMetrics metrics)
appenderator
- appenderatorsegmentAllocator
- segment allocatorhandoffNotifierFactory
- handoff notifier factoryusedSegmentChecker
- used segment checkerobjectMapper
- object mapper, used for serde of commit metadatametrics
- Firedepartment metrics@Nullable public Object startJob()
BaseAppenderatorDriver
Note that this method returns the same metadata you've passed in with your Committers, even though this class stores extra metadata on disk.
startJob
in class BaseAppenderatorDriver
public AppenderatorDriverAddResult add(InputRow row, String sequenceName, com.google.common.base.Supplier<Committer> committerSupplier, boolean skipSegmentLineageCheck, boolean allowIncrementalPersists) throws IOException
row
- the row to addsequenceName
- sequenceName for this row's segmentcommitterSupplier
- supplier of a committer associated with all data that has been added, including this row
if is set to false then this will not be usedskipSegmentLineageCheck
- if true, perform lineage validation using previousSegmentId for this sequence.
Should be set to false if replica tasks would index events in same orderallowIncrementalPersists
- whether to allow persist to happen when maxRowsInMemory or intermediate persist period
threshold is hitAppenderatorDriverAddResult
IOException
- if there is an I/O error while allocating or writing to a segmentpublic void moveSegmentOut(String sequenceName, List<SegmentIdentifier> identifiers)
public Object persist(Committer committer) throws InterruptedException
Should be called after all data has been added through add(InputRow, String, Supplier, boolean, boolean)
.
committer
- committer representing all data that has been added so farInterruptedException
public com.google.common.util.concurrent.ListenableFuture<Object> persistAsync(Committer committer)
Should be called after all data has been added through add(InputRow, String, Supplier, boolean, boolean)
.
committer
- committer representing all data that has been added so farpublic com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> publish(TransactionalSegmentPublisher publisher, Committer committer, Collection<String> sequenceNames)
publisher
- segment publishercommitter
- committersequenceNames
- a collection of sequence names to be publishedListenableFuture
for the submitted task which removes published sequenceNames
from
activeSegments
and publishPendingSegments
public com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> registerHandoff(SegmentsAndMetadata segmentsAndMetadata)
SegmentsAndMetadata
to be handed off and execute a background task which
waits until the hand off completes.segmentsAndMetadata
- the result segments and metadata of
publish(TransactionalSegmentPublisher, Committer, Collection)
ListenableFuture
for the submitted task
which returns SegmentsAndMetadata
containing the segments successfully handed off and the metadata
of the caller of AppenderatorDriverMetadata
public com.google.common.util.concurrent.ListenableFuture<SegmentsAndMetadata> publishAndRegisterHandoff(TransactionalSegmentPublisher publisher, Committer committer, Collection<String> sequenceNames)
public void close()
BaseAppenderatorDriver
close
in interface Closeable
close
in interface AutoCloseable
close
in class BaseAppenderatorDriver
Copyright © 2011–2018. All rights reserved.