public abstract class BaseAppenderatorDriver extends Object implements Closeable
BatchAppenderatorDriver
and StreamAppenderatorDriver
,
which are for batch and streaming ingestion, respectively. This class provides some fundamental methods for making
the child classes' life easier like pushInBackground(io.druid.segment.realtime.appenderator.BaseAppenderatorDriver.WrappedCommitter, java.util.Collection<io.druid.segment.realtime.appenderator.SegmentIdentifier>)
, dropInBackground(io.druid.segment.realtime.appenderator.SegmentsAndMetadata)
, or
publishInBackground(io.druid.segment.realtime.appenderator.SegmentsAndMetadata, io.druid.segment.realtime.appenderator.TransactionalSegmentPublisher)
. The child classes can use these methods to achieve their goal.
Note that the commit metadata stored by this class via the underlying Appenderator is not the same metadata as
you pass in. It's wrapped in some extra metadata needed by the driver.Modifier and Type | Field and Description |
---|---|
protected Appenderator |
appenderator |
protected com.google.common.util.concurrent.ListeningExecutorService |
executor |
protected Map<String,io.druid.segment.realtime.appenderator.BaseAppenderatorDriver.SegmentsForSequence> |
segments |
Modifier and Type | Method and Description |
---|---|
protected AppenderatorDriverAddResult |
append(InputRow row,
String sequenceName,
com.google.common.base.Supplier<Committer> committerSupplier,
boolean skipSegmentLineageCheck,
boolean allowIncrementalPersists)
Add a row.
|
void |
clear()
Clears out all our state and also calls
Appenderator.clear() on the underlying Appenderator. |
void |
close()
Closes this driver.
|
abstract Object |
startJob()
Perform any initial setup and return currently persisted commit metadata.
|
protected final Appenderator appenderator
protected final Map<String,io.druid.segment.realtime.appenderator.BaseAppenderatorDriver.SegmentsForSequence> segments
protected final com.google.common.util.concurrent.ListeningExecutorService executor
@Nullable public abstract Object startJob()
Note that this method returns the same metadata you've passed in with your Committers, even though this class stores extra metadata on disk.
protected AppenderatorDriverAddResult append(InputRow row, String sequenceName, @Nullable com.google.common.base.Supplier<Committer> committerSupplier, boolean skipSegmentLineageCheck, boolean allowIncrementalPersists) throws IOException
row
- the row to addsequenceName
- sequenceName for this row's segmentcommitterSupplier
- supplier of a committer associated with all data that has been added, including this row
if is set to false then this will not be usedskipSegmentLineageCheck
- if true, perform lineage validation using previousSegmentId for this sequence.
Should be set to false if replica tasks would index events in same orderallowIncrementalPersists
- whether to allow persist to happen when maxRowsInMemory or intermediate persist period
threshold is hitAppenderatorDriverAddResult
IOException
- if there is an I/O error while allocating or writing to a segmentpublic void clear() throws InterruptedException
Appenderator.clear()
on the underlying Appenderator.InterruptedException
public void close()
close
in interface Closeable
close
in interface AutoCloseable
Copyright © 2011–2018. All rights reserved.