io.druid.segment.realtime.plumber

Class RealtimePlumber

java.lang.Object

io.druid.segment.realtime.plumber.RealtimePlumber

All Implemented Interfaces:

Plumber

Direct Known Subclasses:

FlushingPlumber

public class RealtimePlumber
extends Object
implements Plumber

Constructor Summary

Constructors

Constructor and Description

RealtimePlumber(DataSchema schema, RealtimeTuningConfig config, FireDepartmentMetrics metrics, com.metamx.emitter.service.ServiceEmitter emitter, QueryRunnerFactoryConglomerate conglomerate, DataSegmentAnnouncer segmentAnnouncer, ExecutorService queryExecutorService, DataSegmentPusher dataSegmentPusher, SegmentPublisher segmentPublisher, SegmentHandoffNotifier handoffNotifier, IndexMerger indexMerger, IndexIO indexIO, Cache cache, CacheConfig cacheConfig, com.fasterxml.jackson.databind.ObjectMapper objectMapper)

Method Summary

Modifier and Type	Method and Description
protected void	abandonSegment(long truncatedTime, Sink sink) Unannounces a given sink and removes all local references to it.
int	add(InputRow row, com.google.common.base.Supplier<Committer> committerSupplier)
protected Object	bootstrapSinksFromDisk()
protected File	computeBaseDir(DataSchema schema)
protected File	computeCorruptedFileDumpDir(File persistDir, DataSchema schema)
protected File	computePersistDir(DataSchema schema, org.joda.time.Interval interval)
void	finishJob() Perform any final processing and clean up after ourselves.
RealtimeTuningConfig	getConfig()
<T> QueryRunner<T>	getQueryRunner(Query<T> query)
RejectionPolicy	getRejectionPolicy()
DataSchema	getSchema()
Map<Long,Sink>	getSinks()
protected void	initializeExecutors()
void	persist(Committer committer) Persist any in-memory indexed data to durable storage.
protected int	persistHydrant(FireHydrant indexToPersist, DataSchema schema, org.joda.time.Interval interval, Map<String,Object> metadataElems) Persists the given hydrant and returns the number of rows persisted
protected void	shutdownExecutors()
Object	startJob() Perform any initial setup.
protected void	startPersistThread()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RealtimePlumber

public RealtimePlumber(DataSchema schema,
                       RealtimeTuningConfig config,
                       FireDepartmentMetrics metrics,
                       com.metamx.emitter.service.ServiceEmitter emitter,
                       QueryRunnerFactoryConglomerate conglomerate,
                       DataSegmentAnnouncer segmentAnnouncer,
                       ExecutorService queryExecutorService,
                       DataSegmentPusher dataSegmentPusher,
                       SegmentPublisher segmentPublisher,
                       SegmentHandoffNotifier handoffNotifier,
                       IndexMerger indexMerger,
                       IndexIO indexIO,
                       Cache cache,
                       CacheConfig cacheConfig,
                       com.fasterxml.jackson.databind.ObjectMapper objectMapper)

Method Detail

getSchema

public DataSchema getSchema()

getConfig

public RealtimeTuningConfig getConfig()

getRejectionPolicy

public RejectionPolicy getRejectionPolicy()

getSinks

public Map<Long,Sink> getSinks()

startJob

public Object startJob()

Description copied from interface: Plumber

Perform any initial setup. Should be called before using any other methods, and should be paired with a corresponding call to Plumber.finishJob().

Specified by:

startJob in interface Plumber

Returns:

the metadata of the "newest" segment that might have previously been persisted

add

public int add(InputRow row,
               com.google.common.base.Supplier<Committer> committerSupplier)
        throws IndexSizeExceededException

Specified by:

add in interface Plumber

Parameters:

row - the row to insert

committerSupplier - supplier of a committer associated with all data that has been added, including this row

Returns:

- positive numbers indicate how many summarized rows exist in the index for that timestamp, -1 means a row was thrown away because it was too late

Throws:

IndexSizeExceededException

getQueryRunner

public <T> QueryRunner<T> getQueryRunner(Query<T> query)

Specified by:

getQueryRunner in interface Plumber

persist

public void persist(Committer committer)

Description copied from interface: Plumber

Persist any in-memory indexed data to durable storage. This may be only somewhat durable, e.g. the machine's local disk.

Specified by:

persist in interface Plumber

Parameters:

committer - committer to use after persisting data

finishJob

public void finishJob()

Description copied from interface: Plumber

Perform any final processing and clean up after ourselves. Should be called after all data has been fed into sinks and persisted.

Specified by:

finishJob in interface Plumber

initializeExecutors

protected void initializeExecutors()

shutdownExecutors

protected void shutdownExecutors()

bootstrapSinksFromDisk

protected Object bootstrapSinksFromDisk()

startPersistThread

protected void startPersistThread()

abandonSegment

protected void abandonSegment(long truncatedTime,
                              Sink sink)

Unannounces a given sink and removes all local references to it. It is important that this is only called from the single-threaded mergeExecutor, since otherwise chaos may ensue if merged segments are deleted while being created.

Parameters:

truncatedTime - sink key

sink - sink to unannounce

computeBaseDir

protected File computeBaseDir(DataSchema schema)

computeCorruptedFileDumpDir

protected File computeCorruptedFileDumpDir(File persistDir,
                                           DataSchema schema)

computePersistDir

protected File computePersistDir(DataSchema schema,
                                 org.joda.time.Interval interval)

persistHydrant

protected int persistHydrant(FireHydrant indexToPersist,
                             DataSchema schema,
                             org.joda.time.Interval interval,
                             Map<String,Object> metadataElems)

Persists the given hydrant and returns the number of rows persisted

Parameters:

indexToPersist - hydrant to persist

schema - datasource schema

interval - interval to persist

Returns:

the number of rows persisted