io.druid.indexing.overlord

Interface IndexerMetadataStorageCoordinator

All Known Implementing Classes:

IndexerSQLMetadataStorageCoordinator

public interface IndexerMetadataStorageCoordinator

Method Summary

Modifier and Type	Method and Description
SegmentIdentifier	allocatePendingSegment(String dataSource, String sequenceName, String previousSegmentId, org.joda.time.Interval interval, String maxVersion) Allocate a new pending segment in the pending segments table.
Set<io.druid.timeline.DataSegment>	announceHistoricalSegments(Set<io.druid.timeline.DataSegment> segments) Attempts to insert a set of segments to the metadata storage.
void	deleteSegments(Set<io.druid.timeline.DataSegment> segments)
List<io.druid.timeline.DataSegment>	getUnusedSegmentsForInterval(String dataSource, org.joda.time.Interval interval) Get all segments which include ONLY data within the given interval and are not flagged as used.
List<io.druid.timeline.DataSegment>	getUsedSegmentsForInterval(String dataSource, org.joda.time.Interval interval) Get all segments which may include any data in the interval and are flagged as used.
List<io.druid.timeline.DataSegment>	getUsedSegmentsForIntervals(String dataSource, List<org.joda.time.Interval> intervals) Get all segments which may include any data in the interval and are flagged as used.
void	updateSegmentMetadata(Set<io.druid.timeline.DataSegment> segments)

Method Detail

getUsedSegmentsForInterval

List<io.druid.timeline.DataSegment> getUsedSegmentsForInterval(String dataSource,
                                                               org.joda.time.Interval interval)
                                                        throws IOException

Get all segments which may include any data in the interval and are flagged as used.

Parameters:

dataSource - The datasource to query

interval - The interval for which all applicable and used datasources are requested. Start is inclusive, end is exclusive

Returns:

The DataSegments which include data in the requested interval. These segments may contain data outside the requested interval.

Throws:

IOException

getUsedSegmentsForIntervals

List<io.druid.timeline.DataSegment> getUsedSegmentsForIntervals(String dataSource,
                                                                List<org.joda.time.Interval> intervals)
                                                         throws IOException

Get all segments which may include any data in the interval and are flagged as used.

Parameters:

dataSource - The datasource to query

intervals - The intervals for which all applicable and used datasources are requested.

Returns:

The DataSegments which include data in the requested intervals. These segments may contain data outside the requested interval.

Throws:

IOException

announceHistoricalSegments

Set<io.druid.timeline.DataSegment> announceHistoricalSegments(Set<io.druid.timeline.DataSegment> segments)
                                                       throws IOException

Attempts to insert a set of segments to the metadata storage. Returns the set of segments actually added (segments with identifiers already in the metadata storage will not be added).

Parameters:

segments - set of segments to add

Returns:

set of segments actually added

Throws:

IOException

allocatePendingSegment

SegmentIdentifier allocatePendingSegment(String dataSource,
                                         String sequenceName,
                                         String previousSegmentId,
                                         org.joda.time.Interval interval,
                                         String maxVersion)
                                  throws IOException

Allocate a new pending segment in the pending segments table. This segment identifier will never be given out again, unless another call is made with the same dataSource, sequenceName, and previousSegmentId.

The sequenceName and previousSegmentId parameters are meant to make it easy for two independent ingestion tasks to produce the same series of segments.

Note that a segment sequence may include segments with a variety of different intervals and versions.

Parameters:

dataSource - dataSource for which to allocate a segment

sequenceName - name of the group of ingestion tasks producing a segment series

previousSegmentId - previous segment in the series; may be null or empty, meaning this is the first segment

interval - interval for which to allocate a segment

maxVersion - use this version if we have no better version to use. The returned segment identifier may have a version lower than this one, but will not have one higher.

Returns:

the pending segment identifier, or null if it was impossible to allocate a new segment

Throws:

IOException

updateSegmentMetadata

void updateSegmentMetadata(Set<io.druid.timeline.DataSegment> segments)
                    throws IOException

Throws:

IOException

deleteSegments

void deleteSegments(Set<io.druid.timeline.DataSegment> segments)
             throws IOException

Throws:

IOException

getUnusedSegmentsForInterval

List<io.druid.timeline.DataSegment> getUnusedSegmentsForInterval(String dataSource,
                                                                 org.joda.time.Interval interval)

Get all segments which include ONLY data within the given interval and are not flagged as used.

Parameters:

dataSource - The datasource the segments belong to

interval - Filter the data segments to ones that include data in this interval exclusively. Start is inclusive, end is exclusive

Returns:

DataSegments which include ONLY data within the requested interval and are not flagged as used. Data segments NOT returned here may include data in the interval