io.druid.storage.s3

Class S3DataSegmentPuller

java.lang.Object

io.druid.storage.s3.S3DataSegmentPuller

All Implemented Interfaces:

io.druid.segment.loading.DataSegmentPuller, io.druid.segment.loading.URIDataPuller

Direct Known Subclasses:

S3TimestampVersionedDataFinder

public class S3DataSegmentPuller
extends Object
implements io.druid.segment.loading.DataSegmentPuller, io.druid.segment.loading.URIDataPuller

A data segment puller that also hanldes URI data pulls.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	S3DataSegmentPuller.S3Coords

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	BUCKET
static int	DEFAULT_RETRY_COUNT
protected static String	KEY
protected org.jets3t.service.impl.rest.httpclient.RestS3Service	s3Client
static String	scheme

Constructor Summary

Constructors

Constructor and Description
S3DataSegmentPuller(org.jets3t.service.impl.rest.httpclient.RestS3Service s3Client)

Method Summary

Methods

Modifier and Type	Method and Description
static FileObject	buildFileObject(URI uri, org.jets3t.service.impl.rest.httpclient.RestS3Service s3Client)
static URI	checkURI(URI uri)
InputStream	getInputStream(URI uri)
void	getSegmentFiles(io.druid.timeline.DataSegment segment, File outDir)
com.metamx.common.FileUtils.FileCopyResult	getSegmentFiles(S3DataSegmentPuller.S3Coords s3Coords, File outDir)
String	getVersion(URI uri) Returns the "version" (aka last modified timestamp) of the URI
com.google.common.base.Predicate<Throwable>	shouldRetryPredicate()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_RETRY_COUNT

public static final int DEFAULT_RETRY_COUNT

See Also:

Constant Field Values

scheme

public static final String scheme

See Also:

Constant Field Values

BUCKET

protected static final String BUCKET

See Also:

Constant Field Values

KEY

protected static final String KEY

See Also:

Constant Field Values

s3Client

protected final org.jets3t.service.impl.rest.httpclient.RestS3Service s3Client

Constructor Detail

S3DataSegmentPuller

@Inject
public S3DataSegmentPuller(org.jets3t.service.impl.rest.httpclient.RestS3Service s3Client)

Method Detail

buildFileObject

public static FileObject buildFileObject(URI uri,
                         org.jets3t.service.impl.rest.httpclient.RestS3Service s3Client)
                                  throws org.jets3t.service.S3ServiceException

Throws:

org.jets3t.service.S3ServiceException

getSegmentFiles

public void getSegmentFiles(io.druid.timeline.DataSegment segment,
                   File outDir)
                     throws io.druid.segment.loading.SegmentLoadingException

Specified by:

getSegmentFiles in interface io.druid.segment.loading.DataSegmentPuller

Throws:

io.druid.segment.loading.SegmentLoadingException

getSegmentFiles

public com.metamx.common.FileUtils.FileCopyResult getSegmentFiles(S3DataSegmentPuller.S3Coords s3Coords,
                                                         File outDir)
                                                           throws io.druid.segment.loading.SegmentLoadingException

Throws:

io.druid.segment.loading.SegmentLoadingException

checkURI

public static URI checkURI(URI uri)

getInputStream

public InputStream getInputStream(URI uri)
                           throws IOException

Specified by:

getInputStream in interface io.druid.segment.loading.URIDataPuller

Throws:

IOException

shouldRetryPredicate

public com.google.common.base.Predicate<Throwable> shouldRetryPredicate()

Specified by:

shouldRetryPredicate in interface io.druid.segment.loading.URIDataPuller

getVersion

public String getVersion(URI uri)
                  throws IOException

Returns the "version" (aka last modified timestamp) of the URI

Specified by:

getVersion in interface io.druid.segment.loading.URIDataPuller

Parameters:

uri - The URI to check the last timestamp

Returns:

The time in ms of the last modification of the URI in String format

Throws:

IOException