io.druid.firehose.s3

Class StaticS3FirehoseFactory

java.lang.Object

io.druid.data.input.impl.AbstractTextFilesFirehoseFactory<T>

io.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactory<org.jets3t.service.model.S3Object>

io.druid.firehose.s3.StaticS3FirehoseFactory

All Implemented Interfaces:

FirehoseFactory<StringInputRowParser>

public class StaticS3FirehoseFactory
extends PrefetchableTextFilesFirehoseFactory<org.jets3t.service.model.S3Object>

Builds firehoses that read from a predefined list of S3 objects and then dry up.

Constructor Summary

Constructors

Constructor and Description
StaticS3FirehoseFactory(org.jets3t.service.impl.rest.httpclient.RestS3Service s3Client, List<URI> uris, List<URI> prefixes, Long maxCacheCapacityBytes, Long maxFetchCapacityBytes, Long prefetchTriggerBytes, Long fetchTimeout, Integer maxFetchRetry)

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object o)
List<URI>	getPrefixes()
List<URI>	getUris()
int	hashCode()
protected Collection<org.jets3t.service.model.S3Object>	initObjects() Initialize objects to be read by this firehose.
protected InputStream	openObjectStream(org.jets3t.service.model.S3Object object) Open an input stream from the given object.
protected InputStream	wrapObjectStream(org.jets3t.service.model.S3Object object, InputStream stream) Wrap the given input stream if needed.

Methods inherited from class io.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactory

connect, getFetchTimeout, getMaxCacheCapacityBytes, getMaxFetchCapacityBytes, getMaxFetchRetry, getPrefetchTriggerBytes

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.druid.data.input.FirehoseFactory

connect

Constructor Detail

StaticS3FirehoseFactory

public StaticS3FirehoseFactory(org.jets3t.service.impl.rest.httpclient.RestS3Service s3Client,
                               List<URI> uris,
                               List<URI> prefixes,
                               Long maxCacheCapacityBytes,
                               Long maxFetchCapacityBytes,
                               Long prefetchTriggerBytes,
                               Long fetchTimeout,
                               Integer maxFetchRetry)

Method Detail

getUris

public List<URI> getUris()

getPrefixes

public List<URI> getPrefixes()

initObjects

protected Collection<org.jets3t.service.model.S3Object> initObjects()
                                                             throws IOException

Description copied from class: AbstractTextFilesFirehoseFactory

Initialize objects to be read by this firehose. Since firehose factories are constructed whenever io.druid.indexing.common.task.Task objects are deserialized, actual initialization of objects is deferred until AbstractTextFilesFirehoseFactory.connect(StringInputRowParser, File) is called.

Specified by:

initObjects in class AbstractTextFilesFirehoseFactory<org.jets3t.service.model.S3Object>

Returns:

a collection of initialized objects.

Throws:

IOException

openObjectStream

protected InputStream openObjectStream(org.jets3t.service.model.S3Object object)
                                throws IOException

Description copied from class: AbstractTextFilesFirehoseFactory

Open an input stream from the given object. If the object is compressed, this method should return a byte stream as it is compressed. The object compression should be handled in AbstractTextFilesFirehoseFactory.wrapObjectStream(Object, InputStream).

Specified by:

openObjectStream in class AbstractTextFilesFirehoseFactory<org.jets3t.service.model.S3Object>

Parameters:

object - an object to be read

Returns:

an input stream for the object

Throws:

IOException

wrapObjectStream

protected InputStream wrapObjectStream(org.jets3t.service.model.S3Object object,
                                       InputStream stream)
                                throws IOException

Description copied from class: AbstractTextFilesFirehoseFactory

Wrap the given input stream if needed. The decompression logic should be applied to the given stream if the object is compressed.

Specified by:

wrapObjectStream in class AbstractTextFilesFirehoseFactory<org.jets3t.service.model.S3Object>

Parameters:

object - an input object

stream - a stream for the object

Returns:

Throws:

IOException

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object