io.druid.data.input.impl

Class AbstractTextFilesFirehoseFactory<T>

java.lang.Object

io.druid.data.input.impl.AbstractTextFilesFirehoseFactory<T>

Type Parameters:

T - object type representing input data

All Implemented Interfaces:

FirehoseFactory<StringInputRowParser>

Direct Known Subclasses:

LocalFirehoseFactory, PrefetchableTextFilesFirehoseFactory

public abstract class AbstractTextFilesFirehoseFactory<T>
extends Object
implements FirehoseFactory<StringInputRowParser>

This is an abstract class for firehose factory for making firehoses reading text files. It provides an unified connect(StringInputRowParser, File) implementation for its subclasses.

Constructor Summary

Constructors

Constructor and Description
AbstractTextFilesFirehoseFactory()

Method Summary

Modifier and Type	Method and Description
Firehose	connect(StringInputRowParser firehoseParser, File temporaryDirectory) Initialization method that connects up the fire hose.
protected abstract Collection<T>	initObjects() Initialize objects to be read by this firehose.
protected abstract InputStream	openObjectStream(T object) Open an input stream from the given object.
protected abstract InputStream	wrapObjectStream(T object, InputStream stream) Wrap the given input stream if needed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.druid.data.input.FirehoseFactory

connect

Constructor Detail

AbstractTextFilesFirehoseFactory

public AbstractTextFilesFirehoseFactory()

Method Detail

connect

public Firehose connect(StringInputRowParser firehoseParser,
                        File temporaryDirectory)
                 throws IOException

Description copied from interface: FirehoseFactory

Initialization method that connects up the fire hose. If this method returns successfully it should be safe to call hasMore() on the returned Firehose (which might subsequently block).

If this method returns null, then any attempt to call hasMore(), nextRow(), commit() and close() on the return value will throw a surprising NPE. Throwing IOException on connection failure or runtime exception on invalid configuration is preferred over returning null.

Some fire hoses like PrefetchableTextFilesFirehoseFactory may use a temporary directory to cache data in it.

Specified by:

connect in interface FirehoseFactory<StringInputRowParser>

Parameters:

firehoseParser - an input row parser

temporaryDirectory - a directory where temporary files are stored

Throws:

IOException

initObjects

protected abstract Collection<T> initObjects()
                                      throws IOException

Initialize objects to be read by this firehose. Since firehose factories are constructed whenever io.druid.indexing.common.task.Task objects are deserialized, actual initialization of objects is deferred until connect(StringInputRowParser, File) is called.

Returns:

a collection of initialized objects.

Throws:

IOException

openObjectStream

protected abstract InputStream openObjectStream(T object)
                                         throws IOException

Open an input stream from the given object. If the object is compressed, this method should return a byte stream as it is compressed. The object compression should be handled in wrapObjectStream(Object, InputStream).

Parameters:

object - an object to be read

Returns:

an input stream for the object

Throws:

IOException

wrapObjectStream

protected abstract InputStream wrapObjectStream(T object,
                                                InputStream stream)
                                         throws IOException

Wrap the given input stream if needed. The decompression logic should be applied to the given stream if the object is compressed.

Parameters:

object - an input object

stream - a stream for the object

Returns:

Throws:

IOException