- All Superinterfaces:
- AutoCloseable, Closeable
public interface FirehoseV2
extends Closeable
This is an interface that holds onto the stream of incoming data. Realtime data ingestion is built around this
abstraction. In order to add a new type of source for realtime data ingestion, all you need to do is implement
one of these and register it with the Main.
In contrast to Firehose v1 version, FirehoseV2 will always operate in a "peek, then advance" manner.
And the intended usage patttern is
1. Call start()
2. Read currRow()
3. Call advance()
4. If index should be committed: commit()
5. GOTO 2
Note that commit() is being called *after* advance.
This object acts a lot like an Iterator, but it doesn't extend the Iterator interface because it extends
Closeable and it is very important that the close() method doesn't get forgotten, which is easy to do if this
gets passed around as an Iterator.
The implementation of this interface only needs to be minimally thread-safe. The methods ##start(), ##advance(),
##currRow() and ##makeCommitter() are all called from the same thread. ##makeCommitter(), however, returns a callback
which will be called on another thread, so the operations inside of that callback must be thread-safe.