io.druid.examples.twitter

Class TwitterSpritzerFirehoseFactory

java.lang.Object

io.druid.examples.twitter.TwitterSpritzerFirehoseFactory

All Implemented Interfaces:

FirehoseFactory<InputRowParser>

public class TwitterSpritzerFirehoseFactory
extends Object
implements FirehoseFactory<InputRowParser>

Twitter "spritzer" Firehose Factory named "twitzer". Builds a Firehose that emits a stream of ?? with timestamps along with ??. The generated tuples have the form (timestamp, ????) where the timestamp is from the twitter event.

Example spec file:

Example query using POST to /druid/v2/?w (where w is an arbitrary parameter and the date and time is UTC):

Notes on twitter.com HTTP (REST) API: v1.0 will be disabled around 2013-03 so v1.1 should be used; twitter4j 3.0 (not yet released) will support the v1.1 api. Specifically, we should be using https://stream.twitter.com/1.1/statuses/sample.json See: http://jira.twitter4j.org/browse/TFJ-186

Notes on JSON parsing: as of twitter4j 2.2.x, the json parser has some bugs (ex: Status.toString() can have number format exceptions), so it might be necessary to extract raw json and process it separately. If so, set twitter4.jsonStoreEnabled=true and look at DataObjectFactory#getRawJSON(); com.fasterxml.jackson.databind.ObjectMapper should be used to parse.

Author:

pbaclace

Constructor Summary

Constructors

Constructor and Description
TwitterSpritzerFirehoseFactory(Integer maxEventCount, Integer maxRunMinutes)

Method Summary

Modifier and Type	Method and Description
Firehose	connect(InputRowParser parser, File temporaryDirectory) Initialization method that connects up the fire hose.
int	getMaxEventCount()
int	getMaxRunMinutes()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.druid.data.input.FirehoseFactory

connect

Constructor Detail

TwitterSpritzerFirehoseFactory

public TwitterSpritzerFirehoseFactory(Integer maxEventCount,
                                      Integer maxRunMinutes)

Method Detail

connect

public Firehose connect(InputRowParser parser,
                        File temporaryDirectory)
                 throws IOException

Description copied from interface: FirehoseFactory

Initialization method that connects up the fire hose. If this method returns successfully it should be safe to call hasMore() on the returned Firehose (which might subsequently block).

If this method returns null, then any attempt to call hasMore(), nextRow(), commit() and close() on the return value will throw a surprising NPE. Throwing IOException on connection failure or runtime exception on invalid configuration is preferred over returning null.

Some fire hoses like PrefetchableTextFilesFirehoseFactory may use a temporary directory to cache data in it.

Specified by:

connect in interface FirehoseFactory<InputRowParser>

Parameters:

parser - an input row parser

temporaryDirectory - a directory where temporary files are stored

Throws:

IOException

getMaxEventCount

public int getMaxEventCount()

getMaxRunMinutes

public int getMaxRunMinutes()