io.druid.examples.rand

Class RandomFirehoseFactory

java.lang.Object

io.druid.examples.rand.RandomFirehoseFactory

All Implemented Interfaces:

io.druid.data.input.FirehoseFactory<io.druid.data.input.impl.InputRowParser>

public class RandomFirehoseFactory
extends Object
implements io.druid.data.input.FirehoseFactory<io.druid.data.input.impl.InputRowParser>

Random value sequence Firehost Factory named "rand". Builds a Firehose that emits a stream of random numbers (outColumn, a positive double) with timestamps along with an associated token (target). This provides a timeseries that requires no network access for demonstration, characterization, and testing. The generated tuples can be thought of as asynchronously produced triples (timestamp, outColumn, target) where the timestamp varies depending on speed of processing.

InputRows are produced as fast as requested, so this can be used to determine the upper rate of ingest if sleepUsec is set to 0; nTokens specifies how many associated target labels are used. Generation is round-robin for nTokens and sleep occurs every nPerSleep values generated. A random number seed can be used by setting the firehose parameter "seed" to a non-zero value so that values can be reproducible (but note that timestamp is not deterministic because timestamps are obtained at the moment an event is delivered.) Values are offset by adding the modulus of the token number to the random number so that token values have distinct, non-overlapping ranges.

Example spec file:

 [{
 "schema" : { "dataSource":"randseq",
 "aggregators":[ {"type":"count", "name":"events"},
 {"type":"doubleSum","name":"outColumn","fieldName":"inColumn"} ],
 "indexGranularity":"minute",
 "shardSpec" : { "type": "none" } },
 "config" : { "maxRowsInMemory" : 50000,
 "intermediatePersistPeriod" : "PT2m" },

 "firehose" : { "type" : "rand",
 "sleepUsec": 100000,
 "maxGeneratedRows" : 5000000,
 "seed" : 0,
 "nTokens" : 19,
 "nPerSleep" : 3
 },

 "plumber" : { "type" : "realtime",
 "windowPeriod" : "PT5m",
 "segmentGranularity":"hour",
 "basePersistDirectory" : "/tmp/realtime/basePersist" }
 }]
 

Example query using POST to /druid/v2/ (where UTC date and time MUST include the current hour):

 {
 "queryType": "groupBy",
 "dataSource": "randSeq",
 "granularity": "all",
 "dimensions": [],
 "aggregations":[
 { "type": "count", "name": "rows"},
 { "type": "doubleSum", "fieldName": "events", "name": "e"},
 { "type": "doubleSum", "fieldName": "outColumn", "name": "randomNumberSum"}
 ],
 "postAggregations":[
 {  "type":"arithmetic",
 "name":"avg_random",
 "fn":"/",
 "fields":[ {"type":"fieldAccess","name":"randomNumberSum","fieldName":"randomNumberSum"},
 {"type":"fieldAccess","name":"rows","fieldName":"rows"} ]}
 ],
 "intervals":["2012-10-01T00:00/2020-01-01T00"]
 }
 

Constructor Summary

Constructors

Constructor and Description
RandomFirehoseFactory(Long sleepUsec, Long maxGeneratedRows, Long seed, Integer nTokens, Integer nPerSleep)

Method Summary

Methods

Modifier and Type	Method and Description
io.druid.data.input.Firehose	connect(io.druid.data.input.impl.InputRowParser parser)
io.druid.data.input.impl.InputRowParser	getParser()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RandomFirehoseFactory

public RandomFirehoseFactory(Long sleepUsec,
                     Long maxGeneratedRows,
                     Long seed,
                     Integer nTokens,
                     Integer nPerSleep)

Method Detail

connect

public io.druid.data.input.Firehose connect(io.druid.data.input.impl.InputRowParser parser)
                                     throws IOException

Specified by:

connect in interface io.druid.data.input.FirehoseFactory<io.druid.data.input.impl.InputRowParser>

Throws:

IOException

getParser

public io.druid.data.input.impl.InputRowParser getParser()

Specified by:

getParser in interface io.druid.data.input.FirehoseFactory<io.druid.data.input.impl.InputRowParser>