io.druid.query.groupby.epinephelinae

Class BufferGrouper<KeyType>

java.lang.Object

io.druid.query.groupby.epinephelinae.BufferGrouper<KeyType>

All Implemented Interfaces:

Grouper<KeyType>, Closeable, AutoCloseable

public class BufferGrouper<KeyType>
extends Object
implements Grouper<KeyType>

Grouper based around a hash table and companion array in a single ByteBuffer. Not thread-safe. The buffer has two parts: a table arena (offset 0 to tableArenaSize) and an array containing pointers objects in the table (tableArenaSize until the end of the buffer). The table uses open addressing with linear probing on collisions. Each bucket contains the key hash (with the high bit set to signify the bucket is used), the serialized key (which are a fixed size) and scratch space for BufferAggregators (which is also fixed size). The actual table is represented by "tableBuffer", which points to the same memory as positions "tableStart" through "tableStart + buckets * bucketSize" of "buffer". Everything else in the table arena is potentially junk. The array of pointers starts out ordered by insertion order, but might be sorted on calls to iterator(boolean). This sorting is done in-place to avoid materializing the full array of pointers. The first "size" pointers in the array of pointers are valid; everything else is potentially junk. The table is periodically grown to accommodate more keys. Even though starting small is not necessary to control memory use (we already have the entire buffer allocated) or iteration speed (iteration is fast due to the array of pointers) it still helps significantly on initialization times. Otherwise, we'd need to clear the used bits of each bucket in the entire buffer, which is a lot of writes if the buckets are small.

Nested Class Summary

Nested classes/interfaces inherited from interface io.druid.query.groupby.epinephelinae.Grouper

Grouper.Entry<T>, Grouper.KeyComparator, Grouper.KeySerde<T>, Grouper.KeySerdeFactory<T>

Constructor Summary

Constructors

Constructor and Description
BufferGrouper(com.google.common.base.Supplier<ByteBuffer> bufferSupplier, Grouper.KeySerde<KeyType> keySerde, ColumnSelectorFactory columnSelectorFactory, AggregatorFactory[] aggregatorFactories, int bufferGrouperMaxSize, float maxLoadFactor, int initialBuckets)

Method Summary

Modifier and Type	Method and Description
boolean	aggregate(KeyType key) Aggregate the current row with the provided key.
boolean	aggregate(KeyType key, int keyHash) Aggregate the current row with the provided key.
void	close() Close the grouper and release associated resources.
void	init() Initialize the grouper.
boolean	isInitialized() Check this grouper is initialized or not.
Iterator<Grouper.Entry<KeyType>>	iterator(boolean sorted) Iterate through entries.
void	reset() Reset the grouper to its initial state.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BufferGrouper

public BufferGrouper(com.google.common.base.Supplier<ByteBuffer> bufferSupplier,
                     Grouper.KeySerde<KeyType> keySerde,
                     ColumnSelectorFactory columnSelectorFactory,
                     AggregatorFactory[] aggregatorFactories,
                     int bufferGrouperMaxSize,
                     float maxLoadFactor,
                     int initialBuckets)

Method Detail

init

public void init()

Description copied from interface: Grouper

Initialize the grouper. This method needs to be called before calling Grouper.aggregate(Object) and Grouper.aggregate(Object, int).

Specified by:

init in interface Grouper<KeyType>

isInitialized

public boolean isInitialized()

Description copied from interface: Grouper

Check this grouper is initialized or not.

Specified by:

isInitialized in interface Grouper<KeyType>

Returns:

true if the grouper is already initialized, otherwise false.

aggregate

public boolean aggregate(KeyType key,
                         int keyHash)

Description copied from interface: Grouper

Aggregate the current row with the provided key. Some implementations are thread-safe and some are not.

Specified by:

aggregate in interface Grouper<KeyType>

Parameters:

key - key object

keyHash - result of Groupers.hash(Object) on the key

Returns:

true if the row was aggregated, false if not due to hitting resource limits

aggregate

public boolean aggregate(KeyType key)

Description copied from interface: Grouper

Aggregate the current row with the provided key. Some implementations are thread-safe and some are not.

Specified by:

aggregate in interface Grouper<KeyType>

Parameters:

key - key

Returns:

true if the row was aggregated, false if not due to hitting resource limits

reset

public void reset()

Description copied from interface: Grouper

Reset the grouper to its initial state.

Specified by:

reset in interface Grouper<KeyType>

iterator

public Iterator<Grouper.Entry<KeyType>> iterator(boolean sorted)

Description copied from interface: Grouper

Iterate through entries. If a comparator is provided, do a sorted iteration.

Once this method is called, writes are no longer safe. After you are done with the iterator returned by this method, you should either call Grouper.close() (if you are done with the Grouper), Grouper.reset() (if you want to reuse it), or Grouper.iterator(boolean) again if you want another iterator.

If "sorted" is true then the iterator will return sorted results. It will use KeyType's natural ordering on deserialized objects, and will use the KeySerde#comparator() on serialized objects. Woe be unto you if these comparators are not equivalent.

Specified by:

iterator in interface Grouper<KeyType>

Parameters:

sorted - return sorted results

Returns:

entry iterator

close

public void close()

Description copied from interface: Grouper

Close the grouper and release associated resources.

Specified by:

close in interface Grouper<KeyType>

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable