public interface DataSegmentPusher
Modifier and Type | Field and Description |
---|---|
static com.google.common.base.Joiner |
JOINER |
Modifier and Type | Method and Description |
---|---|
default List<String> |
getAllowedPropertyPrefixesForHadoop()
Property prefixes that should be added to the "allowedHadoopPrefix" config for passing down to Hadoop jobs.
|
static String |
getDefaultStorageDir(DataSegment segment) |
String |
getPathForHadoop() |
String |
getPathForHadoop(String dataSource)
Deprecated.
|
default String |
getStorageDir(DataSegment dataSegment) |
default String |
makeIndexPathName(DataSegment dataSegment,
String indexName) |
Map<String,Object> |
makeLoadSpec(URI finalIndexZipFilePath) |
DataSegment |
push(File file,
DataSegment segment,
boolean replaceExisting)
Pushes index files and segment descriptor to deep storage.
|
@Deprecated String getPathForHadoop(String dataSource)
String getPathForHadoop()
DataSegment push(File file, DataSegment segment, boolean replaceExisting) throws IOException
file
- directory containing index filessegment
- segment descriptorreplaceExisting
- overwrites existing objects if true, else leaves existing objects unchanged on conflict.
The behavior of the indexer determines whether this should be true or false. For example,
since Tranquility does not guarantee that replica tasks will generate indexes with the same
data, the first segment pushed should be favored since otherwise multiple historicals may
load segments with the same identifier but different contents which is a bad situation. On
the other hand, indexers that maintain exactly-once semantics by storing checkpoint data can
lose or repeat data if it fails to write a segment because it already exists and overwriting
is not permitted. This situation can occur if a task fails after pushing to deep storage but
before writing to the metadata storage, see: https://github.com/druid-io/druid/issues/5161.
If replaceExisting is true, existing objects MUST be overwritten, since failure to do so
will break exactly-once semantics. If replaceExisting is false, existing objects SHOULD be
prioritized but it is acceptable if they are overwritten (deep storages may be eventually
consistent or otherwise unable to support transactional writes).IOException
default String getStorageDir(DataSegment dataSegment)
default String makeIndexPathName(DataSegment dataSegment, String indexName)
default List<String> getAllowedPropertyPrefixesForHadoop()
static String getDefaultStorageDir(DataSegment segment)
Copyright © 2011–2018. All rights reserved.