mapreducer

Type Members

1. case class BridgeStore[A](bridgeStoreId: String, wf: WireReaderWriter, checkpoint: Option[Checkpoint] = None, compression: Option[Compression] = None, pattern: String = BridgeStore.pattern) extends DataSource[NullWritable, ScoobiWritable[A], A] with DataSink[NullWritable, ScoobiWritable[A], A] with Bridge with Product with Serializable

   

   A bridge store is any data that moves between MSCRs.

   A bridge store is any data that moves between MSCRs. It must first be computed, but may be removed once all successor MSCRs have consumed it.

2. class BridgeStoreIterator[A] extends Iterator[A]

   

3. class ChannelOutputFormat extends AnyRef

   

   A class that simplifies writing output to different paths and with different types depending on the output channel and required outputs per channel

4. class ChannelRecordReader[K, V] extends RecordReader[K, V]

   

   A RecordReader that delegates the functionality to the underlying record reader in TaggedInputSplit.

5. class ChannelsInputFormat[K, V] extends InputFormat[K, V]

   

   An input format that delegates to multiple input formats, one for each input channel.

6. class CompositeInputOutputContext extends InputOutputContext

   

   This context either writes to the hadoop context or to an output channel depending on the value tag

   This context either writes to the hadoop context or to an output channel depending on the value tag

   If the result of the map operation must go to the reducer, use the normal hadoop context if it must go to an output file directly (i.e. to a "bypass" output tag), use the corresponding output channel as a reducer

7. class Env extends Environment

   

   A reference to the storage location of a value that represents the "environment" in which a computation is performed within.

   A reference to the storage location of a value that represents the "environment" in which a computation is performed within. "Environment" here refers to the the "side-input" of a "ParallelDo". Computed environments are pushed/pulled to/from the distributed cache.

8. class MscrCombiner extends Reducer[TaggedKey, TaggedValue, TaggedKey, TaggedValue]

   

   Hadoop Combiner class for an MSCR.

9. class MscrMapper extends Mapper[Any, Any, TaggedKey, TaggedValue]

   

   Hadoop Mapper class for an MSCR

   Hadoop Mapper class for an MSCR

   It is composed of several tagged mappers which are taking inputs of a given type on a channel and emitting the result for different tagged outputs

10. class MscrReducer extends Reducer[TaggedKey, TaggedValue, Any, Any]

    

    Hadoop Reducer class for an MSCR.

11. class ScoobiWritableOutputConverter[A] extends OutputConverter[NullWritable, ScoobiWritable[A], A]

    

    OutputConverter for a bridges.

    OutputConverter for a bridges. The expectation is that by the time toKeyValue is called, the Class for 'value' will exist and be known by the ClassLoader.

12. class TaggedInputSplit extends InputSplit with Configurable with Writable

    

    A wrapper around an InputSplit that is tagged with an input channel id.

    A wrapper around an InputSplit that is tagged with an input channel id. Is used with ChannelInputFormat.

13. case class UntaggedValues(tag: Int, values: Iterable[TaggedValue]) extends Iterable[Any] with Product with Serializable

    

14. case class VectorEmitterWriter(context: InputOutputContext) extends EmitterWriter with InputOutputContextScoobiJobContext with Product with Serializable

    

    In memory emitter writer saving the values to a Vector