com.nicta.scoobi.impl.plan.mscr

GbkOutputChannel

case class GbkOutputChannel(groupByKey: GroupByKey, combiner: Option[Combine] = None, reducer: Option[ParallelDo] = None, graph: Graph = Graph(Root(Seq()))) extends MscrOutputChannel with Product with Serializable

Output channel for a GroupByKey.

It can optionally have a reducer and / or a combiner applied to the grouped key/values.

The possible combinations are

There can not be gbk -> reducer -> combiner because in that case the second combiner is transformed as a parallelDo by the Optimiser

Linear Supertypes
Serializable, Serializable, Product, Equals, MscrOutputChannel, OutputChannel, Channel, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. GbkOutputChannel
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. MscrOutputChannel
  7. OutputChannel
  8. Channel
  9. AnyRef
  10. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new GbkOutputChannel(groupByKey: GroupByKey, combiner: Option[Combine] = None, reducer: Option[ParallelDo] = None, graph: Graph = Graph(Root(Seq())))

Value Members

  1. final def !=(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  5. def cleanup(channelOutput: ChannelOutputFormat)(implicit configuration: Configuration): Unit

    invoke the reducer cleanup if there is one

    invoke the reducer cleanup if there is one

    Definition Classes
    GbkOutputChannel → MscrOutputChannel → OutputChannel

  6. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  7. def collectOutputs(outputFiles: Seq[Path])(implicit sc: ScoobiConfiguration, fileSystems: FileSystems): Unit

    copy all outputs files to the destinations specified by sink files

    copy all outputs files to the destinations specified by sink files

    Definition Classes
    MscrOutputChannel → OutputChannel

  8. def collectSuccessFile(successFile: Option[Path])(implicit sc: ScoobiConfiguration, fileSystems: FileSystems): Unit

    copy the success file to the destinations specified by sink files

    copy the success file to the destinations specified by sink files

    Definition Classes
    MscrOutputChannel → OutputChannel

  9. val combiner: Option[Combine]

  10. def convert(sink: Sink, x: Any)(implicit configuration: Configuration): (Any, Any)

    use the output converter of a sink to convert a value to a key/value

    use the output converter of a sink to convert a value to a key/value

    Attributes
    protected
    Definition Classes
    MscrOutputChannel

  11. def createEmitter(channelOutput: ChannelOutputFormat)(implicit configuration1: Configuration): EmitterWriter with InputOutputContextScoobiJobContext

    create an emitter to output values on the current tag for each sink.

    create an emitter to output values on the current tag for each sink. Values are converted to (key, values) using the sink output converter. This emitter is used by both the GbkOutputChannel and the BypassOutputChannel

    Attributes
    protected
    Definition Classes
    MscrOutputChannel

  12. var emitter: EmitterWriter

    Attributes
    protected
    Definition Classes
    MscrOutputChannel

  13. var environment: Any

    store the reducer environment during the setup if there is one

    store the reducer environment during the setup if there is one

    Attributes
    protected

  14. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  15. def equals(a: Any): Boolean

    Definition Classes
    MscrOutputChannel → AnyRef → Any

  16. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  17. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  18. val graph: Graph

    Definition Classes
    GbkOutputChannel → MscrOutputChannel

  19. lazy val graphNodes: Graph

    Definition Classes
    MscrOutputChannel

  20. val groupByKey: GroupByKey

  21. def hashCode(): Int

    Definition Classes
    MscrOutputChannel → AnyRef → Any

  22. lazy val inputNodes: Seq[ValueNode]

    return the reducer environment if there is one

    return the reducer environment if there is one

    Definition Classes
    GbkOutputChannel → OutputChannel

  23. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  24. lazy val lastNode: ProcessNodeImpl with Serializable { ... /* 2 definitions in type refinement */ }

    returns

    the last node of this channel

    Definition Classes
    GbkOutputChannel → MscrOutputChannel

  25. implicit lazy val logger: Log

    Attributes
    protected
    Definition Classes
    MscrOutputChannel

  26. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  27. final def notify(): Unit

    Definition Classes
    AnyRef

  28. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  29. lazy val outputNodes: Seq[CompNode]

    output nodes for this channel

    output nodes for this channel

    Definition Classes
    GbkOutputChannel → OutputChannel

  30. def processNodes: Seq[ProcessNode]

    returns

    all the ProcessNodes for this channel

    Definition Classes
    GbkOutputChannel → Channel

  31. def reduce(key: Any, values: Iterable[Any], channelOutput: ChannelOutputFormat)(implicit configuration: Configuration): Unit

    reduce all the key/values with either the reducer, or the combiner otherwise just emit key/value pairs.

    reduce all the key/values with either the reducer, or the combiner otherwise just emit key/value pairs.

    The key and values are untagged. The emitter is in charge of writing them to the proper tag, which is the channel's tag

    Definition Classes
    GbkOutputChannel → OutputChannel

  32. val reducer: Option[ParallelDo]

  33. def scoobiConfiguration(configuration: Configuration): ScoobiConfiguration

    create a ScoobiConfiguration from a Hadoop one

    create a ScoobiConfiguration from a Hadoop one

    Attributes
    protected
    Definition Classes
    MscrOutputChannel

  34. implicit var scoobiConfiguration: ScoobiConfiguration

    Attributes
    protected

  35. def setup(channelOutput: ChannelOutputFormat)(implicit configuration: Configuration): Unit

    only the reducer needs to be setup if there is one

    only the reducer needs to be setup if there is one

    Definition Classes
    GbkOutputChannel → MscrOutputChannel → OutputChannel

  36. lazy val sinks: Seq[Sink]

    returns

    all the sinks defined by the nodes of the input channel

    Definition Classes
    MscrOutputChannel → OutputChannel

  37. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  38. lazy val tag: Int

    the tag identifying a GbkOutputChannel is the groupByKey id

    the tag identifying a GbkOutputChannel is the groupByKey id

    Definition Classes
    GbkOutputChannel → OutputChannel

  39. def toString(): String

    Definition Classes
    GbkOutputChannel → AnyRef → Any

  40. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  41. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  42. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from MscrOutputChannel

Inherited from OutputChannel

Inherited from Channel

Inherited from AnyRef

Inherited from Any

Ungrouped