com.nicta.scoobi.impl.mapreducer

BridgeStore

case class BridgeStore[A](bridgeStoreId: String, wf: WireReaderWriter, checkpoint: Option[Checkpoint] = None, compression: Option[Compression] = None, pattern: String = BridgeStore.pattern) extends DataSource[NullWritable, ScoobiWritable[A], A] with DataSink[NullWritable, ScoobiWritable[A], A] with Bridge with Product with Serializable

A bridge store is any data that moves between MSCRs. It must first be computed, but may be removed once all successor MSCRs have consumed it.

Linear Supertypes
Serializable, Serializable, Product, Equals, Bridge, SinkSource, DataSink[NullWritable, ScoobiWritable[A], A], Sink, DataSource[NullWritable, ScoobiWritable[A], A], Source, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. BridgeStore
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. Bridge
  7. SinkSource
  8. DataSink
  9. Sink
  10. DataSource
  11. Source
  12. AnyRef
  13. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new BridgeStore(bridgeStoreId: String, wf: WireReaderWriter, checkpoint: Option[Checkpoint] = None, compression: Option[Compression] = None, pattern: String = BridgeStore.pattern)

Value Members

  1. final def !=(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  5. val bridgeStoreId: String

  6. val checkpoint: Option[Checkpoint]

    returns

    the checkpoint parameters if this sink is a Checkpoint

    Definition Classes
    BridgeStore → SinkSource

  7. def checkpointExists(implicit sc: ScoobiConfiguration): Boolean

    returns

    true if this Sink is a checkpoint and has been filled with data

    Definition Classes
    SinkSource

  8. def checkpointPath: Option[String]

    returns

    the path of the checkpoint

    Definition Classes
    SinkSource

  9. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  10. def compress: Sink

    returns

    a new sink with Gzip compression enabled

    Definition Classes
    Sink

  11. def compressWith(codec: CompressionCodec, compressionType: CompressionType = CompressionType.BLOCK): BridgeStore[Nothing]

    returns

    a new sink with compression enabled

    Definition Classes
    BridgeStore → Sink

  12. val compression: Option[Compression]

    returns

    a compression object if this sink is compressed

    Definition Classes
    BridgeStore → DataSink

  13. def configureCompression(configuration: Configuration): DataSink[NullWritable, ScoobiWritable[A], A]

    configure the compression for a given job

    configure the compression for a given job

    Definition Classes
    DataSink → Sink

  14. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  15. def equals(other: Any): Boolean

    Definition Classes
    BridgeStore → Equals → AnyRef → Any

  16. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  17. def freePath(implicit sc: ScoobiConfiguration): Unit

  18. def fromKeyValueConverter: InputConverter[NullWritable, ScoobiWritable[A], A]

    map the key-values of a source's InputFormat to the final type produced by it

    map the key-values of a source's InputFormat to the final type produced by it

    Definition Classes
    DataSource → Source

  19. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  20. def hashCode(): Int

    Definition Classes
    BridgeStore → AnyRef → Any

  21. val id: Int

    unique id for this Sink

    unique id for this Sink

    Definition Classes
    BridgeStore → DataSink → Sink → DataSource → Source

  22. def inputCheck(implicit sc: ScoobiConfiguration): Unit

    check the validity of the source specification

    check the validity of the source specification

    Definition Classes
    BridgeStore → DataSource → Source

  23. def inputConfigure(job: Job)(implicit sc: ScoobiConfiguration): Unit

    configure the source.

    configure the source.

    Definition Classes
    BridgeStore → DataSource → Source

  24. lazy val inputConverter: InputConverter[NullWritable, ScoobiWritable[A], A]

    Definition Classes
    BridgeStore → DataSource

  25. lazy val inputFormat: Class[SequenceFileInputFormat[NullWritable, ScoobiWritable[A]]]

    the InputFormat specifying the type of input for this source

    the InputFormat specifying the type of input for this source

    Definition Classes
    BridgeStore → DataSource → Source

  26. def inputSize(implicit sc: ScoobiConfiguration): Long

    returns

    the size in bytes of the data being input by this source

    Definition Classes
    BridgeStore → DataSource → Source

  27. def isCheckpoint: Boolean

    returns

    true if this sink is a checkpoint

    Definition Classes
    SinkSource

  28. def isCompressed: Boolean

    returns

    true if this Sink is compressed

    Definition Classes
    DataSink → Sink

  29. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  30. def isSinkResult(tag: Int): (Path) ⇒ Boolean

    returns

    true if the file path has the name of an output channel with the proper tag and index or if it is a _SUCCESS file

    Definition Classes
    Sink

  31. lazy val logger: Log

  32. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  33. final def notify(): Unit

    Definition Classes
    AnyRef

  34. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  35. def outputCheck(implicit sc: ScoobiConfiguration): Unit

    Check the validity of the DataSink specification.

    Check the validity of the DataSink specification.

    Definition Classes
    BridgeStore → DataSink → Sink

  36. def outputConfigure(job: Job)(implicit sc: ScoobiConfiguration): Unit

    Configure the DataSink.

    Configure the DataSink.

    Definition Classes
    BridgeStore → DataSink → Sink

  37. lazy val outputConverter: ScoobiWritableOutputConverter[A]

    Maps the type consumed by this DataSink to the key-values of its OutputFormat.

    Maps the type consumed by this DataSink to the key-values of its OutputFormat.

    Definition Classes
    BridgeStore → DataSink → Sink

  38. def outputFormat(implicit sc: ScoobiConfiguration): Class[SequenceFileOutputFormat[NullWritable, ScoobiWritable[A]]]

    The OutputFormat specifying the type of output for this DataSink.

    The OutputFormat specifying the type of output for this DataSink.

    Definition Classes
    BridgeStore → DataSink → Sink

  39. def outputKeyClass(implicit sc: ScoobiConfiguration): Class[NullWritable]

    The Class of the OutputFormat's key.

    The Class of the OutputFormat's key.

    Definition Classes
    BridgeStore → DataSink → Sink

  40. def outputPath(implicit sc: ScoobiConfiguration): Some[Path]

    returns

    the path for this Sink.

    Definition Classes
    BridgeStore → Sink

  41. def outputSetup(implicit sc: ScoobiConfiguration): Unit

    This method is called just before writing data to the sink

    This method is called just before writing data to the sink

    Definition Classes
    SinkSource → Sink

  42. def outputTeardown(implicit sc: ScoobiConfiguration): Unit

    This method is called just after writing data to the sink

    This method is called just after writing data to the sink

    Definition Classes
    DataSink → Sink

  43. def outputValueClass(implicit sc: ScoobiConfiguration): Class[ScoobiWritable[A]]

    The Class of the OutputFormat's value.

    The Class of the OutputFormat's value.

    Definition Classes
    BridgeStore → DataSink → Sink

  44. def path(implicit sc: ScoobiConfiguration): Path

  45. val pattern: String

  46. def readAsIterable(implicit sc: ScoobiConfiguration): Iterable[A]

    Read the contents of this bridge store sequence files as an Iterable collection.

    Read the contents of this bridge store sequence files as an Iterable collection. The underlying Iterator has a lazy implementation and will only bring one element into memory at a time

    Definition Classes
    BridgeStore → Bridge

  47. def readAsValue(implicit sc: ScoobiConfiguration): Any

    Definition Classes
    Bridge

  48. def rtClass(implicit sc: ScoobiConfiguration): RuntimeClass

    rtClass will be created at runtime as part of building the MapReduce job.

  49. lazy val stringId: String

    unique id for this Sink, as a string.

    unique id for this Sink, as a string. Can be used to create a file path

    Definition Classes
    BridgeStore → DataSink → Sink

  50. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  51. def toSource: Source

    Definition Classes
    BridgeStore → SinkSource

  52. def toString(): String

    Definition Classes
    BridgeStore → AnyRef → Any

  53. lazy val typeName: String

    type of the generated class for this Bridge

  54. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  55. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  56. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  57. val wf: WireReaderWriter

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from Bridge

Inherited from SinkSource

Inherited from DataSink[NullWritable, ScoobiWritable[A], A]

Inherited from Sink

Inherited from DataSource[NullWritable, ScoobiWritable[A], A]

Inherited from Source

Inherited from AnyRef

Inherited from Any

Ungrouped