com.nicta.scoobi.impl.mapreducer

BridgeStore

case class BridgeStore[A](bridgeStoreId: String, wf: WireReaderWriter, checkpoint: Option[Checkpoint] = scala.None, compression: Option[Compression] = scala.None, pattern: String = "ch*") extends DataSource[NullWritable, ScoobiWritable[A], A] with DataSink[NullWritable, ScoobiWritable[A], A] with Bridge with Product with Serializable

A bridge store is any data that moves between MSCRs. It must first be computed, but may be removed once all successor MSCRs have consumed it.

Linear Supertypes
Serializable, Serializable, Product, Equals, Bridge, SinkSource, DataSink[NullWritable, ScoobiWritable[A], A], Sink, DataSource[NullWritable, ScoobiWritable[A], A], Source, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. BridgeStore
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. Bridge
  7. SinkSource
  8. DataSink
  9. Sink
  10. DataSource
  11. Source
  12. AnyRef
  13. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new BridgeStore(bridgeStoreId: String, wf: WireReaderWriter, checkpoint: Option[Checkpoint] = scala.None, compression: Option[Compression] = scala.None, pattern: String = "ch*")

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  7. val bridgeStoreId: String

  8. val checkpoint: Option[Checkpoint]

    returns

    the checkpoint parameters if this sink is a Checkpoint

    Definition Classes
    BridgeStore → SinkSource

  9. def checkpointExists(implicit sc: ScoobiConfiguration): Boolean

    returns

    true if this Sink is a checkpoint and has been filled with data

    Definition Classes
    SinkSource

  10. def checkpointPath: Option[String]

    returns

    the path of the checkpoint

    Definition Classes
    SinkSource

  11. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  12. def compress: Sink

    returns

    a new sink with Gzip compression enabled

    Definition Classes
    Sink

  13. def compressWith(codec: CompressionCodec, compressionType: CompressionType = CompressionType.BLOCK): BridgeStore[Nothing]

    returns

    a new sink with compression enabled

    Definition Classes
    BridgeStore → Sink

  14. val compression: Option[Compression]

    returns

    a compression object if this sink is compressed

    Definition Classes
    BridgeStore → DataSink

  15. def configureCompression(configuration: Configuration): DataSink[NullWritable, ScoobiWritable[A], A]

    configure the compression for a given job

    configure the compression for a given job

    Definition Classes
    DataSink → Sink

  16. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  17. def equals(other: Any): Boolean

    Definition Classes
    BridgeStore → Equals → AnyRef → Any

  18. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  19. def freePath(implicit sc: ScoobiConfiguration): Unit

  20. def fromKeyValueConverter: InputConverter[NullWritable, ScoobiWritable[A], A]

    map the key-values of a source's InputFormat to the final type produced by it

    map the key-values of a source's InputFormat to the final type produced by it

    Definition Classes
    DataSource → Source

  21. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  22. def hashCode(): Int

    Definition Classes
    BridgeStore → AnyRef → Any

  23. val id: Int

    unique id for this Sink

    unique id for this Sink

    Definition Classes
    BridgeStore → DataSink → Sink → DataSource → Source

  24. def inputCheck(implicit sc: ScoobiConfiguration): Unit

    check the validity of the source specification

    check the validity of the source specification

    Definition Classes
    BridgeStore → DataSource → Source

  25. def inputConfigure(job: Job)(implicit sc: ScoobiConfiguration): Unit

    configure the source.

    configure the source.

    Definition Classes
    BridgeStore → DataSource → Source

  26. lazy val inputConverter: InputConverter[NullWritable, ScoobiWritable[A], A]

    Definition Classes
    BridgeStore → DataSource

  27. lazy val inputFormat: Class[SequenceFileInputFormat[NullWritable, ScoobiWritable[A]]]

    Definition Classes
    BridgeStore → DataSource → Source

  28. def inputSize(implicit sc: ScoobiConfiguration): Long

    returns

    the size in bytes of the data being input by this source

    Definition Classes
    BridgeStore → DataSource → Source

  29. def isCheckpoint: Boolean

    returns

    true if this sink is a checkpoint

    Definition Classes
    SinkSource

  30. def isCompressed: Boolean

    returns

    true if this Sink is compressed

    Definition Classes
    DataSink → Sink

  31. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  32. lazy val logger: Log

  33. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  34. final def notify(): Unit

    Definition Classes
    AnyRef

  35. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  36. def outputCheck(implicit sc: ScoobiConfiguration): Unit

    Check the validity of the DataSink specification.

    Check the validity of the DataSink specification.

    Definition Classes
    BridgeStore → DataSink → Sink

  37. def outputConfigure(job: Job)(implicit sc: ScoobiConfiguration): Unit

    Configure the DataSink.

    Configure the DataSink.

    Definition Classes
    BridgeStore → DataSink → Sink

  38. lazy val outputConverter: ScoobiWritableOutputConverter[A]

    Definition Classes
    BridgeStore → DataSink → Sink

  39. def outputFormat(implicit sc: ScoobiConfiguration): Class[SequenceFileOutputFormat[NullWritable, ScoobiWritable[A]]]

    The OutputFormat specifying the type of output for this DataSink.

    The OutputFormat specifying the type of output for this DataSink.

    Definition Classes
    BridgeStore → DataSink → Sink

  40. def outputKeyClass(implicit sc: ScoobiConfiguration): Class[NullWritable]

    The Class of the OutputFormat's key.

    The Class of the OutputFormat's key.

    Definition Classes
    BridgeStore → DataSink → Sink

  41. def outputPath(implicit sc: ScoobiConfiguration): Some[Path]

    returns

    the path for this Sink.

    Definition Classes
    BridgeStore → Sink

  42. def outputSetup(implicit sc: ScoobiConfiguration): Unit

    This method is called just before writing data to the sink

    This method is called just before writing data to the sink

    Definition Classes
    SinkSource → Sink

  43. def outputTeardown(implicit sc: ScoobiConfiguration): Unit

    This method is called just after writing data to the sink

    This method is called just after writing data to the sink

    Definition Classes
    DataSink → Sink

  44. def outputValueClass(implicit sc: ScoobiConfiguration): Class[ScoobiWritable[A]]

    The Class of the OutputFormat's value.

    The Class of the OutputFormat's value.

    Definition Classes
    BridgeStore → DataSink → Sink

  45. def path(implicit sc: ScoobiConfiguration): Path

  46. val pattern: String

  47. def readAsIterable(implicit sc: ScoobiConfiguration): Iterable[A]

    Read the contents of this bridge store sequence files as an Iterable collection.

    Read the contents of this bridge store sequence files as an Iterable collection. The underlying Iterator has a lazy implementation and will only bring one element into memory at a time

    Definition Classes
    BridgeStore → Bridge

  48. def readAsValue(implicit sc: ScoobiConfiguration): Any

    Definition Classes
    Bridge

  49. def rtClass(implicit sc: ScoobiConfiguration): RuntimeClass

    rtClass will be created at runtime as part of building the MapReduce job.

  50. lazy val stringId: String

    Definition Classes
    BridgeStore → DataSink → Sink

  51. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  52. def toSource: Source

    Definition Classes
    BridgeStore → SinkSource

  53. def toString(): String

    Definition Classes
    BridgeStore → AnyRef → Any

  54. lazy val typeName: String

    type of the generated class for this Bridge

  55. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  56. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  57. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  58. val wf: WireReaderWriter

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from Bridge

Inherited from SinkSource

Inherited from DataSink[NullWritable, ScoobiWritable[A], A]

Inherited from Sink

Inherited from DataSource[NullWritable, ScoobiWritable[A], A]

Inherited from Source

Inherited from AnyRef

Inherited from Any

Ungrouped