com.nicta.scoobi.io.sequence

SeqSink

case class SeqSink[K, V, B](path: String, keyClass: Class[K], valueClass: Class[V], outputConverter: InputOutputConverter[K, V, B], overwrite: Boolean, check: (Path, Boolean, ScoobiConfiguration) ⇒ Unit = core.Sink.defaultOutputCheck, checkpoint: Option[Checkpoint] = scala.None, compression: Option[Compression] = scala.None) extends DataSink[K, V, B] with SinkSource with Product with Serializable

class that abstracts all the common functionality of persisting to sequence files.

Linear Supertypes
Serializable, Serializable, Product, Equals, SinkSource, DataSink[K, V, B], Sink, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. SeqSink
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. SinkSource
  7. DataSink
  8. Sink
  9. AnyRef
  10. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SeqSink(path: String, keyClass: Class[K], valueClass: Class[V], outputConverter: InputOutputConverter[K, V, B], overwrite: Boolean, check: (Path, Boolean, ScoobiConfiguration) ⇒ Unit = core.Sink.defaultOutputCheck, checkpoint: Option[Checkpoint] = scala.None, compression: Option[Compression] = scala.None)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  7. val check: (Path, Boolean, ScoobiConfiguration) ⇒ Unit

  8. val checkpoint: Option[Checkpoint]

    returns

    the checkpoint parameters if this sink is a Checkpoint

    Definition Classes
    SeqSink → SinkSource

  9. def checkpointExists(implicit sc: ScoobiConfiguration): Boolean

    returns

    true if this Sink is a checkpoint and has been filled with data

    Definition Classes
    SinkSource

  10. def checkpointPath: Option[String]

    returns

    the path of the checkpoint

    Definition Classes
    SinkSource

  11. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  12. def compress: Sink

    returns

    a new sink with Gzip compression enabled

    Definition Classes
    Sink

  13. def compressWith(codec: CompressionCodec, compressionType: CompressionType = CompressionType.BLOCK): SeqSink[K, V, B]

    returns

    a new sink with compression enabled

    Definition Classes
    SeqSink → Sink

  14. val compression: Option[Compression]

    returns

    a compression object if this sink is compressed

    Definition Classes
    SeqSink → DataSink

  15. def configureCompression(configuration: Configuration): DataSink[K, V, B]

    configure the compression for a given job

    configure the compression for a given job

    Definition Classes
    DataSink → Sink

  16. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  17. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  18. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  19. val id: Int

    unique id for this Sink

    unique id for this Sink

    Definition Classes
    DataSink → Sink

  20. def isCheckpoint: Boolean

    returns

    true if this sink is a checkpoint

    Definition Classes
    SinkSource

  21. def isCompressed: Boolean

    returns

    true if this Sink is compressed

    Definition Classes
    DataSink → Sink

  22. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  23. val keyClass: Class[K]

  24. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  25. final def notify(): Unit

    Definition Classes
    AnyRef

  26. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  27. val output: Path

    Attributes
    protected

  28. def outputCheck(implicit sc: ScoobiConfiguration): Unit

    Check the validity of the DataSink specification.

    Check the validity of the DataSink specification.

    Definition Classes
    SeqSink → DataSink → Sink

  29. def outputConfigure(job: Job)(implicit sc: ScoobiConfiguration): Unit

    Configure the DataSink.

    Configure the DataSink.

    Definition Classes
    SeqSink → DataSink → Sink

  30. val outputConverter: InputOutputConverter[K, V, B]

    Maps the type consumed by this DataSink to the key-values of its OutputFormat.

    Maps the type consumed by this DataSink to the key-values of its OutputFormat.

    Definition Classes
    SeqSink → DataSink → Sink

  31. def outputFormat(implicit sc: ScoobiConfiguration): Class[SequenceFileOutputFormat[K, V]]

    The OutputFormat specifying the type of output for this DataSink.

    The OutputFormat specifying the type of output for this DataSink.

    Definition Classes
    SeqSink → DataSink → Sink

  32. def outputKeyClass(implicit sc: ScoobiConfiguration): Class[K]

    The Class of the OutputFormat's key.

    The Class of the OutputFormat's key.

    Definition Classes
    SeqSink → DataSink → Sink

  33. def outputPath(implicit sc: ScoobiConfiguration): Some[Path]

    returns

    the path for this Sink.

    Definition Classes
    SeqSink → Sink

  34. def outputSetup(implicit sc: ScoobiConfiguration): Unit

    This method is called just before writing data to the sink

    This method is called just before writing data to the sink

    Definition Classes
    SeqSink → SinkSource → DataSink → Sink

  35. def outputTeardown(implicit sc: ScoobiConfiguration): Unit

    This method is called just after writing data to the sink

    This method is called just after writing data to the sink

    Definition Classes
    DataSink → Sink

  36. def outputValueClass(implicit sc: ScoobiConfiguration): Class[V]

    The Class of the OutputFormat's value.

    The Class of the OutputFormat's value.

    Definition Classes
    SeqSink → DataSink → Sink

  37. val overwrite: Boolean

  38. val path: String

  39. lazy val stringId: String

    Definition Classes
    DataSink → Sink

  40. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  41. def toSource: SeqSource[K, V, B]

    Definition Classes
    SeqSink → SinkSource

  42. def toString(): String

    Definition Classes
    SeqSink → AnyRef → Any

  43. val valueClass: Class[V]

  44. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  45. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  46. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from SinkSource

Inherited from DataSink[K, V, B]

Inherited from Sink

Inherited from AnyRef

Inherited from Any

Ungrouped