partition

Type Members

1. abstract class PartitionedOutputFormat[P, K, V] extends FileOutputFormat[P, (K, V)]

   

   This format creates a new record writer for each different path that's generated by the partition function Each record writer defines a specific OutputCommitter that will define a different work directory for a given key.

   This format creates a new record writer for each different path that's generated by the partition function Each record writer defines a specific OutputCommitter that will define a different work directory for a given key.

   All the generated paths will be created under temporary dir/sink id in order to collect them more rapidly with just a rename of directories (see OutputChannel)

2. case class PartitionedSink[P, K, V, B](subsink: DataSink[K, V, B], format: Class[_ <: PartitionedOutputFormat[P, K, V]], path: String, partition: (P) ⇒ String, overwrite: Boolean = false, check: OutputCheck = Sink.defaultOutputCheck, compression: Option[Compression] = None) extends DataSink[P, (K, V), (P, B)] with Product with Serializable