core

Type Members

1. sealed trait Association1[+K, +V] extends AnyRef

   

   An association of a key to 1 or many values.

2. case class Checkpoint(path: Path, expiryPolicy: ExpiryPolicy = ExpiryPolicy.default) extends Product with Serializable

   

   store the output path of a Sink as a checkpoint

3. trait CompNode extends Attributable

   

   Base trait for "computation nodes" with no generic type information for easier rewriting

   Base trait for "computation nodes" with no generic type information for easier rewriting

   Each computation node has a unique id and equality of 2 nodes is based on this id.

   CompNodes are Attributable so that they can be used in attribute grammars

4. case class Compression(codec: CompressionCodec, compressionType: CompressionType = CompressionType.BLOCK) extends Product with Serializable

   

   store the compression parameters for sinks

5. trait Counters extends AnyRef

   

6. trait DList[A] extends DataSinks with Persistent[Seq[A]]

   

   A list that is distributed across multiple machines.

   A list that is distributed across multiple machines.

   It supports a few Traversable-like methods:

   - parallelDo: a 'map' operation transforming elements of the list in parallel - ++: to concatenate 2 DLists - groupByKey: to group a list of (key, value) elements by key, so as to get (key, values) - combine: a parallel 'reduce' operation - materialise: transforms a distributed list into a non-distributed list

7. trait DObject[A] extends Persistent[A]

   

   A wrapper around an object that is part of the graph of a distributed computation

8. trait DataSink[K, V, B] extends Sink

   

   An output store from a MapReduce job

9. trait DataSinks extends AnyRef

   

   specify an object on which it is possible to add sinks and to compress them

10. trait DataSource[K, V, A] extends Source

    

    DataSource for a computation graph.

    DataSource for a computation graph.

    It reads key-values (K, V) from the file system and uses an input converter to create a type A of input

11. trait DelegatedScoobiJobContext extends ScoobiJobContext

    

12. trait DoFn[A, B] extends EnvDoFn[A, B, Unit]

    

    Interface for specifying parallel operation over DLists in the absence of an environment

13. trait Emitter[A] extends EmitterWriter

    

    Interface for writing outputs from a DoFn

14. trait EnvDoFn[A, B, E] extends DoFunction

    

    Interface for specifying parallel operation over DLists.

    Interface for specifying parallel operation over DLists. The semantics of DoFn lifecycle are as follows:

    For a given chunk of DList elements: 1. 'setup' will be called; 2. 'process' will be called for each element in the chunk; 3. 'cleanup' will be called.

    These 3 steps encapsulate the entire life-cycle of a DoFn. A DoFn object will not be referenced after these steps

15. trait Environment extends AnyRef

    

    An object holder which can hold a distributed value

16. case class ExpiryPolicy(expiryTime: FiniteDuration = Duration.Zero, archive: (Path, ScoobiConfiguration) ⇒ Unit = ExpiryPolicy.deleteOldFile) extends Product with Serializable

    

    Define the expiry policy for checkpoint files

    Define the expiry policy for checkpoint files

    You can define

    • the expiry time: how long a checkpoint file is valid (long value representing milliseconds)
    • the archiving strategy: what you do with an expired file (delete it, rename it,...)

17. sealed trait Grouped[K, V] extends AnyRef

    

    A distributed list of associations.

    A distributed list of associations.

    See also

    com.nicta.scoobi.core.Association1

18. trait Grouping[K] extends KeyGrouping

    

    Specify the way in which key-values are "shuffled".

    Specify the way in which key-values are "shuffled". Used by groupByKey in DList

    Annotations

    @implicitNotFound( ... )

19. trait GroupingFunctions extends AnyRef

    

20. trait GroupingImplicits extends GroupingImplicits0

    

    Implicit definitions of Grouping instances for common types.

21. trait GroupingImplicits0 extends AnyRef

    

22. trait Heartbeat extends AnyRef

    

23. trait InputConverter[K, V, A] extends FromKeyValueConverter

    

    Convert an InputFormat's key-value types to the type produced by a source

24. class InputOutputContext extends AnyRef

    

25. trait InputOutputContextScoobiJobContext extends ScoobiJobContext

    

26. trait InputOutputConverter[K, V, B] extends OutputConverter[K, V, B] with InputConverter[K, V, B]

    

    fusion of both trait when bi-directional conversion is possible

27. trait Iterable1[+A] extends AnyRef

    

    A non-empty iterable contains at least one element.

    A non-empty iterable contains at least one element. Consequences include:

    - reduceLeft will always produce a value. - head will always produce a value. - tail will always produce a value.

    Some operations on a non-empty iterable result in a non-empty iterable.

    Construction of an Iterable1 is typically performed with the +:: method, defined on Iterable1.RichIterator.

    For example:

    import Iterable1._
    
    // A regular iterator.
    val x: Iterable[Int] = ...
    // Constructs a non-empty iterable with 74 at the head.
    val y: Iterable1[Int] = 74 +:: x

28. trait Iterator1[+A] extends TraversableOnce[A]

    

    A non-empty iterator contains at least one element.

    A non-empty iterator contains at least one element. Consequences include:

    - reduceLeft will always produce a value. - first will always produce a value. - next will always produce a value on its first invocation. - hasNext will always return true on its first invocation. - scanLeft1 will always produce a value.

    Some operations on a non-empty iterator result in a non-empty iterator.

    Construction of an Iterator1 is typically performed with the +:: method, defined on Iterator1.RichIterator.

    For example:

    import Iterator1._
    
    // A regular iterator.
    val x: Iterator[Int] = ...
    // Constructs a non-empty iterator with 74 at the first.
    val y: Iterator1[Int] = 77 +:: x

    NOTE: Most Iterator functions perform SIDE-EFFECTS and so EQUATIONAL REASONING DOES NOT APPLY.

29. trait LowPriorityWireFormatImplicits extends AnyRef

    

30. sealed trait Mode extends AnyRef

    

31. trait NoCounters extends Counters

    

32. trait NoHeartbeat extends Heartbeat

    

33. trait NoScoobiJobContext extends NoCounters with NoHeartbeat

    

34. trait OutputConverter[K, V, B] extends ToKeyValueConverter

    

    Convert the type consumed by a DataSink into an OutputFormat's key-value types.

35. trait Persistent[T] extends DataSinks

    

36. trait ProcessNode extends CompNode

    

37. trait Reduction[A] extends AnyRef

    

38. trait Reductions extends AnyRef

    

39. trait ScoobiConfiguration extends AnyRef

    

    This class wraps the Hadoop (mutable) configuration with additional configuration information such as the jars which should be added to the classpath.

40. trait ScoobiJobContext extends Counters with Heartbeat

    

41. trait SinkSource extends Sink

    

    This is a Sink which can also be used as a Source

42. trait UniqueInt extends AnyRef

    

    Trait that is sub-classed by objects to provide sets of unique identifiers.

43. trait ValueNode extends CompNode

    

44. trait WireFormat[A] extends WireReaderWriter

    

    Typeclass for sending types across the Hadoop wire

    Typeclass for sending types across the Hadoop wire

    Annotations

    @implicitNotFound( ... )

45. trait WireFormatImplicits extends GeneratedWireFormats

    

    Implicit definitions of WireFormat instances for common types.

46. trait WireReaderWriter extends AnyRef