com.nicta.scoobi.testing

HadoopExamples

trait HadoopExamples extends Hadoop with CommandLineScoobiUserArgs with Cluster with HadoopLogFactoryInitialisation

This trait provides an Around context to be used in a Specification

Subclasses can override the context method:

They also need to implement the Cluster trait to specify the location of the remote nodes

Self Type
HadoopExamples
Linear Supertypes
Known Subclasses
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. HadoopExamples
  2. HadoopLogFactoryInitialisation
  3. CommandLineScoobiUserArgs
  4. CommandLineArguments
  5. DelayedInit
  6. Hadoop
  7. LibJars
  8. Cluster
  9. LocalHadoop
  10. InMemoryHadoop
  11. ScoobiUserArgs
  12. ScoobiArgs
  13. AnyRef
  14. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Type Members

  1. class ClusterHadoopContext extends HadoopContext

    Context for running examples on the cluster

  2. trait HadoopContext extends Fixture[ScoobiConfiguration]

    trait for creating contexts having ScoobiConfigurations

    trait for creating contexts having ScoobiConfigurations

    the isLocalOnly method provides a hint to speed-up the execution (because there's no need to upload jars if a run is local)

  3. class InMemoryHadoopContext extends HadoopContext

    Context for running examples in memory

  4. class LocalHadoopContext extends HadoopContext

    Context for running examples locally

  5. class SkippedHadoopContext extends HadoopContext

    Context for showing that an execution is skipped

Abstract Value Members

  1. abstract def frameworkName: String

    returns

    the framework name: yarn, classic or local. This is only relevant for CDH5

    Definition Classes
    Cluster
  2. abstract def fs: String

    returns

    the filesystem address

    Definition Classes
    Cluster
  3. abstract def jobTracker: String

    returns

    the jobtracker address

    Definition Classes
    Cluster

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  4. lazy val arguments: Arguments

    Definition Classes
    CommandLineArguments
  5. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  6. def categories: String

    returns

    the categories to show when logging, as a regular expression

    Definition Classes
    ScoobiUserArgsScoobiArgs
  7. def chain(contexts: Seq[HadoopContext]): HadoopContext { val configuration: com.nicta.scoobi.core.ScoobiConfiguration }

    returns

    a context chaining a sequence of contexts

  8. def classDirs: Seq[String]

    returns

    the classes directories to include on a job classpath

    Definition Classes
    Hadoop
  9. def cleanup(c: ScoobiConfiguration): Unit

    cleanup temporary files after job execution

  10. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  11. def cluster: HadoopContext

    context for cluster execution

  12. def configureArguments(implicit configuration: ScoobiConfiguration): Unit

    set command-line arguments on the configuration object

    set command-line arguments on the configuration object

    Attributes
    protected
    Definition Classes
    InMemoryHadoop
  13. def configureForCluster(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with cluster setup

    Definition Classes
    Hadoop
  14. def configureForInMemory(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with memory setup

    Definition Classes
    InMemoryHadoop
  15. def configureForLocal(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with local setup

    Definition Classes
    LocalHadoop
  16. def configureJars(implicit configuration: ScoobiConfiguration): Unit

    returns

    a configuration where the appropriate properties are set-up for uploaded jars: distributed files + classpath

    Definition Classes
    LibJars
  17. def context: Fixture[ScoobiConfiguration]

    define the context to use: local, cluster, localThenCluster

  18. def contexts: Seq[HadoopContext]

    all contexts to run

  19. def delayedInit(x: ⇒ Unit): Unit

    Definition Classes
    CommandLineArguments → DelayedInit
  20. def deleteJars(implicit configuration: ScoobiConfiguration): Unit

    returns

    delete the remote jars currently on the cluster

    Definition Classes
    LibJars
  21. def deleteLibJars: Boolean

    returns

    true if the libjars must be deleted before the Scoobi job runs

    Definition Classes
    ScoobiUserArgsScoobiArgs
  22. def displayTime(prefix: String): (SimpleTimer) ⇒ Unit

    the execution time will not be displayed with this function, but by adding more information to the execution Result

    the execution time will not be displayed with this function, but by adding more information to the execution Result

    returns

    a function to display execution times. The default uses log messages

    Definition Classes
    HadoopExamplesInMemoryHadoop
  23. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  24. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  25. def executeInMemory[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally

    execute some code locally

    Definition Classes
    InMemoryHadoop
  26. def executeOnCluster[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code on the cluster, setting the filesystem / jobtracker addresses and setting up the classpath

    execute some code on the cluster, setting the filesystem / jobtracker addresses and setting up the classpath

    Definition Classes
    Hadoop
  27. def executeOnLocal[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally

    execute some code locally

    Definition Classes
    LocalHadoop
  28. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  29. implicit def fixtureContext: Fixture[ScoobiConfiguration]

    make the context available implicitly as an Fixture[ScoobiConfiguration] so that examples taking that context as a parameter can be declared

    make the context available implicitly as an Fixture[ScoobiConfiguration] so that examples taking that context as a parameter can be declared

    Attributes
    protected
  30. lazy val fss: FileSystems

    Attributes
    protected[com.nicta.scoobi]
    Definition Classes
    LibJars
  31. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  32. def hadoopArgs: Array[String]

    Definition Classes
    InMemoryHadoop
  33. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  34. def inMemory[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body locally

  35. def inMemory: HadoopContext

    context for in memory execution

  36. def inMemory[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code in memory, using a collection backend, possibly showing execution times

    execute some code in memory, using a collection backend, possibly showing execution times

    Definition Classes
    InMemoryHadoop
  37. def includeLibJars: Boolean

    returns

    true if you want to include the library jars in the jar that is sent to the cluster for each job

    Definition Classes
    Hadoop
  38. def isCluster: Boolean

    Definition Classes
    ScoobiUserArgs
  39. def isClusterOnly: Boolean

    returns

    true if the cluster argument is specified and the local argument is not

    Definition Classes
    ScoobiUserArgs
  40. def isInMemory: Boolean

    tests are always in memory by default, unless !inmemory is passed

    tests are always in memory by default, unless !inmemory is passed

    Definition Classes
    HadoopExamplesScoobiUserArgs
  41. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  42. def isLocal: Boolean

    tests are always local by default, unless !local is passed

    tests are always local by default, unless !local is passed

    Definition Classes
    HadoopExamplesScoobiUserArgs
  43. def isLocalOnly: Boolean

    alias for locally

    alias for locally

    Definition Classes
    ScoobiUserArgs
  44. def jars(implicit configuration: ScoobiConfiguration): Seq[URL]

    returns

    the list of library jars to upload

    Definition Classes
    LibJars
  45. def keepFiles: Boolean

    returns

    false if temporary files and working directory must be cleaned-up after job execution

    Definition Classes
    ScoobiUserArgsScoobiArgs
  46. def level: Level

    returns

    the log level to use when logging

    Definition Classes
    ScoobiUserArgsScoobiArgs
  47. def libjarsDirectory(implicit configuration: ScoobiConfiguration): String

    returns

    the path of the directory to use when loading jars to the filesystem.

    Definition Classes
    LibJars
  48. def local: HadoopContext

    context for local execution

  49. def locally[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body locally

  50. def locally: Boolean

    returns

    true if the Scoobi job must be run locally

    Definition Classes
    ScoobiUserArgsScoobiArgs
  51. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  52. def noLibJars: Boolean

    returns

    false if libjars are used

    Definition Classes
    ScoobiUserArgsScoobiArgs
  53. final def notify(): Unit

    Definition Classes
    AnyRef
  54. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  55. def onCluster[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code on the cluster, possibly showing the execution time

    execute some code on the cluster, possibly showing the execution time

    Definition Classes
    Hadoop
  56. def onHadoop[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code, either locally or on the cluster, depending on the local argument being passed on the commandline

    execute some code, either locally or on the cluster, depending on the local argument being passed on the commandline

    Definition Classes
    Hadoop
  57. def onLocal[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally, possibly showing execution times

    execute some code locally, possibly showing execution times

    Definition Classes
    LocalHadoop
  58. def quiet: Boolean

    returns

    true to suppress log messages

    Definition Classes
    ScoobiUserArgsScoobiArgs
  59. def remotely[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body on the cluster

  60. def runInMemory[T](t: ⇒ T): T

    returns

    the result of the in-memory run

    Definition Classes
    InMemoryHadoop
  61. def runOnCluster[T](t: ⇒ T): T

    returns

    the cluster evaluation of t

    Definition Classes
    Hadoop
  62. def runOnLocal[T](t: ⇒ T): T

    returns

    the result of the local run

    Definition Classes
    LocalHadoop
  63. lazy val scoobiArgs: Seq[String]

    the scoobi arguments passed on the command line

    the scoobi arguments passed on the command line

    Definition Classes
    CommandLineScoobiUserArgsScoobiUserArgs
  64. def set(args: Arguments): Unit

    Definition Classes
    CommandLineArguments
  65. def setLogFactory(name: String = classOf[HadoopLogFactory].getName): Unit

    Static setup to use a testing log factory

    Static setup to use a testing log factory

    Definition Classes
    InMemoryHadoop
  66. def showPlanOnly: Boolean

    returns

    true if the debug logs must show the computation graph and not evaluate it

    Definition Classes
    ScoobiUserArgsScoobiArgs
  67. def showTime[T](t: ⇒ T)(display: (SimpleTimer) ⇒ Unit): T

    measure the time taken by some executed code and display the time with a specific display function

    measure the time taken by some executed code and display the time with a specific display function

    Definition Classes
    InMemoryHadoop
  68. def showTimes: Boolean

    returns

    true to display execution times for each job

    Definition Classes
    ScoobiUserArgsScoobiArgs
  69. def skippedContext(name: String): HadoopContext

    context for showing a skipped execution

  70. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  71. lazy val sysProps: SystemProperties

    Attributes
    protected[com.nicta.scoobi]
    Definition Classes
    LibJars
  72. def toString(): String

    Definition Classes
    AnyRef → Any
  73. def upload: Boolean

    this variable controls if the upload must be done at all

    this variable controls if the upload must be done at all

    Definition Classes
    LibJars
  74. def uploadLibJarsFiles(deleteLibJarsFirst: Boolean = false)(implicit configuration: ScoobiConfiguration): Unit

    upload the jars which don't exist yet in the library directory on the cluster

    upload the jars which don't exist yet in the library directory on the cluster

    Definition Classes
    LibJars
  75. def uploadedJars(implicit configuration: ScoobiConfiguration): Seq[Path]

    returns

    the remote jars currently on the cluster

    Definition Classes
    LibJars
  76. def useHadoopConfDir: Boolean

    returns

    true if cluster configuration must be loaded from Hadoop's configuration directory

    Definition Classes
    ScoobiUserArgsScoobiArgs
  77. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  78. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  79. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  80. def withTimer[T](t: ⇒ T): (T, SimpleTimer)

    returns

    the time for the execution of a piece of code

    Definition Classes
    InMemoryHadoop

Inherited from CommandLineScoobiUserArgs

Inherited from CommandLineArguments

Inherited from DelayedInit

Inherited from Hadoop

Inherited from LibJars

Inherited from Cluster

Inherited from LocalHadoop

Inherited from InMemoryHadoop

Inherited from ScoobiUserArgs

Inherited from ScoobiArgs

Inherited from AnyRef

Inherited from Any

Ungrouped