com.nicta.scoobi.testing

HadoopExamples

trait HadoopExamples extends Hadoop with CommandLineScoobiUserArgs with Cluster with HadoopLogFactoryInitialisation

This trait provides an Around context to be used in a Specification

Subclasses can override the context method:

They also need to implement the Cluster trait to specify the location of the remote nodes

Self Type
HadoopExamples
Linear Supertypes
Known Subclasses
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. HadoopExamples
  2. HadoopLogFactoryInitialisation
  3. CommandLineScoobiUserArgs
  4. CommandLineArguments
  5. DelayedInit
  6. Hadoop
  7. LibJars
  8. Cluster
  9. LocalHadoop
  10. InMemoryHadoop
  11. ScoobiUserArgs
  12. ScoobiArgs
  13. AnyRef
  14. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Type Members

  1. class ClusterHadoopContext extends HadoopContext

    Context for running examples on the cluster

  2. trait HadoopContext extends Fixture[ScoobiConfiguration]

    trait for creating contexts having ScoobiConfigurations

  3. class InMemoryHadoopContext extends HadoopContext

    Context for running examples in memory

  4. class LocalHadoopContext extends HadoopContext

    Context for running examples locally

  5. class SkippedHadoopContext extends HadoopContext

    Context for showing that an execution is skipped

Abstract Value Members

  1. abstract def frameworkName: String

    returns

    the framework name: yarn, classic or local. This is only relevant for CDH5

    Definition Classes
    Cluster
  2. abstract def fs: String

    returns

    the filesystem address

    Definition Classes
    Cluster
  3. abstract def jobTracker: String

    returns

    the jobtracker address

    Definition Classes
    Cluster

Concrete Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. lazy val arguments: Arguments

    Definition Classes
    CommandLineArguments
  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. def categories: String

    returns

    the categories to show when logging, as a regular expression

    Definition Classes
    ScoobiUserArgsScoobiArgs
  9. def chain(contexts: Seq[HadoopContext]): HadoopContext { val configuration: com.nicta.scoobi.core.ScoobiConfiguration }

    returns

    a context chaining a sequence of contexts

  10. def classDirs: Seq[String]

    returns

    the classes directories to include on a job classpath

    Definition Classes
    Hadoop
  11. def cleanup(c: ScoobiConfiguration): Unit

    cleanup temporary files after job execution

  12. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  13. def cluster: HadoopContext

    context for cluster execution

  14. def configureArguments(implicit configuration: ScoobiConfiguration): Unit

    set command-line arguments on the configuration object

    set command-line arguments on the configuration object

    Attributes
    protected
    Definition Classes
    InMemoryHadoop
  15. def configureForCluster(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with cluster setup

    Definition Classes
    Hadoop
  16. def configureForInMemory(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with memory setup

    Definition Classes
    InMemoryHadoop
  17. def configureForLocal(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with local setup

    Definition Classes
    LocalHadoop
  18. def configureJars(implicit configuration: ScoobiConfiguration): Unit

    returns

    a configuration where the appropriate properties are set-up for uploaded jars: distributed files + classpath

    Definition Classes
    LibJars
  19. def context: Fixture[ScoobiConfiguration]

    define the context to use: local, cluster, localThenCluster

  20. def contexts: Seq[HadoopContext]

    all contexts to run

  21. def delayedInit(x: ⇒ Unit): Unit

    Definition Classes
    CommandLineArguments → DelayedInit
  22. def deleteJars(implicit configuration: ScoobiConfiguration): Unit

    returns

    delete the remote jars currently on the cluster

    Definition Classes
    LibJars
  23. def deleteLibJars: Boolean

    returns

    true if the libjars must be deleted before the Scoobi job runs

    Definition Classes
    ScoobiUserArgsScoobiArgs
  24. def displayTime(prefix: String): (SimpleTimer) ⇒ Unit

    the execution time will not be displayed with this function, but by adding more information to the execution Result

    the execution time will not be displayed with this function, but by adding more information to the execution Result

    returns

    a function to display execution times. The default uses log messages

    Definition Classes
    HadoopExamplesInMemoryHadoop
  25. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  26. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  27. def executeInMemory[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally

    execute some code locally

    Definition Classes
    InMemoryHadoop
  28. def executeOnCluster[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code on the cluster, setting the filesystem / jobtracker addresses and setting up the classpath

    execute some code on the cluster, setting the filesystem / jobtracker addresses and setting up the classpath

    Definition Classes
    Hadoop
  29. def executeOnLocal[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally

    execute some code locally

    Definition Classes
    LocalHadoop
  30. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  31. implicit def fixtureContext: Fixture[ScoobiConfiguration]

    make the context available implicitly as an Fixture[ScoobiConfiguration] so that examples taking that context as a parameter can be declared

    make the context available implicitly as an Fixture[ScoobiConfiguration] so that examples taking that context as a parameter can be declared

    Attributes
    protected
  32. lazy val fss: FileSystems

    Attributes
    protected[com.nicta.scoobi]
    Definition Classes
    LibJars
  33. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  34. def hadoopArgs: Array[String]

    Definition Classes
    InMemoryHadoop
  35. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  36. def inMemory[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body locally

  37. def inMemory: HadoopContext

    context for in memory execution

  38. def inMemory[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code in memory, using a collection backend, possibly showing execution times

    execute some code in memory, using a collection backend, possibly showing execution times

    Definition Classes
    InMemoryHadoop
  39. def includeLibJars: Boolean

    returns

    true if you want to include the library jars in the jar that is sent to the cluster for each job

    Definition Classes
    Hadoop
  40. def isCluster: Boolean

    Definition Classes
    ScoobiUserArgs
  41. def isClusterOnly: Boolean

    returns

    true if the cluster argument is specified and the local argument is not

    Definition Classes
    ScoobiUserArgs
  42. def isInMemory: Boolean

    tests are always in memory by default, unless !inmemory is passed

    tests are always in memory by default, unless !inmemory is passed

    Definition Classes
    HadoopExamplesScoobiUserArgs
  43. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  44. def isLocal: Boolean

    tests are always local by default, unless !local is passed

    tests are always local by default, unless !local is passed

    Definition Classes
    HadoopExamplesScoobiUserArgs
  45. def isLocalOnly: Boolean

    alias for locally

    alias for locally

    Definition Classes
    ScoobiUserArgs
  46. def jars(implicit configuration: ScoobiConfiguration): Seq[URL]

    returns

    the list of library jars to upload

    Definition Classes
    LibJars
  47. def keepFiles: Boolean

    returns

    false if temporary files and working directory must be cleaned-up after job execution

    Definition Classes
    ScoobiUserArgsScoobiArgs
  48. def level: Level

    returns

    the log level to use when logging

    Definition Classes
    ScoobiUserArgsScoobiArgs
  49. def libjarsDirectory(implicit configuration: ScoobiConfiguration): String

    returns

    the path of the directory to use when loading jars to the filesystem.

    Definition Classes
    LibJars
  50. def local: HadoopContext

    context for local execution

  51. def locally[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body locally

  52. def locally: Boolean

    returns

    true if the Scoobi job must be run locally

    Definition Classes
    ScoobiUserArgsScoobiArgs
  53. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  54. def noLibJars: Boolean

    returns

    false if libjars are used

    Definition Classes
    ScoobiUserArgsScoobiArgs
  55. final def notify(): Unit

    Definition Classes
    AnyRef
  56. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  57. def onCluster[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code on the cluster, possibly showing the execution time

    execute some code on the cluster, possibly showing the execution time

    Definition Classes
    Hadoop
  58. def onHadoop[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code, either locally or on the cluster, depending on the local argument being passed on the commandline

    execute some code, either locally or on the cluster, depending on the local argument being passed on the commandline

    Definition Classes
    Hadoop
  59. def onLocal[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally, possibly showing execution times

    execute some code locally, possibly showing execution times

    Definition Classes
    LocalHadoop
  60. def quiet: Boolean

    returns

    true to suppress log messages

    Definition Classes
    ScoobiUserArgsScoobiArgs
  61. def remotely[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body on the cluster

  62. def runInMemory[T](t: ⇒ T): T

    returns

    the result of the in-memory run

    Definition Classes
    InMemoryHadoop
  63. def runOnCluster[T](t: ⇒ T): T

    returns

    the cluster evaluation of t

    Definition Classes
    Hadoop
  64. def runOnLocal[T](t: ⇒ T): T

    returns

    the result of the local run

    Definition Classes
    LocalHadoop
  65. lazy val scoobiArgs: Seq[String]

    the scoobi arguments passed on the command line

    the scoobi arguments passed on the command line

    Definition Classes
    CommandLineScoobiUserArgsScoobiUserArgs
  66. def set(args: Arguments): Unit

    Definition Classes
    CommandLineArguments
  67. def setLogFactory(name: String = classOf[HadoopLogFactory].getName): Unit

    Static setup to use a testing log factory

    Static setup to use a testing log factory

    Definition Classes
    InMemoryHadoop
  68. def showPlanOnly: Boolean

    returns

    true if the debug logs must show the computation graph and not evaluate it

    Definition Classes
    ScoobiUserArgsScoobiArgs
  69. def showTime[T](t: ⇒ T)(display: (SimpleTimer) ⇒ Unit): T

    measure the time taken by some executed code and display the time with a specific display function

    measure the time taken by some executed code and display the time with a specific display function

    Definition Classes
    InMemoryHadoop
  70. def showTimes: Boolean

    returns

    true to display execution times for each job

    Definition Classes
    ScoobiUserArgsScoobiArgs
  71. def skippedContext(name: String): HadoopContext

    context for showing a skipped execution

  72. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  73. lazy val sysProps: SystemProperties

    Attributes
    protected[com.nicta.scoobi]
    Definition Classes
    LibJars
  74. def toString(): String

    Definition Classes
    AnyRef → Any
  75. def upload: Boolean

    this variable controls if the upload must be done at all

    this variable controls if the upload must be done at all

    Definition Classes
    LibJars
  76. def uploadLibJarsFiles(deleteLibJarsFirst: Boolean = false)(implicit configuration: ScoobiConfiguration): Unit

    upload the jars which don't exist yet in the library directory on the cluster

    upload the jars which don't exist yet in the library directory on the cluster

    Definition Classes
    LibJars
  77. def uploadedJars(implicit configuration: ScoobiConfiguration): Seq[Path]

    returns

    the remote jars currently on the cluster

    Definition Classes
    LibJars
  78. def useHadoopConfDir: Boolean

    returns

    true if cluster configuration must be loaded from Hadoop's configuration directory

    Definition Classes
    ScoobiUserArgsScoobiArgs
  79. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  80. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  81. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  82. def withTimer[T](t: ⇒ T): (T, SimpleTimer)

    returns

    the time for the execution of a piece of code

    Definition Classes
    InMemoryHadoop

Inherited from CommandLineScoobiUserArgs

Inherited from CommandLineArguments

Inherited from DelayedInit

Inherited from Hadoop

Inherited from LibJars

Inherited from Cluster

Inherited from LocalHadoop

Inherited from InMemoryHadoop

Inherited from ScoobiUserArgs

Inherited from ScoobiArgs

Inherited from AnyRef

Inherited from Any

Ungrouped