com.nicta.scoobi.testing

HadoopSpecificationStructure

trait HadoopSpecificationStructure extends Cluster with HadoopExamples with UploadedLibJars with HadoopLogFactorySetup with CommandLineHadoopLogFactory

You can use this abstract class to create your own specification class, specifying:

class MyHadoopSpec(args: Arguments) extends HadoopSpecificationStructure(args) with MyCluster with mutable.Specification

Linear Supertypes
CommandLineHadoopLogFactory, HadoopLogFactorySetup, UploadedLibJars, SpecificationStructure, DefaultFragmentsFormatting, TagsAssociation, FragmentsFormatting, HadoopExamples, HadoopLogFactoryInitialisation, CommandLineScoobiUserArgs, CommandLineArguments, DelayedInit, Hadoop, LibJars, LocalHadoop, InMemoryHadoop, ScoobiUserArgs, ScoobiArgs, Cluster, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. HadoopSpecificationStructure
  2. CommandLineHadoopLogFactory
  3. HadoopLogFactorySetup
  4. UploadedLibJars
  5. SpecificationStructure
  6. DefaultFragmentsFormatting
  7. TagsAssociation
  8. FragmentsFormatting
  9. HadoopExamples
  10. HadoopLogFactoryInitialisation
  11. CommandLineScoobiUserArgs
  12. CommandLineArguments
  13. DelayedInit
  14. Hadoop
  15. LibJars
  16. LocalHadoop
  17. InMemoryHadoop
  18. ScoobiUserArgs
  19. ScoobiArgs
  20. Cluster
  21. AnyRef
  22. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Type Members

  1. class ClusterHadoopContext extends HadoopContext

    Context for running examples on the cluster

  2. trait HadoopContext extends Fixture[ScoobiConfiguration]

    trait for creating contexts having ScoobiConfigurations

  3. class InMemoryHadoopContext extends HadoopContext

    Context for running examples in memory

  4. class LocalHadoopContext extends HadoopContext

    Context for running examples locally

  5. class SkippedHadoopContext extends HadoopContext

    Context for showing that an execution is skipped

Abstract Value Members

  1. abstract def frameworkName: String

    returns

    the framework name: yarn, classic or local. This is only relevant for CDH5

    Definition Classes
    Cluster
  2. abstract def fs: String

    returns

    the filesystem address

    Definition Classes
    Cluster
  3. abstract def is: Fragments

    Definition Classes
    SpecificationStructure
  4. abstract def jobTracker: String

    returns

    the jobtracker address

    Definition Classes
    Cluster

Concrete Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. lazy val arguments: Arguments

    Definition Classes
    CommandLineArguments
  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. def categories: String

    returns

    the categories to show when logging, as a regular expression

    Definition Classes
    ScoobiUserArgsScoobiArgs
  9. def chain(contexts: Seq[HadoopContext]): HadoopContext { val configuration: com.nicta.scoobi.core.ScoobiConfiguration }

    returns

    a context chaining a sequence of contexts

    Definition Classes
    HadoopExamples
  10. def classDirs: Seq[String]

    returns

    the classes directories to include on a job classpath

    Definition Classes
    Hadoop
  11. def cleanup(c: ScoobiConfiguration): Unit

    cleanup temporary files after job execution

    cleanup temporary files after job execution

    Definition Classes
    HadoopExamples
  12. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  13. def cluster: HadoopContext

    context for cluster execution

    context for cluster execution

    Definition Classes
    HadoopExamples
  14. def configureArguments(implicit configuration: ScoobiConfiguration): Unit

    set command-line arguments on the configuration object

    set command-line arguments on the configuration object

    Attributes
    protected
    Definition Classes
    InMemoryHadoop
  15. def configureForCluster(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with cluster setup

    Definition Classes
    Hadoop
  16. def configureForInMemory(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with memory setup

    Definition Classes
    InMemoryHadoop
  17. def configureForLocal(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with local setup

    Definition Classes
    LocalHadoop
  18. def configureJars(implicit configuration: ScoobiConfiguration): Unit

    returns

    a configuration where the appropriate properties are set-up for uploaded jars: distributed files + classpath

    Definition Classes
    LibJars
  19. def context: Fixture[ScoobiConfiguration]

    define the context to use: local, cluster, localThenCluster

    define the context to use: local, cluster, localThenCluster

    Definition Classes
    HadoopExamples
  20. def contexts: Seq[HadoopContext]

    all contexts to run

    all contexts to run

    Definition Classes
    HadoopExamples
  21. def delayedInit(x: ⇒ Unit): Unit

    Definition Classes
    CommandLineArguments → DelayedInit
  22. def deleteJars(implicit configuration: ScoobiConfiguration): Unit

    returns

    delete the remote jars currently on the cluster

    Definition Classes
    LibJars
  23. def deleteLibJars: Boolean

    returns

    true if the libjars must be deleted before the Scoobi job runs

    Definition Classes
    ScoobiUserArgsScoobiArgs
  24. def displayTime(prefix: String): (SimpleTimer) ⇒ Unit

    the execution time will not be displayed with this function, but by adding more information to the execution Result

    the execution time will not be displayed with this function, but by adding more information to the execution Result

    returns

    a function to display execution times. The default uses log messages

    Definition Classes
    HadoopExamplesInMemoryHadoop
  25. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  26. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  27. def executeInMemory[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally

    execute some code locally

    Definition Classes
    InMemoryHadoop
  28. def executeOnCluster[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code on the cluster, setting the filesystem / jobtracker addresses and setting up the classpath

    execute some code on the cluster, setting the filesystem / jobtracker addresses and setting up the classpath

    Definition Classes
    Hadoop
  29. def executeOnLocal[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally

    execute some code locally

    Definition Classes
    LocalHadoop
  30. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  31. implicit def fixtureContext: Fixture[ScoobiConfiguration]

    make the context available implicitly as an Fixture[ScoobiConfiguration] so that examples taking that context as a parameter can be declared

    make the context available implicitly as an Fixture[ScoobiConfiguration] so that examples taking that context as a parameter can be declared

    Attributes
    protected
    Definition Classes
    HadoopExamples
  32. def formatFragments: (Fragments) ⇒ Fragments

    Definition Classes
    DefaultFragmentsFormatting → FragmentsFormatting
  33. lazy val fss: FileSystems

    Attributes
    protected[com.nicta.scoobi]
    Definition Classes
    LibJars
  34. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  35. def hadoopArgs: Array[String]

    Definition Classes
    InMemoryHadoop
  36. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  37. def identification: SpecIdentification

    Definition Classes
    SpecificationStructure
  38. implicit def identifySpecificationStructure(s: SpecificationStructure): SpecIdentification

    Definition Classes
    SpecificationStructure
  39. def inMemory[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body locally

    execute an example body locally

    Definition Classes
    HadoopExamples
  40. def inMemory: HadoopContext

    context for in memory execution

    context for in memory execution

    Definition Classes
    HadoopExamples
  41. def inMemory[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code in memory, using a collection backend, possibly showing execution times

    execute some code in memory, using a collection backend, possibly showing execution times

    Definition Classes
    InMemoryHadoop
  42. def includeLibJars: Boolean

    returns

    true if you want to include the library jars in the jar that is sent to the cluster for each job

    Definition Classes
    Hadoop
  43. def isCluster: Boolean

    Definition Classes
    ScoobiUserArgs
  44. def isClusterOnly: Boolean

    returns

    true if the cluster argument is specified and the local argument is not

    Definition Classes
    ScoobiUserArgs
  45. def isInMemory: Boolean

    tests are always in memory by default, unless !inmemory is passed

    tests are always in memory by default, unless !inmemory is passed

    Definition Classes
    HadoopExamplesScoobiUserArgs
  46. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  47. def isLocal: Boolean

    tests are always local by default, unless !local is passed

    tests are always local by default, unless !local is passed

    Definition Classes
    HadoopExamplesScoobiUserArgs
  48. def isLocalOnly: Boolean

    alias for locally

    alias for locally

    Definition Classes
    ScoobiUserArgs
  49. def jars(implicit configuration: ScoobiConfiguration): Seq[URL]

    returns

    the list of library jars to upload

    Definition Classes
    LibJars
  50. def keepFiles: Boolean

    returns

    false if temporary files and working directory must be cleaned-up after job execution

    Definition Classes
    ScoobiUserArgsScoobiArgs
  51. def level: Level

    returns

    the log level to use when logging

    Definition Classes
    ScoobiUserArgsScoobiArgs
  52. def libjarsDirectory(implicit configuration: ScoobiConfiguration): String

    returns

    the path of the directory to use when loading jars to the filesystem.

    Definition Classes
    LibJars
  53. def local: HadoopContext

    context for local execution

    context for local execution

    Definition Classes
    HadoopExamples
  54. def locally[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body locally

    execute an example body locally

    Definition Classes
    HadoopExamples
  55. def locally: Boolean

    returns

    true if the Scoobi job must be run locally

    Definition Classes
    ScoobiUserArgsScoobiArgs
  56. def map(fs: ⇒ Fragments): Fragments

    Definition Classes
    HadoopLogFactorySetup → SpecificationStructure
  57. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  58. def noLibJars: Boolean

    returns

    false if libjars are used

    Definition Classes
    ScoobiUserArgsScoobiArgs
  59. final def notify(): Unit

    Definition Classes
    AnyRef
  60. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  61. def onCluster[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code on the cluster, possibly showing the execution time

    execute some code on the cluster, possibly showing the execution time

    Definition Classes
    Hadoop
  62. def onHadoop[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code, either locally or on the cluster, depending on the local argument being passed on the commandline

    execute some code, either locally or on the cluster, depending on the local argument being passed on the commandline

    Definition Classes
    Hadoop
  63. def onLocal[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally, possibly showing execution times

    execute some code locally, possibly showing execution times

    Definition Classes
    LocalHadoop
  64. def quiet: Boolean

    for testing, the output must be quiet by default, unless verbose is specified

    for testing, the output must be quiet by default, unless verbose is specified

    returns

    true to suppress log messages

    Definition Classes
    CommandLineHadoopLogFactoryScoobiUserArgsScoobiArgs
  65. def remotely[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body on the cluster

    execute an example body on the cluster

    Definition Classes
    HadoopExamples
  66. def runInMemory[T](t: ⇒ T): T

    returns

    the result of the in-memory run

    Definition Classes
    InMemoryHadoop
  67. def runOnCluster[T](t: ⇒ T): T

    returns

    the cluster evaluation of t

    Definition Classes
    Hadoop
  68. def runOnLocal[T](t: ⇒ T): T

    returns

    the result of the local run

    Definition Classes
    LocalHadoop
  69. lazy val scoobiArgs: Seq[String]

    the scoobi arguments passed on the command line

    the scoobi arguments passed on the command line

    Definition Classes
    CommandLineScoobiUserArgsScoobiUserArgs
  70. def set(args: Arguments): Unit

    Definition Classes
    CommandLineArguments
  71. def setLogFactory(name: String = classOf[HadoopLogFactory].getName): Unit

    Static setup to use a testing log factory

    Static setup to use a testing log factory

    Definition Classes
    InMemoryHadoop
  72. def showPlanOnly: Boolean

    returns

    true if the debug logs must show the computation graph and not evaluate it

    Definition Classes
    ScoobiUserArgsScoobiArgs
  73. def showTime[T](t: ⇒ T)(display: (SimpleTimer) ⇒ Unit): T

    measure the time taken by some executed code and display the time with a specific display function

    measure the time taken by some executed code and display the time with a specific display function

    Definition Classes
    InMemoryHadoop
  74. def showTimes: Boolean

    returns

    true to display execution times for each job

    Definition Classes
    ScoobiUserArgsScoobiArgs
  75. def skippedContext(name: String): HadoopContext

    context for showing a skipped execution

    context for showing a skipped execution

    Definition Classes
    HadoopExamples
  76. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  77. lazy val sysProps: SystemProperties

    Attributes
    protected[com.nicta.scoobi]
    Definition Classes
    LibJars
  78. def toString(): String

    Definition Classes
    AnyRef → Any
  79. def upload: Boolean

    this variable controls if the upload must be done at all

    this variable controls if the upload must be done at all

    Definition Classes
    LibJars
  80. def uploadLibJarsFiles(deleteLibJarsFirst: Boolean = false)(implicit configuration: ScoobiConfiguration): Unit

    upload the jars which don't exist yet in the library directory on the cluster

    upload the jars which don't exist yet in the library directory on the cluster

    Definition Classes
    LibJars
  81. def uploadStep: Step

    create a Step to upload the jars on the cluster

    create a Step to upload the jars on the cluster

    Definition Classes
    UploadedLibJars
  82. def uploadedJars(implicit configuration: ScoobiConfiguration): Seq[Path]

    returns

    the remote jars currently on the cluster

    Definition Classes
    LibJars
  83. def useHadoopConfDir: Boolean

    returns

    true if cluster configuration must be loaded from Hadoop's configuration directory

    Definition Classes
    ScoobiUserArgsScoobiArgs
  84. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  85. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  86. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  87. def withTimer[T](t: ⇒ T): (T, SimpleTimer)

    returns

    the time for the execution of a piece of code

    Definition Classes
    InMemoryHadoop

Inherited from HadoopLogFactorySetup

Inherited from UploadedLibJars

Inherited from SpecificationStructure

Inherited from DefaultFragmentsFormatting

Inherited from TagsAssociation

Inherited from FragmentsFormatting

Inherited from HadoopExamples

Inherited from CommandLineScoobiUserArgs

Inherited from CommandLineArguments

Inherited from DelayedInit

Inherited from Hadoop

Inherited from LibJars

Inherited from LocalHadoop

Inherited from InMemoryHadoop

Inherited from ScoobiUserArgs

Inherited from ScoobiArgs

Inherited from Cluster

Inherited from AnyRef

Inherited from Any

Ungrouped