com.nicta.scoobi.testing

UploadedLibJars

trait UploadedLibJars extends SpecificationStructure with HadoopExamples with ScoobiUserArgs

This trait can be mixed in a Specification to automatically add a setup step uploading the library jars to the cluster

Linear Supertypes
HadoopExamples, HadoopLogFactoryInitialisation, CommandLineScoobiUserArgs, CommandLineArguments, DelayedInit, Hadoop, LibJars, Cluster, LocalHadoop, InMemoryHadoop, ScoobiUserArgs, ScoobiArgs, SpecificationStructure, DefaultFragmentsFormatting, TagsAssociation, FragmentsFormatting, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. UploadedLibJars
  2. HadoopExamples
  3. HadoopLogFactoryInitialisation
  4. CommandLineScoobiUserArgs
  5. CommandLineArguments
  6. DelayedInit
  7. Hadoop
  8. LibJars
  9. Cluster
  10. LocalHadoop
  11. InMemoryHadoop
  12. ScoobiUserArgs
  13. ScoobiArgs
  14. SpecificationStructure
  15. DefaultFragmentsFormatting
  16. TagsAssociation
  17. FragmentsFormatting
  18. AnyRef
  19. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Type Members

  1. class ClusterHadoopContext extends HadoopContext

    Context for running examples on the cluster

    Context for running examples on the cluster

    Definition Classes
    HadoopExamples
  2. trait HadoopContext extends Fixture[ScoobiConfiguration]

    trait for creating contexts having ScoobiConfigurations

    trait for creating contexts having ScoobiConfigurations

    the isLocalOnly method provides a hint to speed-up the execution (because there's no need to upload jars if a run is local)

    Definition Classes
    HadoopExamples
  3. class InMemoryHadoopContext extends HadoopContext

    Context for running examples in memory

    Context for running examples in memory

    Definition Classes
    HadoopExamples
  4. class LocalHadoopContext extends HadoopContext

    Context for running examples locally

    Context for running examples locally

    Definition Classes
    HadoopExamples
  5. class SkippedHadoopContext extends HadoopContext

    Context for showing that an execution is skipped

    Context for showing that an execution is skipped

    Definition Classes
    HadoopExamples

Abstract Value Members

  1. abstract def frameworkName: String

    returns

    the framework name: yarn, classic or local. This is only relevant for CDH5

    Definition Classes
    Cluster
  2. abstract def fs: String

    returns

    the filesystem address

    Definition Classes
    Cluster
  3. abstract def is: Fragments

    Definition Classes
    SpecificationStructure
  4. abstract def jobTracker: String

    returns

    the jobtracker address

    Definition Classes
    Cluster

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  4. lazy val arguments: Arguments

    Definition Classes
    CommandLineArguments
  5. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  6. def categories: String

    returns

    the categories to show when logging, as a regular expression

    Definition Classes
    ScoobiUserArgsScoobiArgs
  7. def chain(contexts: Seq[HadoopContext]): HadoopContext { val configuration: com.nicta.scoobi.core.ScoobiConfiguration }

    returns

    a context chaining a sequence of contexts

    Definition Classes
    HadoopExamples
  8. def classDirs: Seq[String]

    returns

    the classes directories to include on a job classpath

    Definition Classes
    Hadoop
  9. def cleanup(c: ScoobiConfiguration): Unit

    cleanup temporary files after job execution

    cleanup temporary files after job execution

    Definition Classes
    HadoopExamples
  10. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  11. def cluster: HadoopContext

    context for cluster execution

    context for cluster execution

    Definition Classes
    HadoopExamples
  12. def configureArguments(implicit configuration: ScoobiConfiguration): Unit

    set command-line arguments on the configuration object

    set command-line arguments on the configuration object

    Attributes
    protected
    Definition Classes
    InMemoryHadoop
  13. def configureForCluster(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with cluster setup

    Definition Classes
    Hadoop
  14. def configureForInMemory(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with memory setup

    Definition Classes
    InMemoryHadoop
  15. def configureForLocal(implicit configuration: ScoobiConfiguration): ScoobiConfiguration

    returns

    a configuration with local setup

    Definition Classes
    LocalHadoop
  16. def configureJars(implicit configuration: ScoobiConfiguration): Unit

    returns

    a configuration where the appropriate properties are set-up for uploaded jars: distributed files + classpath

    Definition Classes
    LibJars
  17. def context: Fixture[ScoobiConfiguration]

    define the context to use: local, cluster, localThenCluster

    define the context to use: local, cluster, localThenCluster

    Definition Classes
    HadoopExamples
  18. def contexts: Seq[HadoopContext]

    all contexts to run

    all contexts to run

    Definition Classes
    HadoopExamples
  19. def delayedInit(x: ⇒ Unit): Unit

    Definition Classes
    CommandLineArguments → DelayedInit
  20. def deleteJars(implicit configuration: ScoobiConfiguration): Unit

    returns

    delete the remote jars currently on the cluster

    Definition Classes
    LibJars
  21. def deleteLibJars: Boolean

    returns

    true if the libjars must be deleted before the Scoobi job runs

    Definition Classes
    ScoobiUserArgsScoobiArgs
  22. def displayTime(prefix: String): (SimpleTimer) ⇒ Unit

    the execution time will not be displayed with this function, but by adding more information to the execution Result

    the execution time will not be displayed with this function, but by adding more information to the execution Result

    returns

    a function to display execution times. The default uses log messages

    Definition Classes
    HadoopExamplesInMemoryHadoop
  23. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  24. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  25. def executeInMemory[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally

    execute some code locally

    Definition Classes
    InMemoryHadoop
  26. def executeOnCluster[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code on the cluster, setting the filesystem / jobtracker addresses and setting up the classpath

    execute some code on the cluster, setting the filesystem / jobtracker addresses and setting up the classpath

    Definition Classes
    Hadoop
  27. def executeOnLocal[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally

    execute some code locally

    Definition Classes
    LocalHadoop
  28. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  29. implicit def fixtureContext: Fixture[ScoobiConfiguration]

    make the context available implicitly as an Fixture[ScoobiConfiguration] so that examples taking that context as a parameter can be declared

    make the context available implicitly as an Fixture[ScoobiConfiguration] so that examples taking that context as a parameter can be declared

    Attributes
    protected
    Definition Classes
    HadoopExamples
  30. def formatFragments: (Fragments) ⇒ Fragments

    Definition Classes
    DefaultFragmentsFormatting → FragmentsFormatting
  31. lazy val fss: FileSystems

    Attributes
    protected[com.nicta.scoobi]
    Definition Classes
    LibJars
  32. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  33. def hadoopArgs: Array[String]

    Definition Classes
    InMemoryHadoop
  34. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  35. def identification: SpecIdentification

    Definition Classes
    SpecificationStructure
  36. implicit def identifySpecificationStructure(s: SpecificationStructure): SpecIdentification

    Definition Classes
    SpecificationStructure
  37. def inMemory[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body locally

    execute an example body locally

    Definition Classes
    HadoopExamples
  38. def inMemory: HadoopContext

    context for in memory execution

    context for in memory execution

    Definition Classes
    HadoopExamples
  39. def inMemory[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code in memory, using a collection backend, possibly showing execution times

    execute some code in memory, using a collection backend, possibly showing execution times

    Definition Classes
    InMemoryHadoop
  40. def includeLibJars: Boolean

    returns

    true if you want to include the library jars in the jar that is sent to the cluster for each job

    Definition Classes
    Hadoop
  41. def isCluster: Boolean

    Definition Classes
    ScoobiUserArgs
  42. def isClusterOnly: Boolean

    returns

    true if the cluster argument is specified and the local argument is not

    Definition Classes
    ScoobiUserArgs
  43. def isInMemory: Boolean

    tests are always in memory by default, unless !inmemory is passed

    tests are always in memory by default, unless !inmemory is passed

    Definition Classes
    HadoopExamplesScoobiUserArgs
  44. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  45. def isLocal: Boolean

    tests are always local by default, unless !local is passed

    tests are always local by default, unless !local is passed

    Definition Classes
    HadoopExamplesScoobiUserArgs
  46. def isLocalOnly: Boolean

    alias for locally

    alias for locally

    Definition Classes
    ScoobiUserArgs
  47. def jars(implicit configuration: ScoobiConfiguration): Seq[URL]

    returns

    the list of library jars to upload

    Definition Classes
    LibJars
  48. def keepFiles: Boolean

    returns

    false if temporary files and working directory must be cleaned-up after job execution

    Definition Classes
    ScoobiUserArgsScoobiArgs
  49. def level: Level

    returns

    the log level to use when logging

    Definition Classes
    ScoobiUserArgsScoobiArgs
  50. def libjarsDirectory(implicit configuration: ScoobiConfiguration): String

    returns

    the path of the directory to use when loading jars to the filesystem.

    Definition Classes
    LibJars
  51. def local: HadoopContext

    context for local execution

    context for local execution

    Definition Classes
    HadoopExamples
  52. def locally[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body locally

    execute an example body locally

    Definition Classes
    HadoopExamples
  53. def locally: Boolean

    returns

    true if the Scoobi job must be run locally

    Definition Classes
    ScoobiUserArgsScoobiArgs
  54. def map(fs: ⇒ Fragments): Fragments

    add a first step to upload the library jars before doing anything else

    add a first step to upload the library jars before doing anything else

    Definition Classes
    UploadedLibJars → SpecificationStructure
  55. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  56. def noLibJars: Boolean

    returns

    false if libjars are used

    Definition Classes
    ScoobiUserArgsScoobiArgs
  57. final def notify(): Unit

    Definition Classes
    AnyRef
  58. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  59. def onCluster[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code on the cluster, possibly showing the execution time

    execute some code on the cluster, possibly showing the execution time

    Definition Classes
    Hadoop
  60. def onHadoop[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code, either locally or on the cluster, depending on the local argument being passed on the commandline

    execute some code, either locally or on the cluster, depending on the local argument being passed on the commandline

    Definition Classes
    Hadoop
  61. def onLocal[T](t: ⇒ T)(implicit configuration: ScoobiConfiguration): T

    execute some code locally, possibly showing execution times

    execute some code locally, possibly showing execution times

    Definition Classes
    LocalHadoop
  62. def quiet: Boolean

    returns

    true to suppress log messages

    Definition Classes
    ScoobiUserArgsScoobiArgs
  63. def remotely[R](r: ⇒ R)(implicit arg0: AsResult[R]): Result

    execute an example body on the cluster

    execute an example body on the cluster

    Definition Classes
    HadoopExamples
  64. def runInMemory[T](t: ⇒ T): T

    returns

    the result of the in-memory run

    Definition Classes
    InMemoryHadoop
  65. def runOnCluster[T](t: ⇒ T): T

    returns

    the cluster evaluation of t

    Definition Classes
    Hadoop
  66. def runOnLocal[T](t: ⇒ T): T

    returns

    the result of the local run

    Definition Classes
    LocalHadoop
  67. lazy val scoobiArgs: Seq[String]

    the scoobi arguments passed on the command line

    the scoobi arguments passed on the command line

    Definition Classes
    CommandLineScoobiUserArgsScoobiUserArgs
  68. def set(args: Arguments): Unit

    Definition Classes
    CommandLineArguments
  69. def setLogFactory(name: String = classOf[HadoopLogFactory].getName): Unit

    Static setup to use a testing log factory

    Static setup to use a testing log factory

    Definition Classes
    InMemoryHadoop
  70. def showPlanOnly: Boolean

    returns

    true if the debug logs must show the computation graph and not evaluate it

    Definition Classes
    ScoobiUserArgsScoobiArgs
  71. def showTime[T](t: ⇒ T)(display: (SimpleTimer) ⇒ Unit): T

    measure the time taken by some executed code and display the time with a specific display function

    measure the time taken by some executed code and display the time with a specific display function

    Definition Classes
    InMemoryHadoop
  72. def showTimes: Boolean

    returns

    true to display execution times for each job

    Definition Classes
    ScoobiUserArgsScoobiArgs
  73. def skippedContext(name: String): HadoopContext

    context for showing a skipped execution

    context for showing a skipped execution

    Definition Classes
    HadoopExamples
  74. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  75. lazy val sysProps: SystemProperties

    Attributes
    protected[com.nicta.scoobi]
    Definition Classes
    LibJars
  76. def toString(): String

    Definition Classes
    AnyRef → Any
  77. def upload: Boolean

    this variable controls if the upload must be done at all

    this variable controls if the upload must be done at all

    Definition Classes
    LibJars
  78. def uploadLibJarsFiles(deleteLibJarsFirst: Boolean = false)(implicit configuration: ScoobiConfiguration): Unit

    upload the jars which don't exist yet in the library directory on the cluster

    upload the jars which don't exist yet in the library directory on the cluster

    Definition Classes
    LibJars
  79. def uploadStep: Step

    create a Step to upload the jars on the cluster

  80. def uploadedJars(implicit configuration: ScoobiConfiguration): Seq[Path]

    returns

    the remote jars currently on the cluster

    Definition Classes
    LibJars
  81. def useHadoopConfDir: Boolean

    returns

    true if cluster configuration must be loaded from Hadoop's configuration directory

    Definition Classes
    ScoobiUserArgsScoobiArgs
  82. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  83. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  84. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  85. def withTimer[T](t: ⇒ T): (T, SimpleTimer)

    returns

    the time for the execution of a piece of code

    Definition Classes
    InMemoryHadoop

Inherited from HadoopExamples

Inherited from CommandLineScoobiUserArgs

Inherited from CommandLineArguments

Inherited from DelayedInit

Inherited from Hadoop

Inherited from LibJars

Inherited from Cluster

Inherited from LocalHadoop

Inherited from InMemoryHadoop

Inherited from ScoobiUserArgs

Inherited from ScoobiArgs

Inherited from SpecificationStructure

Inherited from DefaultFragmentsFormatting

Inherited from TagsAssociation

Inherited from FragmentsFormatting

Inherited from AnyRef

Inherited from Any

Ungrouped