com.nicta.scoobi.impl

ScoobiConfigurationImpl

case class ScoobiConfigurationImpl(hadoopConfiguration: Configuration = new Configuration, userJars: Set[String] = Set(), userDirs: Set[String] = Set(), classLoader: Option[AbstractFileClassLoader] = None, counters: Counters = new HadoopCounters) extends ScoobiConfiguration with Product with Serializable

Linear Supertypes
Serializable, Serializable, Product, Equals, ScoobiConfiguration, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. ScoobiConfigurationImpl
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. ScoobiConfiguration
  7. AnyRef
  8. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ScoobiConfigurationImpl(hadoopConfiguration: Configuration = new Configuration, userJars: Set[String] = Set(), userDirs: Set[String] = Set(), classLoader: Option[AbstractFileClassLoader] = None, counters: Counters = new HadoopCounters)

Value Members

  1. final def !=(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  4. def addClassLoader(cl: AbstractFileClassLoader): ScoobiConfigurationImpl

    attach a classloader which classes must be put on the job classpath

    attach a classloader which classes must be put on the job classpath

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  5. def addJar(jar: String): ScoobiConfigurationImpl

    add a new jar url (as a String) to the current configuration

    add a new jar url (as a String) to the current configuration

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  6. def addJarByClass(clazz: Class[_]): ScoobiConfigurationImpl

    add a new jar of a given class, by finding the url in the current classloader, to the current configuration

    add a new jar of a given class, by finding the url in the current classloader, to the current configuration

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  7. def addJars(jars: Seq[String]): ScoobiConfigurationImpl

    add several user jars to the classpath of this configuration

    add several user jars to the classpath of this configuration

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  8. def addUserDir(dir: String): ScoobiConfigurationImpl

    add a user directory to the classpath of this configuration

    add a user directory to the classpath of this configuration

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  9. def addUserDirs(dirs: Seq[String]): ScoobiConfigurationImpl

    add several user directories to the classpath of this configuration

    add several user directories to the classpath of this configuration

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  10. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  11. var classLoader: Option[AbstractFileClassLoader]

  12. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  13. def concurrentJobs: Boolean

    returns

    true if the mscr jobs can be executed concurrently

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  14. lazy val configuration: Configuration

    Scoobi's configuration, initialised with a job id

    Scoobi's configuration, initialised with a job id

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  15. def countValuesPerMapper: Boolean

    returns

    true if we must count the number of values per mapper

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  16. def countValuesPerReducer: Boolean

    returns

    true if we must count the number of values per reducer

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  17. val counters: Counters

  18. def defaultScoobiDir: String

  19. def deleteScoobiDirectory: Boolean

  20. def deleteTemporaryOutputDirectory: Boolean

  21. def deleteWorkingDirectory: Boolean

  22. def disableCombiners: ScoobiConfiguration

    do not use combiners

    do not use combiners

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  23. def disabledCombiners: Boolean

    returns

    true if combiners are disabled

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  24. def duplicate: ScoobiConfigurationImpl

  25. def enableCombiners: ScoobiConfiguration

    use combiners

    use combiners

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  26. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  27. def fileSystem: FileSystem

    returns

    the file system for this configuration, either a local or a remote one

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  28. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  29. def getBytesPerReducer: Long

    Get the number of input bytes per reducer.

    Get the number of input bytes per reducer. Default is 1GiB.

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  30. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  31. def getMaxReducers: Int

    Get the max number of reducers to use in M/R jobs

    Get the max number of reducers to use in M/R jobs

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  32. def getMinReducers: Int

    Get the min number of reducers to use in M/R jobs

    Get the min number of reducers to use in M/R jobs

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  33. def includeLibJars(jars: Seq[URL]): ScoobiConfigurationImpl

    add a list of jars to include as -libjars in this configuration

    add a list of jars to include as -libjars in this configuration

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  34. def isInMemory: Boolean

    returns

    true if this configuration is used for an in memory execution with a collection backend

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  35. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  36. def isLocal: Boolean

    returns

    true if this configuration is used for a local memory execution

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  37. def isRemote: Boolean

    returns

    true if this configuration is used for a remote job execution

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  38. def jobId: String

    The id for the current Scoobi job being (or about to be) executed.

    The id for the current Scoobi job being (or about to be) executed.

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  39. def jobName: Option[String]

    returns

    the job name if one is defined

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  40. def jobNameIs(name: String): Unit

    set a new job name to help recognize the job better

    set a new job name to help recognize the job better

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  41. def jobStep: String

    returns

    the job step

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  42. def jobStepIs(stepId: Int, stepsNumber: Int): String

    The job name for a step in the current Scoobi, i.e.

    The job name for a step in the current Scoobi, i.e. a single MapReduce job

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  43. def loadDefaults: ScoobiConfigurationImpl

    get the default values from the configuration files

    get the default values from the configuration files

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  44. def loadMapredSiteProperties: JobConf

  45. def mode: Mode

    returns

    the current mode

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  46. def modeIs(mode: Mode): ScoobiConfigurationImpl

    set a flag in order to know that this configuration is for a in-memory, local or remote execution,

    set a flag in order to know that this configuration is for a in-memory, local or remote execution,

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  47. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  48. def newEnv(wf: WireReaderWriter): Environment

    returns

    a new environment object

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  49. final def notify(): Unit

    Definition Classes
    AnyRef
  50. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  51. def parse(commandLineArg: String, value: String): ScoobiConfigurationImpl

    use the GenericOptionsParser to parse the value of a command line argument and update the current configuration The command line argument doesn't have to start with a dash.

    use the GenericOptionsParser to parse the value of a command line argument and update the current configuration The command line argument doesn't have to start with a dash.

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  52. def persist[A](o: DObject[A]): A

  53. def persist[A](list: DList[A]): DList[A]

  54. def persist(ps: Seq[Persistent[_]]): Seq[Persistent[_]]

  55. def scoobiClassLoader: ClassLoader

    returns

    the class loader to use with Scoobi classes, including generated ones

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  56. lazy val scoobiDir: String

  57. lazy val scoobiDirectory: Path

  58. def set(key: String, value: Any): Unit

    set a value on the configuration

    set a value on the configuration

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  59. def setAsInMemory: ScoobiConfiguration

    force a configuration to be an in-memory one, currently doing everything as in the local mode

    force a configuration to be an in-memory one, currently doing everything as in the local mode

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  60. def setAsLocal: ScoobiConfiguration

    force a configuration to be a local one

    force a configuration to be a local one

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  61. def setBytesPerReducer(sizeInBytes: Long): Unit

    Set the number of input bytes per reducer.

    Set the number of input bytes per reducer. This is used to control the number of reducers based off the size of the input data to the M/R job.

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  62. def setConcurrentJobs(concurrent: Boolean): ScoobiConfigurationImpl

    set to true if the mscr jobs must be executed concurrently

  63. def setCountValuesPerMapper(b: Boolean): Unit

    if true count the number of values per mapper

    if true count the number of values per mapper

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  64. def setCountValuesPerReducer(b: Boolean): Unit

    if true count the number of values per reducer

    if true count the number of values per reducer

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  65. def setDirectories: ScoobiConfigurationImpl

    this setup needs to be done only after the internal conf object has been set to a local configuration or a cluster one because all the paths will depend on that

    this setup needs to be done only after the internal conf object has been set to a local configuration or a cluster one because all the paths will depend on that

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  66. def setMaxReducers(maxReducers: Int): Unit

    Set an upper bound for the number of reducers to be used in M/R jobs

    Set an upper bound for the number of reducers to be used in M/R jobs

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  67. def setMinReducers(minReducers: Int): Unit

    Set a lower bound for the number of reducers to be used in M/R jobs

    Set a lower bound for the number of reducers to be used in M/R jobs

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  68. def setScoobiDir(dir: String): ScoobiConfigurationImpl

  69. def setUploadedLibJars(uploaded: Boolean): Unit

    set a flag in order to know if jars have been uploaded before jobs are defined

    set a flag in order to know if jars have been uploaded before jobs are defined

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  70. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  71. lazy val tempDirName: String

  72. lazy val temporaryDir: String

  73. lazy val temporaryJarFile: File

  74. def temporaryOutputDirectory: Path

  75. def updateCounters(hadoopCounters: Counters): ScoobiConfigurationImpl

    update the current counters from the result of a job that has just been run

    update the current counters from the result of a job that has just been run

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  76. def uploadedLibJars: Boolean

    returns

    true if the dependent jars have been uploaded

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  77. def userClasses: Map[String, Array[Byte]]

    returns

    user classes to add in the job jar, by class name (and corresponding bytecode)

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  78. var userDirs: Set[String]

  79. var userJars: Set[String]

  80. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  81. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  82. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  83. def withHadoopArgs(args: Array[String])(f: (Array[String]) ⇒ Unit): ScoobiConfiguration

    Parse the generic Hadoop command line arguments, and call the user code with the remaining arguments

    Parse the generic Hadoop command line arguments, and call the user code with the remaining arguments

    Definition Classes
    ScoobiConfigurationImplScoobiConfiguration
  84. lazy val workingDir: String

  85. lazy val workingDirectory: Path

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from ScoobiConfiguration

Inherited from AnyRef

Inherited from Any

Ungrouped