application

Type Members

1. trait Application extends AnyRef

   

2. trait Cluster extends AnyRef

   

   Definition of the Cluster addresses: FileSystem + JobTracker

3. trait ClusterConfiguration extends Cluster

   

   Implementation of the Cluster trait taking the configuration from a ScoobiConfiguration object

4. trait CommandLineScoobiUserArgs extends ScoobiUserArgs with CommandLineArguments

   

5. trait DLists extends AnyRef

   

   This object provides a set of operations to create distributed lists.

6. trait DObjects extends AnyRef

   

   Implicit conversions for DObjects

7. trait Hadoop extends LocalHadoop with Cluster with LibJars

   

   This trait provides methods to execute map-reduce code, either locally or on the cluster.

   This trait provides methods to execute map-reduce code, either locally or on the cluster.

   See also

   LocalHadoop

   To use this trait, you need to provide:

   • the file system address: def fs = "hdfs://svm-hadoop1.ssrg.nicta.com.au"
   • the job tracker address: def jobTracker = "svm-hadoop1.ssrg.nicta.com.au:8021"

8. class HadoopLogFactory extends LogFactory

   

   Log factory used for testing.

   Log factory used for testing.

   It doesn't display any log message by default, unless the QUIET attribute is set to true.

   It can display SCOOBI_TIMES messages if the SHOW_TIMES attributes is true

9. trait HadoopLogFactoryInitialisation extends AnyRef

   

10. trait ILoopCompat extends ILoop

    

11. trait InMemoryHadoop extends ScoobiUserArgs

    

12. trait InputsOutputs extends TextInput with TextOutput with AvroInput with AvroOutput with SequenceInput with SequenceOutput

    

    This trait provides way to create DLists from files and to add sinks to DLists so that the results of computations can be saved to files

13. trait LibJars extends AnyRef

    

    This trait defines:

    This trait defines:

    - the library jars which can be uploaded to the cluster - a method to upload and reference them on the classpath for cluster jobs

14. trait LocalHadoop extends InMemoryHadoop

    

    Execute Hadoop code locally

15. trait Orderings extends AnyRef

    

16. trait Persist extends AnyRef

    

    This trait provides operations and implicit definitions to persist DLists and DObjects.

    This trait provides operations and implicit definitions to persist DLists and DObjects.

    Generally persist on a DList or a DObject executes the computations but doesn't return any value while run return values, triggering the computations if necessary (but not if persist has already been called)

    Several DLists and DObjects which are part of the same logical computation graph can be persisted jointly by calling persist(objects and lists)

17. trait ReplFunctions extends AnyRef

    

18. trait ScoobiApp extends ScoobiCommandLineArgs with ScoobiAppConfiguration with Hadoop with HadoopLogFactoryInitialisation with Persist

    

    This trait can be extended to create an application running Scoobi code.

    This trait can be extended to create an application running Scoobi code.

    Command-line arguments are available in the args attribute (minus the hadoop specific ones) and a default implicit ScoobiConfiguration is also accessible to create DLists.

    A ScoobiApp will be used in 2 different contexts:

    1. with the hadoop script

    In that case you will use hadoop default configuration files or you will need to tell this script where to find the configuration files.

    2. within sbt

    In that case the cluster location can be either defined by:

    • overriding the 'fs' and 'jobTracker' methods
    • using the 'useconfdir' command line argument to add the configuration files found in $HADOOP_HOME/conf

    Then, if it can be determined that the execution will not be local but on the cluster (@see locally), the ScoobiApp trait will attempt to load the dependent jars to the libjars directory on the cluster (if not already there, @see LibJars for the details). This behavior can be switched off by overriding the upload method: override def upload = false or by passing the 'nolibjars' argument on the command line

19. trait ScoobiAppConfiguration extends ClusterConfiguration with ScoobiArgs with SystemProperties

    

    This trait provides a ScoobiConfiguration object initialised with the configuration files found in the $HADOOP_HOME/conf directory.

    This trait provides a ScoobiConfiguration object initialised with the configuration files found in the $HADOOP_HOME/conf directory.

    If the ScoobiArgs indicates a local execution the configuration files are not added to the Hadoop configuration object because it wouldn't be possible to remove them afterwards.

20. trait ScoobiArgs extends AnyRef

    

    This trait defines all the options which can be used to modify the behavior of a Scoobi application

21. trait ScoobiCommandLineArgs extends DelayedInit

    

    This trait can be mixed in an Application trait in order to store command-line arguments before any code is executed

22. class ScoobiILoop extends ILoop with ILoopCompat

    

    definition of the interpreter loop

23. trait ScoobiInterpreter extends ScoobiApp with ReplFunctions

    

    This trait can be extended if you want to provide alternative imports or behaviors

24. trait ScoobiUserArgs extends ScoobiArgs

    

    Implementation of the ScoobiArgs trait taking the values from the command line arguments