Scoobi
checkpoints
checkpoints
Sequence File I/O
Sequence File I/O
Extract a Double from a String.
Extract a Double from a String.
Extract a Float from a String.
Extract a Float from a String.
Extract a Long from a String.
Extract a Long from a String.
Extract an Int from a String.
Extract an Int from a String.
Catch-all implementation of a WireFormat for a type T, using Java serialization.
Catch-all implementation of a WireFormat for a type T, using Java serialization. It is however very inefficient, so for production use, consider creating your own WireFormat instance.
Note that this WireFormat instance definition is *not* implicit, meaning that you will need to explicitely import it when you need it.
WireFormat instance for Avro Specific Records
WireFormat instance for Avro Specific Records
This requires an implicit AvroSchema instance
Avro I/O
Avro I/O
An implicitly Grouping type class instance where sorting is implemented via an Ordering type class instance.
An implicitly Grouping type class instance where sorting is implemented via an Ordering type class instance. Partitioning and grouping use the default implementation.
Java's Date
Java's Date
WireFormat instance for Avro Generic Records
WireFormat instance for Avro Generic Records
No implicit AvroSchema instance is required. A dummy schema is used to start the reading of the file and then the GenericAvroKeyRecordWriter/GenericAvroKeyRecordReader will inspect the schema that is defined by the first generic record that is written / read to instance an appropriate AvroKeyRecordWriter/AvroKeyRecordReader
Map structures
Map structures
Option type.
Option type.
An implicitly Grouping type class instance where sorting is implemented via an Ordering type class instance.
An implicitly Grouping type class instance where sorting is implemented via an Ordering type class instance. Partitioning and grouping use the default implementation.
An implicitly Grouping type class instance where sorting is implemented via an Ordering type class instance.
An implicitly Grouping type class instance where sorting is implemented via an Ordering type class instance. Partitioning and grouping use the default implementation.
WireFormat instance for Avro SpecificFixed
WireFormat instance for Avro SpecificFixed
This requires an implicit AvroSchema instance
Traversable structures
Traversable structures
"Primitive" types.
"Primitive" types.
Hadoop Writable types.
Hadoop Writable types.
make a DList runnable, executing the computation and returning the values
make a DList runnable, executing the computation and returning the values
make a DObject runnable, executing the computation and returning the values
make a DObject runnable, executing the computation and returning the values
INPUT CONVERTERS
INPUT CONVERTERS
Note this is an expensive conversion (it adds an extra map-reduce job), try save the result to reuse if applicable.
Note this is an expensive conversion (it adds an extra map-reduce job), try save the result to reuse if applicable.
Note this is an expensive conversion (it adds an extra map-reduce job), try save the result to reuse if applicable
Note this is an expensive conversion (it adds an extra map-reduce job), try save the result to reuse if applicable
Create a new DList from the contents of a list of one or more Avro files.
Create a new DList from the contents of a list of one or more Avro files. The type of the DList must conform to the schema types allowed by Avro, as constrained by the 'AvroSchema' type class. In the case of a directory being specified, the input forms all the files in that directory.
Create a new DList from the contents of one or more Avro files.
Create a new DList from the contents of one or more Avro files. The type of the DList must conform to the schema types allowed by Avro, as constrained by the 'AvroSchema' type class. In the case of a directory being specified, the input forms all the files in that directory.
Create a distributed list from a text file that is a number of fields delimited by some separator.
Create a distributed list from a text file that is a number of fields delimited by some separator. Use an extractor function to pull out the required fields to create the distributed list.
Create a distributed list from a list of one ore more files or directories (in the case of a directory, the input forms all files in that directory).
Create a distributed list from a list of one ore more files or directories (in the case of a directory, the input forms all files in that directory). The file(s) contain a number of fields delimited by a separator. Use an extractor function to pull out the required fields to create the distributed list.
create a DList from a stream of elements which will only be evaluated on the cluster
create a DList from a stream of elements which will only be evaluated on the cluster
Create a new DList from the contents of a list of one or more Sequence Files.
Create a new DList from the contents of a list of one or more Sequence Files. Note that the type parameters K and V are the "converted" Scala types for the Writable key-value types that must be contained in the the Sequence Files. In the case of a directory being specified, the input forms all the files in that directory.
Create a new DList from the contents of a list of one or more Sequence Files.
Create a new DList from the contents of a list of one or more Sequence Files. Note that the type parameters K and V are the "converted" Scala types for the Writable key-value types that must be contained in the the Sequence Files. In the case of a directory being specified, the input forms all the files in that directory.
SOURCES
SOURCES
Create a DList from one or more files or directories (in the case of a directory, the input forms all files in that directory).
Create a DList from one or more files or directories (in the case of a directory, the input forms all files in that directory).
Create a distributed list from one or more files or directories (in the case of a directory, the input forms all files in that directory).
Create a distributed list from one or more files or directories (in the case of a directory, the input forms all files in that directory). The distributed list is a tuple where the first part is the path of the originating file and the second part is a line of text.
Create a distributed list from a list of one or more files or directories (in the case of a directory, the input forms all files in that directory).
Create a distributed list from a list of one or more files or directories (in the case of a directory, the input forms all files in that directory). The distributed list is a tuple where the first part is the path of the originating file and the second part is a line of text.
TEXT SOURCES
TEXT SOURCES
Create a new DList from the "key" contents of a list of one or more Sequence Files.
Create a new DList from the "key" contents of a list of one or more Sequence Files. Note that the type parameter K is the "converted" Scala type for the Writable key type that must be contained in the the Sequence Files. In the case of a directory being specified, the input forms all the files in that directory.
Create a new DList from the "key" contents of one or more Sequence Files.
Create a new DList from the "key" contents of one or more Sequence Files. Note that the type parameter K is the "converted" Scala type for the Writable key type that must be contained in the the Sequence Files. In the case of a directory being specified, the input forms all the files in that directory.
Create a new DList from the "key" contents of a list of one or more Sequence Files.
Create a new DList from the "key" contents of a list of one or more Sequence Files. Note that the type parameter K is the "converted" Scala type for the Writable key type that must be contained in the the Sequence Files. In the case of a directory being specified, the input forms all the files in that directory. The distributed list is a tuple where the first part is the path of the originating file and the second part is the "key".
Partitioned sinks
Partitioned sinks
create a DObject which will only be evaluated on the cluster
create a DObject which will only be evaluated on the cluster
Persist a distributed lists of 'Products' (e.g.
Persist a distributed lists of 'Products' (e.g. Tuples) as a delimited text file.
Text file I/O
Text file I/O
Persist a distributed object of 'Products' (e.g.
Persist a distributed object of 'Products' (e.g. Tuples) as a delimited text file.
Persisting
Persisting
allow to call list.persist
allow to call list.persist
allow to call object.persist
allow to call object.persist
run a list.
run a list.
This is equivalent to:
val obj = list.materialise
run(obj)
SINKS
SINKS
create a text source
create a text source
implicit conversions to Writables
implicit conversions to Writables
Create a new DList from the "value" contents of a list of one or more Sequence Files.
Create a new DList from the "value" contents of a list of one or more Sequence Files. Note that the type parameter V is the "converted" Scala type for the Writable value type that must be contained in the the Sequence Files. In the case of a directory being specified, the input forms all the files in that directory.
Create a new DList from the "value" contents of one or more Sequence Files.
Create a new DList from the "value" contents of one or more Sequence Files. Note that the type parameter V is the "converted" Scala type for the Writable value type that must be contained in the the Sequence Files. In the case of a directory being specified, the input forms all the files in that directory.
Create a new DList from the "value" contents of a list of one or more Sequence Files.
Create a new DList from the "value" contents of a list of one or more Sequence Files. Note that the type parameter V is the "converted" Scala type for the Writable value type that must be contained in the the Sequence Files. In the case of a directory being specified, the input forms all the files in that directory. The distributed list is a tuple where the first part is the path of the originating file and the second part is the "value".
Java's Map
Java's Map
Global Scoobi functions and values.