com.nicta.scoobi.impl.plan.mscr
MscrsDefinition
A layer contains group by keys and floating nodes defining mscrs so that none of them have dependencies relationship
A layer contains group by keys and floating nodes defining mscrs so that none of them have dependencies relationship
Because of this property they can be executed in parallel
add a map to output values to non-filled sink nodes if there are some
add a map to output values to non-filled sink nodes if there are some
collect all the sinks in the computation graph
collect all the sinks in the computation graph
all the strategies to apply, in sequence
all the strategies to apply, in sequence
the bypass output channels for a given layer
compute the children of a node.
compute the children of a node.
This is similar to calling the initTree method on the node but this stores the information as an attribute instead of storing it as a tree. This is a way to avoid conflicts if we have 2 processes trying to initialise the same graph
Combine nodes which are not the output of a GroupByKey must be transformed to a ParallelDo
Combine nodes which are not the output of a GroupByKey must be transformed to a ParallelDo
get all the non-visited nodes going from an input nodes to an output
get all the non-visited nodes going from an input nodes to an output
find the input and output channels on the layer and create a Mscr from those channels
find the input and output channels on the layer and create a Mscr from those channels
create a Mscr from input nodes, making sure not to use already visited nodes
create a Mscr from input nodes, making sure not to use already visited nodes
From start nodes in the graph and the list of already visited nodes, create new MapReduce jobs
create MapReduce jobs from the computation graph defined by the start node where each map reduce job is independent for the next map reduce jobs in the list
compute all the descendents of a node They are all the recursive children reachable from this node
compute all the descendents of a node They are all the recursive children reachable from this node
duplicate the whole graph by copying all nodes
duplicate the whole graph by copying all nodes
this attribute stores the fact that a Sink has received data
this attribute stores the fact that a Sink has received data
a gbk output channel based on the nodes which are following the gbk
groups of input channels having at least one output tag in common
true if a given Sink has already received data
initialise the parent/child relationship recursively from node s
initialise the parent/child relationship recursively from node s
Process or Load nodes which are children of the nodes parameters but not included in the group these "input nodes" don't include Return nodes or Op nodes because those inputs are retrieved via environments
compute the inputs of a given node For a ParallelDo node this does not consider its environment
compute the inputs of a given node For a ParallelDo node this does not consider its environment
true if a node is an input node for a given layer
true if a node is a candidate for outputing values
return true if a CompNode has a cycle in its graph, this will be detected by Kiama throwing an exception when fetching the descendents of a node
return true if a CompNode has a cycle in its graph, this will be detected by Kiama throwing an exception when fetching the descendents of a node
node at the end of the graph
node at the end of the graph
a node is said to be reducing if it is in a "reducing chain of nodes" after a gbk
a node is said to be reducing if it is in a "reducing chain of nodes" after a gbk
true if a node is used at most once
true if a node is used at most once
Specialisation of the layersOf function taking a selection function in the case where all nodes are selected
Specialisation of the layersOf function taking a selection function in the case where all nodes are selected
mark a sink as filled so it doesn't have to be recomputed
mark a sink as filled so it doesn't have to be recomputed
true if this parallelDo must be read ==> can't be fused
true if a process node has been filled
Optimise a set of CompNodes, starting from the set of outputs
Optimise a set of CompNodes, starting from the set of outputs
all output channels for a given layer
Nested ParallelDos must be fused but only if pd1 is not used anywhere else.
Nested ParallelDos must be fused but only if pd1 is not used anywhere else.
We use somebu to fuse the nodes "bottom-up" starting from all leaves of the tree at the same time
pd1 @ ParallelDo | pd2 @ ParallelDo
This rule is repeated until nothing can be fused anymore
compute the parent of a node.
compute the parent of a node. This relationship is actually maintained while getting the children of a node if the parent node has not been set while recursing for children, then it is None
compute the parents of a node, that is all the chain of parents from this node up to the root of the graph
compute the parents of a node, that is all the chain of parents from this node up to the root of the graph
a nested text representation of the nodes graph
a nested text representation of the nodes graph + graph if it's not too big
non-empty layers of processing nodes
reinitialise all the attributes related to a node, starting from all the parent/children relationships
reinitialise all the attributes related to a node, starting from all the parent/children relationships
reset the attributes, then recreate the parent/children relationships recursively
apply a traversal strategy but make sure that:
apply a traversal strategy but make sure that:
- after each pass the tree is reset in terms of attributable relationships and uses - the strategy to execute is memoised, i.e. if a node has already been processed its result must be reused this ensures that rewritten shared nodes are not duplicated
reinit usages
reinit usages
the root of the graph, computed from a given node
the root of the graph, computed from a given node
Show instance for a CompNode
Show instance for a CompNode
Show is not covariant so it is necessary to add this implicit to prove that we can show subclasses of CompNode
Show is not covariant so it is necessary to add this implicit to prove that we can show subclasses of CompNode
show the structure without the ids or type annotations
show the structure without the ids or type annotations
compute all the indirect uses of a given node, that is all the nodes which have this node as a descendent
compute all the indirect uses of a given node, that is all the nodes which have this node as a descendent
remove nodes from the tree based on a predicate
remove nodes from the tree based on a predicate
the nodes which have this node as a direct child
the nodes which have this node as a direct child
for efficiency, this uses a table of all the nodes using a given one, computed from the root
compute all the nodes which use a given node as an environment
compute all the nodes which use a given node as an environment
This trait processes the computation graph created out of DLists and creates map-reduce jobs from it.
The algorithm consists in:
- building layers of independent nodes in the graph - finding the input nodes for the first layer - reaching "output" nodes from the input nodes - building output channels with those nodes - building input channels connecting the output to the input nodes - aggregating input and output channels as Mscr representing a full map reduce job - iterating on any processing node that is not part of a Mscr