Flow

General purpose graph package for Go

Initial Idea

possibilities

Machine learning
Graphical CI pipeline

Flow package

Using flow package without UI

package main

import (
	"flow"
	"flow/registry"
	"log"
	"strings"
)

func main() {

	r := registry.New()
	r.Add(strings.Split, strings.Join)

	f := flow.New()
	f.SetRegistry(r)

	op := f.Op("Join",
		f.Op("Split", "hello world", " "),
		",",
	)

	res, err := op.Process()
	if err != nil {
		log.Fatal(err)
	}
	log.Println("res:", res)

}

How it works

A flow works by requesting the previous nodes the results of its operation, so the starting node will be the node we want the result of, unattached nodes wont be executed

Registering a function

All nodes are Go functions to create a node src simply add a function like:

package main

import (
	"flow/flowserver"
	"flow/registry"
	"net/http"
)

func main() {

	r := registry.New()
	r.Add("hello", func() string {
		return "hello world"
	})

	http.ListenAndServe(":5000", flowserver.New(r, "storename"))

}

Resulting in

Go funcs

Since every node is a function we can register functions from any Go package

// Adding functions from a go package
r.Add(strings.Split, strings.Join)

Describing funcs

We can describe the inserted functions either by using r.Add methods or grouping functions with the helper registry.Describer

// utility to apply functions to several entries
registry.Describer(
	r.Add(strings.Split).Inputs("str", "sep").Output("slice"),
	r.Add(strings.Join).Inputs("slice", "sep").Output("string"),
).Tags("strings").Extra("style", registry.M{"color": "#a77"})

Special Operations

f.In

Uses an argument passed through the process function

op := f.In(0)
res, err := op.Process("arg")
// res will be "arg"

op := f.Op("sum",f.In(0),f.In(1)
res, err := op.Process(1,2)
// res will be the sum the inputs 1 and 2

f.Var

Var creates an operation that loads data from the flow, if there is no data the second parameter will be used for variable initialization

f.Var("myvar", "initial value or operation")

f.SetVar

SetVar creates an operation that sets a variable to the result of the operation passed as the argument

f.SetVar("myvar", operation)

UI Special nodes

Input

Same as f.In where a property must be set to indicate which argument this node refers to

Var

Same as f.Var passing the variable name as a property of the node

SetVar

Same as f.SetVar passing the variable name as a property of the node

Output

Deprecated
See here

~~Output is a special node that indicates the overall flow output, the UI contains a RUN button that will be visible if there is an output node and the flow will be executed from the output node.~~

~~There can be only one output node~~

Portal From

Portals are helper nodes that allows to connect areas of UI without crossing any links, right click in a node and choose create portal or drag a link from an output socket to an empty area, this will create a portal from the node

Ideas/TODOS

Matching types

in UI we can create a state such as draggingType{in:'io.Writer'} which basically each node can figure a match for each socket

in Server side we can do an analysis on each type to see if it can be implemented by another type and send this as part of the registry with all descriptions, this way we can highlight sockets that can implement the dragging type

Generating code

generate code based on a flow should be simple as we have the function signatures

Portals

Named portal would connect a node to another without any link, this way we can have clean links without crossovers

Recursive registering

Since an operation can be run with op.Process(inputs...) we can simple do registry.Add(op.Process) to register a flow as an operation, UI doesn't retain the registry/flow yet

Multiple node editing

While on multiple selection the inspector can show common properties and editing those will affect all nodes in the selection, RUN would be disabled in multiple selection

Dynamic context menu

Right now the context menu works while right clicking in a node, would be good to have the same by right clicking in other subjects:

Link
- Delete
Trigger
- toggle On: error
- toggle On: success
Editor
- toggle Show Detail
- toggle Show Activity
- toggle Show Triggers
- Reset view

Actions

Instead of having a single Output node that shows the RUN button we would setup nodes to be processed in an action having possible multiple nodes per action

something like inspecting node and defining a list of actions that node would be processed such as [TRAIN, MAIN]

TODO

Readme

Fix any typos bad English etc...
Create an explanation of the project inspiration/goals
describe milestones

UX/UI

UX/UI: Create Undo behaviour, Possibly easy since this is using vuex (centralized state management)
UX/UI: Special visualisers/nodes to display information (images, datatables, graphs);
UX/UI: Ability to group nodes into a single box exposing inputs and outputs;
UX/UI: Implement touch;
UX/UI: Drop link in node to link to the next compatible input;
UX/UI: Find a way to highlight/linked portals for easy user reference
UX/UI: Multiple selection inspecting idea
UX/UI: Dynamic Context menu
UX/UI: link/node animation synchronization on collaborative environment
~~UX/UI: Portals to clean graph crossing~~ (testing);
~~UX/UI: Shift key to merge group selections~~;

Packages

FlowPkg: Create training mechanism;
FlowPkg: Machine learning examples;
FlowPkg: Generate go code to maximize call performance;

Backend

Backend: Export/Deploy a model to a training cluster;
Backend: Expose the model possibly via an Rest API for easy function access;
Backend: Create output for flow-web/flow so we can fetch data for it

Other

Collaboration: Better concurrent editing/message passing;
Flow: Proper documentation;

Changelog

27/02/2018

UI/UX: removed SVGMatrix favouring CSSMatrix and manual matrix calculations Area now rotates around a point, although this is not really useful
UI/UX: improved selection behaviour
UI/UX: Added context menu for editor area, removed some buttons from top button bar

25/02/2018

UI/UX: Added image visualization if the content result is a dataurl
UI/UX: Removed flow-ui Output node source, the button run which is now Process will process and fetch data from selected nodes

18/02/2018

frontend: Refactored nodes and links to improve performance with lots of elements
- Cleaned up some areas while refactoring, less code
- Remove dependencies on $refs and DOM elements, nodes now calculate size based on monospace font
- moved some display logic to respective areas, as nodeProps from editor to node
- Cached node dimensions for performance

15/02/2018

Selection: Added clone using ctrl + drag will clone selected nodes
Selection: Added drag threshold, only start dragging if pointer goes over a threshold value
Selection: Moved selection to vuex state
Styling: Few changes in css for secondary button
Inspector: Added result button to inspector
Activity: Added new activity icon if the node has data

13/02/2018

Selection: Animated selection areas to improve visibility
Portal node: dragging from a output socket to an empty area will create a portal
Portal node: Improved backend portal handling
Dark: Improved styling in the dark mode
Editor: Shift now increases zoom speed, added small transition to smooth zoom

README.md 8.1 KB Permalink History Raw