Why Visi?

Visi is a computer language that allows expression of big data jobs in a syntax that's not frightening to people familiar with Excel formula functions. There is more in Visi than there is in Excel formula functions, but the "more" is adding to what's in Excel.

Let's start with a simple expression… a thing that executes code and returns a value:

1 + 1

That's simple, the expression evaluates to 2. Yay!

In Excel, each cell is a variable… something that can contain a value. In Excel, cells can be constants (values that do not change) or formulas. If a cell contains a formula and that formula refers to another cell, when the predicate cell's value changes, all the dependent cells update their values.

In Visi, you name the placeholders for values. So, to create a names:

x = 5

y = x + 5

The above code created names "x" and "y". "x" is a constant and "y" is dependent on "x".

In Visi and Excel, you can apply a function to a value:

## Define a name that's associated with a String
a1 = "The Brown Cow"

## Calculate the length of the string
length(a1)

13

Visi groups names of functions and other values (in Visi functions are values) into different packages. Functions in packages can be referenced with package::name like:

Math::sin(0.5)

Computes to:

0.479425538604203

In Excel, there are lots and lots of different functions. They are all in the same group, although the user interface (Insert Function) lets you pick functions from categories like "Statistical" or "Financial".

In Visi, functions are grouped in separately named "packages" so that it's easier to find a function you're looking for and sometimes, like in the above example, we have to be explicit about which package the function lives in. The :: between Math and sin tells Visi to look in the Math package for the function.

With Visi, you can use any Java package as part of your Visi notebook. We will cover Java "Host Interoperability" and how to load Java JAR files into your Visi notebook later in this document.

A declaration is the association of a value with a name such that the named declaration can be accessed by other parts of your notebook. For example:

Math::cos(Math::PI / 3) ## expression

cos_third_pi = Math::cos(Math::PI / 3) ## declaration

cos_third_pi ## referencing the name

In Visi, you can create your own functions. So you can use the same logic in many places without having to copy/paste that logic over and over:

calc_tax(income) = income * if(income > 5,000, 40%, 20%)

The above function computes the tax amount depending on the income level.

And functions can call other functions:

tax_rate(income) = if(income > 5,000, 40%, 20%)

calc_tax(income) = income * tax_rate(income)

Visi syntax can span multiple lines and if/then/else can be a formula and it has its own syntax that might be easier to read:

tax_rate(income) =
  if income > 5,000 then
    40%
  else
    20%

calc_tax(income) = income * tax_rate(income)

And Visi code can contain comments so that you and others who read the code can understand what's going on:

tax_rate(income) =
  if income > 5,000 then
    40% ## for high incomes
  else
    20% ## for lower incomes

calc_tax(income) = income * tax_rate(income)

To summarize, Visi syntax is a lot like Excel syntax, but there are extensions to what you are familiar with in Excel to make writing more complex Visi notebooks easier.

In Visi, there are three constructs: expressions, declarations, and housekeeping.

An expression is a value that's computed. Expressions can evaluate to either values or functions. Or more precisely, functions are values just like numbers and strings and collections.

What's a collection? It's a group of other values. For example:

[1, 2, 3, 4]

Is a collection of numbers. The specific collection type is called a Vector. It's a 1 dimensional ordered array of values. Ordered means that the Vector retains the elements in the order that it was originally created.

Visi also supports sets. Sets are unordered collections that contain unique values. So:

#{"foo", "bar", "baz", "foo", "dog", "dog", "baz"}

Only contains 4 elements:

#{"foo", "bar", "dog", "baz"}

Another collection is an association. An association is collection of unique keys and values. Both the keys and values can be any Visi value. Let's create an association of people and ages:

{"David" 51, "Archer" 11, "Tessa" 2}

Note that like sets, the keys in an association are unordered.

If you are familiar with JavaScript, associations in Visi are like Objects in JavaScript… collections of key/value pairs.

Visi also has a keyword type. Keywords are handy for giving a common/shared name to a value in an association. For example:

[{name: "David", type: "Human", age: 51},
 {name: "Archer", type: "Dog", age: 11},
 {name: "Tessa", type: "Cat", age: 2}]

If you're familiar with JavaScript or JSON, the Visi syntax for defining an association looks just like JSON. Familiarity is good.

In Visi, functions are values just like strings and numbers and keywords. The declaration of a function is just fancy syntax, both of the following declarations mean the same thing (don't worry about the function expression syntax for the moment):

## assign a function to a name
plus_one = x => x + 1

another_plus_one(x) = x + 1

z = 99

plus_one(z) == another_plus_one(z)

Are the two functions the same?

true

The first example, assigns the expression x => x + 1 to plus_one. That expression evaluates to a function. The second example does the same thing with different syntax. The latter is "syntactic sugar" for the former.

You may be asking, "why do you have more than one way to say the same thing?" Good question. Visi creates syntactic sugar to give you a more concise or more natural way of expressing the same code. In different contexts, the different syntax may seem more natural. For example, the first declaration looks kind of odd where the second looks like the way we learned functions in math class. We will get to some examples of passing functions as parameters in a little while.

In Visi, top level names (those names defined outside another assignment) can be accessed by any expression in a Visi notebook, just like values in cells in Excel.

Like Excel, Visi doesn't let you change a value once it's computed. This is strange for users of Java and JavaScript. In addition to Visi's syntax which is simple and free from complex stuff from Java like void and class and such, Visi's types can be converted into a form that can be distributed across a cluster of computes. Visi data types can be turned into bytes that get turned back into the same data types on the other side of the network.

Between serialization and immutability, Visi provides tools that make writing distributed computing jobs simple.

Visi sits on top of two host systems. All Visi code is compiled to (well, more technically, transliterated to) Clojure. Then the Clojure code is compiled into Java Virtual Machine byte-code. This means that Visi code works with any Clojure and Java library and Visi code can be used in and called by any JVM-based Big Data platform. Being able to work with any Java and Clojure library means that you have a wealth of libraries to use. But how do you use these libraries in Visi?

First, let's deal with method calls. Java objects all have methods. Rather than using functions like Visi and Clojure, all the actions that can be done to a particular Object is by invoking a method on an Object.

Visi mostly makes method invocation seamless. If there's no function matching the function name, then Visi invokes that named method on the object. For example:

toString([1, 2, 3])

"[1 2 3]"

Boom, the toString method is invoked on the object. In this case, the Vector in Visi is actually represented by a Clojure Vector and we've called the .toString method on it.

You can be explicit about calling a method if there's ambiguity about a name being in scope. By prefixing a function name with $., Visi forces the method invocation:

$.toString($.size([1, 2, 3]))

"3"

Visi also supports Java/JavaScript style method invocation:

x = [1, 2, 3]
  x.size().toString()

"3"

And if you want to force a method invocation rather than a function call (if there is a function in scope with the same name as the method you want to call):

x = [1, 2, 3]
  x$.size()$.toString()

"3"

Why the different ways to do the same thing? Mostly because there will be times when it's cleaner/easier to read/update when you use one style or the other.

Mostly with the $package declaration. Note the $. This means that you're explicitly using some form of host interoperability.

Using $package, you declare the name of the package your code runs in (usually, visi.notebook) and you load any packages you might need off the Internet, for example, the Apache Commons CSV package.

$package(visi.notebook,
  load([org.apache.commons/commons-csv, "1.0"]),
  import(org.apache.commons.csv.CSVParser,
         org.apache.commons.csv.CSVFormat))

(CSVParser::parse(#'''1,2,"hello, moose",33,"thing"''', CSVFormat::DEFAULT))
  |> first ## get the first element
  |> seq ## turn it into a sequence

["1", "2", "hello, moose", "33", "thing"]