A scalable, fast, consistent graph store

Weaver is a distributed graph store that provides horizontal scalability, high-performance, and strong consistency.

Weaver enables users to execute transactional graph updates and queries through a simple python API.

For example, you can create a user (node) and a link (edge) of a specified type in a single transaction:

graph.begin_tx()
graph.create_node('ayush')
graph.create_edge('ayush', 'egs')
graph.set_edge_property('type', 'friend')
graph.end_tx()

You can query a user's friends-of-friends in another transaction:

two_hop_friends = graph.traverse('ayush'). \
                  out_edge([('type', 'friend')]).node(). \
                  out_edge([('type', 'friend')]).node().execute()

Weaver dynamically migrates portions of the graph across shards to maintain graph locality and minimize communication.

Weaver also enables users to cache results of graph computation at the nodes.

You can learn more about the API by reading the tutorial.

Install (Ubuntu 14.04 LTS)

Install HyperDex Warp.

Run the following commands as root for Ubuntu 14.04:

$ wget -O - http://weaver.systems/repos/apt/ubuntu/weaver.gpg.key | apt-key add -
$ wget -O /etc/apt/sources.list.d/weaver.list http://weaver.systems/repos/apt/ubuntu/trusty.list
$ apt-get update
$ apt-get install weaver

Install (from source)

Dependencies:

$ apt-get install autoconf autoconf-archive build-essential \
    pkg-config libtool python-dev cython libyaml-dev libpopt-dev \
    libgoogle-glog-dev libpugixml-dev sparsehash gcc-4.8 g++-4.8

Dependencies from HyperDex repository:

$ apt-get install libpo6-dev libe-dev libbusybee-dev \
    libhyperleveldb-dev replicant librsm-dev libreplicant-dev \
    libhyperdex-dev-warp libhyperdex-client-dev-warp hyperdex-warp

Get the code:

$ git clone https://github.com/dubey/weaver.git
$ cd weaver
$ autoreconf -i
$ ./configure
$ make && make install

Weaver on Docker

Weaver has a pre-built Docker image that makes it easier to deploy the system. To get the image:

$ docker pull weaver/weaver

Setup a Weaver cluster

Weaver needs running instances of Hyperdex, the Weaver timeline oracle, and the Weaver coordinator. To start all of this, get the code and execute the start_weaver.sh script. This script will ssh into the machines which will host the various processes, so you need a running ssh-server on those machines.

$ git clone https://github.com/dubey/weaver.git
$ cd weaver/
$ startup_scripts/start_weaver.sh

The script takes a single argument: path of the weaver.yaml config file, which defaults to /etc/weaver.yaml or /usr/local/etc/weaver.yaml. The config file specifies the IP addresses and ports to which the Hyperdex coordinator, the timeline oracle, and the Weaver coordinator will bind.

To start a Weaver shard:

$ weaver shard

To start a Weaver timestamper:

$ weaver timestamper

You must start as many timestampers as listed in the weaver.yaml config file, at which point you have a working cluster. You should additionally start backup timestampers and shards by appending the -b option to weaver shard and weaver timestamper. You need f backups for each shard and timestamper to tolerate f failures. For additional startup options such as specifying the IP address and port to bind to, see the output of weaver shard --help and weaver timestamper --help.

Test the cluster

A simple sanity check program which creates and traverses a small graph is given in tests/python/correctness/simple_test.py. Give it a try by running:

$ python tests/python/correctness/simple_test.py

Kill the cluster

Each of the timestampers and shards must be killed individually on the machines they are running. You can use the kill_weaver.sh script to kill Hyperdex, the timeline oracle, and the coordinator.

$ startup_scripts/kill_weaver.sh

Setup using Docker

The weaver/weaver Docker image includes a Weaver installation and scripts that setup a local Weaver instance. To run Weaver on Docker:

$ docker run --net="host" -itd weaver/weaver /start_weaver.sh
$ docker run --net="host" -itd weaver/weaver /weaver_shard.sh
$ docker run --net="host" -itd weaver/weaver /weaver_timestamper.sh

$ docker run --net="host" -it weaver/weaver python weaver/tests/python/correctness/simple_test.py

To kill the cluster simply kill all the Docker containers. In order to setup the cluster with another configuration, you will have to modify the /etc/config.yaml file by running an interactive Docker container with a shell program. You can learn more by reading the Docker documentation.

Tutorial

A typical Weaver program creates a Weaver client by providing the IP address and port of the Weaver coordinator.

from weaver import client
c = client.Client('127.0.0.1', 2002)

Weaver clients provide functions to create nodes and edges, and to assign them properties. All such updates take place in the context of a transaction. Multiple clients can issue concurrent transactions, which are guaranteed to execute in isolation.

from weaver import client
c = client.Client('127.0.0.1', 2002)

c.begin_tx()
c.create_node('ayush')
new_node = c.create_node() # weaver creates a unique node handle
new_edge = c.create_edge('ayush', new_node)
c.set_node_property(node='ayush', key='type', value='person')
c.set_node_property(node=new_node, key='type', value='content')
c.set_edge_property(node='ayush', edge=new_edge, key='rel', value='likes')
c.end_tx()

c.begin_tx()
c.delete_edge(edge=new_edge, node='ayush')
c.delete_node('ayush')
c.end_tx()

Apart from creating the graph, you can also bulk-load an existing graph while starting the Weaver shards.

weaver shard -l 127.0.0.1 -p 5201 \
             --graph-file=data.txt --graph-format=snap

Weaver clients also provide query methods that trigger the execution of snippets of code called node programs. Each query is fully-specified by the client and is executed entirely at the shards as a transaction. The binaries include a number of precompiled node programs such as get node neighbors, get node properties, get edge properties, count edges, BFS traversal, reachability request, and computation of clustering coefficient.

from weaver import client
c = client.Client('127.0.0.1', 2002)

# read color of nodes
prog_params = c.ReadNodePropsParams(keys=['color']) # prepare program parameters
response = c.read_node_props([('ayush',prog_params)])
print response.node_props

# syntactic sugar for traversals
links = c.traverse('egs').out_edge().execute() # list all out-links from node egs
friends = c.traverse('egs').out_edge().node().execute # list all of egs's friends
all = c.traverse('egs').out_edge().node().collect() # links + friends

There are a number of sample python client programs in the tests/python/* directory in the source.

If you are interested in implementing your own node program, take a look at the node_prog/ directory.

Users

FAQ

Join the community

Weaver Discussion List

Source Code on Github

Weaver Repo

The Weaver Team

Contact us at Turn on JavaScript to view email address