diff --git a/README.txt b/README.md similarity index 52% rename from README.txt rename to README.md index 09f9fa2..86de155 100644 --- a/README.txt +++ b/README.md @@ -1,31 +1,42 @@ -Installation steps ------------------- +# Overview +Naiad is a distributed, parallel dataflow computation framework. It supports +both batch and streaming computations, and is described in detail in the +[SOSP 2013 paper](http://dl.acm.org/citation.cfm?id=2522738). +# Installation steps + +## Compatibility +Naiad works on Windows using the Microsoft .NET framework (v4.5+) and on +Linux using Mono 3.0+. Other configurations (e.g. the Linux version of the .NET +framework) *may* work, but are untested. + +## Components Naiad is distributed as a Visual Studio 2013 solution, comprising the following projects: -* "Naiad", which contains the core Naiad system and support libraries, - and builds Microsoft.Research.Naiad.dll. If you create your own project that +* *Naiad*, which contains the core Naiad system and support libraries, + and builds `Microsoft.Research.Naiad.dll`. If you create your own project that uses Naiad, you must add a reference to this library. -* "Lindi", a LINQ-like library for data-parallel iterative computation. +* *Lindi*, a LINQ-like library for data-parallel iterative computation. -* "DifferentialDataflow", a LINQ-like library supporting incremental +* *DifferentialDataflow*, a LINQ-like library supporting incremental execution of iterative computation. -* "AzureSupport", a library of Naiad components that supports writing to - and reading from Windows Azure Storage. +* *AzureSupport*, a library of Naiad components that supports writing to + and reading from Windows Azure Storage (Windows only). -* "Examples", which contains several example Naiad applications and - builds Examples.exe. +* *Examples*, which contains several example Naiad applications and + builds `Examples.exe`. -* In addition, the "ClusterSubmission" solution contains experimental support +* In addition, the *ClusterSubmission* solution contains experimental support for launching Naiad programs on Azure and YARN clusters. These projects can be built and executed using any of the following implementations of the .NET framework: * Microsoft .NET Framework 4.5 +* Mono v3.0+ To build Naiad, follow the steps below that correspond to your setup. @@ -50,43 +61,50 @@ To build Naiad, follow the steps below that correspond to your setup. 3. At the command prompt, run MSBuild.exe. -Getting started with the example programs ------------------------------------------ +* Build using Mono (on Linux, without Visual Studio) + + 1. Open a terminal and navigate to the root directory of the Naiad + repository. + + 2. Run `./build_mono.sh`. + + +# Getting started with the example programs As part of this source distribution, the Examples project demonstrates how to write simple applications using Naiad. After building the Naiad -solution, you will find the executable Examples.exe in the -Examples\bin\Debug directory. +solution, you will find the executable `Examples.exe` in the +`Examples\bin\Debug directory`. -Examples.exe uses command-line arguments to select an example, and it +`Examples.exe` uses command-line arguments to select an example, and it is recommended that you run it from a command prompt or terminal. For example, to run the word count example, follow these steps: 1. Open a command prompt or terminal, and change to the - Examples\bin\Debug directory. + `Examples\bin\Debug` directory. -2. At the command prompt, type "Examples.exe" to see the usage and +2. At the command prompt, type `Examples.exe` to see the usage and available examples. -3. At the command prompt, type "Examples.exe wordcount" to get started +3. At the command prompt, type `Examples.exe wordcount` to get started with the wordcount example. -Known issues with Naiad ------------------------ + +# Known issues This release of Naiad is an alpha release. The amount of testing the code -has been subjected to is limited, mostly to programs we have written. The -email address naiadquestions@microsoft.com goes to each of the project -members, and is a great place to send questions and comments. +has been subjected to is limited, mostly to programs we have written. If you +encounter any issues or unexpected behaviour, the Github issue tracker is a good +way to raise them with the project members. There are several specific issues we know about now, and want to point out to avoid headaches for everyone involved: 1. In distributed mode, we strongly recommend that each process call - OnNext the same numbers of times before calling OnCompleted. + `OnNext` the same numbers of times before calling `OnCompleted`. Several aspects of the graph manager logic consult the epoch numbers - associated with local inputs. Ideally, all process call OnNext the - same number of times; passing no parameters to OnNext causes the + associated with local inputs. Ideally, all process call `OnNext` the + same number of times; passing no parameters to `OnNext` causes the epoch to advance with no input supplied, and is a great way to do this. 2. Naiad currently requires custom datatypes to be structs, rather @@ -98,29 +116,31 @@ out to avoid headaches for everyone involved: auto-generated Serialization/Deserialization code requires all of your fields to be public and mutable (ie: not readonly). This is because the generated code attempts to set these values on a newly - minted struct, and can�t do this without these settings. More + minted struct, and can't do this without these settings. More details on writing custom serialization code in upcoming blog posts. As Naiad is a research prototype, there are many parts of it that are -likely to change. We�ve tried hard to put interfaces in place that will +likely to change. We've tried hard to put interfaces in place that will remain relatively stable, but there are certainly aspects of the -interfaces we expect will change. Information about programming examples, -the current state of the code, known problems, and upcoming fixes and -features should be available at the project web page: +interfaces we expect will change. - http://research.microsoft.com/Naiad/, +Some information about programming examples, the state of the code, known +problems, and fixes and features is available at the project web page: -or at the related blog: +http://research.microsoft.com/Naiad/, - http://bigdataatsvc.wordpress.com/ +or at the related blog: +http://bigdataatsvc.wordpress.com/ -Contacting us -------------- +Note, however, that the original authors no longer work at Microsoft Research, +and thus the website and the blog are not currently kept up to date. -If you have questions or comments, please feel welcome to email us at +# Contacting us - naiadquestions@microsoft.com +The former email address for questions (`naiadquestions@microsoft.com`) is no +longer actively monitored, as the original authors no longer work at Microsoft +Research. Raising issues on Github is the best way of getting in touch. We'd love to hear your thoughts on the project. Thanks!