diff --git a/docs/reference-docs/concepts-overview.md b/docs/reference-docs/concepts-overview.md new file mode 100644 index 00000000..48efc611 --- /dev/null +++ b/docs/reference-docs/concepts-overview.md @@ -0,0 +1,53 @@ +--- +sidebar_position: 1 +--- + +# Overview + +There are four primitives that are used with Conductor as you build your applications with it. + +## [Tasks](tasks-and-workers.md#Conductor-Tasks) +Tasks are the basic logical unit of work. It has defined inputs and outputs, does a specific activity and returns an execution status of success or failure. + +Conceptually, Conductor is oblivious to the actual work a task does, and similarly the task is oblivious to the state of the execution of the overall application which it expects Conductor to manage and ensure high levels of reliability. + +The actual execution of a Task can happen in multiple ways: +* **[External Tasks](tasks-and-workers.md#Conductor-Tasks):** This category is where the task is run external to Conductor as a micro service (e.g. in a container managed through Kubernetes in a public cloud) or it can run as a serverless function at a cloud provider. You can also have your task execute as part of a monolith that you have in your on-premise datacenter. As long as it can connect to Conductor server through REST of gRPC, it will work! +* **[System Tasks](system-tasks.md)** This category of tasks runs within the Conductor server. A good example is the [HTTP System Task](http-task.md) which makes an outbound call (internal or Internet routed) to a HTTP service and returns the results from that call. When you use System Tasks, you do not need to worry about provisioning, deploying and managing the application code for your tasks. +> Note: The current Conductor nomenclature includes [Operators](operators.md) in the Tasks category. In this documentation, for clarity purposes, we treat them as a separate primitive of Conductor +## [Operators](operators.md) +Operators allows you to wire up your workflows to capture the business logic in an intuitive and flexible way. Specific examples include [SWITCH](switch-task.md), [FORK](fork-task.md) and [JOIN](join-task.md) operators. Leveraging these operators gives you programmability that is also visual and easy to manage. Compare this with embedding your business logic deep in codebases that are distributed among many modules, languages and versions - and having to scramble for hours to understand how logic flows in your application, for example during a live-site issue! + +## [Workflows](workflows.md) +A workflow is the central entity that wires up different tasks and operataors to represent an application's business logic. It specifies whcih tasks to schedule when, how their results are combined and how failure scenarios are handled. + +The execution of a workflow that is defined in Conductor involves not only executing the tasks, but also doing so with high levels of reliability at scale. This is possible because Conductor keeps tracks of the execution status for each invocation of a workflow, even when millions of them are running. If a Task is not responding or is failing, Conductor will retry the execution at the specific point, potentially with a different Task Worker. + +Workflows in Conductor are also highly durable. You can have a workflow that stays waiting for as long as needed *(think weeks, months, years)* in a specific point in the execution for a long time *(e.g. monthly subscription billing that happens on the 1st of every month)*. + +Conductor's workflows are defined using JSON and results in a visual and intuitive graph representation of the application's components, relationships and flows. This abstraction of business logic is key to making developers & teams more empowered in multiple ways. +* Developers can move fast by easily adding new tasks or logic flows to a workflow and let Conductor ensure high levels of reliability, scalability and durability. +* A new team member can be easily ramped up since these workflow definitions inherently serve as easy to understand documentation of the applications a team owns +* During livesite issues, a DevOps engineer can quickly see which workflow execution is failing, poinpoint the exact task having issues and get actionable information about what happened and which specific worker instance the problem code was running. This reduces debugging time for critical issues from hours or days to minutes + + + +## [Workers](tasks-and-workers.md#Conductor-Workers) +Workers are the external entities that host and run the code for a Task that is defined in Conductor. + +Once the tasks' code is written to implement a task - you can choose the language you prefer for that, and for [Java](../how-tos/java-sdk.md), [Python](../how-tos/python-sdk.md), [Node.js](../how-tos/nodejs-sdk.md) and [Go](../how-tos/golang-sdk.md), we have SDKs that make it easy - it is deployed, and will poll the Conductor server to see if there are any Tasks queued for execution. If there are any, Conductor will allocate that for a Worker to execute and will wait for it to respond back with the result. In the event of a failure or a non-response from a Worker, Conductor will retry or gracefully fail in accordance with the behaviour the task creater has specified. + +There are multiple ways in which you can run a Worker +* **Micro Services:** The most common way a worker is imlpemented by Conductor users is by building a micro service to do that and deploying it as a container. This option allows you to have granualirity in your application development, scale your tasks independently, and contain the blast radius of any task level failures. +* **Serverless Functions:** Your Task code can be written as a stateless and serverless function that can be hosted with a serverless provider such as [AWS Lambda](https://aws.amazon.com/lambda/), [Azure Functions](https://azure.microsoft.com/en-us/services/functions/), [Google Cloud Functions](https://cloud.google.com/functions) etc. Choosing this option to run your Tasks allows you to not have to spend tiem and resources managing a server footprint +* **Monolith:** Many applications run as part of a monolith that encompasses other applications or even entirely different business lines. Conductor fully supports executing workers in these stacks. This gives developers the flexibility to have parts of their application running in micro services and others in a monolith. Customers have used this flexibility to plan and execute their micro services isolation strategy (sometimes along with a cloud migration strategy) and also for their ongoing hybrid operational strategy whcih different parts of their business logic lies in different parts of their backend ecosystem. + +## Further Reading +* [Tasks](tasks-and-workers.md#Conductor-Tasks) +* [Workers](tasks-and-workers.md#Conductor-Workers) +* [System Tasks](system-tasks.md) +* [Operators](operators.md) +* [Workflows](workflows.md) + + + diff --git a/docs/reference-docs/tasks-and-workers.md b/docs/reference-docs/tasks-and-workers.md index bf3d575b..615f486c 100644 --- a/docs/reference-docs/tasks-and-workers.md +++ b/docs/reference-docs/tasks-and-workers.md @@ -6,6 +6,12 @@ sidebar_position: 1 TODO -## Summary +## Conductor Tasks + +TODO + +## Conductor Workers +TODO +## Further Reading TODO diff --git a/docs/reference-docs/workflows.md b/docs/reference-docs/workflows.md index 7105b13b..b8ed6b91 100644 --- a/docs/reference-docs/workflows.md +++ b/docs/reference-docs/workflows.md @@ -4,8 +4,66 @@ sidebar_position: 1 # Workflows -TODO +## What are Workflows? +At a high level, a workflow is the Conductor primitive that encompasses the definition and flow of your business logic. It is through a workflow definition that you specify what are the [Tasks](tasks-and-workers.md) that you want Conductor to execute, the ordering of execution flows across these Tasks and how results from different Tasks should be combined together to give you the final result. This orchestration of Tasks can happen a hybrid ecosystem that includes micro services, serverless functions, monolithic applications that spans public cloud and on-premise datacenter footprints. + +One key benefit of this approach is that you can build a complex application using simple and granular tasks that do not need to be aware of or keep track of the state of your application's execution flow. Conductor will keep track of that, calls tasks in the right order (sequentially or parallelly, as defined by you), retry calls if needed, handle failure scenarios gracefully and outputs the final result. + +Leveraging workflows in Conductor enables developers to truly focus on their core mission - building their application code in the languages of their choice. Conductor meanwhile does the heavy lifting associated with ensuring high reliability, transactional consistency and long durability of their workflows. Simply put, wherever your application's component lives and whichever languages they were written in, you can build a workflow in Conductor to orchestrate their execution in a reliable & scalable manner. + +## How does a Workflow look like? +Lets start with a basic workflow and understand what are the different aspects of it. In particular, we will talk about two stages of a workflow, *defining* a workflow and *executing* a workflow +### *Simple Workflow Example* +Assume your business logic is to simply to get some shipping information and then do the shipping. You start by logically partitioning them into two tasks +* **shipping_info** +* **shipping_task** + +The next step is to [create a workflow in Conductor using JSON](../running-workflows/create-workflow.md) and then [create the associated tasks in Conductor](../running-workflows/create-task.md) for the above identified ones. + +After that, you [add those tasks into the workflow](../running-workflows/adding-tasks.md) to have **shipping_info** called fist and then, if it is successful, call **shipping_task**. You now have a *definition* of the workflow in Conductor and Conductor will then generate an easy to understand visual representation of this workflow + +![Simple Shipping Workflow - Visual Representation](../../static/img/tutorial/ShippingWorkflow.png) + +### *Multiple Paths Workflow Example* + +Next lets see a more complex example where you want to support multiple shipping vendors (e.g. FedEx, DHL, UPS) and the code for each of them ive in separate services. This is a good design pattern to follow since you can now independtly change each of them without having to worry about breaking others. Usually this means you now have to take on the work of wiring up many different services into your primary execution path. But with Conductor, this just means adding a [switch operator](../system-tasks/switch-task.md) to decide which vendor to call depending on an incoming parameter, and then during execution time the right one will be called! + +![Multi-vendor Shipping Workflow - Visual Representation of Design](../../static/img/tutorial/Switch_Workflow.png) + +Furthermore, with Conductor, in addition to the above design view of the workflow, you can also see the execution view of the workflow. In this particular example, the workflow picked UPS at runtime and as seen from the green color of the tasks in the execution path, this workflow completed successfully. If a particular task had failed, it would be show in red. + +![Multi-vendor Shipping Workflow - Visual Representation of Execution](../../static/img/tutorial/Switch_UPS.png) + + +> ### The Power of Seeing +> These visual representations of workflows are key to how Conductor turbocharges the productivity of engineering teams. +> * Workflows definitions serve as the enduring documentation for all the different applications a team owns and this benefit becomes even more powerful as the team scales in size and scope. Furthermore, it allows anyone new to the team to quickly get ramped up . +> * The execution visualization allows you to quickly identify problem areas and provides you details on the error responses received, details on where the failing task was executing etc. This makes debugging much faster than digging across distributed logs and events amking Conductor's approach to workflows relevant not only during the creation time but also during live operations of the workflows in production + +## How do I use Workflows? + +### *Starting Workflows* +Once a workflow is defined in Conductor, it is ready to be invoked. An invocation executes the workflow and passes in any arguments that were provided by the caller. There are three ways in which a workflow can be invoked. +* [Calling the Conductor API via REST or gRPC](../running-workflows/execute-workflow.md#Start-a-workflow-by-calling-an-API). An example of how to do this is also in the [running workflows](../running-workflows/running-first-workflow.md#Running-our-First-Workflow) article +* [Posting an event to a queue that Conductor is listening to for incoming workflow invocation requests](../running-workflows/execute-workflow.md#Start-a-workflow-by-posting-an-event) +* [Scheduling a time at which Conductor should invoke the workflow](../running-workflows/execute-workflow.md#Schedule-a-workflow-for-later) + +### *View Workflows* +Once a workflow is invoked, it starts running and you can [view details of its execution status](../how-tos/view-workflow-executions.md) + +### *Update Workflows* + +When your application's business logic evolves or you need to fix an error in your workflow definition, you can [udpate your workflows](../how-tos/updating-workflows.md) in Conductor with built-in support for versioning. + +> ### The Power of Versioning +> Conductor's native support for versioning allows developers to rapidly iterate on new features even with multiple invocations of the same workflow are in-flight. Unlike other platforms where you either need to wait till those in-flight executions finsh or forcefully error them out, with Conductor you can have both versions in-flight at the same time. In addition to increase in developer agility, this also unlocks other benefits +> * Experiment new features for a small subset of users +> * Safely test changes in production while containing any issue's blast raidus to a known value + +## Further Reading +* [Learn more about tasks and workers](../reference-docs/tasks-and-workers.md) +* [Learn more about system tasks](../reference-docs/system-tasks.md) +* [Learn more about operators](../reference-docs/operators.md) +* [Run your first workflow](../running-workflows/running-first-workflow.md) -## Summary -TODO diff --git a/docs/running-workflows/create-task.md b/docs/running-workflows/create-task.md new file mode 100644 index 00000000..de44be19 --- /dev/null +++ b/docs/running-workflows/create-task.md @@ -0,0 +1,7 @@ +# Create a Task + +TODO + +## Summary + +TODO diff --git a/docs/running-workflows/execute-workflow.md b/docs/running-workflows/execute-workflow.md index 418ba477..0bec3729 100644 --- a/docs/running-workflows/execute-workflow.md +++ b/docs/running-workflows/execute-workflow.md @@ -4,9 +4,13 @@ sidebar_position: 1 # Executing Workflows +A workflow can be exexcuted in three different ways as listed below. In all of these, once the invocation is made, Conductor will return a WorkflowID that you can use to [view the execution status of the invocation](docs/how-tos/view-workflow-executions.md) +## Start a workflow by calling an API TODO - -Let's learn how to run workflows that we have defined +## Start a workflow by posting an event +TODO +## Schedule a workflow for later +TODO ## Summary diff --git a/sidebars.js b/sidebars.js index 0c7bd8db..ff485049 100644 --- a/sidebars.js +++ b/sidebars.js @@ -104,6 +104,7 @@ module.exports = { items: [ 'running-workflows/create-workflow', 'running-workflows/execute-workflow', + 'running-workflows/create-task', 'running-workflows/adding-tasks', 'running-workflows/adding-system-tasks', 'running-workflows/running-task-workers', @@ -124,6 +125,7 @@ module.exports = { label:'Concepts', collapsed:false, items:[ + 'reference-docs/concepts-overview', 'reference-docs/workflows', 'reference-docs/tasks-and-workers', 'reference-docs/operators',