DOCSP-31922 Filtered Sync Improvements (#154)

kennethdyer · ajhuh-mdb · mdb-ashley · web-flow · commit 17f26dc514f0 · 2023-09-14T12:35:23.000-04:00
* DOCSP-31922 Regex and Exclude Filter Sync * Fixes build error * Removes fitler syntax from table * Adds filter syntax * Adds intro * Minor fix to trigger build * Adjusts intro text * Nests intro under versionchanged * Adjusts intro text * Adjusts intro text * Adjusts intro text * Adjusts intro text * Adjusts intro text * Expands Configure Filter step * Expands Use Filter step * Test alternate syntax highlighter * Fixes syntax highlighting options * Stubs Filter Regex * Stubs Filter Regex * Moves regex to new directory * Fixes build issue * Fixes build issue * Fixes build issue * Fixes build issue * Populates syntax documentation * Populates syntax documentation * Adds regex options * Adds regex options * Adds learn more link * Adds use case * Fixes build error * Fixes build error * Adds release notes * Fixes per Ali * Fixes per Ali Co-authored-by: Alison Huh <112565127+ajhuh-mdb@users.noreply.github.com> * Fixes per Ali * Fixes per Ali Co-authored-by: Alison Huh <112565127+ajhuh-mdb@users.noreply.github.com> * Fixes per Ali * Fixes per Ali * Fixes per Tim and Jessica * Fixes per Jessica * Fixes per Jessica * Fixes per Ashley Co-authored-by: Ashley Brown <98361885+mdb-ashley@users.noreply.github.com> --------- Co-authored-by: Alison Huh <112565127+ajhuh-mdb@users.noreply.github.com> Co-authored-by: Ashley Brown <98361885+mdb-ashley@users.noreply.github.com>
diff --git a/snooty.toml b/snooty.toml
@@ -19,7 +19,8 @@ toc_landing_pages = ["/quickstart",
                      "/using-mongosync",
                      "/multiple-mongosyncs",
                      "/release-notes/release-notes",
-                     "/faq"
+                     "/faq",
+                     "/reference/collection-level-filtering"
                     ]
 
 [constants]
diff --git a/source/includes/api/facts/filter-regex.rst b/source/includes/api/facts/filter-regex.rst
@@ -0,0 +1,6 @@
+
+Starting in 1.6, the :ref:`c2c-api-start` API now supports the use of
+Regular Expressions to configure filters for the ``includeNamespaces``
+and ``excludeNamespaces`` parameters used in :ref:`c2c-filtered-sync`.
+
+
diff --git a/source/includes/api/facts/namespace-explanation.rst b/source/includes/api/facts/namespace-explanation.rst
@@ -0,0 +1,14 @@
+
+If you configure a filter on a source cluster that has multiple
+databases, ``mongosync`` only syncs the databases specified in
+the filter definition. ``mongosync`` will not sync the other
+existing databases.
+       
+If you want to modify the filter to add a newly created database,
+you have to :ref:`restart the filtered sync <c2c-change-filter>`
+from the beginning.
+
+For more details, see :ref:`c2c-filtered-sync`.
+
+For current limitations, see :ref:`c2c-filtering-limitations`.
+
diff --git a/source/includes/api/facts/namespaces-syntax.rst b/source/includes/api/facts/namespaces-syntax.rst
diff --git a/source/includes/api/requests/start-filtered.sh b/source/includes/api/requests/start-filtered.sh
@@ -2,13 +2,15 @@ curl -X POST "http://localhost:27182/api/v1/start" --data '
    {
       "source": "cluster0",
       "destination": "cluster1",
-      "includeNamespaces" : [
+      "includeNamespaces": [
          {
-             "database" : "sales",
-             "collections": [ "EMEA", "APAC" ]
-         },
-         {
-             "database" : "marketing"
+             "database": "sales",
+             "collectionRegex": {
+                "pattern": "^accounts_.+$",
+                "options": "i"
+             }
+         }, {
+            "database": "marketing"
          }
       ]
-   } '
+   } '
diff --git a/source/includes/example-filter-collection-with-renaming.rst b/source/includes/example-filter-collection-with-renaming.rst
@@ -11,7 +11,7 @@ databases.
 The ``includeNamespaces`` array in this example defines a filter on two
 of the databases:
 
-.. code-block:: shell
+.. code-block:: json
 
    {
       "source": "cluster0",
@@ -52,7 +52,7 @@ Renaming a Collection
 
 You can rename any collection in the ``staff`` database. 
 
-.. code-block:: shell
+.. code-block:: javascript
 
    // This code works 
    use admin
@@ -62,7 +62,7 @@ You can only rename a collection within the ``students`` database if the
 new and old names are both in the filter. If either of the names is not
 in the filter, ``monogsync`` reports an error and exists.
 
-.. code-block:: shell
+.. code-block:: javascript
 
    // This code works 
    use admin
@@ -71,7 +71,7 @@ in the filter, ``monogsync`` reports an error and exists.
 If a collection is specified in the filter, you can drop it, but you
 cannot rename it to remove it from the filter.
 
-.. code-block:: shell
+.. code-block:: javascript
    :copyable: false 
 
    // This code produces an error and mongosync stops syncing 
@@ -83,22 +83,22 @@ collections to add them to the filter:
 
 - Source collection is specified in the filter
 
-  .. code-block:: shell
+  .. code-block:: javascript
 
      use admin
      db.runCommand( { renameCollection: "students.adjuncts", to: "staff.adjuncts" } )
 
 - Source collection is not specified in the filter
 
-  .. code-block:: shell
+  .. code-block:: javascript
 
      use admin
      db.runCommand( { renameCollection: "prospects.current", to: "staff.newHires" } )
 
 You can also rename collections in the source database when the whole
 target database is in the filter:
 
-.. code-block:: shell
+.. code-block:: javascript
 
    use admin
    db.runCommand( { renameCollection: "staff.employees", to: "staff.onPayroll" } )
diff --git a/source/includes/example-filter-collection.rst b/source/includes/example-filter-collection.rst
@@ -12,7 +12,7 @@ of the databases, ``sales`` and ``marketing``.
 The ``sales`` database also filters on the ``EMEA`` and ``APAC``
 collections. 
 
-.. code-block:: javascript
+.. code-block:: json
 
    "includeNamespaces" : [
          {
diff --git a/source/reference/api/start.txt b/source/reference/api/start.txt
@@ -10,7 +10,7 @@
 .. contents:: On this page
    :local:
    :backlinks: none
-   :depth: 2
+   :depth: 1
    :class: singlecol
 
 Description
@@ -137,25 +137,20 @@ Request Body Parameters
    * - ``includeNamespaces``
      - array
      - Optional
-     - Filters the databases or collections to sync. The filter syntax
-       is:
+     - Filters the databases or collections to include in sync. 
 
-       .. include:: /includes/api/facts/includeNamespaces-syntax.rst
+       .. include:: /includes/api/facts/namespace-explanation.rst
 
        .. versionadded:: 1.1
 
-       If you configure a filter on a source cluster that has multiple
-       databases, ``mongosync`` only syncs the databases specified in
-       the filter definition. ``mongosync`` will not sync the other
-       existing databases.
-       
-       If you want to modify the filter to add a newly created database,
-       you have to :ref:`restart the filtered sync <c2c-change-filter>`
-       from the beginning.
+   * - ``excludeNamespaces``
+     - array
+     - Optional
+     - Filters the databases or collections to exclude from sync.
 
-       For more details, see: :ref:`c2c-filtered-sync`.
+       .. include:: /includes/api/facts/namespace-explanation.rst
 
-       For current limitations, see: :ref:`c2c-filtering-limitations`.
+       .. versionadded:: 1.6
 
    * - ``reversible``
      - boolean
diff --git a/source/reference/collection-level-filtering.txt b/source/reference/collection-level-filtering.txt
@@ -17,6 +17,80 @@ Filtered Sync
 
 .. include:: /includes/api/facts/filtering-intro.rst
 
+Starting in 1.1, ``mongosync`` supports inclusion filters to specify which
+databases and collections to include in sync.  Starting in 1.6, ``mongosync``
+also supports exclusion filters and regular expressions.
+
+- With inclusion filters, ``mongosync`` syncs matching
+  databases and collections.
+- With exclusion filters, ``mongosync`` syncs all databases
+  and collections, except for those that match the filters.
+- With both inclusion and exclusion filters, ``mongosync`` only syncs
+  databases and collections that match the inclusion filters then excludes
+  any that also match the exclusion filters.
+- With no filters, ``mongosync`` syncs all databases and collections.
+
+.. _c2c-filter-syntax:
+
+Filter Syntax
+-------------
+
+The :ref:`c2c-api-start` API endpoint accepts two fields that configure 
+filtered sync: ``includeNamespaces`` and ``excludeNamespaces``.
+Each field takes an array of filters that specify the databases and collections 
+to include or exclude from sync.
+
+.. note::
+
+   If the :ref:`c2c-api-start` call uses both ``includeNamespaces`` and 
+   ``excludeNamespaces`` parameters, ``mongosync`` first matches databases
+   and collections from the inclusion filters, then excludes those that
+   also match an exclusion filter.
+
+Filters have the following syntax:
+
+.. code-block:: json
+   :copyable: false
+
+   "includeNamespaces": [
+       {
+          "database": "<database-name>",
+          "collections": [
+             "<collection-name>"
+          ]
+          "databaseRegex": {
+             "pattern": "<regex-pattern>",
+             "options": "<options>"
+          },
+          "collectionRegex": {
+             "pattern": "<regex-pattern>",
+             "options": "<options>"
+          }
+       }
+    ],
+    "excludeNamespaces": [
+       {
+          "database": "<database-name>",
+          "collections": [
+             "<collection-name>"
+          ]
+          "databaseRegex": {
+             "pattern": "<regex-pattern>",
+             "options": "<options>"
+          },
+          "collectionRegex": {
+             "pattern": "<regex-pattern>",
+             "options": "<options>"
+          }
+       }
+    ]
+
+Filters must include either the ``database`` field or the ``databaseRegex`` field.
+
+If you need the filter to match specific collections, you can use either 
+the ``collections`` array to specify collections individually or define 
+a regular expression using the ``collectionRegex`` field.
+
 .. _c2c-configure-filter:
 
 Configure a Filter
@@ -42,31 +116,54 @@ Configure a Filter
 
    .. step:: Create a Filter.
 
-      The ``includeNamespaces`` parameter specifies an optional filter
-      that you can pass to the :ref:`/start <c2c-api-start>` API.
+      The :ref:`c2c-api-start` API accepts two parameters that configure
+      optional filters: 
       
-      If you don't specify a filter, ``mongosync`` does a full cluster
+      - The ``includeNamespaces`` parameter takes an array of filters, which
+        are used to determines which databases and collections ``mongosync`` 
+        should include in the sync.
+      - The ``excludeNamespaces`` parameter takes an array of filters, which
+        are used to determine which databases and collections ``mongosync``
+        should exclude from the sync.
+     
+      If you don't specify a filter, ``mongosync`` performs a full cluster
       sync.
 
-      The filter syntax is: 
-
-      .. include:: /includes/api/facts/includeNamespaces-syntax.rst
-
-      Create an entry in the ``includeNamespaces`` array for each
-      database that you identified in step 1. Use the ``"database"``
-      field to specify the database name.
-      
-      If you want to filter on collections within a database, add those
-      collections to the list in the ``"collections"`` field for that
-      database entry.
+      Create inclusion and/or exclusion filters to identify the databases and
+      collections you want to sync.
+
+      For example, this inclusion filter would configure ``mongosync`` to only
+      sync collections whose names begin with ``accounts_`` from the ``sales`` 
+      database, except for the ``accounts_old`` collection:
+
+      .. code-block:: json
+
+         "includeNamespaces": [
+            {
+               "database": "sales",
+               "collectionRegex": {
+                  "pattern": "^accounts_.+$",
+                  "options": "ms"
+               }
+         ],
+         "excludeNamespaces": [
+             {
+                 "database": "sales",
+                 "collections": [
+                     "accounts_old"
+                 ]
+             }
+         ]
+
+      For more information on filters, see :ref:`c2c-filter-syntax`. 
 
    .. step:: Use the Filter.
 
       To use the filter, attach the filter json when you make the
       :ref:`/start <c2c-api-start>` API call to begin syncing.
 
       .. literalinclude:: /includes/api/requests/start-filtered.sh
-         :language: shell
+         :language: bash
 
 For an example configuration, see: :ref:`c2c-example-start-with-filter`.
 
@@ -253,3 +350,8 @@ Adding and Renaming Collections While Syncing
 
 .. include:: /includes/example-filter-collection-with-renaming.rst
 
+.. toctree::
+   :hidden:
+
+   /reference/collection-level-filtering/filter-regex
+
diff --git a/source/reference/collection-level-filtering/filter-regex.txt b/source/reference/collection-level-filtering/filter-regex.txt
diff --git a/source/release-notes/1.6.txt b/source/release-notes/1.6.txt

Original file line number	Diff line number	Diff line change
`@@ -19,7 +19,8 @@ toc_landing_pages = ["/quickstart",`
`19`	`19`	`"/using-mongosync",`
`20`	`20`	`"/multiple-mongosyncs",`
`21`	`21`	`"/release-notes/release-notes",`
`22`		`- "/faq"`
	`22`	`+ "/faq",`
	`23`	`+ "/reference/collection-level-filtering"`
`23`	`24`	`]`
`24`	`25`
`25`	`26`	`[constants]`
Original file line number	Diff line number	Diff line change
`@@ -2,13 +2,15 @@ curl -X POST "http://localhost:27182/api/v1/start" --data '`
`2`	`2`	`{`
`3`	`3`	`"source": "cluster0",`
`4`	`4`	`"destination": "cluster1",`
`5`		`- "includeNamespaces" : [`
	`5`	`+ "includeNamespaces": [`
`6`	`6`	`{`
`7`		`- "database" : "sales",`
`8`		`- "collections": [ "EMEA", "APAC" ]`
`9`		`- },`
`10`		`- {`
`11`		`- "database" : "marketing"`
	`7`	`+ "database": "sales",`
	`8`	`+ "collectionRegex": {`
	`9`	`+ "pattern": "^accounts_.+$",`
	`10`	`+ "options": "i"`
	`11`	`+ }`
	`12`	`+ }, {`
	`13`	`+ "database": "marketing"`
`12`	`14`	`}`
`13`	`15`	`]`
`14`		`- } '`
	`16`	`+ } '`
Original file line number	Diff line number	Diff line change
@@ -12,7 +12,7 @@ of the databases, ``sales`` and ``marketing``.
`12`	`12`	The ``sales`` database also filters on the ``EMEA`` and ``APAC``
`13`	`13`	`collections.`
`14`	`14`
`15`		`-.. code-block:: javascript`
	`15`	`+.. code-block:: json`
`16`	`16`
`17`	`17`	`"includeNamespaces" : [`
`18`	`18`	`{`