Track upstream changes

README.md

@@ -1,30 +1,18 @@
 # Amazon DocumentDB Tools
 
-This repo contains the following tools.
-
-## Amazon DocumentDB Index Tool 
-
-The `DocumentDB Index Tool` makes it easier to migrate only indexes (not data) between a source MongoDB deployment and an Amazon DocumentDB cluster. The Index Tool can also help you find potential compatibility issues between your source databases and Amazon DocumentDB. You can use the Index Tool to dump indexes and database metadata, or you can use the tool against an existing dump created with the mongodump tool.
-
-For more information about this tool, checkout the [Amazon DocumentDB Index Tool README](./index-tool/README.md) file.
+This repository contains several tools to help users with Amazon DocumentDB including migration, monitoring, and performance. A few of the most popular tools are listed below but there are additional tools in the [migration](./migration), [monitoring](./monitoring), [operations](./operations), and [performance](./performance) folders.
 
 ## Amazon DocumentDB Compatibility Tool 
 
-The `DocumentDB Compatibility Tool` examines log files from MongoDB or source code from MongoDB applications to determine if there are any queries which use operators that are not supported in Amazon DocumentDB. This tool produces a simple report of unsupported operators and file names with line numbers for further investigation.
+The [DocumentDB Compatibility Tool](./compat-tool) examines log files from MongoDB or source code from MongoDB applications to determine if there are any queries which use operators that are not supported in Amazon DocumentDB.
 
-For more information about this tool, checkout the [Amazon DocumentDB Compatibility Tool README](./compat-tool/README.md) file.
-
-## Cosmos DB Migration Utility
-
-The `Cosmos DB Migration Utility` is an application created to help live migrate the Azure Cosmos DB for MongoDB API databases to Amazon DocumentDB with very little downtime. It keeps the target Amazon DocumentDB cluster in sync with the source Microsoft Azure Cosmos DB until the client applications are cut over to the DocumentDB cluster. 
-
-For more information about the Cosmos DB Migrator tool, checkout the [Cosmos DB Migration Utility README](./cosmos-db-migration-utility/README.md) file.
+## Amazon DocumentDB Index Tool 
 
-## Amazon DocumentDB Global Clusters Automation Tool
+The [DocumentDB Index Tool](./index-tool) makes it easy to migrate only indexes (not data) between a source MongoDB deployment and an Amazon DocumentDB cluster.
 
-The `global-clusters-automation` is a tool created to automate the global cluster failover process for Disaster Recovery (DR) and Business Continuity Planning (BCP) use cases. It uses AWS lambda functions to trigger failover process and convert a standalone regional cluster to a global cluster.Amazon Route53 private hosted zone is used to manage cluster endpoints changes for applications. 
+## Support
 
-For more information about the Global Clusters Automation Tool, checkout the [Global Clusters Automation Tool](./global-clusters-automation/README.md) file.
+The contents of this repository are maintained by Amazon DocumentDB Specialist SAs and are not officially supported by AWS. Please file a [Github Issue](https://github.com/awslabs/amazon-documentdb-tools/issues) if you experience any problems.
 
 ## License
 

compat-tool/README.md

@@ -1,8 +1,58 @@
 # Amazon DocumentDB Compatibility Tool
-The tool examines MongoDB log files or source code from MongoDB applications to determine if there are any queries which use operators that are not supported in Amazon DocumentDB. This tool produces a simple report of unsupported operators and file names with line numbers for further investigation.
+The tool examines **MongoDB serverStatus() counters**, **MongoDB log files**, or **application source code** from MongoDB applications to determine if there are any operators in use that are not supported in Amazon DocumentDB. It produces a simple report of both supported and unsupported operator usage.
+
+## Recommended compatibility testing for MongoDB 5.0 and newer
+* run the tool once using directConnection=true to the primary instance in your cluster
+  * ```python3 compat.py --uri "mongodb://<username>:<password>@<primary-hostname>:<port>/admin?directConnection=true"```
+* if using secondary instances for read-scale run the tool again on one secondary (again using &directConnection)
+  * ```python3 compat.py --uri "mongodb://<username>:<password>@<secondary-hostname>:<port>/admin?directConnection=true"```
+
+## Prerequisites for MongoDB Log File Analysis
+
+### Enable query logging in MongoDB
+#### For local, on-premise, or self-managed installations:
+By default, MongoDB logs the slow queries, over the 100ms threshold, to the configured log file.
+
+1. Check current profiling status and note the `slowms` value:
+```
+> db.getProfilingStatus()
+{
+  "was": 0,
+  "slowms": 100,
+  "sampleRate": 1
+}
+```
+
+2. Enable logging of all queries by setting `slowms` to `-1`:
+```
+> db.setProfilingLevel(0, -1)
+```
+3. **After completing the compatibility analysis**, reset to the original profiling level (using the `slowms` value from step 1):
+```
+> db.setProfilingLevel(0, 100)
+```
+
+#### For MongoDB Atlas:
+MongoDB Atlas dynamically adjusts slow query threshold based on the execution time of operations across the cluster. The Atlas-managed slow operation threshold is enabled by default and must be disabled using the Atlas CLI, Atlas Administration API, or Atlas UI.
+
+1. Disable the Atlas-Managed Slow Operation Threshold by going to Project Settings for the cluster project and toggling Managed Slow Operations to Off. Please refer to the documentation here: https://www.mongodb.com/docs/atlas/performance-advisor/
+2. Connect to your cluster using MongoDB shell or compass and enable full logging:
+   ```
+   db.setProfilingLevel(0, -1)
+   ```
+3. [Download the logs](https://www.mongodb.com/docs/atlas/mongodb-logs/) from the Atlas console
+4. **After completing the compatibility analysis**, enable the Atlas-Managed Slow Operation Threshold
+
+If Atlas-managed slow operation threshold is not enabled, follow the same steps as local or on-premise installations above.
+
+#### NOTE:
+Query profiling can cause additional overhead, it is recommended to use a dev/test environment to capture the queries.
+See the MongoDB [documentation](https://www.mongodb.com/docs/manual/reference/method/db.setProfilingLevel/) for additional information.
 
 ## Requirements
-Python 3.6 or later
+- Python 3.6 or later
+- pymongo (if testing compatibility using the --uri option)
+
 
 ## Installation
 Clone the repository and go to the tool folder:
@@ -12,17 +62,40 @@ cd amazon-documentdb-tools/compat-tool/
 ```
 
 ## Usage/Examples
-This tool supports examining compatibility with either the 3.6, 4.0 or 5.0 versions of Amazon DocumentDB. The script has the following arguments:
+This tool supports examining compatibility with either the 3.6, 4.0, 5.0, or Elastic Clusters 5.0 versions of Amazon DocumentDB. The script has the following arguments:
 ```
---version {3.6,4.0,5.0} -> Check for DocumentDB version compatibility (default is 5.0)
---directory SCANDIR     -> Directory containing files to scan for compatibility
---file SCANFILE         -> Specific file to scan for compatibility
---excluded-extensions EXCLUDEDEXTENSIONS -> Filename extensions to exclude from scanning, comma separated
---included-extensions INCLUDEDEXTENSIONS -> Filename extensions to include in scanning, comma separated
---show-supported        -> Include supported operators in the report
+--uri MONGO_URI                             -> Use db.serverStatus() output to scan for compatibility (requires MongoDB 5.0+)
+--version {3.6,4.0,5.0,EC5.0}               -> Check for DocumentDB version compatibility (default is 5.0)
+--directory SCANDIR                         -> Directory containing files to scan for compatibility
+--file SCANFILE                             -> Specific file to scan for compatibility
+--excluded-extensions EXCLUDEDEXTENSIONS    -> Filename extensions to exclude from scanning, comma separated
+--excluded-directories EXCLUDEDDIRECTORIES  -> Fully qualified path to directory to exclude, comma separated
+--included-extensions INCLUDEDEXTENSIONS    -> Filename extensions to include in scanning, comma separated
 ```
 
 #### Example 1:
+Check for compatibility using a MongoDB 5.0+ instance:
+```
+python compat.py --uri "mongodb://<username>:<password>@<hostname>:<port>/admin?directConnection=true" 
+
+connecting to the server at <hostname>:27017
+database server major version is 5
+checking compatibility using db.serverStatus()
+
+The following 2 unsupported operators were found:
+  $bucket | executed 1 time(s)
+  $facet | executed 1 time(s)
+
+The following 6 supported operators were found:
+  $eq | executed 12 time(s)
+  $group | executed 3 time(s)
+  $gt | executed 6 time(s)
+  $match | executed 1 time(s)
+  $switch | executed 1 time(s)
+  $unwind | executed 1 time(s)
+```
+
+#### Example 2:
 Check for compatibility with Amazon DocumentDB version 5.0, files from the folder called test, excluding the ones with extension `.txt`:
 ```
 python3 compat.py --version 5.0 --directory test --excluded-extensions txt
@@ -61,11 +134,11 @@ List of skipped files - excluded extensions
   test/testlog2.txt
 ```
 
-#### Example 2:
+#### Example 3:
 Check a specific file and show the supported operators found:
 
 ```
-python3 compat.py --file test/testlog.txt --show-supported
+python3 compat.py --file test/testlog.txt
 
 processing file test/testlog.txt
 Processed 1 files, skipped 0 files
@@ -95,39 +168,93 @@ The following 9 supported operators were found
   - $sum | found 1 time(s)
 ```
 
-#### NOTES:
-* All files scanned by this utility are opened read-only and scanned in memory. For large files, make sure you have enough available RAM or split the files accordingly.
-* With the exception of operators used, there is no logging of the file contents.
-* Using the `--directory` argument will scan all the files, including subdirectories which will be scanned resursively.
-
-### Enable query logging in MongoDB
-#### For local or on-premise installations:
-By default, MongoDB logs the slow queries, over the 100ms threshold, to the configured log file.
-To view the current profiling status, use the `getProfilingStatus()` in MongoDB shell:
+### Example 4:
+Check for compatibility with Amazon DocumentDB, files from the folder called test, excluding the ones with extension `.txt` and excluding directories `exclude1` and `exclude2`:
 
 ```
-> db.getProfilingStatus()
-{
-  "was": 0,
-  "slowms": 100,
-  "sampleRate": 1
-}
-```
+python3 compat.py --version 5.0 --directory test --excluded-extensions txt --excluded-directories /path/to/directory/exclude1,/path/do/directory/exclude2
 
-To enable logging of all queries, set the `slowms` parameter to `-1`:
+processing file test/mongod.log.2020-11-10T19-33-14
+processing file test/mongodb.log
+processing file test/sample-5-0-features.py
+processing file test/sample-python-1.py
+processing file test/sample-python-2.py
 
-```
-> db.setProfilingLevel(0, -1)
-```
+Processed 5 files, skipped 3 files
 
-To set the slow logging threshold to the prvious level:
-```
-> db.setProfilingLevel(0, 100)
+The following 5 unsupported operators were found:
+  $facet | found 2 time(s)
+  $sortByCount | found 2 time(s)
+  $bucket | found 1 time(s)
+  $bucketAuto | found 1 time(s)
+  $expr | found 1 time(s)
+
+Unsupported operators by filename and line number:
+  $facet | lines = found 2 time(s)
+    test/mongodb.log | lines = [80, 82]
+  $sortByCount | lines = found 2 time(s)
+    test/mongod.log.2020-11-10T19-33-14 | lines = [83]
+    test/sample-python-2.py | lines = [29]
+  $bucket | lines = found 1 time(s)
+    test/mongodb.log | lines = [80]
+  $bucketAuto | lines = found 1 time(s)
+    test/mongodb.log | lines = [82]
+  $expr | lines = found 1 time(s)
+    test/mongod.log.2020-11-10T19-33-14 | lines = [107]
+
+The following 34 supported operators were found:
+  - $match | found 34 time(s)
+  - $gt | found 15 time(s)
+  - $project | found 15 time(s)
+  - $lte | found 14 time(s)
+  - $group | found 13 time(s)
+  - $gte | found 13 time(s)
+  - $sum | found 11 time(s)
+  - $in | found 10 time(s)
+  - $count | found 7 time(s)
+  - $ne | found 7 time(s)
+  - $lookup | found 6 time(s)
+  - $unwind | found 4 time(s)
+  - $eq | found 3 time(s)
+  - $sort | found 3 time(s)
+  - $nin | found 2 time(s)
+  - $nor | found 2 time(s)
+  - $set | found 2 time(s)
+  - $skip | found 2 time(s)
+  - $addToSet | found 1 time(s)
+  - $and | found 1 time(s)
+  - $arrayElemAt | found 1 time(s)
+  - $avg | found 1 time(s)
+  - $dateAdd | found 1 time(s)
+  - $dateSubtract | found 1 time(s)
+  - $elemMatch | found 1 time(s)
+  - $first | found 1 time(s)
+  - $inc | found 1 time(s)
+  - $last | found 1 time(s)
+  - $limit | found 1 time(s)
+  - $lt | found 1 time(s)
+  - $max | found 1 time(s)
+  - $min | found 1 time(s)
+  - $not | found 1 time(s)
+  - $or | found 1 time(s)
+
+List of skipped files - excluded extensions
+  test/not_a_log_file.txt
+  test/testlog.txt
+  test/testlog2.txt
+
+List of skipped directories - excluded directories
+  test/exclude1
+  test/exclude2
+
 ```
+#### NOTES:
+* All files scanned by this utility are opened read-only and scanned in memory. For large files, make sure you have enough available RAM or split the files accordingly.
+* With the exception of operators used, there is no logging of the file contents.
+* Using the `--directory` argument will scan all the files, including subdirectories which will be scanned resursively.
 
-#### For MongoDB Atlas:
-Check the MongoDB Atlas documentation for how to [enable profiling](https://www.mongodb.com/docs/atlas/tutorial/profile-database/#access-the-query-profiler) and [download the logs](https://www.mongodb.com/docs/atlas/mongodb-logs/).
+## Contributing
+Contributions are always welcome! See the [contributing page](https://github.com/awslabs/amazon-documentdb-tools/blob/master/CONTRIBUTING.md) for ways to get involved.
 
-#### NOTE:
-Query profiling can cause additional overhead, it is recommended to use a dev/test environment to capture the queries.
-See the MongoDB [documentation](https://www.mongodb.com/docs/manual/reference/method/db.setProfilingLevel/) for additional information.
+## License
+Apache 2.0

compat-tool/add-new-version.py

@@ -0,0 +1,16 @@
+#!/usr/bin/python3
+
+import compat
+
+def main():
+    existingVersion = "5.0"
+    newVersion = "8.0"
+
+    keywords = compat.load_keywords()
+
+    for thisKeyword in keywords.keys():
+        keywords[thisKeyword][newVersion] = keywords[thisKeyword][existingVersion]
+        print("        \"{}\":{},".format(thisKeyword,keywords[thisKeyword]))
+
+if __name__ == '__main__':
+    main()

compat-tool/check-percentages.py

@@ -0,0 +1,57 @@
+#!/usr/bin/python3
+
+import compat
+
+def main():
+    versions = ['3.6','4.0','5.0','8.0','EC5.0']
+    keywords = compat.load_keywords()
+
+    totOps = 0
+    numOps = {}
+    numOpsSupported = {}
+
+    for thisKeyword in keywords.keys():
+        # get counts by mongodb version
+        totOps += 1
+        thisMongodbVersion = keywords[thisKeyword]["mongodbversion"]
+        if thisMongodbVersion in numOps:
+            numOps[thisMongodbVersion] += 1
+        else:
+            numOps[thisMongodbVersion] = 1
+
+        # get supported count by documentdb version
+        for docDbVersion in versions:
+            if keywords[thisKeyword][docDbVersion] == "Yes":
+                if docDbVersion in numOpsSupported:
+                    numOpsSupported[docDbVersion] += 1
+                else:
+                    numOpsSupported[docDbVersion] = 1
+
+    print("")
+    print("MongoDB Operations By Version, total = {}".format(totOps))
+    for thisVersion in sorted(numOps.keys()):
+        print("  {} in version {}".format(numOps[thisVersion],thisVersion))
+
+    print("")
+    print("DocumentDB Supported Operations By Version")
+    for thisVersion in sorted(numOpsSupported.keys()):
+        print("  {} supported by DocumentDB version {} ({:.1f}%)".format(numOpsSupported[thisVersion],thisVersion,numOpsSupported[thisVersion]/totOps*100))
+    print("")
+
+    print("")
+    print("DocumentDB EC Compat Check")
+    for thisKeyword in sorted(keywords.keys()):
+        if keywords[thisKeyword]["5.0"] == "Yes" and keywords[thisKeyword]["EC5.0"] == "No":
+            print("  {}".format(thisKeyword))
+    print("")
+
+    #print("")
+    #print("DocumentDB 5.0 Check")
+    #for thisKeyword in sorted(keywords.keys()):
+    #    if keywords[thisKeyword]["5.0"] == "No":
+    #        print("  {} from MongoDB {}".format(thisKeyword,keywords[thisKeyword]["mongodbversion"]))
+    #print("")
+
+
+if __name__ == '__main__':
+    main()