Skip to content
Merged
Show file tree
Hide file tree
Changes from 99 commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
251535c
Fresh Mahara based on https://github.com/catalyst/Azure_moodle from c…
darrinhodges May 15, 2018
a371134
Update to note that mssql is not supported by Mahara
darrinhodges May 15, 2018
a9e9f1f
Update for Mahara
darrinhodges May 15, 2018
0e4f21a
fix shell syntax errors
darrinhodges May 15, 2018
01a435e
Fix nginx html root for mahrar
darrinhodges May 15, 2018
5907c38
Fix for additional serveroot hard coding in mahara
darrinhodges May 15, 2018
099371b
Remove mssql option from other templates
darrinhodges May 15, 2018
c3b75d5
move comment down below bash init
darrinhodges May 15, 2018
5a4feb3
move comment down below bash init
darrinhodges May 15, 2018
f8336af
Remove tika and tika references
darrinhodges May 21, 2018
c610012
Update diagram and README.md
darrinhodges May 22, 2018
2e8c2da
Remove Redis as Azure supported Mahara versions do not support Redis
darrinhodges May 22, 2018
597ab35
updated for testing
darrinhodges May 22, 2018
7e7fd03
fix syntax error in network.json
darrinhodges May 22, 2018
715a7a1
Removed GDPR,office365 and azuresearch reference and configs
darrinhodges May 22, 2018
08cabcb
remove references to azuresearch
darrinhodges May 22, 2018
84b99bb
Fix syntax error in azuredploy.json
darrinhodges May 22, 2018
a72c078
Fix syntax error in azuredploy.json
darrinhodges May 22, 2018
cff5f55
Fix terminology for Mahara
darrinhodges May 22, 2018
a8e3a54
Remove all code,references and configs for mssql - Mahara does not su…
darrinhodges May 22, 2018
e02f9e2
enable selectable search functionality
darrinhodges May 22, 2018
2288ab6
add sql updates for elastic search options
darrinhodges May 22, 2018
9689daf
fix syntax error
darrinhodges May 22, 2018
0acb212
fix syntax error
darrinhodges May 22, 2018
6dbae23
fix script logic
darrinhodges May 22, 2018
cc9cc76
Update terminology for Mahara
darrinhodges May 22, 2018
82c4e88
remove references to objectfs as it requires backporting to versions …
darrinhodges May 22, 2018
20f2b8b
Fix link to RG deletion
darrinhodges May 22, 2018
df154a7
remove remaining redis references
darrinhodges May 22, 2018
be70446
update Azure content url
darrinhodges May 22, 2018
f795df4
Remove inappropriate files and documentation
darrinhodges May 24, 2018
486c604
Updates to various documents, removal of other files
darrinhodges May 25, 2018
94d75bd
Remove inappropriate documentation, update README.md to fit into quic…
darrinhodges May 25, 2018
f446bf1
update stack diagram
darrinhodges May 25, 2018
36aa24c
update azure deploy parameters template
darrinhodges May 25, 2018
3cebb15
update url content to a different value
darrinhodges May 25, 2018
00c9e06
reset default db
darrinhodges May 25, 2018
ebacf68
minor tidy
darrinhodges May 25, 2018
8753f37
fix syntax error in template
darrinhodges May 27, 2018
775feb6
add more options to gatewaysubnet var
darrinhodges May 27, 2018
ab3e27c
remove unused variable
darrinhodges May 27, 2018
a5ec5be
update metadata file
darrinhodges May 27, 2018
cb7c2cf
update api as validation command suggests
darrinhodges May 27, 2018
d467afb
Revert "update api as validation command suggests"
darrinhodges May 27, 2018
61f8686
re-edit metadata.json
darrinhodges May 27, 2018
40bd576
Updates to README.md
darrinhodges May 27, 2018
3da2ce3
Revise deploy estimate in README.md
darrinhodges May 27, 2018
5a957eb
remove env.json
darrinhodges May 28, 2018
7377a60
added autogen for dblogin
darrinhodges May 28, 2018
053d7d3
use gen-unique for db passwords
darrinhodges May 28, 2018
a990994
add variable
darrinhodges May 28, 2018
1e5b035
db passwords now autogen
darrinhodges May 28, 2018
a9c4a33
update metadata.json
darrinhodges May 28, 2018
6c0701f
remove extra comma
darrinhodges May 28, 2018
7605b14
Fix syntax
darrinhodges May 28, 2018
48ffa38
pick of another rogue comma
darrinhodges May 28, 2018
d6c4525
fix elasticsearch settings
darrinhodges May 28, 2018
4c6b07b
remove autogen passwords
darrinhodges May 28, 2018
b0a2a76
update search name value for mahara
darrinhodges May 28, 2018
df284d3
Add note on Mahara to readme.md
darrinhodges May 28, 2018
e1bba0a
reset mahara db password
darrinhodges May 28, 2018
21a67b8
reset azure db password
darrinhodges May 28, 2018
ee0df81
Add undocumented schema property to metadata.json
darrinhodges May 29, 2018
b1d2b26
Text update as per https://github.com/Azure/azure-quickstart-template…
darrinhodges Jul 16, 2018
83daa04
Text update as per https://github.com/Azure/azure-quickstart-template…
darrinhodges Jul 16, 2018
d76a6a8
Text update as per https://github.com/Azure/azure-quickstart-template…
darrinhodges Jul 16, 2018
115e943
Text update as per https://github.com/Azure/azure-quickstart-template…
darrinhodges Jul 16, 2018
6c09e74
Text update as per https://github.com/Azure/azure-quickstart-template…
darrinhodges Jul 16, 2018
789ee58
Text update as per https://github.com/Azure/azure-quickstart-template…
darrinhodges Jul 16, 2018
f18c5b0
Text update as per https://github.com/Azure/azure-quickstart-template…
darrinhodges Jul 16, 2018
8769d73
Update as per https://github.com/Azure/azure-quickstart-templates/pul…
darrinhodges Jul 16, 2018
8fb126a
update as per https://github.com/Azure/azure-quickstart-templates/pul…
darrinhodges Jul 16, 2018
466ea6d
update as per https://github.com/Azure/azure-quickstart-templates/pul…
darrinhodges Jul 16, 2018
84c6e03
Further updates to https://github.com/Azure/azure-quickstart-template…
darrinhodges Jul 16, 2018
579a073
Additional updates for https://github.com/Azure/azure-quickstart-temp…
darrinhodges Jul 17, 2018
66b6a02
Text update
darrinhodges Jul 17, 2018
48d2e62
Update api version for nested template deployment as per Travis CI me…
darrinhodges Jul 18, 2018
c8ac66a
Update generated usernames with fixed length as per Travis CI
darrinhodges Jul 18, 2018
64f1344
Remove default values as per https://github.com/Azure/azure-quickstar…
darrinhodges Jul 18, 2018
91bf572
Remove hard coded endpoint as per https://github.com/Azure/azure-quic…
darrinhodges Jul 18, 2018
1ed08bf
Altering to output lb FQDN
darrinhodges Jul 18, 2018
5f1f00a
Removing static end points
darrinhodges Jul 18, 2018
9cbdc6e
Remove static endpoint for lb fqdn.
darrinhodges Jul 18, 2018
586cd05
Fix JSON syntax error
darrinhodges Jul 18, 2018
705e8e1
replace missing variable
darrinhodges Jul 18, 2018
bbdb989
Fix issue with siteURL var
darrinhodges Jul 18, 2018
287346d
Output ssh username from parent template
darrinhodges Jul 18, 2018
f576f11
Revert "Fix issue with siteURL var"
darrinhodges Jul 18, 2018
ad3e737
Generate FQDN and pass through to scaleset and controller templates.
darrinhodges Jul 18, 2018
2654375
Add variable to deploy template
darrinhodges Jul 18, 2018
d74c9cc
Fix syntax error
darrinhodges Jul 18, 2018
96f0c6c
Fix parameters for nested templates
darrinhodges Jul 18, 2018
a5ed253
Insert missing quote character
darrinhodges Jul 18, 2018
f1ec59a
Move siteurl config into scripts as its not relevant to the Azure tem…
darrinhodges Jul 19, 2018
a163444
Fix cli parameters
darrinhodges Jul 19, 2018
bf92296
Fix cli parameters
darrinhodges Jul 19, 2018
96c616f
Fix cli parameters
darrinhodges Jul 20, 2018
ad5d47d
Resolve conflict
darrinhodges Jul 20, 2018
ab6ae1b
Text update as per https://github.com/Azure/azure-quickstart-template…
darrinhodges Jul 20, 2018
f20c4d0
remove deployment resource
bmoore-msft Aug 4, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 133 additions & 0 deletions mahara-autoscale-cache/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# *Deploy and manage a Scalable Mahara Cluster on Azure*

After deploying, these templates will provide you with a new Mahara site with caching for speed and scaling frontends to handle PHP load. The filesystem behind it is mirrored for high availability. Filesystem permissions and options have been tuned to make Mahara more secure than the default install.

[![Deploy to Azure Minimally](http://azuredeploy.net/deploybutton.png)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com/Azure/azure-quickstart-templates/master/mahara-autoscale-cache/azuredeploy.json) [![Visualize](https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/1-CONTRIBUTION-GUIDE/images/visualizebutton.png)](http://armviz.io/#/?load=https%3A%2F%2Fraw.githubusercontent.com/Azure/azure-quickstart-templates/master/mahara-autoscale-cache/azuredeploy.json)

`Tags: cluster, ha, mahara, autoscale, linux, ubuntu`


## *What this stack will give you*

This template set deploys the following infrastructure:
- Autoscaling web frontend layer (Nginx for https termination, Varnish for caching, Apache/php or nginx/php-fpm)
- Private virtual network for frontend instances
- Controller instance running cron and handling syslog for the autoscaled site
- Load balancer to balance across the autoscaled instances
- [Azure Database for MySQL](https://azure.microsoft.com/en-us/services/mysql/) or [Azure Database for PostgreSQL](https://azure.microsoft.com/en-us/services/postgresql/)
- Three Elasticsearch VMs for search indexing in Mahara (optional)*
- Dual Gluster nodes for high availability access to Mahara files

* Note: You will need to login into Mahara as 'admin' and configure the artifacts you want indexed after the installation has completed.

![network_diagram](images/stack_diagram.png "Diagram of deployed stack")

## *Deployment steps*

You can click the "deploy to Azure" button at the beginning of this document.

## *Using the created stack*

In testing, stacks typically took between 1 and 1 and a half hours to finish, depending on spec. Once this is done, you will receive JSON data with outputs needed to continue setup. You can also retrieve these from the portal or the CLI, more information below. The available parameters are:

- siteURL: If you provided a `siteURL` parameter when deploying this will be set to the supplied value. Otherwise it will be the same as the loadBalancerDNS, see below.
- loadBalancerDNS: This is the address of your load balancer. If you provided a `siteURL` parameter when deploying you'll need to add a DNS entry CNAMEs to this.
- maharaAdminPassword: The password for the "admin" user in your Mahara install.
- controllerinstanceIP: This is the address of the controller. You will need to SSH into this to make changes to your Mahara code or view logs.
- databaseDNS: This is the public DNS of your database instance. If you wish to set up local backups or access the db directly, you'll need to use this.
- databaseAdminUsername: The master account (not Mahara) username for your database.
- databaseAdminPassword: The master account password for your database.

Once Mahara has been created, and (if necessary) with your custom `siteURL` DNS pointing to the load balancer, you should be able to load the `siteURL` and login with "admin" and the password suppliedin the maharaAdminPassword.

#### Retrieving Deployment Configuration

The outputs provided by your deployment should include everything you need to manage your Mahara deployment. These are available in the portal by clicking on the deployment for your resource group. They are also available via the Azure CLI. For example:

Retrieve all the outputs in JSON format:

```
az group deployment show --resource-group $MAHARA_RG_NAME --name $MAHARA_DEPLOYMENT_NAME --out json --query *.outputs
```

Retrieve just the database password:

```
az group deployment show --resource-group $MAHARA_RG_NAME --name $MAHARA_DEPLOYMENT_NAME --out tsv --query *.outputs.databaseAdminPassword.value
```

Retrieve the public URL (if you did not provide your own URL):

```
az group deployment show --resource-group $MAHARA_RG_NAME --name $MAHARA_DEPLOYMENT_NAME --out tsv --query *.outputs.siteURL.value
```

### *Updating Mahara code/settings*

Your controller VM has Mahara code and data stored on /mahara. The code is stored in /mahara/html/mahara/. This is also mounted to your autoscaled frontends so all changes are instance. Depending on how large your Gluster disks are sized, it may be helpful to keep multiple older versions (/mahara/html1,/mahara/html2, etc) to roll back if needed.

### *Getting an SQL dump*

A daily SQL dump of your database is taken at 02:22 and saved to /mahara/db-backup.sql(.gz). If your database is small enough to fit, you may be able to get a more current SQL dump of your Mahara db by dumping it to /mahara/. Otherwise, you'll want to do this remotely by connecting to the hostname shown in the database-dns output using the database-admin-username and database-admin-password.

While Azure does not currently backup up Postgres/MySQL database, by dumping it to /mahara it is included in the Gluster VM backups should you enable Recovery Services in your parameters.

### *Azure Recovery Services*

If you have set azureBackupSwitch to 1 then Azure will provide VM backups of your Gluster node. This is recommended as it contains both your Mahara code and your sitedata. Restoring a backed up VM is outside the scope of this dos, but Azure's documentation on Recovery Services can be found here: https://docs.microsoft.com/en-us/azure/backup/backup-azure-vms-first-look-arm


### *Resizing your database*

Note: This involves a lengthy site downtime.

As mentioned above, Azure does not currently support resizing database. You can, however, create a new database instance and change your config to point to that. To get a different size database you'll need to:

1. Place your Mahara site into maintenance mode. You can do this either via the web interface or the command line on the controller VM.
2. Perform an SQL dump of your database, either to /mahara or remotely to your machine.
3. Create a new Azure database of the size you want inside your existing resource group.
4. Using the details in your /mahara/html/mahara/htdocs/config.php create a new user and database matching the details in config.php. Make sure to grant all rights on the db to the user.
5. On the controller instance, change the db setting in /mahara/html/mahara/htdocs/config.php to point to the new database.
6. Take Mahara out of maintenance mode.
7. Once confirmed working, delete the previous database instance.

How long this takes depends entirely on the size your database and the speed of your VM tier. It will always be a large enough window to make a noticeable outage.

### *Change the SSL cert*

The self-signed certificate generated by the template is suitable for very basic testing, but a public website will require a real certificate. After purchasing a trusted certificate, it can be copied to the following files to be ready immediately:

- /mahara/certs/nginx.key: Your certificate's private key
- /mahara/certs/nginx.crt: Your combined signed certificate and trust chain certificate(s).

Once replace these changes become effective immediately.

### *Sizing Considerations and Limitations*

Depending on what you're doing with Mahara, there are several considerations to make when configuring. The defaults included produce a cluster that is inexpensive but probably too low spec to use beyond single-user Mahara testing.

It should be noted that as of the time of writing both Postgres and MySQL databases are in preview at Azure. In the future larger DB sizes for different VM sizes will be available. The templates will allow you to select whatever sizeyou want, but there are restrictions in place (VMs with certain storage types, disk size for database tiers, etc) that may prevent certain selections from working together.

### *Database sizing*

As of the time of writing, Azure supports "Basic" and "Standard" tiers for database instances. In addition the skuCapacityDTU defines Compute Units, and the number of those you can use is limited by databas tier:

- Basic: 50, 100
- Standard: 100, 200, 400

This value also limits the maximum number of connections, as defined here: https://docs.microsoft.com/en-us/azure/mysql/concepts-limits

As the Mahara database will handle cron processes as well as the website, any public facing websites with than 10 users will likely require upgrading to 100. Once the site reaches 30+ users it will require upgrading to Standard for more compute units. This depends entirely on the individual site. As MySQL databases cannot change (or be restored to a different tier) once deployed it is a good idea to slightly overspec your database.

Standard instances have a minimum storage requirement of 128GB. All database storage, regardless of tier, has a hard upper limit of 1 terrabyte. After 128GB you gain additional iops for each GB, so if you're expecting a heavy amount of traffic you will want to oversize your storage. The current maximum iops with a 1TB disk is 3000.

### *Controller instance sizing*

The controller handles both syslog and cron duties. Depending on how big you Mahara cron runs are this may not be sufficent. If cron jobs are delayed and cron processess are building up on the controller then an upgrade in tier is needed.

### *Frontend instances*

In general the frontend instances will not be the source of any bottlenecs unless they are severly undersized versus the rest of the cluster. More powerful instances will be needed should fpm processess spawn and exhaust memory during periods of heavy site load. This can also be mitigated against by increasing the number of VMs but spawning new VMs is slower (and potentially more expensive) than having that capacity already available.

It is worth noting that the memory allowances on these instances allow for more memory than they may be able to provide with lower instance tiers. This is intentional as you can opt to run larger VMs with more memory and not require manual configuration. FPM also allows for a very large number of threads with prevents the systerm from failing during many small jobs.

Loading