Skip to content

Commit b976077

Browse files
artongeartonge
committed
Add high level filesystem documention
From nextcloud/server#26982 Adds some documentation for the filesystem layer, both a high level overview of how the various pieces interact and a high level guide for apps interacting with the filesystem. Hopefully this will be useful to anyone trying to either use the filesystem or work on the filesystem. Signed-off-by: Louis Chemineau <[email protected]>
1 parent 15c3ff2 commit b976077

2 files changed

Lines changed: 230 additions & 0 deletions

File tree

‎developer_manual/core/architecture/files.rst‎

Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
============
2+
Nextcloud filesystem API
3+
============
4+
5+
High level overview
6+
-------------------
7+
8+
The Nextcloud filesystem is roughly based on the unix filesystem, consisting of multiple storages
9+
mounted at various locations.
10+
11+
.. code-block:: txt
12+
13+
┌──────────────────────────────────┐
14+
│Code wanting to use the filesystem│
15+
└─────────┬─────────────────────┬──┘
16+
│ │
17+
│ │
18+
┌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐
19+
╎Filesystem │ │ ╎
20+
╎layer │new │legacy ╎
21+
╎ │ │ ╎
22+
╎ ▼ ▼ ╎
23+
╎ ┌────────┐ Partly build on ┌─┴──────┐ ╎
24+
╎ │Node API├─────────────────►│View API│ ╎
25+
╎ └───────┬┘ └─┬──────┘ ╎
26+
╎ │ │ ╎
27+
└╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘
28+
│ │
29+
┌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐
30+
╎Storage layer │ │ ╎
31+
╎ ├─────────────────────┤ ╎
32+
╎ │ │ ╎
33+
╎ ▼ ▼ ╎
34+
╎ ┌───────┐ ┌───────┐ ┌──────┐ ╎
35+
╎ │Storage│═══>│Scanner│═══>│Cache │ ╎
36+
╎ └───────┘ └───────┘ └──────┘ ╎
37+
╎ ╎
38+
╎ ╎
39+
└╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘
40+
41+
Filesystem layer
42+
^^^^^^^^^^^^^^^^
43+
44+
Any code that wants to use the filesystem has two API options to use, the new ``Node`` api and the old ``View`` api.
45+
New code should preferably use the ``Node`` api as it allows building systems with less overhead than the old api.
46+
47+
Besides the filesystem apis, this layer also manages the available mounts, containing the logic to allow apps
48+
to setup their mounts and translating filesystem paths into a mountpoint + "internal" path.
49+
50+
### Storage layer
51+
52+
The storage implementation handles the details of communicating with the filesystem or remote storage api
53+
and provide a uniform api for Nextcloud to use the storage.
54+
55+
For each storage a metadata cache/index is maintained to allow reading metadata of the storage without having
56+
to talk to the (potentially) slow storage backend. The scanner is responsible for updating the cache with
57+
information from the storage backend.
58+
59+
## Storage/Cache wrappers
60+
61+
To allow apps to customize the behavior of a storage without requiring the app to implement this for every
62+
possible storage backend, a ``Wrapper`` system is used.
63+
64+
A ``Wrapper`` encapsulates an inner storage and allows overwriting any method to customize its behavior, with
65+
all other methods being passed through to the inner storage.
66+
67+
Generally search storage wrapper has an equivalent cache wrapper encapsulating the cache of the inner storage
68+
to provide the same behavior modifications when reading metadata from the cache.
69+
70+
Wrappers can be layered to stack the behavior of the wrappers, for example the ``groupfolders`` app works by
71+
stacking a wrapper to provide access to a single folder on the root storage with a wrapper to limit the permissions
72+
of the storage.
73+
74+
.. code-block:: txt
75+
76+
┌───────────────┐ ┌────────────────────┐
77+
│PermissionsMask├─────►│CachePermissionsMask│ PermissionsMask applies a mask to the permissions of a storage
78+
└───────┬───────┘ └─────────┬──────────┘ to provide less-privilaged access to a storage
79+
│ │
80+
▼ ▼
81+
┌───────────────┐ ┌────────────────────┐
82+
│Jail ├─────►│CacheJail │ Jail restricts access to a file or folder of a storage providing
83+
└───────┬───────┘ └─────────┬──────────┘ a limited view into the storage (think unix chroot or bind mount)
84+
│ │
85+
▼ ▼
86+
┌───────────────┐ ┌────────────────────┐
87+
│Base Storage ├─────►│Base Cache │
88+
└───────────────┘ └────────────────────┘
89+
Code Map
90+
--------
91+
92+
Approximate overview of the significant filesystem code
93+
94+
AppData
95+
~~~~~~~
96+
97+
High level api for accessing "appdata" folders, based on the ``Node`` API
98+
99+
Cache
100+
~~~~~
101+
102+
- ``Cache`` implementation
103+
- Cache wrappers
104+
- Scanner and cache update logic
105+
- Search infrastructure
106+
107+
Mount
108+
~~~~~
109+
110+
Mountpoint management and setup
111+
112+
Node
113+
~~~~
114+
115+
``Node`` filesystem api implementation
116+
117+
ObjectStorage
118+
~~~~~~~~~~~~~
119+
120+
Implementation of the various supported object store storage backends
121+
122+
SimpleFS
123+
~~~~~~~~
124+
125+
Simplified version of the Node api, for providing a more limited api for some filesystem bits
126+
127+
Storage
128+
~~~~~~~
129+
130+
Implementation of various storage backends and wrappers
131+
132+
Streams
133+
~~~~~~~
134+
135+
Various low-level php stream wrapper used in storage implementations
136+
137+
Type
138+
~~~~
139+
140+
Mimetype management and detection
141+
142+
View.php
143+
~~~~~~~~
144+
145+
Legacy View api

‎developer_manual/digging_deeper/files.rst‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
============
2+
Nextcloud filesystem API
3+
============
4+
5+
High level guide to using the Nextcloud filesystem API
6+
7+
Node API
8+
--------
9+
10+
The "Node API" is the primary api for apps to access the Nextcloud filesystem, each item in the filesystem is
11+
represented as either a File or Folder node with each node providing access to the relevant filesystem information
12+
and actions for the node.
13+
14+
Getting access
15+
^^^^^^^^^^^^^^
16+
17+
Access to the filesystem is provided by the ``IRootFolder`` which can be injected into your class.
18+
From the root folder you can either access a user's home folder or access a file or folder by its absolute path.
19+
20+
.. code-block:: php
21+
22+
use OCP\Files\IRootFolder;
23+
use OCP\IUserSession;
24+
class FileCreator {
25+
/** @var IUserSession */
26+
private $userSession;
27+
/** @var IRootFolder */
28+
private $rootFolder;
29+
30+
public function __constructor(IUserSession $userSession, IRootFolder $rootFolder) {
31+
$this->userSession = $userSession;
32+
$this->rootFolder = $rootFolder;
33+
}
34+
35+
/**
36+
* Create a new file with specified content in the home folder of the current user
37+
* returning the size of the resulting file.
38+
*/
39+
public function createUserFile(string $path, string $content): int {
40+
$user = $this->userSession->getUser();
41+
if ($user !== null) {
42+
// the "user folder" corresponds to the root of the user visible files
43+
$userFolder = $this->rootFolder->getUserFolder($user->getUID());
44+
// paths passed to a folder method are relative to that folder
45+
$file = $userFolder->newFile($path, $content);
46+
return $file->getSize();
47+
} else {
48+
return 0;
49+
}
50+
}
51+
}
52+
53+
For details on the specific methods provided by file and folder nodes see the method documentation from the ``OCP\Files\File`` and ``OCP\Files\Folder`` interfaces.
54+
55+
Direct storage access
56+
---------------------
57+
58+
While it should be generally avoided in favor of the higher level apis,
59+
sometimes an app needs to talk directly to the storage implementation of it's metadata cache.
60+
61+
You can get access to the underlying storage of a file or folder by calling ``getStorage`` on the node or first getting
62+
the mountpoint by calling ``getMountPoint`` and getting the storage from there.
63+
64+
Once you have the storage instance you can use the storage api from ``OCP\Files\Storage\IStorage``, note however that
65+
all paths used in the storage api are internal to the storage, the ``IMountPoint`` returned from ``getMountPoint`` provides
66+
methods for translating between absolute filesystem paths and internal storage paths.
67+
68+
If you need to query the cached metadata directory you can get the ``OCP\Files\Cache\ICache`` from the storage by calling ``getCache`.
69+
70+
Implementing a storage
71+
----------------------
72+
73+
The recommended way for implementing a storage backend is by sub-classing ``OC\Files\Storage\Common`` which provides
74+
fallback implementations for various methods, reducing the amount of work required to implement the full storage api.
75+
Note however that various of these fallback implementations are likely to be significantly less efficient than an
76+
implementation of the method optimized for the abilities of the storage backend.
77+
78+
Adding mounts to the filesystem
79+
-------------------------------
80+
81+
The recommended way of adding your own mounts to the filesystem from an app is implementing ``OCP\Files\Config\IMountProvider``
82+
and registering the provider using ``OCP\Files\Config\IMountProviderCollection::registerProvider``.
83+
84+
Once registered, your provider will be called every time the filesystem is being setup for a user and your mount provider
85+
can return a list of mounts to add for that user.

0 commit comments

Comments
 (0)