BasefsAFuseFSforDecoding
From BononWiki
This text is about the compilation and the abilities of basefs, a FUSE fs.
Contents |
Purpose
The purpose is to test the behaviour of the use of a decoder to convert files from one format to another on the fly, like mp3fs and aifffffs, of which you'll find info at the FUSE mainpage/filesystems using FUSE.
Recent discussions on the maillingslist pointed at the importance of serialize the request to a file, when the fs is multithreaded. The decoders are not thread safe, and seeking in a file is expensive (causing extra unnecessary CPU time).
To make the read call threadsafe a lock is used (pthread_mutex_lock). The next thread which gets access to the file does not have to be the one which is send by the context. So this lock is causing the order of the requests to be scrambled, which is not a good thing.
There has been a patch to adjust the behaviour of FUSE, to serialize the access per inode. Now in my opinion it's also very good to investigate the location to do this serializing in the fs, and not in FUSE. Now this fs is meant mainly for this purpose. I'm not ready with this fs, I have to build a decoder in it. This will be not so difficult since there is already aiffffs, and it's very good written, so it will be not that hard to take the required functions from there.
Also I've tested Gstfs, a Gstreamer fs, using the abilities of Gstreamer to decode files.
Compile and use of basefs
Requirements are of course FUSE. The latest version 2.8.5 (at this moment 2011-06-24) is sufficient.
Unpack the source if not already, or use git:
git clone git://gitorious.org/basefs/basefs.git basefs cd basefs
Compile with:
> make
You'll get some warnings. I've tried to do something about these, but some cannot be avoided. Use it there or move it to a location you can use it in your path:
mv basefs ~/bin
or
mv basefs /usr/bin
Use it as follows:
basefs -o bind-directory=DIR,logging=NR MOUNTPOINT
will make DIR (just like fusexmp) to appear under mountpoint. This looks like a bind mount.
- If bind-directory is empty, it will take the root /.
- If logging is empty, it will default to 0. There are different loglevels, 0 means only errors, higher means more logging, up to 3.
For example:
mkdir ~/testmount basefs -o bind-directory=/tmp ~/testmount
This will create a copy of /tmp under ~/testmount.
Abilities
Use of Extended Attributes to adjust behaviour.
The fs supports Extended Attributes, ie a call to for example system.posix_acl_access is supported, and will lead to a valid answer when the backend fs supports Xattr.
But also when the underlying fs does not support it, basefs supports various setfattr/getfattr calls, meant to get detailed information and ti adjust to behaviour of the filesystem.
Basefs uses various Extended Attributes like:
. system.workspace_logging, get/set the logging level (0...3) . system.workspace_global_nohide, get/set the hiding of entries globally (0/1) . system.workspace_entry_hide, get/set the hiding of an entry (0/1) . system.workspace_abs_path, get the abs path, the abs path on the underlying fs. . system.workspace_use_directio, get/set the use of direct io (0/1)
Some of these ( logging, global_nohide, use_directio) are only available on a system entry.
This works as follows:
:> mkdir ~/testmount :> basefs -o logging=3,bind-directory=/tmp testmount :> cd ~/testmount :> ls -al .... ... .contents of /tmp ... .... :> getfattr --dump --match=system . # file: . system.workspace_abs_path="/tmp/." system.workspace_entry_hide="0" system.workspace_global_nohide="0" system.workspace_logging="3"
Now the xattrs on the "." are globally, you can also get/set per file or directory.
It's possible to add more xattributes, of course. When doing so, take care to add it in the list (listxattr4workspace), and to take care for setting (setxattr4workspace) and getting (getxattr4workspace) with the right values. There are some helper functions for getting simple integers and strings.
Creation of virtual entries
The purpose of this fs is to provide the decoded contents of media files on the fly, from one format to a desired other. The way it does that is to create a virtual entry for every "to be decoded candidate".
This fs looks at the extension to determine the type of the file. If the extension does not reflect the contents (a mp3 file with a .ogg extension for example) the fs will be fooled at this moment (20110702).
Maybe in future add a test to inspect the contents. Most formats use a header, but to look at that, it's required to open and read the first x bytes.
For every to be decoded file a "twin" virtual entry is created.
This fs has a administration of entries and inodes, for every entry found in the backend a "copy" of that is created in the fs. So you can call these "real" entries: they present a real file in the backend. The new "twin" virtual entries introduces here do not represent such a file in the backend, and thus the name "virtual".
The name "twin" I have chosen here because it's closely related to the "real" entry, and there can be only one decoded entry per image file.
Maybe in future there can be more than one "decoded" virtual entries per media file, for each chosen different format one.
The virtual entry has the same attributes as the real entry it's related to, except (of course) the size, the name (other extension!) and inode.
The entry structure looks like:
struct basefs_entry_struct {
char *name;
struct basefs_inode_struct *inode;
struct basefs_entry_struct *name_next;
struct basefs_entry_struct *name_prev;
struct basefs_entry_struct *parent;
size_t namehash;
int hide;
unsigned char type;
unsigned char virtualtype;
struct directory_data_struct *directory_data;
struct virtual_data_struct *virtual_data;
};
for every "to be decoded" entry, extra virtual_data is created, looks like:
struct virtual_data_struct {
struct basefs_entry_struct *entry;
struct basefs_entry_struct *virtual_entry;
int status;
size_t size;
unsigned char cached;
pathstring cached_decoded_file;
unsigned char type;
union {
struct aiff *flac_to_aiff;
} decoder;
};
The realentry points to this virtual entry:
realentry->virtual_data=virtual_data virtual_data->entry=realentry
and a twin entry is created, which also points to this virtual_entry:
virtual_entry->virtual_data=virtual_data virtual_data->virtual_entry=virtual_entry
So this virtual_data is a bridge between the two entries. Futher it contains information about the decode file and type decoder. The struct virtual_data (20110702) is likely to change, like other decoders, Gstreamer for example.
Pathresolution and caching of directories
The most expensive tasks (so far) is the determination of the path in the underlying fs, after a opendir/readdir call. After this call, for every direntry a lookup and a getattr will be done. This means a lot of determing of almost the same path, for every dir entry again.
This fs has a construction to make that path resolution go much faster. Especially when a directory has a lot of entries, this counts.
Normally the path is determined by walking back the entries to the root, by just taking the parent, and pasting the name:
With a "burst" of lookup/getattr calls, the dirname is done every time over and over again. Now here the fs caches the path for the dir, and when it resolves the path, and it finds this cached path, it uses that.
In the case of the image above, this will result in:
. keep the path of the first parent by create a cached path (directory data) and make the parent entry point to that. This path is already known, it's required for the opendir call on the backend.
. when resolving the path for the "children entries" at a lookup or a getfattr(=stat) call , it finds this cache when walking back, and uses that.
Right now this is only done for opendir calls. I guess that's causing the most work, it's call typically use by the user and not the system. (well there are scripts/utilities wanting to know the contents of a directory, but overall it's still a user call..)
The cached values do not expire. Once created, it stays there. There is some code present for removing "expired" directory data, but is not complete yet. Every time a directory is opened, the score if that directory_data is increased. This fs has a special io thread, which can take care of removing directory_data with a low score/period not used. This needs more work.
Usefull resource is the manpage of path_resoultion:
