Put Jail /tmp and /var/tmp in memory?

Discussion in 'Plugins' started by Milhouse, Oct 23, 2012.

  1. Offline

    Milhouse FreeNAS Aware

    Member Since:
    Jun 1, 2011
    Messages:
    555
    Message Count:
    555
    Likes Received:
    5
    Trophy Points:
    18
    Milhouse, Oct 23, 2012

    Is it possible, within an 8.3.0-RC1 Jail, to have /tmp (and /var/tmp) mounted in memory (ie. tmpfs)?

    MySQL will, by default, create temporary files in /var/tmp- even when performing read-only queries (ie. SELECT). This is causing excessive disk writes to ZFS (which is where /var/tmp lives inside a Jail), and prevents the disks from spinning down (assuming ataidle worked, which isn't actually the case in 8.3.0-RC1, but I digress).

    Unfortunately, the Jail seems to have no access to volatile storage such as tmpfs, so activity that wouldn't normally trigger disk activity outside of a jail, do so inside of a jail. The only obvious solution appears to be to lump the entire Jail on to a USB memory stick, but if possible I'd rather shift /var/tmp and /tmp into RAM.

    Anyone got any ideas? :)
  2. Offline

    paleoN FreeNAS Guru

    Member Since:
    Apr 22, 2012
    Messages:
    1,403
    Message Count:
    1,403
    Likes Received:
    15
    Trophy Points:
    38
    paleoN, Oct 23, 2012

    I haven't tried it or even bothered to think about it any further.
  3. Offline

    Milhouse FreeNAS Aware

    Member Since:
    Jun 1, 2011
    Messages:
    555
    Message Count:
    555
    Likes Received:
    5
    Trophy Points:
    18
    Milhouse, Oct 23, 2012

    Hi, yes - I identified several of the cron jobs as part of the problem, the default cron setup doesn't seem particularly appropriate for the jail.

    I've just created a memory-backed disk, and mounted it so that it can be written to within the jail, and that's solved the MySQL excessive write problem.

    Next step is to work out how I can create the memory backed disk and mount it automatically with each boot...
  4. Offline

    Milhouse FreeNAS Aware

    Member Since:
    Jun 1, 2011
    Messages:
    555
    Message Count:
    555
    Likes Received:
    5
    Trophy Points:
    18
    Milhouse, Oct 23, 2012

    Right, here's a patch to add support for a memory backed /var/tmp in 8.3.0-RC1. Its rather quick & dirty, but it works for me.

    Apply the attached patch to /conf/base/etc/rc.d/ix-jail:

    Code (text):
    1.  
    2. mount -uw /
    3. cd /conf/base/etc/rc.d/
    4. patch ix-jail /path/to/ix-jail.patch
    5. mount -ur /
    6.  
    Enable the memory backed modification by specifying a value for gaol_tmp_fs_size in /conf/base/etc/rc.conf (or /etc/rc.conf, if only testing).

    Example: gaol_tmp_fs_size="64m".

    The modified ix-jail will create a memory backed device (always unit 3, ie. /dev/md3) of the specified size, that will be mounted on /var/tmp for the relevant plugin/jail. This device will be unmounted and then destroyed when the plugin is stopped. Obviously, this patch assumes there will only ever be one plugin/jail per server... not sure if multiple plugins are currently supported, but this patch will need additional work to support multiple memory backed devices.

    Note that I've had to use "gaol_" as an identifier because ix-jail strips out from rc.conf every entry that contains the prefix "jail_". As an aside, ix-jail should probably be changed to use the regex "^jail_" so that it removes only the entries that begin with "jail_".

    Attached Files:

  5. Offline

    paleoN FreeNAS Guru

    Member Since:
    Apr 22, 2012
    Messages:
    1,403
    Message Count:
    1,403
    Likes Received:
    15
    Trophy Points:
    38
    paleoN, Oct 25, 2012

    Glad the idea seems to pan out. :) I will have to play with this myself some. Though I won't likely get a chance to do so all that soon.
  6. Offline

    Milhouse FreeNAS Aware

    Member Since:
    Jun 1, 2011
    Messages:
    555
    Message Count:
    555
    Likes Received:
    5
    Trophy Points:
    18
    Milhouse, Oct 25, 2012

    Patch updated (see post #4) with support for dynamic memory disk unit allocation and multiple jails per server.

    Changelog:

    • Jail-specific gaol_tmp_fs_size added: Optionally specify "global" gaol_tmp_fs_size (as before), or gaol_<name>_tmp_fs_size (if present, latter supersedes the former)
    • Dynamically determines memory disk unit rather than assume md3 (so potentially each jail can have it's own memory backed device)
    • Changed memory back disk to use swap instead of malloc

    Eg., in rc.conf:

    gaol_tmp_fs_size=64M
    gaol_myjail1_tmp_fs_size=32M
    gaol_myjail2_tmp_fs_size=48M

    If you had 3 jails, myjail1 would be allocated 32M, myjail2 48M and jail #3 (and any other jails) 64M.

    If you remove the global gaol_tmp_fs_size setting, then jail #3 would not be allocated a memory backed device at all.
  7. Offline

    Milhouse FreeNAS Aware

    Member Since:
    Jun 1, 2011
    Messages:
    555
    Message Count:
    555
    Likes Received:
    5
    Trophy Points:
    18
    Milhouse, Oct 27, 2012

    Patch updated (see post #4), now with support for /var/log and additional directories. Tested with 8.3.0-RELEASE.

    Due to continued logging activity within the Jail preventing disk spin-down, I've decided to cut my losses and include /var/log within the memory backed storage area.

    To support multiple directories (tmp, log etc.), the memory backed storage will have its own mount point /var/mdisk, and the relevant directories automatically created within by searching for and locating symbolic links in /var that have their content defined as being within /var/mdisk.

    If you want to relocate any other directories from /var to memory backed disk, simply create a symbolic link to /var/mdisk. If you don't want log or tmp to be in memory backed storage, keep them as physical directories rather than symbolic links.

    The easiest way to create the required symbolic links is to stop the plugin, cd directly to the /var directory in the jail (ie. cd /mnt/<tank>/jail/software/var) and:
    Code (text):
    1.  
    2. rm -fr log tmp
    3. mkdir mdisk
    4. ln -s ./mdisk/log log
    5. ln -s ./mdisk/tmp tmp
    6.  
    As you start the plugin, the memory backed storage will be mounted on /var/mdisk, and within /var/mdisk the tmp and log directories will be created.

    If, you also wanted to relocate /var/run into memory backed storage: stop the plugin, remove the /var/run physical directory, create a symbolic link for /var/run to ./mdisk/run (or if you prefer, /var/mdisk/run or even just mdisk/run), and start the plugin.

    Changelog:

    • Use own mount point, /var/mdisk
    • Support multiple directories based on symbolic links to /var/mdisk that are located in /var
  8. Offline

    joeschmuck Old Man

    Member Since:
    May 28, 2011
    Messages:
    2,609
    Message Count:
    2,609
    Likes Received:
    86
    Trophy Points:
    48
    Occupation:
    Electrical Engineer, Data Analysis, and Management
    Location:
    Virginia
    joeschmuck, Nov 30, 2012

    Good posting. I'll give it a shot tomorrow.

    EDIT: I got this to work fine in a VM but not on the real machine. I will try it again when my head is clear and the Christmas lights are on the house.
  9. Offline

    vvanderm

    Member Since:
    Jan 14, 2013
    Messages:
    1
    Message Count:
    1
    Likes Received:
    0
    Trophy Points:
    0
    vvanderm, Jan 14, 2013

    Any luck with testing this joeschmuck? Thoughts as to which version of freenas this would make it into?

    Thanks!
  10. Offline

    joeschmuck Old Man

    Member Since:
    May 28, 2011
    Messages:
    2,609
    Message Count:
    2,609
    Likes Received:
    86
    Trophy Points:
    48
    Occupation:
    Electrical Engineer, Data Analysis, and Management
    Location:
    Virginia
    joeschmuck, Jan 15, 2013

    I tested it out and it worked from what I recalled but I didn't leave it installed, only because I don't have any issue with my NAS drives spinning down.

    As for it becoming part of FreeNAS, I haven't heard that before so I don't know if it ever will be part or not. Also keep in mind that the patch is only for the FreeNAS version it was built for, you could probably create one for a future version as well. But if this is something you would like to see implemented into FreeNAS, you should submit a ticket requesting the modification but it should be an optional modification.
  11. Offline

    Milhouse FreeNAS Aware

    Member Since:
    Jun 1, 2011
    Messages:
    555
    Message Count:
    555
    Likes Received:
    5
    Trophy Points:
    18
    Milhouse, Jan 15, 2013

    Patch still works with 8.3.0-RELEASE-p1.

    Very unlikely this would be accepted officially, and frankly I don't even bother asking for such stuff now, I've decided the only way to fix stuff is do it yourself (hence patches like mine).
  12. Offline

    joeschmuck Old Man

    Member Since:
    May 28, 2011
    Messages:
    2,609
    Message Count:
    2,609
    Likes Received:
    86
    Trophy Points:
    48
    Occupation:
    Electrical Engineer, Data Analysis, and Management
    Location:
    Virginia
    joeschmuck, Jan 15, 2013

    And your patches are welcomed.
  13. Offline

    fracai FreeNAS Aware

    Member Since:
    Aug 22, 2012
    Messages:
    533
    Message Count:
    533
    Likes Received:
    17
    Trophy Points:
    18
    fracai, Feb 26, 2013

    Could this also be implemented as a startup script? In other words, is there anything that can't be run from within the jail? That would seem more stable than patching each FreeNAS release, if this isn't added to a proper release that is.
  14. Offline

    Milhouse FreeNAS Aware

    Member Since:
    Jun 1, 2011
    Messages:
    555
    Message Count:
    555
    Likes Received:
    5
    Trophy Points:
    18
    Milhouse, Feb 26, 2013

    No, you need to patch FreeNAS.
  15. Offline

    fracai FreeNAS Aware

    Member Since:
    Aug 22, 2012
    Messages:
    533
    Message Count:
    533
    Likes Received:
    17
    Trophy Points:
    18
    fracai, Feb 26, 2013

    Could you elaborate on that?

    I've seen people getting tmpfs running in FreeBSD jails. Are the tools not available in the FreeNAS jail? Or are the actions (mounting a filesystem) simply not permitted in the jail?
  16. Offline

    Milhouse FreeNAS Aware

    Member Since:
    Jun 1, 2011
    Messages:
    555
    Message Count:
    555
    Likes Received:
    5
    Trophy Points:
    18
    Milhouse, Feb 26, 2013

    Look at the patch to see what it does. The way in which the jail is created has to be modified to support the memory backed storage, it's not about creating tempfs mounts within the jail as tempfs is created outside of the jail so that the jail will use it automatically without modifying any addons running within the jail. Possibly you could modify addons to use tempfs on an add-hoc basis but you'll be modifying individual addons each time rather than doing it once at the "system" level.

    I will backtrack slightly and say that yes, you could possibly apply the functionality discussed in this thread by not patching the system but instead making the changes at startup, but you would need a startup script that shuts down all jails, replaces the jail creation scripts with the memory-backed version and then restarts the jails. If you're going to do that then you might as well patch the system...
  17. Offline

    fracai FreeNAS Aware

    Member Since:
    Aug 22, 2012
    Messages:
    533
    Message Count:
    533
    Likes Received:
    17
    Trophy Points:
    18
    fracai, Feb 26, 2013

    I did look at the script and it didn't look like anything that wouldn't be possible without patching. We're probably talking past each other though. I'd be fine with one tmpfs mount in the only jail. I don't need one per plugin. So I'll probably play around with it tonight if I get the chance to see if I can just put together an rc script that would run when the jail starts up.
  18. Offline

    Milhouse FreeNAS Aware

    Member Since:
    Jun 1, 2011
    Messages:
    555
    Message Count:
    555
    Likes Received:
    5
    Trophy Points:
    18
    Milhouse, Feb 26, 2013

    OK, I see what you are getting at, let us know how you get on.
  19. Offline

    smufu

    Member Since:
    Apr 17, 2013
    Messages:
    4
    Message Count:
    4
    Likes Received:
    0
    Trophy Points:
    0
    smufu, Apr 17, 2013

    I tested the fix with the following version: FreeNAS-8.3.1-RELEASE-p2-x64 (r12686+b770da6_dirty), and seems like the fix isn't working.
    I can see the following message in the boot-log:
    Code (text):
    1. Apr 17 21:41:17 FreeNAS ix-jail: Created memory backed disk /dev/md3 with size 64M
    But the disk don't spin down if the plugin-service is running.
    Can anybody confirm this?
    Can I check by myself if the fix is working as expected?

    Regfards,
    smufu
  20. Offline

    Milhouse FreeNAS Aware

    Member Since:
    Jun 1, 2011
    Messages:
    555
    Message Count:
    555
    Likes Received:
    5
    Trophy Points:
    18
    Milhouse, Apr 17, 2013

    Sure, write to the directories you think are mounted from mdisk (/tmp, /var/tmp, /var/log etc.) and see if the act of writing to these directories causes your disks to spin up - if not, then the "fix" is working as planned.

    If however your disks continue to spin up even with this fix in place, then you need to track down what process is reading/writing to your disks, and where, then see if there is a way to prevent it.

Share This Page