OpenBSD: Apache VirtualHosts with suexec and chroot



If you're anything like me, you lead a crazy life. When you set up a multi-user web server you pretty much want to forget about it once you're done, so it's got to be secure. So of course you want to use OpenBSD, and if you run Apache then you want it to stay in a chroot jail, and you want to use suexec so that your users won't be able to overwrite each other's stuff.

But wait! It turns out that the default OpenBSD install won't work with both suexec and chroot enabled! Perhaps you've come to this page after seeing an error like these in your error.log:

[error] Premature end of script headers

And then after some digging perhaps you figured out how to get suexec's logging to work, you found an error like this:

command not in docroot

What mockery of life is this? Don't worry. After days of frustration and kernel tracing, I've found the problem(s).


Why, it's only the most secure operating system known to middle earth! Install it on your computer as soon as you can, check out the OpenBSD Journal, or read about the curious history of OpenBSD.


suexec is a feature of the Apache web server. It means "Switch User For Exec," and basically it allows Apache to run cgi programs for your VirtualHosts as different users, so that one VirtualHost's cgi program can't trash your other VirtualHosts. This piece of software consists of a few lines of code in Apache itself, plus a binary executable (/usr/sbin/suexec) that is called from Apache as needed.

You see, normally any CGI scripts are run as the user/group defined in /var/www/conf/httpd.conf, which by default in OpenBSD are set to user = www and group = www. So even if you have different users and different Virtual Hosts, all your cgi programs are normally run under the same user and group as the server. This can be bad because if you allow cgi programs to read and write files, it means that any user's cgi programs will be able to read and write files on other user's web sites.

With suexec though, when a user's script runs, it runs under her/his user id. This allows the administrator (that's you!) to set permissions correctly on the cgi-bin directories so that your users' cgi programs can read/write to their own cgi-bin directory, but not to anyone else's.


A chroot is an operation which changes the root directory for a given program. A program that is running under a chroot cannot access files outside of the chroot directory. This provides a convenient way to make a "sandbox" for an untrusted program to run in.

Apache is one of those programs you might not want to trust. It takes its input from web browsers all around the world, including people who might be trying to break into your computer through Apache. By putting Apache inside a chroot, you limit the amount of damage that someone could do if they broke into your computer through Apache.


Often, system administrators are called upon to host multiple web sites on a single server. To do this, you will need your web server software to examine the requests that come in, and send them to the right place on the file system. You might have five users with their own web directories, each representing a different web site. You need your server software to examine incoming http requests to see which domain the request is for, and serve the request from the appropriate user's web directory. Apache's Virtual Hosts do just this.

To get this working, all you need to do is edit your /var/www/conf/httpd.conf file and put in a <VirtualHost> section for each of your domains.

The Problem


In short, when suexec is compiled, a bunch of configuration options get compiled right in there, and after that you can't change them. Run /usr/sbin/suexec -V and you'll see that something called the DOC_ROOT got compiled right into there. The DOC_ROOT specifies a directory that all cgi programs must reside under in order for suexec to work.

So here's the quirk: when you've got Apache chroot-ed (for example, to /var/www), it becomes unaware of anything outside of that chroot. You know this already though, don't you? But look in /var/www (or whatever your chroot is). Do you see a directory in there called var? Probably not. So while suexec is expecting your documents to be somewhere under the DOC_ROOT directory (/var/www/htdocs by default), your documents are effectively now at /htdocs thanks to the chroot.

The first thing I tried to fix this was going into /var/www and creating a var subdirectory with a www link inside it pointing back to /var/www, but I found that it still won't work and the suexec error log now says:

emerg: cannot get current working directory

Does that make any sense? No it doesn't. Well, not until you look through the source code for suexec, and find that it's using getcwd() to find the current working directory, and that function specifically passes through symlinks as if they weren't there. So now you're stuck in quite a ball of mud, yes?

The Solution


For starters you've got to recompile suexec, and set the DOC_ROOT to /cgi-bin (which is living inside the chroot). So just go over to /usr/ports and... WAIT A MINUTE! suexec isn't there! That's because suexec is part of Apache, so you're going to have to recompile Apache as well. There's no port for Apache though, because it's part of the default OpenBSD installation. We can find Apache, then, in the source tree for OpenBSD. No worries, Apache is easy enough to compile (even with SSL support!):

# cd /usr
# export CVSROOT=:pserver:anoncvs@www.openbsd.org:/cvs
# cvs checkout src/usr.sbin/httpd && cvs logout

now you should see all this crazy stuff downloading...

# cd src/usr.sbin/httpd
# ./configure --with-layout="OpenBSD" --suexec-docroot="/cgi-bin" --suexec-logfile="/logs/suexec_log" --enable-suexec \
--enable-module=ssl --enable-module=so --enable-module=auth_anon --enable-shared=auth_anon \
--enable-module=expires --enable-shared=expires --enable-module=headers --enable-shared=headers \
--enable-module=auth_db --enable-shared=auth_db --enable-module=auth_dbm --enable-shared=auth_dbm \
--enable-module=auth_digest --enable-shared=auth_digest --enable-module=cern_meta --enable-shared=cern_meta \
--enable-module=define --enable-shared=define --enable-module=digest --enable-shared=digest \
--enable-module=info --enable-shared=info --enable-module=log_agent --enable-shared=log_agent \
--enable-module=log_referer --enable-shared=log_referer --enable-module=mime_magic --enable-shared=mime_magic \
--enable-module=mmap_static --enable-shared=mmap_static --enable-module=proxy --enable-shared=proxy \
--enable-module=rewrite --enable-shared=rewrite --enable-module=speling --enable-shared=speling \
--enable-module=unique_id --enable-shared=unique_id --enable-module=usertrack --enable-shared=usertrack \
--enable-module=vhost_alias --enable-shared=vhost_alias

if no errors went by, you're ready to compile...

# make
# make install

Did it finish without any errors? Good work! Now that the new "suexec" is in place, you should be able to run "/usr/sbin/suexec -V" and the DOC_ROOT should now be /cgi-bin.


Congratulations, you've recompiled Apache. You're on your way to a better life. Well, wait a minute friend! There's still some minor hurdles to get over before you can go make your coffee and sit on the porch.

First, suexec needs to be owned by root and it needs to have its sticky bit set. When a sticky bit is set on a program, then when it runs it gets the privileges of the owner (instead of the privileges of whomever happens to be running the program). This combination of being owned by the root user and having its sticky bit set is what allows suexec to become any user (in order to run cgi programs as that user). So go ahead with something like this:

# chown root:wheel /usr/sbin/suexec
# chmod 4755 /usr/sbin/suexec

Christian Pedaschus wrote in with this word of caution: By default in OpenBSD, if you created a separate partition for /var, it will be mounted with the "nosuid" flag set. This will spell disaster for suexec, which won't be allowed to change its user id. To fix this, edit your /etc/fstab file and delete "nosuid" from the line that mentions your /var directory. Thanks Christian, for this most excellent tip!

Next, a little housekeeping. It's likely that you want your users to own their cgi-bin directories, and the "users" group (gid 10) is probably the group owner on those directories. This is how you're supposed to use suexec. There's a problem though... suexec wants to prevent privileged users and groups from executing things, so it has this concept of a "minimum uid" and a "minimum gid" - which by default are set to 1000 in the OpenBSD Apache source tree. It won't execute any scripts if the owner's uid or gid is lower than 1000. This prevents scripts from ever running as the "root" user or the "wheel" (admin) group, so that's good in terms of security. However... if any cgi scripts are owned by your "users" group (gid 10), then suexec is going to refuse to run them! The right way to fix this is to modify your "users" group to have a higher gid (say 2020), like so:

# groupmod -g 2020 users

As a bonus hassle, you'll need to go find everything that now says it's owned by group 10 (the old "users" group gid), and reset the ownership correctly for each of those files. So for every file that now says it's owned by group "10", do something like this:

#chown :users <filename>

Now let's get suexec running inside the chroot. It turns out that suexec needs to actually be copied into the chroot. This is an interesting situation. suexec needs to be in /usr/sbin so that Apache sees it and enables it (this happens *before* httpd actually chroot's itself) -- and then suexec also needs to be in /var/www/usr/sbin so that Apache can find it *after* it has chrooted. So we figure out which libraries suexec uses, and copy those over to the chroot along with the suexec binary:

# ldd /usr/sbin/suexec
        Start    End      Type Ref Name
        00000000 00000000 exe   1  /usr/sbin/suexec
        030fc000 23103000 rlib  1  /usr/lib/libm.so.2.0
        0e97a000 2e985000 rlib  1  /usr/lib/libssl.so.10.0
        0e605000 2e633000 rlib  1  /usr/lib/libcrypto.so.12.0
        0f766000 2f797000 rlib  1  /usr/lib/libc.so.38.2
        0366c000 0366c000 rtld  1  /usr/libexec/ld.so

Ok so let's copy all that over to the chroot:

# mkdir -p /var/www/usr/lib
# cp /usr/lib/libm.so.* /var/www/usr/lib
# cp /usr/lib/libssl.so.* /var/www/usr/lib
# cp /usr/lib/libcrypto.so.* /var/www/usr/lib
# cp /usr/lib/libc.so.* /var/www/usr/lib
# mkdir -p /var/www/usr/libexec
# cp /usr/libexec/ld.so /var/www/usr/libexec
# mkdir -p /var/www/usr/sbin
# cp /usr/sbin/suexec /var/www/usr/sbin

Now for some real tricks. suexec needs a few files to do its work. First and foremost, from its little chroot jail, it needs to figure out which groups equal which gid's, and which users equal which uid's, and then a few more things like that. If suexec can't find this information in the chroot jail, it gives errors like these:

invalid target user name:
invalid target user id:
invalid target group name:
invalid target group id:
failed to setgid:
failed to setuid:

To find out which files are needed, you have no choice but to run the Apache httpd daemon through ktrace (which traces system calls), and watch to see which files it fails to open. Just your luck though, because I've already done all that messy stuff for you, and I don't mind telling you which files you'll need to copy over to make suexec and Apache happy:

# mkdir -p /var/www/etc/
# cp /etc/group /var/www/etc/
# cp /etc/localtime /var/www/etc/
# cp /etc/login.conf /var/www/etc/
# cp /etc/passwd /var/www/etc/
# cp /etc/pwd.db /var/www/etc/
# mkdir -p /var/www/usr/share
# cp -R /usr/share/nls /var/www/usr/share


Now most people use perl or python or something to run their cgi programs. If you want perl to work, you're going to have to copy it into the chroot:

# mkdir -p /var/www/usr/bin
# cp /usr/bin/perl /var/www/usr/bin/
# cp /usr/bin/perl5.* /var/www/usr/bin/

And of course perl is linked to some libraries:

# ldd /usr/bin/perl
        Start    End      Type Ref Name
        00000000 00000000 exe   1  /usr/bin/perl
        01a1a000 21a3b000 rlib  1  /usr/lib/libperl.so.10.0
        0eded000 2edf4000 rlib  1  /usr/lib/libm.so.2.0
        07b99000 27b9d000 rlib  1  /usr/lib/libutil.so.11.0
        01999000 219ca000 rlib  1  /usr/lib/libc.so.38.2
        0adac000 0adac000 rtld  1  /usr/libexec/ld.so

So we copy those over too (note some are already there):

# cp /usr/lib/libperl.so.* /var/www/usr/lib/
# cp /usr/lib/libutil.so.* /var/www/usr/lib/

Phew! Now let's create a perl script to test with. Go into /var/www/cgi-bin and create a file called test.pl (make sure it's executable by the "www" user or the "www" group!)


print "Content-type: text/plain\n\n";
opendir(DIRHANDLE, "/");
@filenames = readdir(DIRHANDLE);
foreach $file (@filenames) { print "$file\n"; }

So let's see if it works!

# lynx http://localhost/cgi-bin/test.pl


Sweet! The chroot is working - we asked perl to open up the / directory and all it saw was the stuff inside our chroot at /var/www. Now you might want to copy some of those neat perl modules over to your chroot, and then things to keep those perl modules happy (especially the "net" modules):

# cp /etc/resolv.conf /var/www/etc/
# cp /etc/services /var/www/etc/
# mkdir -p /var/www/usr/libdata /var/www/usr/local
# cp -R /usr/lib/Apache /var/www/usr/lib/
# cp -R /usr/libdata/perl5 /var/www/usr/libdata/
# mkdir -p /var/www/usr/local/libdata
# cp -R /usr/local/libdata/perl5 /var/www/usr/local/libdata/
# mkdir -p /var/www/usr/share
# cp -R /usr/share/zoneinfo /var/www/usr/share/


Let's pretend we wanted to have a VirtualHost running on port 81 with its own cgi-bin and htdocs directories, all managed by the "nobody" user. We'll create /var/www/htdocs/www.host.com owned by user "nobody" and group "www" (so the web server can read it even without suexec) and we'll create /var/www/cgi-bin/www.host.com owned by user "nobody" and group "nobody" (so other users can't write to it). Also we'll copy our test.pl into this new cgi-bin, so we have something to test with:

# mkdir -p /var/www/htdocs/www.host.com
# echo "hello world" > /var/www/htdocs/www.host.com/index.html
# chown -R nobody:www /var/www/htdocs/www.host.com
# chmod 750 /var/www/htdocs/www.host.com

# mkdir -p /var/www/cgi-bin/www.host.com
# cp /var/www/cgi-bin/test.pl /var/www/cgi-bin/www.host.com/
# chown -R nobody:nobody /var/www/cgi-bin/www.host.com
# chmod -R 700 /var/www/cgi-bin/www.host.com/test.pl

And add a <VirtualHost> section to your /var/www/conf/httpd.conf like so:

Listen 81
<VirtualHost *:81>
    DocumentRoot /htdocs/www.host.com
    ServerName www.host.com
    User nobody
    Group nobody
    ErrorLog /var/www/logs/www.host.com-ERROR
    CustomLog /var/www/logs/www.host.com-ACCESS combined
    ScriptAlias /cgi-bin/ "/cgi-bin/www.host.com/"

First thing you'll notice is that we set the DocumentRoot to /htdocs/www.host.com - but of course there is no "htdocs" subdirectory in your root directory. Well, the DocumentRoot is accessed *after* Apache has done it's chroot to /var/www, so when it tries to access /htdocs it will actually be accessing /var/www/htdocs - just like we want it to. As you can see in the above configuration, the same thing is true for the cgi-bin directory.

But what's the deal with those log files? Why do they reference /var/www? Well, for whatever reason, Apache opens up those files *before* it does its chroot, so we need to have that /var/www in there. Curious, no? I'm sure there's some reason that this is true, and it would become obvious if we read through the Apache source... but that's like a million lines and I don't think any of us have that much coffee on hand.

Ok, but what is www.host.com? Should you change that to your domain? Surprisingly, you don't need to bother. Let's temporarily add it to your /etc/hosts file and point it towards localhost. You'll need to add this line to your /etc/hosts file: www.host.com host

So now on your machine if you ping www.host.com you should see replies coming from Neat, right? So now restart Apache (just run "Apachect restart"), and your VirtualHost should be active:

lynx http://www.host.com:81/cgi-bin/test.pl



Contact the Author

Ian Charnas lives alone atop a mountain. Once a week he comes down to see other human faces, and to check his email. You can go ahead and email him, the address is icc - at - case - dot - edu.

- Phew! What a day, eh? Time to get some fresh air -