[Perl Modules] Installing in non-standard (isolated) locations.

[Brandon Luhring]

Hello,

I'm pretty new to Perl and especially the server requirements of it. But I've inherited a site that was built with Perl/CGI and we were forced to switch servers (with no help from the previous developer).

On the original server they had a Perl Module located here:
/home/www/AppCode/app.pm

And they had a staging area similarly with module here:
/home/staging/AppCode/app.pm

Each of those accessible through either the "www" or "staging" sub-domains, respectively.

We could edit the .pm file and FTP it up, and changes would be immediate.

On this new server, the host wanted to install the .pm files to this location:
/usr/lib/perl5/site_perl/app.pm

This allowed the host to get our site up and running. However, now we can't easily just FTP up an update.

They also installed the "staging" sub-directory under the main site's location like this:
/home/www/staging/

The sub-domains to access them work just like you'd want them too, but now both www and staging are using the Module from the "perl5" location, and the version of that file that is in the site files doesn't get loaded up.

It's important that staging loads its own version of the Perl Module because it contains a database connection string which differs between the two environments.

I've explained this to my host, but they seem to not fully understand what I need to do.

I have SSH access, so if someone could point me in the right direction I could just do it myself and not go back-and-forth with the host.

I've been here:
Perl Module Mechanics

But even their "non-standard location" section seems to not fit my needs with keeping the staging area running its own version.

Thanks for any help!
Brandon

[Stomme poes]

I want to start sending these Perl threads to others in the Perl community in hopes that someone can give some decent answers.

In the meantime, it would not be a bad idea to cross-post this to PerlMonks - The Monastery Gates and/or stackoverflow (where the Perl community has been making a more concentrated effort to be there for Perl questions). If anyone complains about cross-forum posting, tell 'em I said do it : )

I had some people at YAPC who said they'd maybe be interested in threads over here so I'm sending out mails now. But if this needs faster attention do copy this out.

I'm never on SO but perlmonks is an excellent resource despite how it looks. : )

[dsheroh]

Hi, Brandon! My name is Dave, and I'm... a perloholic.

I'm not entirely clear on your server configuration and how the application is invoked. You mention that you're using a module called app.pm, but you didn't say whether it's run directly by the server (probably using mod_perl or similar) or if there's some other (probably CGI/fCGI) script which you control that loads app.pm, then accesses its functions.

So I'm going to make these assumptions, which would fit a fairly common configuration:

Assumption 1) You have a CGI (or similar) script which I'll call index.cgi.

Assumption 2) There are two separate copies of index.cgi, one in /home/www/AppCode and one in /home/staging/AppCode.

Assumption 3) Each index.cgi has its corresponding app.pm in the same directory as the index.cgi itself.

If these assumptions are accurate, you have two options for how to get this working the way you want, both of which are fairly simple:

Option 1) Similar to what's described in the Perl Module Mechanics document you linked, you can set the environment variable

PERL5LIB=.

in the web server config for each of the virtual hosts. It might be a little tricky to work out how to make this setting in your environment, but it doesn't require changing the index.cgi source code at all.

Option 2) Edit index.cgi and, somewhere near the top (not the very first line, but before app.pm is referenced in any way), add the line

use lib '.';

This is dead simple and independent of your web server configuration, but it requires a (minor) change to index.cgi at a time when you might want to avoid making any changes.

Both options work by telling perl to look in the same directory as the script (index.cgi) itself first, before going off to /usr/lib/perl5 and looking there. If index.cgi and app.pm are in different directories, replacing the "." with the appropriate relative path (../MyApp/lib or whatever) should handle that. Just don't use an absolute path (one that starts with "/") or else both versions will look in the same place, which is what we're trying to avoid here.

If you have any problems, or if any of my assumptions are wrong, just let me know and we can try to work the details out.

[Brandon Luhring]

The "use lib" option would work, I'm sure, but the original developers didn't set this up in a way that would allow us to add that code once and have it inherited for use on all pages... We'd have to update quite a few code files with that.

The Perl5Lib environment variable sounds promising. I'm not sure how the host has setup the virtual hosts, so I've asked them how we can set that variable in the respective environments.

I'll let you know if that works.

Thanks,
B

[Brandon Luhring]

Not quite there yet, but still working with my host. Because I have more faith in this forum, I'll post where I currently am.

In the httpd.conf file we now have things like this:

<VirtualHost 000.000.000.000>
ServerName myapp.com
ServerAlias www.myapp.com
DocumentRoot /home
#...omitting some other values for brevity...#
PerlSetEnv PERL5LIB /home/AppCode
</VirtualHost>

And for the staging site's virtualhost (in the same httpd.conf) it's more like this:

<VirtualHost 000.000.000.000>
ServerName staging.myapp.com
ServerAlias www.staging.myapp.com
DocumentRoot /home/staging
#...omitting some other values for brevity...#
PerlSetEnv PERL5LIB /home/staging/AppCode
</VirtualHost>

But just adding these didn't change anything. Still works like it did before where both www and staging are still pulling from the original location:

/usr/lib/perl5/site_perl/app.pm

I renamed the module here (both file name and package name inside the file) just to see if it was overwriting the modules set by the file found within the sites. As soon as I did that, the sites blew up with errors about not being able to find the "app" module.

Any thoughts?

Thanks,
Brandon

[dsheroh]

The first thing that comes to mind is that the PerlSetEnv directive comes from mod_perl and I believe that it only affects Perl code that runs embedded within the Apache server process as a mod_perl handler. If your code is running under CGI, you should use the plain SetEnv directive instead.

(Since you said that the "use lib" approach would require modifying a large number of source files, I'm assuming that it's CGI-based code because using a separate script to handle each page/URI was a common way to do CGI once upon a time, but I've never heard of anyone doing that with mod_perl handlers.)

[Brandon Luhring]

I've tried both of these lines:

PerlSetEnv PERL5LIB /home/staging/AppCode

SetEnv PERL5LIB /home/staging/AppCode

I've tried each and both at the same time.

I've had it in the httpd.config file under the <VirtualHost> node, and I've tried putting them in the root site files in the .htaccess file (I saw others online talking about putting them there).

No matter what, the staging server appears to always require that the app.pm file be located in the "perl5/site_perl" directory, or the site breaks.

[dsheroh]

I've had it in the httpd.config file under the <VirtualHost> node, and I've tried putting them in the root site files in the .htaccess file (I saw others online talking about putting them there).

Changes to httpd.conf generally aren't recognized until apache reloads its config ("apachectl graceful" or "apache2ctl graceful" - you need to be an admin to run this command).

Changes made in .htaccess need to be allowed by the server config.

So those are some theories on why your changes may not be getting recognized.

As for a further debugging step, you can insert this script (I call it "env.cgi") into each site's directory and try running it through your browser to see the environments that apache is providing. @INC is the list of directories that perl will check when looking for modules, in the listed order; you want your site-specific directory to be at the top of that list. %ENV is the overall environment, which is where those SetEnv commands are trying to set PERL5LIB, so this will show whether they're doing so successfully or not.

#!/usr/bin/env perl

use strict;
use warnings;

print "Content-Type: text/plain\n\n";
print "\@INC:\n";
print join "\n", @INC;
print "\n\n";

print "\%ENV:\n";
print "$_ = $ENV{$_}\n" for sort keys %ENV;

[Stomme poes]

Changes to httpd.conf generally aren't recognized until apache reloads its config ("apachectl graceful" or "apache2ctl graceful" - you need to be an admin to run this command).

Just a quick side note, Debian systems seem to go out of their way to avoid restart commands as seen at apache.org : )

[Brandon Luhring]

dsheroh,

I put in the env.cgi and it looks like this:

@INC:
/usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi
/usr/lib/perl5/site_perl/5.8.8
/usr/lib/perl5/site_perl
/usr/lib/perl5/vendor_perl/5.8.8/i386-linux-thread-multi
/usr/lib/perl5/vendor_perl/5.8.8
/usr/lib/perl5/vendor_perl
/usr/lib/perl5/5.8.8/i386-linux-thread-multi
/usr/lib/perl5/5.8.8
.

%ENV:
DOCUMENT_ROOT = /home/staging
GATEWAY_INTERFACE = CGI/1.1
HTTP_ACCEPT = text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
HTTP_ACCEPT_CHARSET = ISO-8859-1,utf-8;q=0.7,*;q=0.7
HTTP_ACCEPT_ENCODING = gzip,deflate
HTTP_ACCEPT_LANGUAGE = en-us,en;q=0.5
HTTP_CACHE_CONTROL = max-age=0
HTTP_CONNECTION = keep-alive
HTTP_COOKIE = __utma=229519070.1722601203.1290612113.1314835609.1314894735.54; __utmz=229519070.1310752647.40.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utma=175732791.35145943.1311275562.1314835597.1314894747.23; __utmz=175732791.1311275562.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); cookies=true; __utmb=229519070.1.10.1314894735; __utmc=229519070; __utmb=175732791.3.10.1314894747; __utmc=175732791
HTTP_HOST = staging.myapp.com
HTTP_KEEP_ALIVE = 115
HTTP_USER_AGENT = Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9.2.21) Gecko/20110830 Firefox/3.6.21
PATH = /usr/local/bin:/usr/bin:/bin
QUERY_STRING = 
REMOTE_ADDR = 000000000
REMOTE_PORT = 50836
REQUEST_METHOD = GET
REQUEST_URI = /env.cgi
SCRIPT_FILENAME = /home/staging/env.cgi
SCRIPT_NAME = /env.cgi
SERVER_ADDR = 0000000000
SERVER_ADMIN = webmaster@staging.myapp.com
SERVER_NAME = staging.myapp.com
SERVER_PORT = 80
SERVER_PROTOCOL = HTTP/1.1
SERVER_SIGNATURE = <address>Apache/2.2.3 (CentOS) Server at staging.myapp.com Port 80</address>

SERVER_SOFTWARE = Apache/2.2.3 (CentOS)

But I see no PERL5LIB variables from either the SetEnv or PerlSetEnv in either httpd.conf or .htaccess.

I ran the apachectl graceful. It just went to another line with no messages at all...so do I assume it worked and let me do it?

Thanks,
Brandon

[dsheroh]

But I see no PERL5LIB variables from either the SetEnv or PerlSetEnv in either httpd.conf or .htaccess.

Nor do I, and your custom module directory isn't showing in @INC, either. I'm afraid you'll have to either work with your hosting company to figure out how to get PERL5LIB set correctly for each vhost or go through all the files adding the appropriate "use lib" command.

I ran the apachectl graceful. It just went to another line with no messages at all...so do I assume it worked and let me do it?

Yeah, that's normal. apachectl doesn't normally print any output on the command line, but you can watch the apache error_log and it will log the message "SIGUSR1 received. Doing graceful restart" if you want to verify that it's doing something.

[Brandon Luhring]

OK, just to put this thread to rest...

I honestly don't know what ended up happening, but we eventually went back in to "punt" and create our own method of letting the perl5 location file serve as the class file for both staging and production environments.

In so doing, we ended up doing a last round of tests, and for whatever reason it appeared to be working. So we tested further, and ended up renaming the perl5 file and proving that it was no longer used. We deleted the file in the perl5 location all together, and now our site is running as we desired.

I'm thinking that we had some sort of caching problem before where our changes were made, but we weren't seeing the effect of them. Over time possibly this cleared itself out, and the above attempts actually worked.... That's my only assumption to go on...

Thanks for your help!

Brandon

[LesleyB]

Sunday sitepoint time !

1. Perl Module Mechanics was last edited in 2007. It is reasonable to consider it out of date by now. Stommespoes is right, you really should refer to perlmonks.org, stackoverflow.com and, one of my favourite reference sites, perl.org.

2. It would really help to understand your server arrangement better. From what you have written you seem to have access to the apache config files which implies, in the minimum, a virtual machine.

3. I have never had a problem using apache2ctl on a Debian server as an admin. You won't get far using it as a normal user. The startup script, /etc/init.d/apache, should also work but here's the options for apache2ctl

apache2ctl
Usage: /usr/sbin/apache2ctl start|stop|restart|graceful|graceful-stop|configtest|status|fullstatus|help
       /usr/sbin/apache2ctl <apache2 args>
       /usr/sbin/apache2ctl -h            (for help on <apache2 args>)

apache2ctl -h
Usage: /usr/sbin/apache2 [-D name] [-d directory] [-f file]
                         [-C "directive"] [-c "directive"]
                         [-k start|restart|graceful|graceful-stop|stop]
                         [-v] [-V] [-h] [-l] [-L] [-t] [-T] [-S] [-X]
Options:
  -D name            : define a name for use in <IfDefine name> directives
  -d directory       : specify an alternate initial ServerRoot
  -f file            : specify an alternate ServerConfigFile
  -C "directive"     : process directive before reading config files
  -c "directive"     : process directive after reading config files
  -e level           : show startup errors of level (see LogLevel)
  -E file            : log startup errors to file
  -v                 : show version number
  -V                 : show compile settings
  -h                 : list available command line options (this page)
  -l                 : list compiled in modules
  -L                 : list available configuration directives
  -t -D DUMP_VHOSTS  : show parsed settings (currently only vhost settings)
  -S                 : a synonym for -t -D DUMP_VHOSTS
  -t -D DUMP_MODULES : show all loaded modules 
  -M                 : a synonym for -t -D DUMP_MODULES
  -t                 : run syntax check for config files
  -T                 : start without DocumentRoot(s) check
  -X                 : debug mode (only one worker, do not detach)

All taken from a Debian server.
Historical note: back in the day when Apache 2.x came out, /usr/sbin/apachectl and /usr/bin/apache2ctl both existed independently on Debian systems. One restarted the Apache 2 server, and the other any Apache 1.x server. The Apache server on a Debian machine is still identified as apache2 in a process listing, and I think the Apache 1.x were identified by http. Now, /usr/sbin/apachectl, on my systems, at least, is linked to /usr/bin/apache2ctl but I have got into the habit of using /usr/sbin/apache2ctl directly.

4. When you make changes to your apache config files, use

apache2ctl -t

to check the config files are at least syntactically correct before restarting your server. It should come up with either an error message or 'Syntax OK'.

5. On my system, the graceful and restart options do not change the process ID number of the apache process. The stop andn start approach does. I can detect what is happening by looking at /var/log/apache2/error.log. (In which we'll all notice that FastCGI perhaps isn't so graceful !)

[Sun Nov 20 20:52:05 2011] [notice] Graceful restart requested, doing restart
[Sun Nov 20 20:52:05 2011] [alert] (2)No such file or directory: FastCGI: read() from pipe failed (0)
[Sun Nov 20 20:52:05 2011] [alert] (2)No such file or directory: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye
[Sun Nov 20 20:52:05 2011] [notice] FastCGI: process manager initialized (pid 562)
[Sun Nov 20 20:52:05 2011] [notice] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
[Sun Nov 20 20:52:05 2011] [notice] mod_python: using mutex_directory /tmp
[Sun Nov 20 20:52:05 2011] [notice] Apache/2.2.21 (Debian) mod_fastcgi/2.4.6 PHP/5.3.8-1 with Suhosin-Patch mod_python/3.3.1 Python/2.7.2+ mod_perl/2.0.5 Perl/v5.12.4 configured -- resuming normal operations
[Sun Nov 20 20:52:28 2011] [notice] SIGHUP received.  Attempting to restart
[Sun Nov 20 20:52:28 2011] [notice] FastCGI: process manager initialized (pid 581)
[Sun Nov 20 20:52:28 2011] [notice] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
[Sun Nov 20 20:52:28 2011] [notice] mod_python: using mutex_directory /tmp
[Sun Nov 20 20:52:28 2011] [notice] Apache/2.2.21 (Debian) mod_fastcgi/2.4.6 PHP/5.3.8-1 with Suhosin-Patch mod_python/3.3.1 Python/2.7.2+ mod_perl/2.0.5 Perl/v5.12.4 configured -- resuming normal operations
[Sun Nov 20 20:52:39 2011] [notice] caught SIGTERM, shutting down
[Sun Nov 20 20:52:48 2011] [notice] FastCGI: process manager initialized (pid 599)
[Sun Nov 20 20:52:48 2011] [notice] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
[Sun Nov 20 20:52:48 2011] [notice] mod_python: using mutex_directory /tmp
[Sun Nov 20 20:52:48 2011] [notice] Apache/2.2.21 (Debian) mod_fastcgi/2.4.6 PHP/5.3.8-1 with Suhosin-Patch mod_python/3.3.1 Python/2.7.2+ mod_perl/2.0.5 Perl/v5.12.4 configured -- resuming normal operations

The last part shows the effect of apache2ctl stop followed by the effect of apache2ctl start command.

Here endeth the apache lesson - possibly

6a.. Your original setup looks like it used mod_perl. This is okay. mod_perl is cool but it is rare to have it unless you have a sysadmin that knows what they are doing.
6b. Your current arrangement may not have mod_perl installed or may be configuring it differently.
7. Your current arrangement at least has a user 'staging'. I deduce that from

DOCUMENT_ROOT = /home/staging

8. http://www.perlmonks.org/?node_id=867860 has a fairly recent discussion on PERL5LIB. Particularly note the fourth reply, from toolic

perl does not require that the PERL5LIB environment variable be set. Typically, the variable will not be set to any value.
One common usage scenario is to set the variable to a directory which contains Perl modules that you have developed.

And I have just noticed the OP has fixed the problem according to his satisfaction
I should wake up and read the full thread!

I'll post this anyway in case it helps anyone else looking here.

[Stomme poes]

While there aren't too many "Debian-encumbered" Apache users (Rich Bowen's term, lawlz) in the Apache section, you should check it out regardless. We needs m0aR Apache lessons over there. :)