Troubleshooting

Site logs

Server access and error logs are available to the site owner on Troubleshooting link, under the Advanced section.

Select Type: Error to view the error logs.

HTTP Error 403

If the EOS site returns a 403 - Forbidden error, this is usually due to the following reasons:

1. double-check that the EOS folder is shared with a:wwweos. See point 3 of the prerequisites.
2. this error also happens if there is no default page in the site's root folder. Solution: add an index.html file, or enable Directory Browsing to auto-generate a page that lists folder contents.
3. Web Application Firewall identified the request as a threat and blocked it. Requests blocked by Web Application Firewall

Common problems and resolutions

Description of common problems encountered by websites after the migration to the new infrastructure and their solutions are available at the dedicated Discourse topic in the Web hosting and development category.

Some common problems seen with web sites migrated from AFS:

php_value in .htaccess

PHP settings (php_value, php_admin_value and php_flag) currently set in .htaccess will stop working and should be moved to a .user.ini file instead: https://www.php.net/manual/en/configuration.file.per-user.php.

When accessing the preview of your website, you'll get a 500 - Internal server error:

In the error logs of your website, the error will be similar to:

[test-dchatzic-webeos-php.web.cern.ch] [Fri Oct 20 13:52:23.461941 2023] [alert] [pid 128804] config.c(2045): [client 2a01:cb15:31a:4600:a882:dae8:9bc3:4eb4:0] /eos/user/d/dchatzic/tests/www/.htaccess: Invalid command 'php_value', perhaps misspelled or defined by a module not included in the server configuration

Example of an .htaccess file with PHP settings:

php_value max_input_vars 9000
php_value max_execution_time 60

To fix the error, you need first to remove the lines with php_value in the .htaccess file and move them in a .user.ini file:

max_input_vars=9000
max_execution_time=60

Shebang in Python 2 scripts

Python 2 scripts using #!/usr/bin/python shebang will fail on execution.

When accessing the preview of your website, you'll get a 500 - Internal server error:

In the error logs of your website, the error will be similar to:

[test-dchatzic-webeos-php.web.cern.ch] [Fri Oct 20 14:38:17.760238 2023] [error] [pid 130577] mod_cgid.c(1196): [client 2a01:cb15:31a:4600:a882:dae8:9bc3:4eb4:0] AH01241: error spawning CGI child: exec of '/eos/user/d/dchatzic/tests/www/cgi-bin/hello.py' failed (No such file or directory): /eos/user/d/dchatzic/tests/www/cgi-bin/hello.py
[test-dchatzic-webeos-php.web.cern.ch] [Fri Oct 20 14:38:17.761481 2023] [error] [pid 130577] util_script.c(501): [client 2a01:cb15:31a:4600:a882:dae8:9bc3:4eb4:0] End of script output before headers: hello.py

To fix this, you need to replace #!/usr/bin/python with #!/usr/bin/python3 on the top of your python scripts.

create_function in PHP scripts

create_function has been removed in PHP 8: https://www.php.net/manual/en/function.create-function.php.

Example of create_function usage in a php script:

create_function('$a,$b', 'return filemtime($b) - filemtime($a);');

In the error logs of your website, the error will include:

Got error 'PHP message: PHP Fatal error: Uncaught Error: Call to undefined function create_function()

To fix the error, you need to define an anonymous function instead:

function($a,$b) {return filemtime($b) - filemtime($a);};

Undefined arguments in PHP function with type declarations

Passing undefined arguments in a PHP function can cause TypeError: https://www.php.net/manual/en/language.types.declarations.php.

An example is provided that the argument of the function is expected to be of type array. However, we are giving the variable $b as an input, which is undefined:

<?php
function f(array $a) {
 // function implementation
}
f($b); // call the function
?>

In the error logs of your website, the error will include:

PHP Fatal error:  Uncaught TypeError: f(): Argument #1 ($b) must be of type array, null given

To fix the error, we need to define the variables before passing them to the corresponding functions. In the example, this could be done by defining $b to an empty array:

<?php
$b = []; // define $b
f($b);

PHP scripts in CGI-enabled folders

Example of non-functional .htaccess file in CGI-enabled folder:

# *** This htaccess file has been generated automatically on November 24, 2020 ***
# Due to the end-of-life of SLC6 and Apache 2.2, web sites get migrated to a new infratructure running
# CC7 and Apache 2.4. Some settings previously managed in the old webservices site management interface
# have been moved to .htaccess files.
# See https://cern.service-now.com/service-portal/?id=outage&n=OTG0059516

<IfVersion >= 2.3>

# Enable CGI scripts execution in this folder in the new Apache 2.4 infrastructure.
# This folder was previously CGI-enabled using a setting in the old webservices site management interface.
Options +ExecCGI
SetHandler cgi-script
# The old infrastructure supported php scripts in CGI-enabled folders, preserve that behavior for migrated sites.
<FilesMatch \.php$>
SetHandler application/x-httpd-php
</FilesMatch>


</IfVersion>

The above configuration should be simplified into:

Options +ExecCGI
SetHandler cgi-script

without special directives for PHP files since the current httpd configuration in EL9 supports the execution of PHP scripts in CGI-enabled folders.

Requests blocked by Web Application Firewall

When ModSecurity blocks a legitimate request (a false positive), use this guide to investigate and fix the issue using logs from your web services portal. This will be translated in 403 responses from the webeos server.

1. Identify the Blocked Request

• Log into your web services portal
• Click on "View Site Logs" in the Advanced -> Troubleshooting section
• Select Type to "Error"
• Log entries with pattern COMBINED_SCORE=X where X is greater than 5 will be effectively blocked.

2. Examine the Log Details

Example log entry:

[webeos-dev.web.cern.ch] [Wed Apr 23 08:56:26.801405 2025] [error] [pid 1973761] apache2_util.c(271): [client 2001:1458:204:1::101:a647:0] [client 2001:1458:204:1::101:a647] ModSecurity: Warning. Unconditional match in SecAction. [file "/etc/crs4/rules/RESPONSE-980-CORRELATION.conf"] [line "98"] [id "980170"] [msg "Anomaly Scores: (Inbound Scores: blocking=14, detection=14, per_pl=14-0-0-0, threshold=5) - (Outbound Scores: blocking=0, detection=0, per_pl=0-0-0-0, threshold=4) - (SQLI=0, XSS=0, RFI=0, LFI=12, RCE=2, PHPI=0, HTTP=0, SESS=0, COMBINED_SCORE=14)"] [ver "OWASP_CRS/4.7.0"] [tag "reporting"] [tag "OWASP_CRS"] [hostname "jbreinli-webeos-dev.web.cern.ch"] [uri "/index.html"] [unique_id "aAiPGrO78kToItAXqiIqjQAAAEM"] `

Request analysis

• The COMBINED_SCORE is the total of all rule scores that matched the request.
• The unique_id helps identify all rules that matched the specific request.
• This request will be blocked because COMBINED_SCORE is greater than 5

Search for additional related entries using the unique_id:

[jbreinli-webeos-dev.web.cern.ch] [Wed Apr 23 08:56:26.787094 2025] [error] [pid 1973761] apache2_util.c(271): [client 2001:1458:204:1::101:a647:0] [client 2001:1458:204:1::101:a647] ModSecurity: Warning. Matched phrase "bin/passwd" at ARGS:exec. [file "/etc/crs4/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "615"] [id "932160"] [msg "Remote Command Execution: Unix Shell Code Found"] [data "Matched Data: bin/passwd found within ARGS:exec: ../../cat_/bin/passwd"] [severity "CRITICAL"] [ver "OWASP_CRS/4.7.0"] [tag "application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag "attack-rce"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/88"] [tag "PCI/6.5.2"] [hostname "jbreinli-webeos-dev.web.cern.ch"] [uri "/"] [unique_id "aAiPGrO78kToItAXqiIqjQAAAEM"]

We can clearly see a potential Remote Code Execution attack

3. Fix false positives

• Understand the request: Was this a normal user action that was misinterpreted?
• Evaluate the context: Does the input genuinely pose a threat, or does it just resemble attack patterns?
• Update your application code:
• Sanitize inputs.
• Avoid sending suspicious patterns (e.g., bin/passwd, <script>, or SQL keywords).
• Use POST instead of GET when sending structured or sensitive data.

4. Common Scenarios Where WAF Can Detect a False Positive

• SQL keywords (SELECT, DROP) anywhere in the request, like query parameters, cookies or headers or data
• Scripting tags (<script>) anywhere in the request
• Encoded payloads (%3Cscript%3E)UST** make anywhere in the request
• Avoid Sending Raw JSON in Query Strings