1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Nginx reverse proxy microcaching not working

Discussion in 'Nginx' started by mumer91, Feb 19, 2019.

  1. #2
    My current config is Nginx webserver as a reverse proxy with Apache at backend for dynamic pages and Apache is configured with PHP-FPM.

    My dynamic pages don't change often so I am trying to use Nginx microcaching but can't seem to get working.

    Here is my nginx.conf (possible unnecessary stuff excluded):

    http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    client_header_timeout 10s;
    client_body_timeout 10s;
    client_max_body_size 200k;
    client_header_buffer_size 2k;
    client_body_buffer_size 200k;
    large_client_header_buffers 3 1k;
    send_timeout 3m;
    keepalive_timeout 60 60;
    reset_timedout_connection on;
    ignore_invalid_headers on;
    connection_pool_size 256;
    request_pool_size 4k;
    output_buffers 4 32k;
    postpone_output 1460;
    
    include mime.types;
    default_type application/octet-stream;
    
    # Compression gzip
    gzip on;
    gzip_vary on;
    gzip_disable "MSIE [1-6]\.";
    gzip_proxied any;
    gzip_min_length 512;
    gzip_comp_level 6;
    gzip_buffers 8 64k;
    gzip_types text/plain text/xml text/css text/js application/x-javascript application/xml image/png image/x-icon image/gif image/jpeg image/svg+xml application/xml+rss text/javascript application/atom+xml application$
    
    # Proxy settings
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    #proxy_pass_header Set-Cookie;
    proxy_connect_timeout 90;
    proxy_send_timeout 90;
    proxy_read_timeout 90;
    proxy_buffers 32 4k;
    
    # SSL Settings
    ssl_session_cache shared:SSL:10m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA!RC4:EECDH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS";
    
    # Logs
    log_format bytes '$body_bytes_sent';
    #access_log /var/log/nginx/access.log main;
    access_log off;
    
    ## Advanced microcaching
    ## set cache dir, hierarchy, max key size and total size
    fastcgi_cache_path /dev/shm/microcache levels=1:2 keys_zone=microcache:50M max_size=2000M inactive=2h;
    
    ## set cache log location, so you can evaluate hits
    log_format cache '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $upstream_cache_status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';
    
    ## Set a cache_uid variable for authenticated users.
    map $http_cookie $cache_uid {
    default nil;
    ~SESS[[:alnum:]]+=(?<session_id>[[:alnum:]]+) $session_id;
    }
    
    ## mapping cache to request method
    map $request_method $no_cache {
    default 1; # by default do not cache
    HEAD 0; # cache HEAD requests
    GET 0; # cache GET requests
    }
    
    # Include additional configuration
    include /etc/nginx/cloudflare.inc;
    include /etc/nginx/conf.d/*.conf;
    }
    Code (markup):
    My vhost config (relevant stuff only) is:

    location / {
    location ~.*\.(3gp|gif|jpg|jpeg|png|ico|wmv|avi|asf|asx|mpg|mpeg|mp4|pls|mp3|mid|wav|swf|flv|html|htm|txt|js|css|exe|zip|tar|rar|gz|tgz|bz2|uha|7z|doc|docx|xls|xlsx|pdf|iso|woff|ttf|svg|eot|sh)$ {
    root /home/xtrefish/public_html;
    expires max;
    try_files $uri @backend;
    }
    
    error_page 405 = @backend;
    error_page 500 = @custom;
    add_header X-Cache "HIT from Backend";
    add_header Strict-Transport-Security "max-age=31536000";
    add_header X-Content-Type-Options nosniff;
    proxy_pass IPADDRESS:8181;
    include proxy.inc;
    }
    
    location @backend {
    internal;
    proxy_pass IPADDRESS:8181;
    include proxy.inc;
    }
    
    location @custom {
    internal;
    proxy_pass IPADDRESS:8181;
    include proxy.inc;
    
    location @custom {
    internal;
    proxy_pass IPADDRESS:8181;
    include proxy.inc;
    }
    
    location ~ .*\.(php|jsp|cgi|pl|py)?$ {
    
    ## Caching stuff is here
    # Setup var defaults
    set $no_cache "";
    # If non GET/HEAD, don't cache & mark user as uncacheable for 1 second via cookie
    if ($request_method !~ ^(GET|HEAD)$) {
    set $no_cache "1";
    }
    # Drop no cache cookie if need be
    # (for some reason, add_header fails if included in prior if-block)
    if ($no_cache = "1") {
    add_header Set-Cookie "_mcnc=1; Max-Age=2; Path=/";
    add_header X-Microcachable "0";
    }
    # Bypass cache if no-cache cookie is set
    if ($http_cookie ~* "_mcnc") {
    set $no_cache "1";
    }
    
    # Bypass cache if flag is set
    fastcgi_no_cache $no_cache;
    fastcgi_cache_bypass $no_cache;
    fastcgi_cache microcache;
    fastcgi_cache_key $scheme$host$request_uri$request_method;
    fastcgi_cache_valid 200 301 302 10m;
    fastcgi_cache_use_stale updating error timeout invalid_header http_500;
    fastcgi_pass_header Set-Cookie;
    fastcgi_pass_header Cookie;
    fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
    
    proxy_pass IPADDRESS:8181;
    include proxy.inc;
    }
    Code (markup):
    I can confirm I am using PHP-FPM as info.php responds

    Server API FPM/FastCGI
    Code (markup):
    To test this, I have a microcache_test.php containing

    <?php
    /**
    * Display microtime 
    * Requires PHP 5, preferably PHP 5.4+
    * This function is only available on operating systems that 
    * support the gettimeofday() system call. 
    * 
    * Returns float in format seconds.microseconds
    */
    echo ' Current microtime: ' . microtime(true);
    Code (markup):
    And /microcache_test.php responds:

    Request URL: https://example.com/microcache_test.php
    Request Method: GET
    Status Code: 200 
    Remote Address: 82.102.16.22:443
    Referrer Policy: no-referrer-when-downgrade
    cache-control: public
    content-encoding: gzip
    content-type: text/html; charset=UTF-8
    date: Tue, 19 Feb 2019 09:13:58 GMT
    server: nginx/1.15.8
    status: 200
    vary: Accept-Encoding
    :authority: example.com
    :method: GET
    :path: /mictest.php
    :scheme: https
    accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
    accept-encoding: gzip, deflate, br
    accept-language: en-US,en;q=0.9
    cache-control: max-age=0
    cookie: PHPSESSID=864e0ajkc6l1rbhepr651dp1t2
    save-data: on
    upgrade-insecure-requests: 1
    user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36
    Code (markup):
    So caching is certainly not working. My caching directories are 777 and I tried setting cache in htaccess in apache as well but to no avail (Nginx is set to ignore the headers anyway).

    Any ideas what am I missing?
     
    Last edited: Feb 20, 2019
    mumer91, Feb 19, 2019 IP
  2. bountysite

    bountysite Active Member

    Messages:
    71
    Likes Received:
    4
    Best Answers:
    1
    Trophy Points:
    73
    #3
    Your test file is /microcache_test.php but the request is for /mictest.php. I am guessing they are the same files.
    Your configs are hard to read. Put it inside code.

    You are trying bypass caching on:-
    • POST request - when you are updating something, you would obviously want the modified(which is not cached content)
    • If _mcnc cookie(2 seconds expiry) is found
    My question would be, does your POST request redirect to a GET request of the same page or does POST request directly render the modified page?
    If POST redirects to GET, then you need to set cookie to bypass caching. Just want to confirm if this is the case.

    Assuming POST redirects to GET to fetch new contents.
    
    set $no_cache "0" ;
    if ( $request_method = POST  ) {
        add_header Set-Cookie "_mcnc=1; Max-Age=3; Path=/";
    }
    # Bypass cache if no-cache cookie is set
    if ($http_cookie ~* "_mcnc") {
       set $no_cache "1";
    }
    
    
    Code (ApacheConf):

    If POST method also renders content then you dont need to maintain a cookie.
    
    set $no_cache "0";
    if ( $request_method = POST  ) {
        set $no_cache "1";
    }
    
    
    Code (ApacheConf):

    If the above is not relevant, then
    • start stripping conditions and enable caching for all requests
    • then add condition for cache bypass
    • use httpie to troubleshoot. Showing header values are irrelevant for readers
    • try changing configs one by one and see results on httpie and do it methodically
     
    bountysite, Feb 19, 2019 IP
  3. mumer91

    mumer91 Peon

    Messages:
    2
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #4
    Thanks for your response.

    Actually I would also like to cache POST along with GET (for now at least).
    Can you suggest caching parameters that will just microcache everything regardless of cookie or cache-headers?
     
    mumer91, Feb 20, 2019 IP
  4. bountysite

    bountysite Active Member

    Messages:
    71
    Likes Received:
    4
    Best Answers:
    1
    Trophy Points:
    73
    #5
    • First remove all ifs and start with minimal approach
    • I have never used dynamic content caching. You will have to research on own.
    • This link has some info to start from scratch
     
    bountysite, Feb 20, 2019 IP
  5. hostechsupport

    hostechsupport Well-Known Member

    Messages:
    413
    Likes Received:
    23
    Best Answers:
    7
    Trophy Points:
    138
    #6
    Make sure you have this cache configuration in nginx config..
    proxy_cache_path /tmp/cache levels=1:2 keys_zone=default_cache:10m max_size=2g inactive=120m use_temp_path=off; proxy_cache_key "$scheme$request_method$host$request_uri"; proxy_cache_valid 200 302 60m;
     
    hostechsupport, Feb 21, 2019 IP