Activities

December 2013
M T W T F S S
« Nov   Jan »
 1
2345678
9101112131415
16171819202122
23242526272829
3031  

Optimizing Magento hosting and Boosting performance

Recently I had to work with a eCommerce web application which sells products by showing it’s images. This will be hosted on a heavy duty server where there are many admins/suppliers are adding products in back end to this system. This app uses AWS service.

After spending/testing in Goggling few days and hours in b, I log eating got lots of information to boost the site performance. Simply you can not run the Magento application on 2GB RAM. Lots of links have created a confusion where do I start. Obviously only a programmer can identify the application area where performance can be drastically improved and only Sys-Admin can say how to make it faster in Internet cloud 🙂 sadly.

So definitely an admin requires a back end support to do this :-). Before touching the config. you need to check te existing site performance with http://gtmetrix.com/

Server infrastructure :

1. EC2 Webserver (c3.large) : 1 Nos
2. RDS DB server ( c1.medium)
3. Amazon S3 bucket : 1
4. Cloudfront
5. SES for smtp.

The Approach I had taken,

1. Use cloudfront CDN service to distribute the images under cache folder. Expiry headers and Cache expiry time meta data are set with each S3 image objects while uploading. Synchronizing “cache” folder from media folder to CDN through a bash script.

2. Install and configure Varnish cache.

3. Use Turpentine (Magento plugins for Varnish Cache ) to create/manage ACL for varnish config file.
4. Using amazon S3 and CDN service (OnePica ImageCDN).Product image will automatically uploaded to S3 and then service it through Cloudfront CDN.

5. Use High CPU capacity instance (C3.large) which driven on SSD drive comes up with moderate I/O. Cost will be $109/Mon.

6. Set expiry headers and enabled gzip compression for JS/CSS scripts those are serve from web server. Merging jss to a single file a good idea. But loading them all on landing page is a bad idea. So landing page should be load fast and use the js/css only required to load that page.

7. Use dedicated hardware for Database. I uses RDS service for MySQL database.

8. Enable APC cache in PHP.

9. Optimize Apache settings for Magento.

############# Descriptions and usage ##################

1. Use cloudfront CDN service to distribute the images under cache folder.

I have created new s3 bucket and then formed cloudfront service associated with it. Also create new CNAME for accessing CDN url. like media.mydomain.com. Also I set a cronjob to synchronize the files under “media/catalog/product/cache” as well as /home/youspred/public_html/media/avatar/. Based on the discussion with devop, I noticed thatonly cache folder need to move to CDN since all the files are displayed from it.

Here is my S3 command which I set for mirroring cache folder to Amazon S3 on every 10 mts interval. Also I did the same for “avatar” folder too using the same script file.

/usr/bin/s3cmd  sync  --recursive  setacl --acl-public --recursive  --add-header="Expires:`date -u +"%a, %d %b %Y %H:%M:%S GMT" --date "+4 years"`" --add-header='Cache-Control:max-age=31536000, public'  /home/domain/public_html/media/catalog/product/cache/ s3://domain.com/catalog/product/cache/

2. Install and configure Varnish cache

This is the major brake point of boosting the site performance I believe. I installed varnish simpy using yum command.

[root@mail ]# yum install varnish
Dependencies Resolved
==================================================================================================================================================
 Package                             Arch                        Version                                Repository                           Size
==================================================================================================================================================
Installing:
 varnish                             i686                        3.0.3-1.3.amzn1                        amzn-updates                        348 k
Installing for dependencies:
 jemalloc                            i686                        3.0.0-2.3.amzn1                        amzn-main                            89 k
 varnish-libs                        i686                        3.0.3-1.3.amzn1                        amzn-updates                        171 k
Transaction Summary
==================================================================================================================================================
Install       3 Package(s)
Upgrade       0 Package(s)
Total download size: 607 k
Installed size: 1.3 M
Is this ok [y/N]:

Configuration files for Varnish

1. /etc/varnish/default.vcl (cache control file)
2. /etc/sysconfig/varnish ( varnish server config file)

Configure Varnish for your environment,

a. Switch Apache to listen on 8000 port
First you need to change the Apache Lister port and NameVirtualHost entry to become another port. Say 8000

After changing the settings my httpd.conf file will have the following entry.

a.NameVirtualHost *:8000
b. Listen *:8000
c. Updated VirtaulHost config file

Note : Ensure the your site displayed correctly on the port 8000. Don’t allow this port has public access.

  <VirtualHost *:8000>
ServerName uat.domain.com
DocumentRoot /home/domain/public_html
DirectoryIndex index.php index.html
<Directory /home/domain/public_html/>
Options -Indexes IncludesNOEXEC FollowSymLinks SymLinksIfOwnerMatch
 AllowOverride All
</Directory>
</VirtualHost>

2. Update the Varnish config settings

Now you need to update the varnish config file (/etc/sysconfig/varnish) and change the following variables.

VARNISH_LISTEN_ADDRESS=0.0.0.0
VARNISH_LISTEN_PORT=80

Varnish config file can be found here

3. Restarting Apache and Varnish

[root@rc-026 ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
[root@sf-qa01 ~]# service varnish restart
Stopping Varnish Cache:                                    [  OK  ]
Starting Varnish Cache:                                    [  OK  ]
[root@sf-qa01 ~]#

b. Installing Turpentine Magento plugin

This is a very good magento plug-ins which is designing Varnish ACL list. ie what are the urls to be cached or not cached. Especially anything under “/admin” shall not be cached by Varnish. Once you can able to install this plug-in on Magento, you can see a menu “Turpentine” under system main menu.

You can see 2 main menu under this plugin, Varnish options and Cache options.

Varnish options : Information about the Varnish server, admin port and secret key. you can see the secret key under “/etc/varnish/secret” file.

1

Cache Options : Information about back end server port and adjusting ACL parameters and timed out values.

2

Cool ! I see whenever we change varnish options, it will create a file “default.vcl” under var folder. So what I did was I have created a symbolic link to this file from original config location to become Varnish to use the latest setting set by the Turpentine plug-ins.

[root@sf-host1 ~]#mv /etc/varnish/default.vcl /etc/varnish/default.vcl-bk
[root@sf-host1 ~]#ln -s /home/domain.com/public-html/var/default.vcl /etc/varnish/default.vcl
cat /dev/null > /var/lib/varnish/varnish_storage.bin
[root@sf-host1 ~]# service varnish restart

You can see the default varnish settings here

5.5 Installing OnePica ImageCDN from Magento Connect

This is one of the Awsome plugin which can speedup your store loading. This plugin uses to upload all the images those are cached are uploaded to Amazon S3 instantly. All these uploaded images shared over CDN url. This makes your platform loading quickly.

As you are using AWS hosting, I would like to choose this option because of,

1. You product images will be loaded faster on diff. geographical location since it is distributed over AWS cloudfront CDN service.
2. Can choose low server configurations like single core with 4G RAM . This CDN service will free up the server resource not to stream the binary files over Apache. Because Apache has to utilize the same resource for serving an image each and every time.
3. Reduce the server cost by choosing lesser resource config.
4. Easy setup.

a. How do I create Amazon S3 user
a. Open Amazon Account
b. Create IAM account for OnePica ImageCDN
c. Apply full permission to both S3 and CDN to this IAM user.
IAM user credential like this

Access Key ID: AKIAYTGJGJHHKHJLQSMCQ
Secret Access Key: 7p3FKsMGHGJGHHHGUbuhFXo4RJZmfU

b. Enable Image CD plugin
1. Go to settings –> choose “image CDN” from the left side on the menu tree”
2. Select the Current Adapter to Amazon S3/CDN
3. Provide the bucket name
4. Provide your Access Key and Secret Key
5. Provide the Image non-secure (http) CDN url
6. provide Image secure (https) CDN url
7. Then Save settings.

clound_front_config

Image_CDN-config

c. Update the Magento Image url from Web settings

Login to Admin account and navigate to “settings –> Web and update the “URL to media File” with your CDN location. See the image below,
magento-main-settings

6. Installing APC cache

This is one easy way to accomplish. You can install it by “yum install php-apc” in centOS platform. APC config. file will be /ect/php.d/apc.ini”. We need to modify the apc default value with as showing below,

Edit the file “/etc/php.d/apc.ini”

 extension = apc.so
    apc.enabled = 1
    apc.shm_segments = 1
    apc.shm_size = 256M
    apc.num_files_hint = 10000
    apc.user_entries_hint = 10000
    apc.max_file_size = 5M
    apc.optimization = 0
    apc.ttl = 0
    apc.user_ttl = 0
    apc.gc_ttl = 600
    apc.cache_by_default = 1
    apc.filters = "apc\.php$"
    apc.slam_defense = 0
    apc.use_request_time = 1
    apc.mmap_file_mask = /tmp/apc-dummy.XXXXXX
    apc.file_update_protection = 2
    apc.enable_cli = 0
    apc.stat = 1
    apc.write_lock = 1
    apc.report_autofilter = 0
    apc.include_once_override = 0
    apc.rfc1867 = 0
    apc.rfc1867_prefix = "upload_"
    apc.rfc1867_name = "APC_UPLOAD_PROGRESS"
    apc.rfc1867_freq = 0
    apc.localcache = 1
    apc.localcache.size = 512
    apc.coredump_unmap = 0
    apc.stat_ctime = 0

You can get the php file apc.php from APC source code to monitor the php file caching statistics through browser. See the sample screen below,

apc-pics

Note: You need to enable APC in app/etc/local.xml.additional under config > global > cache > backend:

7. Apache optimization

a. Add or update the following values in your “httpd.conf” file.

 KeepAlive On
   KeepAliveTimeout 2
   ServerTokens Prod
   ServerSignature Off
   TraceEnable off
   Timeout 60
   php_admin_value open_basedir none

2. Choose appropriate MPM and Worker properties on Apache main config. file.

MaxClients : (Total Memory – Critical Services Memory) / Size Per Apache process

Size of each apache process can be calculated by using a small script(taken from another site)

Create a bash script named as “process_mem_calc

#!/bin/bash
ps -C $1 -O rss | gawk '{ count ++; sum += $2 }; END {count --; print "Number of processes =",count; print "Memory usage per process =",sum/1024/count, "MB"; print "Total memory usage =", sum/1024, "MB" ;};'

Usage : sh process_mem_calc

[root@sf-host1 ~]# sh /home/installation/scripts/mem_check_process.sh httpd
Number of processes = 18
Memory usage per process = 82.7114 MB
Total memory usage = 1488.8 MB
[root@sf-host1 ~]#

The above output shows apache took 83Mb for each process out of 18 Nos.

Server memory : 3.7GB
OS required : 700MB
Varnish required : 100MB
Other process : 200MB
Minimal RAM required for system : 1GB
So available memory for Apache : 2.7GB
Each Apache process required : 83Mb

Value for MaxClients is,

MaxClients = (3.7GB -1GB)/83Mb = 32 Nos.

<IfModule prefork.c>
StartServers       10
MinSpareServers    5
MaxSpareServers    50
ServerLimit       256
MaxClients        256
MaxRequestsPerChild  800
</IfModule>

Worker properties.

<IfModule worker.c>
StartServers         4
MaxClients           400
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

You can test these above memfork values are property adjusted to your server configuration by running a storm test using Apache Benchmark tool.

# ab -k -n 10000 -c 100  http://www.mydomain:8000/db_test.php

See how fast the response are receiving in ab tool and check the load status using top command. You can see how many worker process are created using the following commands

[root@sf-host1 ~]#  ps -ef | grep -i http | awk '{print $1}' | wc -l
52
[root@sf-host1 ~]# ps -ylC httpd --sort:rss | wc -l
52

Securing public writable folders protected from hacking.

Magento uses var and media folders to be writable by the web server. The following lines would help not to execute any php files from these folders suppose a hacker can create a file inside it through script injection.

php-security

The best Virtualhost settings can be downloaded here

Now check the site status using gtmatrix.com site and see the results.

Hope you have enjoyed my X’Mas day special blog 🙂

2 comments to Optimizing Magento hosting and Boosting performance

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>