某nginx+php fcgi服务器优化记录

上一篇 / 下一篇 2008-05-23 00:11:31 / 个人分类：LAMP

服务器梳理及优化记录


通过apachetop发现两台服务器负载不均： A 280-295/s
                                                                            B 310-320/s


一、初步优化：
        nginx：keepalive=0
        系统： net.ipv4.tcp_max_tw_buckets = 5000

        系统情况变化情况：
        1. [root@xx_xx_API_37_42 <37.42>]: /data1/nginx
           0> netstat -ant | awk '{A[$6]++}END{ for (a in A){print a,A[a]}}'
       LAST_ACK 19
       LISTEN 7
       SYN_RECV 76
       ESTABLISHED 3781
       established) 1
       FIN_WAIT1 129
       FIN_WAIT2 4876
       Foreign 1
       CLOSING 3
       TIME_WAIT 6098

                       PM    totsck    tcpsck    udpsck    rawsck   ip-frag
         4:50:01 PM      4306      9331         3         0         0
         05:00:01 PM      4030      8968         3         0         0
       05:10:01 PM      3988      8596         3         0         0

       Cpu(s): 21.2% us, 2.8% sy, 0.0% ni, 75.8% id, 0.2% wa, 0.0% hi, 0.0% si

      修改后变化：
          0> !net
        netstat -ant | awk '{A[$6]++}END{ for (a in A){print a,A[a]}}'
        LAST_ACK 2
        LISTEN 7
        SYN_RECV 75
        ESTABLISHED 217
        established) 1
        FIN_WAIT1 80
        FIN_WAIT2 162
        Foreign 1
        CLOSING 56
        TIME_WAIT 5014

        03:00:01 PM    totsck    tcpsck    udpsck    rawsck   ip-frag
        05:00:01 PM      4030      8968         3         0         0
        05:10:01 PM      3988      8596         3         0         0
        05:20:01 PM        47      1865         3         0         0
        05:30:01 PM       410      2548         3         0         0
        05:40:01 PM       400      1388         3         0         0
        05:50:01 PM       401       499         3         0         0
        06:00:01 PM       407       507         3         0         0

        2.2% syCpu(s): 17.3% u, 0.0% ni, 80.3% id, 0.2% wa, 0.0% hi, 0.0% si

      结果：负载降低5%左右。
      原则：apachetop显示的每秒处理请求数与连接数相差不多，连接数更少一些（由后面ab测试结果得知每次请求消耗的时间小于1s，established的数量严重大于ngxin正在处理的量，是不合理的）

二、性能测试及深入优化
        ab 测试：
        [root@xx_xx_API_37_43 <37.43>]: ~
    130> ab -n 100000 -c 200   http://xx.xx.37.42/myblog/xxxxxxxwd_image.php
    This is ApacheBench, Version 1.3d <$Revision: 1.73 $> apache-1.3
    Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Copyright (c) 2006 The Apache Software Foundation, http://www.apache.org/

    Benchmarking xx.xx.37.42 (be patient)
    Completed 10000 requests
    Completed 20000 requests
    Completed 30000 requests
    Completed 40000 requests
    Completed 50000 requests
    Completed 60000 requests
    Completed 70000 requests
    Completed 80000 requests
    Completed 90000 requests
    Finished 100000 requests
    Server Software:        nginx/0.5.35
    Server Hostname:        xx.xx.37.42
    Server Port:            80

    Document Path:          /myblog/xxxxxxxwd_image.php
    Document Length:        575 bytes

    Concurrency Level:      200
    Time taken for tests:   129.500 seconds
    Complete requests:      100000
    Failed requests:        97359
       (Connect: 0, Length: 97359, Exceptions: 0)
    Broken pipe errors:     0
    Total transferred:      82307953 bytes
    HTML transferred:       57107953 bytes
    Requests per second:    772.20 [#/sec] (mean)
    Time per request:       259.00 [ms] (mean)
    Time per request:       1.29 [ms] (mean, across all concurrent requests)
    Transfer rate:          635.58 [Kbytes/sec] received

    Connnection Times (ms)
                  min mean[+/-sd] median   max
    Connect:        1   124   29.3    117 3142
    Processing:    29   135   17.7    129   775
    Waiting:       28   134   17.7    128   775
    Total:         29   259   39.0    243 3305

    Percentage of the requests served within a certain time (ms)
      50%    243
      66%    274
      75%    290
      80%    295
      90%    305
      95%    310
      98%    315
      99%    318
     100%   3305 (last request)

     cpu性能：
     Cpu(s): 50.4% us, 6.6% sy, 0.0% ni, 42.9% id, 0.2% wa, 0.0% hi, 0.0% si

     分析：
     如上面数据：
             ab每秒增加700多个请求（正常业务带来200-340，测试当时为250左右），apachetop显示每秒正确处理（返回code 200）400-480个
             请求的失败率为10%左右。

             即：在业务请求每秒达到900-1000的时候，apachetop显示的正确处理个数为400-480，错误处理（非200）数量增加
       问题存在：
           1，后端的处理能力
           2，fcgi的处理能力（进程数量）
                       目前通过top显示每个php进程设定了5532 k byte的内存，使用了3824 （不到4M），
                       php.ini 中 memory_limit = 8M

     调整：
             修改 php fcgi 的worker 为 200后测试情况:
            0> ab -n 100000 -c 200   http://xx.xx.37.42/myblog/xxxxxxxwd_image.php
          This is ApacheBench, Version 1.3d <$Revision: 1.73 $> apache-1.3
          Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
          Copyright (c) 2006 The Apache Software Foundation, http://www.apache.org/

          Benchmarking xx.xx.37.42 (be patient)
          Completed 10000 requests
          Completed 20000 requests
          Completed 30000 requests
          Completed 40000 requests
          Completed 50000 requests
          Completed 60000 requests
          Completed 70000 requests
          Completed 80000 requests
          Completed 90000 requests
          Finished 100000 requests
          Server Software:        nginx/0.5.35
          Server Hostname:        xx.xx.37.42
          Server Port:            80

          Document Path:          /myblog/xxxxxxxwd_image.php
          Document Length:        545 bytes

          Concurrency Level:      200
          Time taken for tests:   129.350 seconds
          Complete requests:      100000
          Failed requests:        99415
             (Connect: 0, Length: 99415, Exceptions: 0)
          Broken pipe errors:     0
          Total transferred:      82315072 bytes
          HTML transferred:       57115072 bytes
          Requests per second:    773.10 [#/sec] (mean)
          Time per request:       258.70 [ms] (mean)
          Time per request:       1.29 [ms] (mean, across all concurrent requests)
          Transfer rate:          636.37 [Kbytes/sec] received

          Connnection Times (ms)
                        min mean[+/-sd] median   max
          Connect:        0   125 106.5    117 3049
          Processing:    19   133   21.0    128   513
          Waiting:       18   132   21.0    128   513
          Total:         19   258 111.2    244 3271

          Percentage of the requests served within a certain time (ms)
            50%    244
            66%    272
            75%    288
            80%    293
            90%    303
            95%    309
            98%    313
            99%    317
           100%   3271 (last request)



           [root@xx_xx_API_37_42 <37.42>]: /data1/nginx
           0> top
           top - 14:24:52 up 132 days, 23:10, 5 users, load average: 6.80, 3.60, 2.37
           Tasks: 325 total, 16 running, 308 sleeping,   1 stopped,   0 zombie
           Cpu(s): 54.9% us, 7.9% sy, 0.0% ni, 37.0% id, 0.1% wa, 0.0% hi, 0.0% si
           Mem:   4144692k total, 2994984k used, 1149708k free,   188240k buffers
           Swap: 2096440k total,      160k used, 2096280k free, 2341820k cached

         分析：
             分析：apachetop显示并发处理的正常的请求数为800-900（此时所有的请求数大约是900-1000），是跟据php 进程数量进行了线性的增长
                    得出结论：此次调整合适，php cgi的进程数量可再调高，ab测试的并发数可再提高一些。

                调整：将php cgi的进程数增加到300（php共使用内存1.5G），增大ab测试的并发数量
             [root@xx_xx_API_37_42 <37.42>]: /data1/nginx
             0> logs/apachetop -f logs/blog.xxx.com.cn-access_log
             2xx: 1s)   6:31:52%) 3xx: 0.0/s0 ( 0.0%) 4xx:     0 ( 025%) 5xx:     0 ( 0.053
                         25374       1015.0              19.8M7.8 809.2K          816.4
                    25363 (100.0%) 3xx:       0 ( 0.0%) 4xx:    11
                 2553746     1006.7   915.0              19.8M7.8 809.2K          816.4
             25367 1014.7 20225 803xx:blog/c 0 ( 0.0%) 4xx:    11

             [root@xx_xx_API_37_42 <37.42>]: /data1/nginx
             0> logs/apachetop -f logs/blog.xxx.com.cn-access_log
             2xx: 1s)   6:34:58%) 3xx: 0.0/s0 ( 0.0%) 4xx:     0 (2:50%) 5xx:     0 ( 04:59
                        187738       1104.3             146.31.20 811.2K          817.1
                   187678 (100.0%) 3xx:       1 ( 0.0%) 4xx:    59
                 3031919 3408018.1   113.6              26.087.9 898.6K          817.1
             33132 110404 26440 88*/myblog/ch0ckwd_image.php    4

             增加ab并发请求 1000（1100 失败）
             [root@xxx_xx_API_37_42 <37.42>]: /data1/nginx
             130> top
             top - 14:42:53 up 132 days, 23:28, 5 users, load average: 7.51, 8.14, 6.23
             Tasks: 382 total, 13 running, 368 sleeping,   1 stopped,   0 zombie
             Cpu(s): 55.5% us, 7.6% sy, 0.0% ni, 36.6% id, 0.2% wa, 0.0% hi, 0.0% si
             Mem:   4144692k total, 3244744k used,   899948k free,   188852k buffers
             Swap: 2096440k total,      160k used, 2096280k free, 2495908k cached

             0> logs/apachetop -f logs/blog.xxx.com.cn-access_log
             2xx: 1s)   6:44:33%) 3xx: 0.0/s0 ( 0.0%) 4xx:     0 (1:10%) 5xx:     0 ( 04:34
                         78944     &