目录
- .chglogRelease v0.2.02年前
- .githubFix issue template2个月前
- sharefix encodeURI into template2个月前
- srcIntroduce vhost_traffic_status_stats_by_upstream (#293)2个月前
- tSupport query string encoding for control API #279 (#287)2个月前
- utilCompatibility: fixed some issues for the nginx-module-sts/issues/18年前
- .gitignoreTest: describe how to test and fix failed test case5年前
- CHANGELOG.mdRelease v0.2.32个月前
- LICENSEFeature: added vhost_traffic_status_display_jsonp to support JSONP9年前
- README.mdUpdate README.md2个月前
- configFeature: added support for implementing format/prometheus6年前
邀请码
Nginx virtual host traffic status module
Nginx virtual host traffic status module
Table of Contents
Version
See the GitHub Releases for the latest tagged release.
Test
Run
sudo prove -r tafter you have installed this module. Thesudois required because the test requires Nginx to listen on port 80.Dependencies
Compatibility
Earlier versions is not tested.
Screenshots
Installation
Add the module to the build configuration by adding
--add-module=/path/to/nginx-module-vtsBuild the nginx binary.
Install the nginx binary.
Installtion with Profile-Guided Optimization
It can be built with Profile-Guided Optimization (PGO) using gcc
fprofileoptions. The detail of the PGO mechanisms has refer to the section 7.4 of this paper. Here is an example of the process to make a PGO supported binary. Please use at your own risk.Synopsis
Description
This is an Nginx module that provides access to virtual host status information. It contains the current status such as servers, upstreams, caches. This is similar to the live activity monitoring of nginx plus. The built-in html is also taken from the demo page of old version.
First of all, the directive
vhost_traffic_status_zoneis required, and then if the directivevhost_traffic_status_displayis set, can be access to as follows:/status/format/json, will respond with a JSON document containing the current activity data for using in live dashboards and third-party monitoring tools./status/format/html, will respond with the built-in live dashboard in HTML that requests internally to/status/format/json./status/format/jsonp, will respond with a JSONP callback function containing the current activity data for using in live dashboards and third-party monitoring tools./status/format/prometheus, will respond with a prometheus document containing the current activity data./status/control, will respond with a JSON document after it reset or delete zones through a query string. See the Control.JSON document contains as follows:
*) and hit ratiovhost_traffic_status_filter_by_set_keydirective*) and hit ratio filtered through thevhost_traffic_status_filter_by_set_keydirectiveThe
overCountsobjects in JSON document are mostly for 32bit system and will be increment by 1 if its value is overflowed. The directivevhost_traffic_status_display_formatsets the default ouput format that is one of json, jsonp, html, prometheus. (Default: json)Traffic calculation as follows:
All calculations are working in log processing phase of Nginx. Internal redirects(X-Accel-Redirect or error_page) does not calculate in the UpstreamZones.
Caveats:this module relies on nginx logging system(NGX_HTTP_LOG_PHASE:last phase of the nginx http), so the traffic may be in certain cirumstances different that real bandwidth traffic. Websocket, canceled downloads may be cause of inaccuracies. The working of the module doesn’t matter at all whether the access_log directive “on” or “off”. Again, this module works well on “access_log off”. When using several domains it sets to be first domain(left) of server_name directive. If you don’t want it, see the vhost_traffic_status_filter_by_host, vhost_traffic_status_filter_by_set_key directive.See the following modules for the
streamtraffic statistics:Calculations and Intervals
Averages
All averages are currently calculated as AMM(Arithmetic Mean) over the last 64 values.
Control
It is able to reset or delete traffic zones through a query string. The request responds with a JSON document.
{status_uri}/control?cmd={command}&group={group}&zone={name}If it set as above, then the control uri is like
example.org/status/control.The available request arguments are as follows:
status|reset|delete>status/format/json.server|filter|upstream@alone|upstream@group|cache|*>To get status of traffic zones on the fly
This is similar to the
status/format/jsonexcept that it can get each zones.To get fully zones
status/format/json.To get group zones
The mainZones values are default status values including
hostName,moduleVersion,nginxVersion,loadMsec,nowMsec,connections.To get each zones
namefilter_group@nameupstream_group@namenamenameTo reset traffic zones on the fly
It reset the values of specified zones to 0.
To reset fully zones
To reset group zones
To reset each zones
namefilter_group@nameupstream_group@namenamenameTo delete traffic zones on the fly
It delete the specified zones in shared memory.
To delete fully zones
To delete group zones
To delete each zones
namefilter_group@nameupstream_group@namenamenameSet
It can get the status values in nginx configuration separately using
vhost_traffic_status_set_by_filterdirective. It can acquire almost all status values and the obtained value is stored in user-defined-variable which is first argument.The above settings are as follows:
Please see the vhost_traffic_status_set_by_filter directive for detailed usage.
JSON
The following status information is provided in the JSON format:
Json used by status
/
{status_uri}/format/json/
{status_uri}/control?cmd=status&…{version}(|.dev.{commit})format.vhost_traffic_status)vhost_traffic_status_histogram_bucketsdirective.serverZonesexcept that it included group names.vhost_traffic_status_histogram_bucketsdirective.vhost_traffic_status_histogram_bucketsdirective.weightsetting of the server.max_failssetting of the server.fail_timeoutsetting of the server.backupsetting of the server.downsetting of the server. Basically, this is just a mark the ngx_http_upstream_module‘s server down(eg.server backend3.example.com down), not actual upstream server state. It will changed to actual state if you enabled the upstream zone directive.max_sizeinproxy_cache_pathdirective is not specified, the system dependent valueNGX_MAX_OFF_T_VALUEis assigned by default. In other words, this value is from nginx, not what I specified.maxSizevalue.Json used by control
/
{status_uri}/control?cmd=reset&…/
{status_uri}/control?cmd=delete&…Variables
The following embedded variables are provided:
Limit
It is able to limit total traffic per each host by using the directive
vhost_traffic_status_limit_traffic. It also is able to limit all traffic by using the directivevhost_traffic_status_limit_traffic_by_set_key. When the limit is exceeded, the server will return the 503 (Service Temporarily Unavailable) error in reply to a request. The return code can be changeable.To limit traffic for server
*.example.orgto 64G and 1024G respectively. It works individually per each domain ifvhost_traffic_status_filter_by_hostdirective is enabled.To limit traffic for filter
example.orgto 1024G and 2048G respectively.To limit traffic for upstream
example.orgto 512G and 1024G per each peer.Caveats:Traffic is the cumulative transfer or counter, not a bandwidth.Use cases
It is able to calculate the user defined individual stats by using the directive
vhost_traffic_status_filter_by_set_key.To calculate traffic for individual country using GeoIP
Basically, country flags image is built-in in HTML. The country flags image is enabled if the
countrystring is included in group name which is second argument ofvhost_traffic_status_filter_by_set_keydirective.To calculate traffic for individual storage volume
To calculate traffic for individual user agent
http_user_agentTo calculate traffic for detailed http status code
http status codeCaveats:$status variable is available in nginx-(1.3.2, 1.2.2).To calculate traffic for dynamic dns
If the domain has multiple DNS A records, you can calculate traffic for individual IPs for the domain using the filter feature or a variable in proxy_pass.
elb.example.org. Ifelb.example.orghas multiple DNS A records, will be display all IPs infilterZones. In the above settings, as NGINX starts up or reloads it configuration, it queries a DNS server to resolve domain and DNS A records is cached in memory. Therefore the DNS A records are not changed in memory even if DNS A records are chagned by DNS administrator unless NGINX re-starts up or reloads.elb.example.org. Ifelb.example.org‘s DNS A record is changed, will be display both the old IP and the new IP in::nogroups. Unlike the first upstream group setting, the second setting works well even if DNS A records are chagned by DNS administrator.Caveats:Please more details about NGINX DNS see the dns-service-discovery-nginx-plus.To calculate traffic except for status page
/statusuri is excluded from the status traffic calculation and limit feature. See the following directives:To maintain statistics data permanently
vhost_traffic_status_dumpdirective maintains statistics data permanently even if system has been rebooted or nginx has been restarted. Please see the vhost_traffic_status_dump directive for detailed usage.Customizing
To customize after the module installed
You need to change the
{{uri}}string to your status uri in status.template.html as follows:And then, customizing and copy status.template.html to server root directory as follows:
Configure
nginx.conf```Nginx server {}
http://example.org/status.html
shell> cd util shell> ./tplToDefine.sh ../share/status.template.html > ../src/ngx_http_vhost_traffic_status_module_html.h
It provides the same function that set
vhost_traffic_status_filter_by_set_key $host.vhost_traffic_status_filter_by_set_key
Description:Enables the keys by user defined variable. The key is a key string to calculate traffic. The name is a group string to calculate traffic. The key and name can contain variables such as $host, $server_name. The name‘s group belongs tofilterZonesif specified. The key‘s group belongs toserverZonesif not specified second argument name. The example with geoip module is as follows:vhost_traffic_status_filter_check_duplicate
Description:Enables or disables the deduplication of vhost_traffic_status_filter_by_set_key. It is processed only one of duplicate values(key+name) in each directives(http, server, location) if this option is enabled.vhost_traffic_status_filter_max_node
Description:Enables the limit of filter size using the specified number and string values. If the number is exceeded, the existing nodes are deleted by the LRU algorithm. The number argument is the size of the node that will be limited. The default value0does not limit filters. The one node is an object infilterZonesin JSON document. The string arguments are the matching string values for the group string value set byvhost_traffic_status_filter_by_set_keydirective. Even if only the first part matches, matching is successful like the regular expression/^string.*/. By default, If you do not set string arguments then it applied for all filters.For examples:
$ vi nginx.conf$ for i in {0..1000}; do curl -H 'Host: example.org' -i "http://localhost:80/test$i"; doneIn the above example, the
/^uris.*/and/^client::ports.*/group string patterns are limited to a total of 16 nodes. The other filters likecountry::.*are not limited.vhost_traffic_status_limit
Description:Enables or disables the limit features.vhost_traffic_status_limit_traffic
Description:Enables the traffic limit for specified member. The member is a member string to limit traffic. The size is a size(k/m/g) to limit traffic. The code is a code to return in response to rejected requests.(Default: 503)The available
memberstrings are as follows:vhost_traffic_status_limit_traffic_by_set_key
Description:Enables the traffic limit for specified key and member. The key is a key string to limit traffic. The member is a member string to limit traffic. The size is a size(k/m/g) to limit traffic. The code is a code to return in response to rejected requests.(Default: 503)The
keysyntax is as follows:group@[subgroup@]nameThe available
groupstrings are as follows:subgroup)subgroup)The available
memberstrings are as follows:The member is the same as
vhost_traffic_status_limit_trafficdirective.vhost_traffic_status_limit_check_duplicate
Description:Enables or disables the deduplication of vhost_traffic_status_limit_by_set_key. It is processed only one of duplicate values(member|key+member) in each directives(http, server, location) if this option is enabled.vhost_traffic_status_set_by_filter
Description:Get the specified status value stored in shared memory. It can acquire almost all status values and the obtained value is stored in $variable which is first argument.Caveats:The name is case sensitive. All return values take the integer type.For examples:
$requestCounterserver/example.org/requestCounter$requestCounterfilter/country::example.org@KR/requestCounter$requestCounterupstream@group/backend@10.10.10.11:80/requestCounter$requestCounterupstream@alone/10.10.10.11:80/requestCounter$cacheHitcache/my_cache_name/cacheHitvhost_traffic_status_average_method
Description:Sets the method which is a formula that calculate the average of response processing times. The period is an effective time of the values used for the average calculation.(Default: 60s) If period set to 0, effective time is ignored. In this case, the last average value is displayed even if there is no requests and after the elapse of time. The corresponding values arerequestMsecandresponseMsecin JSON.vhost_traffic_status_histogram_buckets
Description:Sets the observe buckets to be used in the histograms. By default, if you do not set this directive, it will not work. The second can be expressed in decimal places with a minimum value of 0.001(1ms). The maximum size of the buckets is 32. If this value is insufficient for you, change theNGX_HTTP_VHOST_TRAFFIC_STATUS_DEFAULT_BUCKET_LENin thesrc/ngx_http_vhost_traffic_status_node.hFor examples:
0.0050.010.050.10.515100.0050.010.050.1Caveats:By default, if you do not set this directive, the histogram statistics does not work. The restored histograms byvhost_traffic_status_dumpdirective have no affected by changes to the buckets byvhost_traffic_status_histogram_bucketsdirective. So you must first delete the zone or the dump file before changing the buckets byvhost_traffic_status_histogram_bucketsdirective. Similar to the above, delete the dump file when using the histogram for the first time.vhost_traffic_status_bypass_limit
Description:Enables or disables to bypassvhost_traffic_status_limitdirectives. The limit features is bypassed if this option is enabled. This is mostly useful if you want to connect the status web page like/statusregardless ofvhost_traffic_status_limitdirectives as follows:vhost_traffic_status_bypass_stats
Description:Enables or disables to bypassvhost_traffic_status. The traffic status stats features is bypassed if this option is enabled. In other words, it is excluded from the traffic status stats. This is mostly useful if you want to ignore your request in status web page like/statusas follows:vhost_traffic_status_stats_by_upstream
Description:Enables or disables to statsupstreamZone. TheupstreamZonein the traffic status stats features is bypassed if this option is disabled. In other words, it is excluded from the traffic status stats. This is mostly useful if you want to be disable statistics collection for upstream servers to reduce CPU load.Releases
To cut a release, create a changelog entry PR with git-chglog
After the PR is merged, create the new tag and release on the GitHub Releases.
See Also
Stream traffic status
Prometheus
System protection
TODO
Author
YoungJoo.Kim(김영주) [vozltx@gmail.com]