Quantcast
Channel: Nginx Forum
Viewing all 53287 articles
Browse latest View live
↧

Re: error logging with nginx and uWSGI

May 5, 2015, 7:04 am
$
0
0
Hello!

On Tue, May 05, 2015 at 08:08:42AM -0400, Larry Martell wrote:

> On Sun, May 3, 2015 at 8:18 AM, Maxim Dounin <mdounin@mdounin.ru> wrote:
> > Hello!
> >
> > On Fri, May 01, 2015 at 12:04:51PM -0400, Larry Martell wrote:
> >
> >> Prior to now, all the django projects I've worked on have used apache
> >> and WSGI. With those, when an error occurred I went to
> >> /var/log/httpd/error_log and details of the error were clearly there.
> >>
> >> Now for the first time I am working on a project using nginx and
> >> uWSGI. Here, the /var/log/nginx/error_log is always empty. And the
> >> uWSGI logs have some messages, but nothing about any errors. In this
> >> setup where would I go to find the errors that
> >> /var/log/httpd/error_log logs? Is there some config setting that is
> >> suppressing the nginx errors?
> >
> > The uwsgi protocol doesn't include an error stream from an
> > application to nginx. That is, if you are looking for errors
> > generated by your application, you should look into uWSGI logs.
> >
> > Own nginx logs can be controlled using the error_log directive,
> > see http://nginx.org/r/error_log. But I suspect it's not what are
> > you looking for, see above.
>
> When my app has, for example, a syntax error, then yes, that appears
> in the uWSGI log. But what I was talking about are the HTTP errors
> like a 500 or a 400. When I get those there's nothing in the logs.

If an error is returned by your application and/or uWSGI, then its
reasons are expected to be in your application logs (or the uWSGI
logs).

If an error is returned by nginx (e.g., because a client sent an
invalid request and nginx returned 400), then reasons should be in
nginx error log. Client-related errors, though, are usually
logged at the "info" level, and won't be visible in error log by
default, see http://nginx.org/r/error_log.

Note well that nginx error logs are highly customizeable, and it
is possible that you are looking into a wrong file. In
particular, please note that default error log can be redefined
during compilation (see "nginx -V" output to find out which one is
used by default), and can also be redefined on a per-server or a
per-location basis (check your configs to find out if it's the
case).

--
Maxim Dounin
http://nginx.org/

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
↧
Search

Re: How to block fake google spider and fake web browser access?

May 5, 2015, 7:06 am
$
0
0
The only way you can stop people from mirroring your site is to pull the
plug. Anything you set up can be bypassed like a normal user would. If you
put CAPTCHAs on every page, someone motivated can get really smart people
in poor countries to type in the letters, click the blue box, complete the
pattern, etc. on the cheap.

However, that being said the legit Googlebot operates from a well defined
subset of IP blocks and always identifies itself and honors robots.txt, so
you can look those up and white list them.

Any traffic from Amazon EC2, Google Clould, and Digital Ocean is
immediately suspect, you can filter them out by IP block because they are
probably not going to identify themselves as a bot. However you may lose
traffic from real people running VPNs and proxies though those sites as a
consequence so think it through before you act.

And there are no shortage of other providers for people to turn to if you
block the big clouds, so it comes back to pulling the plug if you want to
keep your content locked down.


On Tue, May 5, 2015 at 8:38 AM, meteor8488 <nginx-forum@nginx.us> wrote:

> Hi All,
>
> Recently I found that someguys are trying to mirror my website. They are
> doing this in two ways:
>
> 1. Pretend to be google spiders . Access logs are as following:
>
> 89.85.93.235 - - [05/May/2015:20:23:16 +0800] "GET /robots.txt HTTP/1.0"
> 444
> 0 "http://www.example.com" "Mozilla/5.0 (compatible; Googlebot/2.1;
> +http://www.google.com/bot.html)" "66.249.79.138"
> 79.85.93.235 - - [05/May/2015:20:23:34 +0800] "GET /robots.txt HTTP/1.0"
> 444
> 0 "http://www.example.com" "Mozilla/5.0 (compatible; Googlebot/2.1;
> +http://www.google.com/bot.html)" "66.249.79.154"
>
> The http_x_forwarded_for address are google addresses.
>
> 2. Pretend to be a normal web browser.
>
>
> I'm trying to use below configuration to block their access:
>
>
>
> For 1 above, I'll check X_forward_for address. If user agent is spider, and
> X_forward_for is not null. Then block.
> I'm using
>
> map $http_x_forwarded_for $xf {
> default 1;
> "" 0;
> }
> map $http_user_agent $fakebots {
> default 0;
> "~*bot" $xf;
> "~*bing" $xf;
> "~*search" $xf;
> }
> if ($fakebots) {
> return 444;
> }
>
> With this configuration, it seems the fake google spider can't access the
> root of my website. But they can still access my php files, and they can't
> access and js or css files. Very strange. I don't know what's wrong.
>
> 2. For user-agent who declare they are not spiders. I'll use ngx_lua to
> generate a random value and add the value into cookie, and then check
> whether they can send this value back or not. If they can't send it back,
> then it means that they are robot and block access.
>
> map $http_user_agent $ifbot {
> default 0;
> "~*Yahoo" 1;
> "~*archive" 1;
> "~*search" 1;
> "~*Googlebot" 1;
> "~Mediapartners-Google" 1;
> "~*bingbot" 1;
> "~*msn" 1;
> "~*rogerbot" 3;
> "~*ChinasoSpider" 3;
> }
>
> if ($ifbot = "0") {
> set $humanfilter 1;
> }
> #below section is to exclude flash upload
> if ( $request_uri !~ "~mod\=swfupload\&action\=swfupload" ) {
> set $humanfilter "${humanfilter}1";
> }
>
> if ($humanfilter = "11"){
> rewrite_by_lua '
> local random = ngx.var.cookie_random
> if(random == nil) then
> random = math.random(999999)
> end
> local token = ngx.md5("hello" .. ngx.var.remote_addr .. random)
> if (ngx.var.cookie_token ~= token) then
> ngx.header["Set-Cookie"] = {"token=" .. token, "random=" .. random}
> return ngx.redirect(ngx.var.scheme .. "://" .. ngx.var.host ..
> ngx.var.request_uri)
> end
> ';
> }
> But it seems that with above configuration, google bot is also blocked
> while
> it shouldn't.
>
>
> Any one can help?
>
> Thanks
>
> Posted at Nginx Forum:
> http://forum.nginx.org/read.php?2,258659,258659#msg-258659
>
> _______________________________________________
> nginx mailing list
> nginx@nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx
>
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
↧
↧

Websockets max connections with SSL + slow

May 5, 2015, 7:54 am
$
0
0
Hi guys,

We set up a basic reverse proxy configuration (that you can find below this thread).
Our main app is using websocket, and the reverse proxy works fine when no using SSL.

But when SSL is enabled, we noticed a big performance issue making our app very slow. Moreover, the most important: we get a problem when reaching the 50th websocket alive connection for a given user: it crashes our app.

Could you help us finding what's wrong in the following?

App server conf:
- ubuntu v.14.10

Nginx server conf:
- nginx v1.9.0
- ubuntu v.14.04

and the conf file is the following:

###################################

user nginx_user nginx_user;
daemon off;
worker_processes 2;

pid logs/nginx.pid;

events {
worker_connections 1024;
}


http {
include mime.types;
default_type application/octet-stream;

server {
listen 8443 ssl;
server_name ourapp.com;

ssl_certificate ../ssl/cacert.pem;
ssl_certificate_key ../ssl/privkey.pem;

ssl_session_cache shared:SSL:1m;
ssl_session_timeout 5m;

ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;

location / {
proxy_pass http://ourapp.com:8800;
}

location /our_ws_location {
proxy_pass http://ourapp.com:8801;

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_set_header X-NginX-Proxy true;

# WebSocket support
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
}

###################################

Thanks in advance,

Regards,

Z
↧

Re: how to separate robot access log and human access log

May 5, 2015, 10:50 am
$
0
0
On Tue, May 05, 2015 at 08:20:35AM -0400, meteor8488 wrote:

Hi there,

> if $spiderbot=0, then log to location_access.log

Set a variable which is non-zero when $spiderbot=0, and which is zero
or blank otherwise. Use that as the access_log if=$variable for
location_access.log.

> if $spiderbot=1, then log to spider_access.log.

Set a variable which is non-zero when $spiderbot=1, and which is zero
or blank otherwise. ($spiderbot is probably perfect for this as-is.) Use
that as the access_log if=$variable for spider_access.log.

> And I don't want the same logs write to different files.

For each loggable request, make sure that exactly one of your if=$variable
variables is non-zero.

f
--
Francis Daly francis@daoine.org

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
↧

Re: How to block fake google spider and fake web browser access?

May 5, 2015, 10:56 am
$
0
0
On Tue, May 05, 2015 at 09:07:41AM -0400, meteor8488 wrote:

Hi there,

> I tried to use "deny" to deny access from an IP. But it seems that it can
> still access my server.
>
> In my http part:
>
> deny 69.85.92.0/23;
> deny 69.85.93.235;

A request comes in to nginx. nginx chooses one server{} block in its
configuration to handle it. nginx chooses one location{} block in that
server{} configuration to handle it. Only configuration directives in,
or inherited into, that location{} are relevant.

(If you use any rewrite-module directives, things may be different.)

> 69.85.93.235 - - [05/May/2015:19:44:22 +0800] "GET /thread-1251687-1-1.html
> HTTP/1.0" 302 154 "http://www.example.com" "Mozilla/5.0 (compatible;
> Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
> "123.125.71.107"

What is the one location{} that handles this request? What "allow" and
"deny" directives are in that location{}? And in the enclosing server{}?


Can you provide a complete nginx.conf that shows the behaviour you report?

(It doesn't have to be your production config. Something smaller
that shows this problem on a test machine, may make obvious where the
problem is.)

Thanks,

f
--
Francis Daly francis@daoine.org

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
↧
↧

Re: Connection timeout from work, working anywhere else

May 5, 2015, 11:12 am
$
0
0
On Mon, May 04, 2015 at 02:19:45PM -0400, GuiPoM wrote:

Hi there,

> Thank you for your answer. I can reproduce on demand ! BUTI am not familiar
> with nginx.
> Could you give me some hints what to activate in order to provide useful
> information for debugging ?

You could follow http://nginx.org/en/docs/debugging_log.html to get all
sorts of information out of nginx -- but I suspect that that will not
be immediately useful.

When things work, what is the sequence of requests made? access_log will
have that.

When things fail, what is the sequence of requests made? access_log will
also have that.

What is the first request in the sequence that fails, or otherwise does
not get the expected response, in the "fail" case?

Can you arrange a single "curl" command that works in one case and fails
in the other? That may help you analyse where things go wrong.

> Next follows the config. Hope it will help !

Once the "failing" request is identified, the matching server{} and
location{} can be analysed to see what should happen.

(If it turns out that the "failure" happens before the request gets to
nginx -- for example, during ssl negotiation -- then the details of the
request are less important.)

But that config is presumably in one of the files mentioned in an
"include" directive.

> include /etc/nginx/conf.d/*.conf;
> include /etc/nginx/sites-enabled/*;

Good luck with it,

f
--
Francis Daly francis@daoine.org

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
↧

Re: Connection timeout from work, working anywhere else

May 5, 2015, 1:00 pm
$
0
0
I will do so.

Two questions:
1/ In my config file /etc/nginx/nginx.conf, in section http, I already have the logging entries defined :
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

I just tried to put debug as for example : "access_log /var/log/nginx/access.log debug;"

But:
Restarting nginx: nginx: [emerg] unknown log format "debug" in /etc/nginx/nginx.conf:36
nginx: configuration file /etc/nginx/nginx.conf test failed

I also tried to adapt to add the server section that I don't have in my own config file, as your link mention (http://nginx.org/en/docs/debugging_log.html#memory).

I did the same in one mentionned file : default_ssl
#access_log off;
#error_log /usr/share/nginx/www/jeedom/log/nginx.error;
access_log memory:32m debug;
error_log memory:32m debug;


Restarting nginx: nginx: [emerg] unknown log format "debug" in /etc/nginx/sites-enabled/default_ssl:8
nginx: configuration file /etc/nginx/nginx.conf test failed

I a doing something wrong ?

2/ In error log, even if I can't set a debug level, there is already something strange I would like to change:

2015/05/02 13:25:05 [error] 2144#0: *4926 upstream prematurely closed connection while reading response header from upstream, client: XX.XX.XXX.XXX, server: , request: "GET /socket.io/?EIO=3&transport=polling&t=1430565786187-49&sid=IIJ1gX_E4Ny_ojN8AACB HTTP/1.1", upstream: "http://127.0.0.1:8070/socket.io/?EIO=3&transport=polling&t=1430565786187-49&sid=IIJ1gX_E4Ny_ojN8AACB", host: "hostname.dtdns.net:9876", referrer: "https://hostname.dtdns.net:9876/jeedom/index.php?v=m&"

How host and referrer can be filled with a dynamic dns name ? How nginx is aware of this information ? I am requesting with an IP address, so no chance this information come from the sender.

Could this configuration be erroneous ? (/etc/nginx/sites-enabled/default_ssl)

location /socket.io/ {
proxy_pass http://127.0.0.1:8070/socket.io/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
proxy_redirect off;
proxy_read_timeout 6000;
}


Thx.
GuiPoM
↧

Re: Connection timeout from work, working anywhere else

May 5, 2015, 2:08 pm
$
0
0
On Tue, May 05, 2015 at 04:00:11PM -0400, GuiPoM wrote:

Hi there,

> I just tried to put debug as for example : "access_log
> /var/log/nginx/access.log debug;"
>
> But:
> Restarting nginx: nginx: [emerg] unknown log format "debug" in
> /etc/nginx/nginx.conf:36
> nginx: configuration file /etc/nginx/nginx.conf test failed

Does "nginx -V" show "--with-debug"?

"debug" is for error_log, not for access_log.

> 2/ In error log, even if I can't set a debug level, there is already
> something strange I would like to change:
>
> 2015/05/02 13:25:05 [error] 2144#0: *4926 upstream prematurely closed
> connection while reading response header from upstream, client:
> XX.XX.XXX.XXX, server: , request: "GET
> /socket.io/?EIO=3&transport=polling&t=1430565786187-49&sid=IIJ1gX_E4Ny_ojN8AACB
> HTTP/1.1", upstream:
> "http://127.0.0.1:8070/socket.io/?EIO=3&transport=polling&t=1430565786187-49&sid=IIJ1gX_E4Ny_ojN8AACB",
> host: "hostname.dtdns.net:9876", referrer:
> "https://hostname.dtdns.net:9876/jeedom/index.php?v=m&"
>
> How host and referrer can be filled with a dynamic dns name ? How nginx is
> aware of this information ? I am requesting with an IP address, so no chance
> this information come from the sender.

When you copy-paste the commands issued and the responses gathered,
it may become clearer where all of the information is coming from.

My guess is that you are issuing one request with an ip address, and
that is returning a http redirect to a hostname; and then you are issuing
the next request to that hostname.

But until you show your work, all anyone can do here is guess.

> Could this configuration be erroneous ?
> (/etc/nginx/sites-enabled/default_ssl)
>
> location /socket.io/ {
> proxy_pass http://127.0.0.1:8070/socket.io/;
> proxy_http_version 1.1;
> proxy_set_header Upgrade $http_upgrade;
> proxy_set_header Connection "Upgrade";
> proxy_set_header Host $host;
> proxy_redirect off;
> proxy_read_timeout 6000;
> }

This looks like the connection is using WebSockets.

Does your proxy server at work allow WebSocket connections to pass
through it?

Can you successfully connect to any WebSocket service anywhere from
work? If not, the problem may not be on the nginx side.

Good luck with it,

f
--
Francis Daly francis@daoine.org

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
↧
Search

Re: How to block fake google spider and fake web browser access?

May 5, 2015, 4:05 pm
$
0
0
Hi Francis,

I put the "deny" directives in http{} part.

Here is my nginx.conf.

http {

deny 4.176.128.153;
deny 23.105.85.0/24;
deny 36.44.146.99;
deny 42.62.36.167;
deny 42.62.74.0/24;
deny 50.116.28.209;
deny 50.116.30.23;
deny 52.0.0.0/11;
deny 54.72.0.0/13;
deny 54.80.0.0/12;
deny 54.160.0.0/12;
deny 54.176.0.0/12;
deny 54.176.195.13;
deny 54.193.0.0/16;
deny 54.193.212.129;
deny 54.208.0.0/15;
deny 54.212.0.0/15;
deny 54.219.0.0/16;
deny 54.224.0.0/12;
deny 58.208.0.0/12;
deny 61.135.219.2;
deny 61.173.11.234;
deny 61.177.134.164;
deny 61.178.110.42;
deny 69.85.92.0/23;
deny 69.85.93.235;
deny 101.226.62.63;
deny 101.226.167.237;
deny 101.226.168.225;
deny 101.231.74.38;
deny 101.231.74.40;
deny 103.19.84.0/22;
deny 106.186.112.0/21;
deny 111.20.18.224;
deny 111.20.19.148;
deny 111.67.200.68;
deny 112.90.51.35;
deny 112.235.133.139;
deny 113.74.83.46;
deny 113.120.156.252;
deny 114.80.109.30;
deny 114.80.116.164;
deny 114.86.54.43;
deny 114.87.109.129;
deny 114.112.103.46;
deny 115.226.236.69;
deny 116.7.169.91;
deny 116.208.12.74;
deny 116.228.41.122;
deny 116.232.27.33;
deny 116.234.130.64;
deny 117.27.152.197;
deny 117.27.152.198;
deny 117.151.97.223;
deny 118.144.32.66;
deny 119.85.190.7;
deny 119.147.225.177;
deny 119.254.64.12;
deny 119.254.86.240;
deny 119.254.86.246;
deny 121.202.22.154;
deny 122.4.149.168;
deny 122.49.5.11;
deny 122.49.5.14;
deny 122.49.5.15;
deny 122.96.36.167;
deny 123.151.176.198;
deny 124.156.6.198;
deny 124.226.42.78;
deny 125.125.41.167;
deny 128.199.153.220;
deny 128.199.78.7;
deny 136.243.36.95;
deny 139.200.132.233;
deny 171.108.67.30;
deny 171.112.242.65;
deny 174.2.171.84;
deny 180.153.72.92;
deny 180.153.211.148;
deny 180.153.229.0/24;
deny 180.171.146.137;
deny 182.16.44.26;
deny 182.33.66.29;
deny 182.41.45.241;
deny 182.240.7.79;
deny 183.8.83.248;
deny 183.129.200.250;
deny 183.156.102.146;
deny 183.156.108.133;
deny 183.157.68.141;
deny 183.250.40.194;
deny 188.143.232.40;
deny 188.143.232.72;
deny 198.58.96.215;
deny 198.58.99.82;
deny 198.58.102.117;
deny 198.58.102.155;
deny 198.58.102.156;
deny 198.58.102.158;
deny 198.58.102.49;
deny 198.58.102.95;
deny 198.58.102.96;
deny 198.58.103.102;
deny 198.58.103.114;
deny 198.58.103.115;
deny 198.58.103.158;
deny 198.58.103.160;
deny 198.58.103.28;
deny 198.58.103.36;
deny 198.58.103.91;
deny 198.58.103.92;
deny 202.1.232.243;
deny 203.195.219.37;
deny 204.236.128.0/17;
deny 209.141.40.22;
deny 211.97.148.191;
deny 218.148.90.164;
deny 220.240.235.158;
deny 222.73.68.103;
deny 222.95.129.93;
deny 222.175.185.14;
deny 222.175.186.18;
geo $geo {
ranges;
111.67.200.68-111.67.200.68 badip;
58.213.119.20-58.213.119.21 badip;
54.208.0.0-54.209.255.255 badip;
54.176.0.0-54.191.255.255 badip;
54.219.0.0-54.219.255.255 badip;
54.193.0.0-54.193.255.255 badip;
54.160.0.0-54.175.255.255 badip;
106.145.17.0-106.145.17.255 badip;
112.235.133.139-112.235.133.139 spider;
5.255.253.77-5.255.253.77 spider;
69.85.93.235-69.85.93.235 spider;
54.160.105.130-54.160.105.130 spider;
95.108.158.146-95.108.158.146 spider;
131.253.21.0-131.253.47.255 spider;
157.54.0.0-157.60.255.255 spider;
202.160.176.0-202.160.191.255 spider;
207.46.0.0-207.46.255.255 spider;
207.68.128.0-207.68.207.255 spider;
209.191.64.0-209.191.127.255 spider;
209.85.128.0-209.85.255.255 spider;
216.239.32.0-216.239.63.255 spider;
64.233.160.0-64.233.191.255 spider;
64.4.0.0-64.4.63.255 spider;
65.52.0.0-65.55.255.255 spider;
66.102.0.0-66.102.15.255 spider;
66.196.64.0-66.196.127.255 spider;
66.228.160.0-66.228.191.255 spider;
66.249.64.0-66.249.95.255 spider;
67.195.0.0-67.195.255.255 spider;
68.142.192.0-68.142.255.255 spider;
72.14.192.0-72.14.255.255 spider;
72.30.0.0-72.30.255.255 spider;
74.125.0.0-74.125.255.255 spider;
74.6.0.0-74.6.255.255 spider;
8.12.144.0-8.12.144.255 spider;
98.136.0.0-98.139.255.255 spider;
203.208.32.0-203.208.63.255 spider;
}

map $request_method $bad_method {
default 1;
~(?i)(GET|HEAD|POST) 0;
}

map $http_referer $bad_referer {
default 0;
~(?i)(babes|click|forsale|jewelry|nudit|organic|poker|porn|amnesty|poweroversoftware|webcam|zippo|casino|replica|CDR) 1;
}

map $query_string $spam {
default 0;
~*"\b(ultram|unicauca|valium|viagra|vicodin|xanax|ypxaieo)\b" 1;
~*"\b(erections|hoodia|huronriveracres|impotence|levitra|libido)\b" 1;
~*"\b(ambien|blue\spill|cialis|cocaine|ejaculation|erectile)\b" 1;
~*"\b(lipitor|phentermin|pro[sz]ac|sandyauer|tramadol|troyhamby)\b" 1;
}

map $http_x_forwarded_for $xf {
default 1;
"" 0;
}
map $http_user_agent $fakebots {
default 0;
"~*bot" $xf;
"~*bing" $xf;
"~*search" $xf;
"~*Baidu" $xf;
}

map $http_user_agent $ifbot {
default 0;
"~*rogerbot" 3;
"~*ChinasoSpider" 3;
"~*Yahoo" 1;
"~*archive" 1;
"~*search" 1;
"~*Googlebot" 1;
"~Mediapartners-Google" 1;
"~*bingbot" 1;
"~*YandexBot" 1;
"~*Baiduspider" 1;
"~*Feedly" 2;
"~*Superfeedr" 2;
"~*QuiteRSS" 2;
"~*g2reader" 2;
"~*Digg" 2;
"~*AhrefsBot" 3;
"~*ia_archiver" 3;
"~*trendiction" 3;
"~*AhrefsBot" 3;
"~*curl" 3;
"~*Ruby" 3;
"~*Player" 3;
"~*Go\ http\ package" 3;
"~*Lynx" 3;
"~*Sleuth" 3;
"~*Python" 3;
"~*Wget" 3;
"~*perl" 3;
"~*httrack" 3;
"~*JikeSpider" 3;
"~*PHP" 3;
"~*WebIndex" 3;
"~*magpie-crawler" 3;
"~*JUC" 3;
"~*Scrapy" 3;
"~*libfetch" 3;
"~*WinHTTrack" 3;
"~*htmlparser" 3;
"~*urllib" 3;
"~*Zeus" 3;
"~*scan" 3;
"~*Indy\ Library" 3;
"~*libwww-perl" 3;
"~*GetRight" 3;
"~*GetWeb!" 3;
"~*Go!Zilla" 3;
"~*Go-Ahead-Got-It" 3;
"~*Download\ Demon" 3;
"~*TurnitinBot" 3;
"~*WebscanSpider" 3;
"~*WebBench" 3;
"~*YisouSpider" 3;
"~*check_http" 3;
"~*webmeup-crawler" 3;
"~*omgili" 3;
"~*blah" 3;
"~*fountainfo" 3;
"~*MicroMessenger" 3;
"~*QQDownload" 3;
"~*shoulu.jike.com" 3;
"~*omgilibot" 3;
"~*pyspider" 3;
"~*mysite" 3;
}

......

server {
listen 80 accept_filter=httpready;
index index.html index.htm index.php;
access_log /var/log/server_access.log main;

location / {
root /var/www;

if ( $geo = "badip" ) {
return 444;
}
if ( $geo = "spider" ) {
set $spiderip 1;
}
if ($bad_method = 1) {
return 444;
}
if ($spam = 1) {
return 444;
}
set $humanfilter 0;
if ($ifbot = "0") {
set $humanfilter 1;
}
if ( $request_uri !~ "~mod\=swfupload\&action\=swfupload" ) {
set $humanfilter "${humanfilter}1";
}
if ($humanfilter = "11"){
rewrite_by_lua '
local random = ngx.var.cookie_random
if(random == nil) then
random = math.random(999999)
end
local token = ngx.md5("guessguess" .. ngx.var.remote_addr .. random)
if (ngx.var.cookie_token ~= token) then
ngx.header["Set-Cookie"] = {"token=" .. token, "random=" .. random}
return ngx.redirect(ngx.var.scheme .. "://" .. ngx.var.host .. ngx.var.request_uri)
end
';
}

if ($ifbot = "1") {
set $spiderbot 1;
}
if ($ifbot = "2") {
set $rssbot 1;
}
if ($ifbot = "3") {
return 444;
}

if ($fakebots) {
return 444;
}

if ($bad_referer = 1) {
return 410;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_pass backend;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
include fastcgi_params;
access_log /web/log/php.log main;
}
}
}
↧
↧

Re: FastCGI sent in stderr: "Primary script unknown"

May 5, 2015, 7:12 pm
$
0
0
Hi,

On Tue, May 5, 2015 at 7:01 AM, vincent123456 <nginx-forum@nginx.us> wrote:
> Hi,
>
> I try to configure a vhost with Nginx and PHP-FPM.
>
> I have an application with Symfony2.6, i followed this tutorial :
> http://symfony.com/doc/current/cookbook/configuration/web_server_configuration.html#nginx
>
This tutorial has helped me ->
https://www.digitalocean.com/community/tutorials/how-to-install-linux-nginx-mysql-php-lemp-stack-on-debian-7.
Maybe it can help you as well.

Hope that helps,

Regards,

--
Thiago Farina

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
↧

Unbuffered POST requests and/or uWSGI in nginx 1.8

May 5, 2015, 9:38 pm
$
0
0
Hi,

it's stated in a lot of IT news websites, that nginx 1.8 supports
unbuffered uploads. However, I could not find it in thechangelog
http://nginx.org/en/CHANGES-1.8, and also, did not find any new relevant
options in the nginx core module options.

Is it really so? If yes, how do you enable unbuffered uploads? Is that
supported for uwsgi?

Thanks!
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
↧

Re: Unbuffered POST requests and/or uWSGI in nginx 1.8

May 5, 2015, 10:08 pm
$
0
0
Hello,

On 06 May 2015, at 07:36, Igor Katson <igor.katson@gmail.com> wrote:

> Hi,
>
> it's stated in a lot of IT news websites, that nginx 1.8 supports unbuffered uploads. However, I could not find it in thechangelog, and also, did not find any new relevant options in the nginx core module options.
> Is it really so? If yes, how do you enable unbuffered uploads? Is that supported for uwsgi?


http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering

--
Roman Arutyunyan
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
↧

[1.8.0 stable] bug when install on old linux version

May 6, 2015, 12:36 am
$
0
0
Hello,

I tested new stable version (1.8.0) with simple option: ./configure —add-module=./ngx_enhance_mp4_module (https://github.com/whatvn/ngx_http_enhance_mp4_module https://github.com/whatvn/ngx_http_enhance_mp4_module) , then start nginx and it fail to spawn child process (with error similar to compiling with —with-file-aio on old linux kernel):

2015/05/06 14:22:28 [emerg] 19004#0: eventfd() failed (38: Function not implemented)
2015/05/06 14:22:28 [emerg] 19005#0: eventfd() failed (38: Function not implemented)
2015/05/06 14:22:28 [emerg] 19006#0: eventfd() failed (38: Function not implemented)
2015/05/06 14:22:28 [alert] 18999#0: worker process 19000 exited with fatal code 2 and cannot be respawned
2015/05/06 14:22:28 [alert] 18999#0: worker process 19002 exited with fatal code 2 and cannot be respawned
2015/05/06 14:22:28 [alert] 18999#0: worker process 19003 exited with fatal code 2 and cannot be respawned
2015/05/06 14:22:28 [alert] 18999#0: worker process 19004 exited with fatal code 2 and cannot be respawned
2015/05/06 14:22:28 [alert] 18999#0: worker process 19005 exited with fatal code 2 and cannot be respawned
2015/05/06 14:22:28 [alert] 18999#0: worker process 19006 exited with fatal code 2 and cannot be respawned



with same configure options, nginx stable 1.6.3 work fine. Maybe a bug?

System details:

Centos 5, kernel: 2.6.18-164.el5



--
Hưng
Email: hungnv@opensource.com.vn



_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
↧
↧

RE: [1.8.0 stable] bug when install on old linux version

May 6, 2015, 1:54 am
$
0
0
Hi ,

I quickly review your code .

Please check this :

while (1) {
…
free(ftyp_atom);
ftyp_atom = ngx_palloc(r->connection->pool, ftyp_atom_size);
// ftyp_atom = malloc(ftyp_atom_size);

I see others ngx_palloc/free mix-up…

You also may merge 6d468b45f40e change (rev 5807) .

Regards,
Filipe



De : nginx-devel-bounces@nginx.org [mailto:nginx-devel-bounces@nginx.org] De la part de hungnv@opensource.com.vn
Envoyé : mercredi 6 mai 2015 09:36
À : nginx-devel@nginx.org
Objet : [1.8.0 stable] bug when install on old linux version


Hello,

I tested new stable version (1.8.0) with simple option: ./configure —add-module=./ngx_enhance_mp4_module (https://github.com/whatvn/ngx_http_enhance_mp4_module) , then start nginx and it fail to spawn child process (with error similar to compiling with —with-file-aio on old linux kernel):

2015/05/06 14:22:28 [emerg] 19004#0: eventfd() failed (38: Function not implemented)
2015/05/06 14:22:28 [emerg] 19005#0: eventfd() failed (38: Function not implemented)
2015/05/06 14:22:28 [emerg] 19006#0: eventfd() failed (38: Function not implemented)
2015/05/06 14:22:28 [alert] 18999#0: worker process 19000 exited with fatal code 2 and cannot be respawned
2015/05/06 14:22:28 [alert] 18999#0: worker process 19002 exited with fatal code 2 and cannot be respawned
2015/05/06 14:22:28 [alert] 18999#0: worker process 19003 exited with fatal code 2 and cannot be respawned
2015/05/06 14:22:28 [alert] 18999#0: worker process 19004 exited with fatal code 2 and cannot be respawned
2015/05/06 14:22:28 [alert] 18999#0: worker process 19005 exited with fatal code 2 and cannot be respawned
2015/05/06 14:22:28 [alert] 18999#0: worker process 19006 exited with fatal code 2 and cannot be respawned



with same configure options, nginx stable 1.6.3 work fine. Maybe a bug?

System details:

Centos 5, kernel: 2.6.18-164.el5



--
Hưng
Email: hungnv@opensource.com.vn<mailto:hungnv@opensource.com.vn>



_______________________________________________
nginx-devel mailing list
nginx-devel@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
↧

proxy_cache

May 6, 2015, 3:10 am
$
0
0
_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
↧
Search

Re: Неконтроллируемый объем кеша Nginx

May 6, 2015, 3:39 am
$
0
0
Здравствуйте.

Дебаг лог, во время повторения инцидента можно получить по ссылке: https://mega.co.nz/#!VI5DBbiI!GZDPBbfkyTCCmY8J0r9KFuov4UYldNegvN1SvtOYPVs (1.3 GB)

Проблема была обнаружена 5 мая около 22:50 (лог в себ включает сообщения на временном промежутке 22:45 - 22:55).

В самом логе вижу множество следующих сообщений:
2015/05/05 22:49:42 [debug] 6696#0: http file cache forced expire: #1 1 31dd5198
2015/05/05 22:49:42 [debug] 6696#0: http file cache forced expire: #1 1 bdb466de
2015/05/05 22:49:42 [debug] 6696#0: http file cache forced expire: #1 1 3207dd39
2015/05/05 22:49:42 [debug] 6696#0: http file cache forced expire: #1 1 b417f542
2015/05/05 22:49:42 [debug] 6696#0: http file cache forced expire: #1 1 14a73e73

При чем неоднократно повторяющихся. Если искать по hash id (напр., 31dd5198), нахожу только эти сообщения и ничего больше.
Меня интересует, что в вышеприведенных логах означает 4-е поле (6696#0). Могу предположить, что это привязка к процессу cache manager, но не уверен.

Если фильтровать логи по этому полю, нахожу следующее:
2015/05/05 22:48:16 [debug] 6696#0: shmtx lock
2015/05/05 22:48:16 [debug] 6696#0: shmtx unlock
2015/05/05 22:48:16 [debug] 6696#0: shmtx wake 634
2015/05/05 22:48:16 [debug] 6696#0: http file cache size: 20975604
2015/05/05 22:48:16 [debug] 6696#0: http file cache forced expire
2015/05/05 22:48:16 [debug] 6696#0: malloc: 00000000026BAA20:65
2015/05/05 22:48:16 [debug] 6696#0: shmtx lock
2015/05/05 22:48:16 [debug] 6696#0: http file cache forced expire: #0 1 a3397f6c
2015/05/05 22:48:16 [debug] 6696#0: shmtx unlock

Я так понимаю, это удачная очистка кеша. И следующее:
2015/05/05 22:48:16 [debug] 6696#0: shmtx lock
2015/05/05 22:48:16 [debug] 6696#0: shmtx unlock
2015/05/05 22:48:16 [debug] 6696#0: shmtx wake 625
2015/05/05 22:48:16 [debug] 6696#0: http file cache size: 20975581
2015/05/05 22:48:16 [debug] 6696#0: http file cache forced expire
2015/05/05 22:48:16 [debug] 6696#0: malloc: 00000000026BAA20:65
2015/05/05 22:48:16 [debug] 6696#0: shmtx lock
2015/05/05 22:48:16 [debug] 6696#0: http file cache forced expire: #1 1 dd87e60a
2015/05/05 22:48:16 [debug] 6696#0: http file cache forced expire: #0 1 93a2ff07
2015/05/05 22:48:16 [debug] 6696#0: shmtx unlock

И еще:
2015/05/05 22:48:16 [debug] 6696#0: shmtx wake 571
2015/05/05 22:48:16 [debug] 6696#0: http file cache expire: "/opt2/nginx-cache-images1/2f/ee/093821fba4c1bf6f947a291d0dcfee2f"
2015/05/05 22:48:16 [debug] 6696#0: shmtx lock
2015/05/05 22:48:16 [debug] 6696#0: slab free: 00007F66BE5FF380
2015/05/05 22:48:16 [debug] 6696#0: shmtx unlock
2015/05/05 22:48:16 [debug] 6696#0: shmtx wake 570
2015/05/05 22:48:16 [debug] 6696#0: shmtx lock
2015/05/05 22:48:16 [debug] 6696#0: shmtx unlock
2015/05/05 22:48:16 [debug] 6696#0: shmtx wake 569
2015/05/05 22:48:16 [debug] 6696#0: http file cache size: 20975519
2015/05/05 22:48:16 [debug] 6696#0: http file cache forced expire
2015/05/05 22:48:16 [debug] 6696#0: malloc: 00000000026BAA20:65
2015/05/05 22:48:16 [debug] 6696#0: shmtx lock
2015/05/05 22:48:16 [debug] 6696#0: http file cache forced expire: #1 1 dd87e60a
2015/05/05 22:48:16 [debug] 6696#0: http file cache forced expire: #1 1 8bc7932f
2015/05/05 22:48:16 [debug] 6696#0: http file cache forced expire: #0 1 91e3b5df
2015/05/05 22:48:16 [debug] 6696#0: shmtx unlock

Во время инцидента с uptime'мом воркеров аномалий не обнаружено.

P.S. Такая же версия nginx работает также на Ubuntu 12 LTS. В этом случае проблем не было обнаружено.
↧

Re: Неконтроллируемый объем кеша Nginx

May 6, 2015, 3:45 am
$
0
0
Еще доп. информация.

Тут было упомянуто, что cache manager делает порядка 20 попыток удалить "устаревший" кеш. Хотя если искать, скажем по hash id 31dd5198 (http file cache forced expire: #1 1 31dd5198), за 10 мин. логов нахожу 515 сообщений такого рода.
↧
↧

Re: Unbuffered POST requests and/or uWSGI in nginx 1.8

May 6, 2015, 4:20 am
$
0
0
On Tuesday 05 May 2015 21:36:26 Igor Katson wrote:
> Hi,
>
> it's stated in a lot of IT news websites, that nginx 1.8 supports
> unbuffered uploads. However, I could not find it in thechangelog
> http://nginx.org/en/CHANGES-1.8, and also, did not find any new relevant
> options in the nginx core module options.
>
> Is it really so? If yes, how do you enable unbuffered uploads? Is that
> supported for uwsgi?
>

http://nginx.org/r/uwsgi_request_buffering

uwsgi_request_buffering off;


wbr, Valentin V. Bartenev

_______________________________________________
nginx mailing list
nginx@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx
↧

simple Nginx config not working

May 6, 2015, 5:54 am
$
0
0
hello,

i am a developer with absolutely no server experience, this project im working on is my first foray into the world of servers. Basically i have to host an API online, on a Linode server running Nginx on Ubuntu. And im having trouble getting a simple server block set up to host a html index page, with just a h1 tag saying 'hello'. I've followed and read various tutorials on getting a simple server block working, i've even got these blocks working on my local machine, but im having issues getting it to work on the remote server.

il show you some blocks that have failed to work for me on the Linode:

first, a simple block to server a static html page, this works on my local machine:

server {

listen 8005;
index index.html;
root /srv/www/site;

}


another simple block:

server {

listen [server ip]:8006;
server_name "";
index index.html;
root /srv/www/site;

location ~/ {

root /srv/www/site;

}

}


a block for serving php (works locally):

server {

listen [server ip]:8007;
server_name "";
index index.php index.html index.htm;
root /srv/www/site;

location / {

try_files $uri $uri/ /index.php?q=$uri&$args;

}

location ~* \.php$ {

try_files $uri /index.php;
fastcgi_index index.php;
fastcgi_pass 127.0.0.1:9000;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;

}

}


whenever i tried to access them through the browser i got server connection timeouts. I do not have any DNS setup with it so i was trying to access it using the server IP and port number:

[server ip]:[port]

or

[server ip]:[port]/index.html

but got nothing.


So one thing that might cause the problem is the fact that another application is being hosted on the server already. It has its own block and is running on its own port. It has its own DNS name pointing to it. But none of this should matter right? as each server block is basically a virtual server or creates a virtual domain. Accessing it through ip and port should work fine, but it aint working!

thats about all i can think of, my knowledge in this area is maxed out and i need help from smart people!

any help is greatly appreciated, thanks!
↧

Re: Неконтроллируемый объем кеша Nginx

May 6, 2015, 5:58 am
$
0
0
On 06 May 2015, at 13:45, vlakas <nginx-forum@nginx.us> wrote:

> Еще доп. информация.
>
> Тут было упомянуто, что cache manager делает порядка 20 попыток удалить
> "устаревший" кеш. Хотя если искать, скажем по hash id 31dd5198 (http file
> cache forced expire: #1 1 31dd5198), за 10 мин. логов нахожу 515 сообщений
> такого рода.

Имелось в виду, что за один раз он делает 20 попыток.
_______________________________________________
nginx-ru mailing list
nginx-ru@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-ru
↧
Viewing all 53287 articles
Browse latest View live
Search
<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>