aha/net/sunrpc
Neil Brown 61d0a8e6a8 NFS/RPC: fix problems with reestablish_timeout and related code.
[[resending with correct cc:  - "vfs.kernel.org" just isn't right!]]

xprt->reestablish_timeout is used to cause TCP connection attempts to
back off if the connection fails so as not to hammer the network,
but to still allow immediate connections when there is no reason to
believe there is a problem.

It is not used for the first connection (when transport->sock is NULL)
but only on reconnects.

It is currently set:

 a/ to 0 when xs_tcp_state_change finds a state of TCP_FIN_WAIT1
    on the assumption that the client has closed the connection
    so the reconnect should be immediate when needed.
 b/ to at least XS_TCP_INIT_REEST_TO when xs_tcp_state_change
    detects TCP_CLOSING or TCP_CLOSE_WAIT on the assumption that the
    server closed the connection so a small delay at least is
    required.
 c/ as above when xs_tcp_state_change detects TCP_SYN_SENT, so that
    it is never 0 while a connection has been attempted, else
    the doubling will produce 0 and there will be no backoff.
 d/ to double is value (up to a limit) when delaying a connection,
    thus providing exponential backoff and
 e/ to XS_TCP_INIT_REEST_TO in xs_setup_tcp as simple initialisation.

So you can see it is highly dependant on xs_tcp_state_change being
called as expected.  However experimental evidence shows that
xs_tcp_state_change does not see all state changes.
("rpcdebug -m rpc trans" can help show what actually happens).

Results show:
 TCP_ESTABLISHED is reported when a connection is made.  TCP_SYN_SENT
 is never reported, so rule 'c' above is never effective.

 When the server closes the connection, TCP_CLOSE_WAIT and
 TCP_LAST_ACK *might* be reported, and TCP_CLOSE is always
 reported.  This rule 'b' above will sometimes be effective, but
 not reliably.

 When the client closes the connection, it used to result in
 TCP_FIN_WAIT1, TCP_FIN_WAIT2, TCP_CLOSE.  However since commit
 f75e674 (SUNRPC: Fix the problem of EADDRNOTAVAIL syslog floods on
 reconnect) we don't see *any* events on client-close.  I think this
 is because xs_restore_old_callbacks is called to disconnect
 xs_tcp_state_change before the socket is closed.
 In any case, rule 'a' no longer applies.

So all that is left are rule d, which successfully doubles the
timeout which is never rest, and rule e which initialises the timeout.

Even if the rules worked as expected, there would be a problem because
a successful connection does not reset the timeout, so a sequence
of events where the server closes the connection (e.g. during failover
testing) will cause longer and longer timeouts with no good reason.

This patch:

 - sets reestablish_timeout to 0 in xs_close thus effecting rule 'a'
 - sets it to 0 in xs_tcp_data_ready to ensure that a successful
   connection resets the timeout
 - sets it to at least XS_TCP_INIT_REEST_TO after it is doubled,
   thus effecting rule c

I have not reimplemented rule b and the new version of rule c
seems sufficient.

I suspect other code in xs_tcp_data_ready needs to be revised as well.
For example I don't think connect_cookie is being incremented as often
as it should be.

Signed-off-by: NeilBrown <neilb@suse.de>
Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
2009-09-23 14:36:37 -04:00
..
auth_gss sunrpc: reply AUTH_BADCRED to RPCSEC_GSS with unknown service 2009-08-25 17:39:43 -04:00
xprtrdma Merge branch 'nfs-for-2.6.32' of git://git.linux-nfs.org/projects/trondmy/nfs-2.6 into for-2.6.32-incoming 2009-08-21 11:27:29 -04:00
addr.c SUNRPC: Provide functions for managing universal addresses 2009-08-09 15:09:34 -04:00
auth.c SUNRPC: Defer the auth_gss upcall when the RPC call is asynchronous 2009-09-15 20:49:33 -04:00
auth_generic.c SUNRPC: Defer the auth_gss upcall when the RPC call is asynchronous 2009-09-15 20:49:33 -04:00
auth_null.c NFSv4: Don't use cred->cr_ops->cr_name in nfs4_proc_setclientid() 2008-04-19 16:54:53 -04:00
auth_unix.c SUNRPC: Use GFP_NOFS when allocating credentials 2008-07-09 12:08:48 -04:00
backchannel_rqst.c nfs41: sunrpc: xprt_alloc_bc_request() should not use spin_lock_bh() 2009-06-20 14:55:39 -04:00
bc_svc.c nfs41: Backchannel callback service helper routines 2009-06-17 14:11:28 -07:00
cache.c sunrpc/cache: avoid variable over-loading in cache_defer_req 2009-09-18 17:01:12 -04:00
clnt.c Merge branch 'for-2.6.32' of git://linux-nfs.org/~bfields/linux 2009-09-22 07:54:33 -07:00
Kconfig Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial 2009-04-03 15:24:35 -07:00
Makefile SUNRPC: Provide functions for managing universal addresses 2009-08-09 15:09:34 -04:00
rpc_pipe.c const: mark remaining super_operations const 2009-09-22 07:17:24 -07:00
rpcb_clnt.c SUNRPC: Eliminate PROC macro from rpcb_clnt 2009-08-09 15:09:44 -04:00
sched.c nfsd41: sunrpc: move struct rpc_buffer def into sunrpc.h 2009-09-10 12:09:06 -04:00
socklib.c SUNRPC: temp var should match return type of xdr_skb_read_actor 2008-01-30 02:05:43 -05:00
stats.c nfs41: Rename rq_received to rq_reply_bytes_recvd 2009-06-17 14:11:40 -07:00
sunrpc.h nfsd41: sunrpc: Added rpc server-side backchannel handling 2009-09-11 15:04:16 -04:00
sunrpc_syms.c SUNRPC: Ensure that sunrpc gets initialised before nfs, lockd, etc... 2009-08-21 08:17:56 -04:00
svc.c Merge commit 'linux-pnfs/nfs41-for-2.6.31' into nfsv41-for-2.6.31 2009-06-17 17:59:58 -07:00
svc_xprt.c nfsd41: sunrpc: Added rpc server-side backchannel handling 2009-09-11 15:04:16 -04:00
svcauth.c SUNRPC: The sunrpc server code should not be used by out-of-tree modules 2009-01-07 17:18:42 -05:00
svcauth_unix.c Merge branch 'nfs-for-2.6.32' of git://git.linux-nfs.org/projects/trondmy/nfs-2.6 into for-2.6.32-incoming 2009-08-21 11:27:29 -04:00
svcsock.c nfsd41: sunrpc: Added rpc server-side backchannel handling 2009-09-11 15:04:16 -04:00
sysctl.c sunrpc: fix possible overrun on read of /proc/sys/sunrpc/transports 2008-09-01 14:24:24 -04:00
timer.c SUNRPC: Add documenting comments in net/sunrpc/timer.c 2009-08-09 15:09:47 -04:00
xdr.c sunrpc: ntoh -> be*_to_cpu 2009-08-14 13:12:52 -04:00
xprt.c nfsd41: sunrpc: Added rpc server-side backchannel handling 2009-09-11 15:04:16 -04:00
xprtsock.c NFS/RPC: fix problems with reestablish_timeout and related code. 2009-09-23 14:36:37 -04:00