[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Xen-devel] Re: [PATCH] XML-RPC: Cope with large integers on x86-64 systems
Ewan Mellor wrote:
On Mon, Mar 27, 2006 at 08:29:13PM -0600, Anthony Liguori wrote:Attached patch fixes a problem in marshaling large integers. It seems to only occur on systems with very large memory. However, on those systems, it's a show-stopper. I see this as a temporary solution until we can sanitize the XML-RPC functions and get rid of all the S-Expression stuff.Anthony, This code is clearly plagiarised from Lib/xmlrpclib.py in the Python distribution, without appropriate credit, copyright and licence statements in the code, and without a patch to the documentation as per the licence agreement. I can't accept this patch as it stands.
Sorry, I didn't realize I hadn't put that in there. Attached is another patch with the appropriate references.
Regardless, I don't see what is to be gained in marshalling ints >32bits as strings, but those <=32 bits as integers; the receiving code is going to have to convert the incoming argument unconditionally, so why not just send all integers as strings? They're going to be stringified on the wire anyway, so there's no efficiency loss, and I don't think that anyone fancies auditing Xend to figure out which integers are strict 32 bit integers on all platforms and which aren't, so it would be safer just to pass them all as strings.
This is a pretty big hack which based on Aravindh's note may not be as important as I thought it was. Aravindh bumped NR_CPUS which increased the size of vcpu_avail. I've audited the interface and the only place that large ints can be passed around are in the S-Expressions. The S-Expressions are untyped so the user always has to manually do type conversion. Passing the S-Expressions as strings is always safe.
I think this is the least invasive and safest way to address this problem. Since it will only occur if you bump NR_CPUS, we could perhaps take the time to think of a more clever solution.
Regards, Anthony Liguori
# HG changeset patch # User anthony@xxxxxxxxxxxxxxxxxxxxx # Node ID 16b3267df4ebbfeeab91aadb5a48ed89609e3622 # Parent 8b5a752167a17785b998a080a5c1ce1991b9379b In some cases, Xend can return a very large integer. This so far only happens on 64bit systems where the sizeof the Python integer is > 32bit. Presumably, this is occuring because of the fact that we return PFNs as part of the domain info so on a system with greater than 2G of memory we get into the invalid integer range. For now, the work-around is to overload the xmlrpclib Marshaller and instead of throwing an exception when we get out-of-range integers, we marshal them as strings. In the future, we can handle this in a more elegant way by introducing higher-level types. We also won't have to deal with things like PFNs once we clean up the XML-RPC interface. Signed-off-by: Anthony Liguori <aliguori@xxxxxxxxxx> diff -r 8b5a752167a1 -r 16b3267df4eb tools/python/xen/util/xmlrpclib2.py --- a/tools/python/xen/util/xmlrpclib2.py Mon Mar 27 10:16:36 2006 +++ b/tools/python/xen/util/xmlrpclib2.py Tue Mar 28 15:02:39 2006 @@ -15,15 +15,45 @@ # Copyright (C) 2006 Anthony Liguori <aliguori@xxxxxxxxxx> # Copyright (C) 2006 XenSource Ltd. #============================================================================ +# +# This file contains portions of the xmlrpclib.py that is distributed with +# Python. Those portions are: +# +# Copyright (c) 1999-2002 by Secret Labs AB +# Copyright (c) 1999-2002 by Fredrik Lundh +# +# By obtaining, using, and/or copying this software and/or its +# associated documentation, you agree that you have read, understood, +# and will comply with the following terms and conditions: +# +# Permission to use, copy, modify, and distribute this software and +# its associated documentation for any purpose and without fee is +# hereby granted, provided that the above copyright notice appears in +# all copies, and that both that copyright notice and this permission +# notice appear in supporting documentation, and that the name of +# Secret Labs AB or the author not be used in advertising or publicity +# pertaining to distribution of the software without specific, written +# prior permission. +# +# SECRET LABS AB AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD +# TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANT- +# ABILITY AND FITNESS. IN NO EVENT SHALL SECRET LABS AB OR THE AUTHOR +# BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY +# DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, +# WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS +# ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE +# OF THIS SOFTWARE. +# -------------------------------------------------------------------- """ An enhanced XML-RPC client/server interface for Python. """ from httplib import HTTPConnection, HTTP -from xmlrpclib import Transport +from xmlrpclib import Transport, Fault from SimpleXMLRPCServer import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler -import xmlrpclib, socket, os, traceback +import xmlrpclib, socket, os, traceback, string +from types import * # A new ServerProxy that also supports httpu urls. An http URL comes in the # form: @@ -31,6 +61,9 @@ # httpu:///absolute/path/to/socket.sock # # It assumes that the RPC handler is /RPC2. This probably needs to be improved + +MAXINT = 2L**31-1 +MININT = -2L**31 class HTTPUnixConnection(HTTPConnection): def connect(self): @@ -58,6 +91,67 @@ xmlrpclib.ServerProxy.__init__(self, uri, transport, encoding, verbose, allow_none) +class Marshaller(xmlrpclib.Marshaller): + def __init__(self, encoding=None, allow_none=0): + xmlrpclib.Marshaller.__init__(self, encoding, allow_none) + self.dispatch[IntType] = Marshaller.dump_int + self.dispatch[LongType] = Marshaller.dump_long + + def dump_int(self, value, write): + if value > MAXINT or value < MININT: + self.dispatch[StringType](self, str(value), write) + else: + xmlrpclib.Marshaller.dump_int(self, value, write) + + def dump_long(self, value, write): + if value > MAXINT or value < MININT: + self.dispatch[StringType](self, str(value), write) + else: + xmlrpclib.Marshaller.dump_long(self, value, write) + +def dumps(params, methodname=None, methodresponse=None, encoding=None, + allow_none=0): + assert isinstance(params, TupleType) or isinstance(params, Fault),\ + "argument must be tuple or Fault instance" + + if isinstance(params, Fault): + methodresponse = 1 + elif methodresponse and isinstance(params, TupleType): + assert len(params) == 1, "response tuple must be a singleton" + + if not encoding: + encoding = "utf-8" + + m = Marshaller(encoding, allow_none) + + data = m.dumps(params) + + if encoding != "utf-8": + xmlheader = "<?xml version='1.0' encoding='%s'?>\n" % str(encoding) + else: + xmlheader = "<?xml version='1.0'?>\n" # utf-8 is default + + if methodname: + if not isinstance(methodname, StringType): + methodname = methodname.encode(encoding) + data = ( + xmlheader, + "<methodCall>\n" + "<methodName>", methodname, "</methodName>\n", + data, + "</methodCall>\n" + ) + elif methodresponse: + data = ( + xmlheader, + "<methodResponse>\n", + data, + "</methodResponse>\n" + ) + else: + return data + return string.join(data, "") + # This is a base XML-RPC server for TCP. It sets allow_reuse_address to # true, and has an improved marshaller that serializes unknown exceptions # with full traceback information. @@ -74,13 +168,13 @@ response = self._dispatch(method, params) response = (response,) - response = xmlrpclib.dumps(response, - methodresponse=1, - allow_none=1) + response = dumps(response, + methodresponse=1, + allow_none=1) except xmlrpclib.Fault, fault: - response = xmlrpclib.dumps(fault) + response = dumps(fault) except: - response = xmlrpclib.dumps( + response = dumps( xmlrpclib.Fault(1, traceback.format_exc()) )
_______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-devel
Lists.xenproject.org is hosted with RackSpace, monitoring our