pyfarm.agent.http.core.client module

HTTP Client

The client library the manager uses to communicate with the master server.

class pyfarm.agent.http.core.client.HTTPLog

Bases: object

Provides a wrapper around the http logger so requests and responses can be logged in a standardized fashion.

static queue(method, url, uid=None)

Logs the request we’re asking treq to queue

static response(response, uid=None)

Logs the return code of a request that treq completed

static error(failure, uid=None, method=None, url=None)

Called when the treq request experiences an error and calls the errback method.

pyfarm.agent.http.core.client.build_url(url, params=None)

Builds the full url when provided the base url and some url parameters:

>>> build_url("/foobar", {"first": "foo", "second": "bar"})
'/foobar?first=foo&second=bar'
>>> build_url("/foobar bar/")
''/foobar%20bar/'

Parameters:

• url (str) – The url to build off of.
• params (dict) – A dictionary of parameters that should be added on to url. If this value is not provided url will be returned by itself. Arguments to a url are unordered by default however they will be sorted alphabetically so the results are repeatable from call to call.

pyfarm.agent.http.core.client.http_retry_delay(offset=None, factor=None, rand=None)

Returns a floating point value that can be used to delay an http request. The main purpose of this is to ensure that not all requests are run with the same interval between them.

The basic formula for the retry delay is:

offset * (random() * factor)

Parameters:

• factor (int or float) –

  The factor to multiply the output from random() by.

  This defaults to the agent_http_retry_delay_factor configuration variable.

• offset –

  The initial offset to start the calculation at.

  This defaults to the agent_http_retry_delay_offset configuration variable.

• rand – A callable to determine randomness, defaulting to random(). This is mainly used for testing purposes.

class pyfarm.agent.http.core.client.Request

Bases: pyfarm.agent.http.core.client.Request

Contains all the information used to perform a request such as the method, url, and original keyword arguments (kwargs). These values contain the basic information necessary in order to retry() a request.

retry(**kwargs)

When called this will rerun the original request with all of the original arguments to request()

Parameters:kwargs – Additional keyword arguments which should override the original keyword argument(s).

class pyfarm.agent.http.core.client.Response(deferred, response, request)

Bases: twisted.internet.protocol.Protocol

This class receives the incoming response body from a request constructs some convenience methods and attributes around the data.

Parameters:

• deferred (Deferred) – The deferred object which contains the target callback and errback.
• response – The initial response object which will be passed along to the target deferred.
• request (Request) – Named tuple object containing the method name, url, headers, and data.

data()

Returns the data currently contained in the buffer.

Raises:RuntimeError – Raised if this method id called before all data has been received.

json(loader=<function loads>)

Returns the json data from the incoming request

Raises:

• RuntimeError – Raised if this method id called before all data has been received.
• ValueError – Raised if the content type for this request is not application/json.

dataReceived(data)

Overrides Protocol.dataReceived() and appends data to _body.

connectionLost(reason=<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>)

Overrides Protocol.connectionLost() and sets the _done when complete. When called with ResponseDone for reason this method will call the callback on _deferred

pyfarm.agent.http.core.client.request(method, url, **kwargs)

Wrapper around treq.request() with some added arguments and validation.

Parameters:

• method (str) – The HTTP method to use when making the request.
• url (str) – The url this request will be made to.
• data (str, list, tuple, set, dict) – The data to send along with some types of requests such as POST or PUT
• headers (dict) – The headers to send along with the request to url. Currently only single values per header are supported.
• callback (function) – The function to deliver an instance of Response once we receive and unpack a response.
• errback (function) – The function to deliver an error message to. By default this will use log.err().
• response_class (class) – The class to use to unpack the internal response. This is mainly used by the unittests but could be used elsewhere to add some custom behavior to the unpack process for the incoming response.

Raises:

NotImplementedError – Raised whenever a request is made of this function that we can’t implement such as an invalid http scheme, request method or a problem constructing data to an api.

pyfarm.agent.http.core.client.random() → x in the interval [0, 1).