pyfarm.core.config module¶
Configuration Object¶
Basic module used for reading configuration data into PyFarm in various forms.
const BOOLEAN_TRUE: | |
---|---|
set of values which will return a True boolean value from
read_env_bool() |
|
const BOOLEAN_FALSE: | |
set of values which will return a False boolean value from
read_env_bool() |
-
class
pyfarm.core.config.
Configuration
(name, version=None, cwd=None)¶ Bases:
dict
Main object responsible for finding, loading, and merging configuration data. By default this class does nothing until
load()
is called. Once this method is calledConfiguration
class will populate itself with data loaded from the configuration files. The configuration files themselves can be loaded from multiple location depending on the system’s setup. For example on Linux you might end up attempting to load these files for pyfarm.agent v1.2.3:Override paths set by
DEFAULT_ENVIRONMENT_PATH_VARIABLE
. By default this path will not be set, this is only an example./tmp/pyfarm/agent/1.2.3/agent.yml
/tmp/pyfarm/agent/1.2/agent.yml
/tmp/pyfarm/agent/1/agent.yml
/tmp/pyfarm/agent/agent.yml
Paths relative to the current working directory or the directory provided to
cwd
whenConfiguration
was instanced.etc/pyfarm/agent/1.2.3/agent.yml
etc/pyfarm/agent/1.2/agent.yml
etc/pyfarm/agent/1/agent.yml
etc/pyfarm/agent/agent.yml
User’s home directory
~/.pyfarm/agent/1.2.3/agent.yml
~/.pyfarm/agent/1.2/agent.yml
~/.pyfarm/agent/1/agent.yml
~/.pyfarm/agent/agent.yml
System level paths
/etc/pyfarm/agent/1.2.3/agent.yml
/etc/pyfarm/agent/1.2/agent.yml
/etc/pyfarm/agent/1/agent.yml
/etc/pyfarm/agent/agent.yml
Finally, if we don’t locate a configuration file in any of the above paths we’ll use the file which was installed along side the source code.
Configuration
will only attempt to load data from files which exist on the file system whenload()
is called. If multiple files exist the data will be loaded from each file with the successive data overwriting the value from the previously loaded configuration file. So if you have two files containing the same data:/etc/pyfarm/agent/agent.yml
env: a: 0 foo: 1 bar: true
etc/pyfarm/agent/1.2.3/agent.yml
env: a: 1 b: 1 foo: 0
You’ll end up with a single merged configuration. Please note that the only keys which will be merged in the configuration are the
env
key. Configuration files are meant to store simple data and while it can be used to store more complicate data it won’t merge any other data structures.env: a: 1 b: 1 foo: 0 bar: true
Variables: - DEFAULT_SYSTEM_ROOT (string) –
The system level directory that we should look for configuration files in. This path is platform dependent:
- Linux - /etc/
- Mac - /Library/
- Windows - %ProgramData%. An environment variable that varies depending on the Windows version. See Microsoft’s docs: https://www.microsoft.com/security/portal/mmpc/shared/variables.aspx
The value built here will be copied onto the instance as
system_root
- DEFAULT_USER_ROOT (string) –
The user level directory that we should look for configuration files in. This path is platform dependent:
- Linux/Mac - ~ (home directory)
- Windows - %APPDATA%. An environment variable that varies depending on the Windows version. See Microsoft’s docs: https://www.microsoft.com/security/portal/mmpc/shared/variables.aspx
The value built here will be copied onto the instance as
user_root
- DEFAULT_FILE_EXTENSION (string) – The default file extension of the configuration files. This will
default to
.yml
and will be copied tofile_extension
when the class is instanced. - DEFAULT_LOCAL_DIRECTORY_NAME (string) – A directory local to the current process which we should search
for configuration files in. This will default to
etc
and will be copied tolocal_dir
when the class is instanced. - DEFAULT_PARENT_APPLICATION_NAME (string) – The base name of the parent application. This used used to build
child directories and will default to
pyfarm
. - DEFAULT_ENVIRONMENT_PATH_VARIABLE (string) – A environment variable to search for a configuration path in. The value
defined here, which defaults to
PYFARM_CONFIG_ROOT
, will be read from the environment whenConfiguration
is instanced. This allows for an non-standard configuration location to be loaded first for testing forced-override of the configuration. - DEFAULT_TEMP_DIRECTORY_ROOT – The directory which will store any temporary files.
Parameters: - name (string) – The name of the configuration itself, typically ‘master’ or ‘agent’. This may also be the name of a package such as ‘pyfarm.agent’. When the package name is provided we can usually automatically determine the version number.
- version (string) – The version the version of the program running.
- cwd (string) – The current working directory to construct the local
path from. If not provided then we’ll use
os.getcwd()
to determine the current working directory.
-
_expandvars
(value)¶ Performs variable expansion for
value
. This method is run when a string value is returned fromget()
or__getitem__()
. The default behavior of this method is to recursively expand variables using sources in the following order:- The environment,
os.environ
- The environment (from the configuration),
env
- Other values in the configuration
~
to the user’s home directory
For example, the following configuration:
foo: foo bar: bar foobar: $foo/$bar path: ~/$foobar/$TEST
Would result in the following assuming
$TEST
is an environment variable set tosomevalue
and the current user’s name isuser
:{ "foo": "foo", "bar": "bar", "foobar": "foo/bar", "path": "/home/user/foo/bar/somevalue" }
- The environment,
-
DEFAULT_ENVIRONMENT_PATH_VARIABLE
= 'PYFARM_CONFIG_ROOT'¶
-
DEFAULT_FILE_EXTENSION
= '.yml'¶
-
DEFAULT_LOCAL_DIRECTORY_NAME
= 'etc'¶
-
DEFAULT_PARENT_APPLICATION_NAME
= 'pyfarm'¶
-
DEFAULT_SYSTEM_ROOT
= '/etc'¶
-
DEFAULT_TEMP_DIRECTORY_ROOT
= '/tmp/pyfarm'¶
-
DEFAULT_USER_ROOT
= '/home/docs'¶
-
MAX_EXPANSION_RECURSION
= 10¶
-
directories
(validate=True, unversioned_only=False)¶ Returns a list of platform dependent directories which may contain configuration files.
Parameters:
-
files
(validate=True, unversioned_only=False)¶ Returns a list of configuration files.
Parameters: - validate (bool) –
When
True
this method will only return files which exist on disk.Note
This method calls
directories()
and will be passed the value that is provided tovalidate
here. - unversioned_only (bool) – See the keyword documentation for
unversioned_only
indirectories`()
- validate (bool) –
-
get
(key, default=None)¶ Overrides
dict.get()
to provide internal variable expansion through_expandvars()
.
-
load
(environment=None)¶ Loads data from the configuration files. Any data present in the
env
key in the configuration files will updateenvironment
Parameters: environment (dict) – A dictionary to load data in the env
key from the configuration files into. This would typically be set toos.environ
so the environment itself could be updated.
-
split_version
(sep='.')¶ Splits
self.version
into a tuple of individual versions. For example1.2.3
would be split into['1', '1.2', '1.2.3']
-
pyfarm.core.config.
read_env
(envvar, default=<object object>, warn_if_unset=False, eval_literal=False, raise_eval_exception=True, log_result=True, desc=None, log_defaults=False)¶ Lookup and evaluate an environment variable.
Parameters: - envvar (string) – The environment variable to lookup in
os.environ
- default (object) – Alternate value to return if
envvar
is not present. If this is instead set toNOTSET
then an exception will be raised ifenvvar
is not found. - warn_if_unset (bool) – If True, log a warning if the value being returned is the same
as
default
- eval_literal – if True, run
literal_eval()
on the value retrieved from the environment - raise_eval_exception (bool) – If True and we failed to parse
envvar
withliteral_eval()
then raise aEnvironmentKeyError
- log_result (bool) – If True, log the query and result to INFO. If False, only log the
query itself to DEBUG. This keyword mainly exists so environment
variables such as
PYFARM_SECRET
orPYFARM_DATABASE_URI
stay out of log files. - desc (string) – Describes the purpose of the value being returned. This may also be read in at the time the documentation is built.
- log_defaults (bool) – If False, queries for envvars that have not actually been set and will
just return
default
will not be logged
- envvar (string) – The environment variable to lookup in
-
pyfarm.core.config.
read_env_bool
(*args, **kwargs)¶ Wrapper around
read_env()
which converts environment variables to boolean values. Please see the documentation forread_env()
for additional information on exceptions and input arguments.Raises: - AssertionError – raised if a default value is not provided
- TypeError – raised if the environment variable found was a string and could not be converted to a boolean.
-
pyfarm.core.config.
read_env_number
(*args, **kwargs)¶ Wrapper around
read_env()
which will read a numerical value from an environment variable. Please see the documentation forread_env()
for additional information on exceptions and input arguments.Raises: TypeError – raised if we either failed to convert the value from the environment variable or the value was not a float, integer, or long
-
pyfarm.core.config.
read_env_strict_number
(*args, **kwargs)¶ Strict version of
read_env_number()
which will only return an integerParameters: number_type – the type of number(s) this function must return
Raises: - AssertionError – raised if the number_type keyword is not provided (required to check the type on output)
- TypeError – raised if the type of the result is not an instance of number_type