CHANfiG¶
Introduction¶
CHANfiG aims to make your configuration easier.
There are tons of configurable parameters in training a Machine Learning model.
To configure all these parameters, researchers usually need to write gigantic config files, sometimes even thousands of lines.
Most of the configs are just replicates of the default arguments of certain functions, resulting in many unnecessary declarations.
It is also very hard to alter the configurations.
One needs to navigate and open the right configuration file, make changes, save and exit.
These had wasted an uncountable1 amount of precious time and are no doubt a crime.
Using argparse
could relieve the burdens to some extent.
However, it takes a lot of work to make it compatible with existing config files, and its lack of nesting limits its potential.
CHANfiG would like to make a change.
You just type the alternations in the command line, and leave everything else to CHANfiG.
CHANfiG is highly inspired by YACS.
Different from the paradigm of YACS(
your code + a YACS config for experiment E (+ external dependencies + hardware + other nuisance terms ...) = reproducible experiment E
),
The paradigm of CHANfiG is:
your code + command line arguments (+ optional CHANfiG config + external dependencies + hardware + other nuisance terms ...) = reproducible experiment E (+ optional CHANfiG config for experiment E)
Components¶
A Config is basically a nested dict structure.
However, the default Python dict is hard to manipulate.
The only way to access a dict member is through dict['name']
, which is obviously extremely complex.
Even worse, if the dict is nested like a config, member access could be something like dict['parent']['children']['name']
.
Enough is enough, it is time to make a change.
We need attribute-style access, and we need it now.
dict.name
and dict.parent.children.name
is all you need.
Although there have been some other works that achieve a similar functionality of attribute-style access to dict members.
Their Config object either uses a separate dict to store information from attribute-style access (EasyDict), which may lead to inconsistency between attribute-style access and dict-style access;
or re-use the existing __dict__
and redirect dict-style access (ml_collections), which may result in confliction between attributes and members of Config.
To overcome the aforementioned limitations, we inherit the Python built-in dict
to create FlatDict
, DefaultDict
, NestedDict
, Config
, and Registry
.
We also introduce Variable
to allow sharing a value across multiple places, and ConfigParser
to parse command line arguments.
FlatDict¶
FlatDict
improves the default dict
in 3 aspects.
Dict Operations¶
FlatDict
incorporates a merge
method that allows you to merge a Mapping
, an Iterable
, or a path to the FlatDict
.
Different from built-in update
, merge
assign values instead of replace, which makes it works better with DefaultDict
.
dict
in Python is ordered since Python 3.7, but there isn’t a built-in method to help you sort a dict
. FlatDict
supportssort
to help you manage your dict.
Moreover, FlatDict
comes with difference
and intersect
, which makes it very easy to compare a FlatDict
with other Mapping
, Iterable
, or a path.
ML Operations¶
FlatDict
supports to
method similar to PyTorch Tensors.
You can simply convert all member values of FlatDict
to a certain type or pass to a device in the same way.
FlatDict
also integrates cpu
, gpu
(cuda
), and tpu
(xla
) methods for easier access.
IO Operations¶
FlatDict
provides json
, jsons
, yaml
and yamls
methods to dump FlatDict
to a file or string.
It also provides from_json
, from_jsons
, from_yaml
and from_yamls
methods to build a FlatDict
from a string or file.
FlatDict
also includes dump
and load
methods which determines the type by its extension and dump/load FlatDict
to/from a file.
DefaultDict¶
To facility the needs of default values, we incorporate DefaultDict
which accepts default_factory
and works just like a collections.defaultdict
.
NestedDict¶
Since most Configs are in a nested structure, we further propose a NestedDict
.
Based on DefaultDict
, NestedDict
provides all_keys
, all_values
, and all_items
methods to allow iterating over the whole nested structure at once.
NestedDict
also comes with apply
and apply_
methods, which made it easier to manipulate the nested structure.
Config¶
Config
extends the functionality by supporting freeze
and defrost
, and by adding a built-in ConfigParser
to pare command line arguments.
Note that Config
also has default_factory=Config()
by default for convenience.
Registry¶
Registry
extends the NestedDict
and supports register
, lookup
, and build
to help you register constructors and build objects from a Config
.
Variable¶
Have one value for multiple names at multiple places? We got you covered.
Just wrap the value with Variable
, and one alteration will be reflected everywhere.
Variable
also supports type
, choices
, validator
, and required
to ensure the correctness of the value.
To make it even easier, Variable
also supports help
to provide a description when using ConfigParser
.
ConfigParser¶
ConfigParser
extends ArgumentParser
and provides parse
and parse_config
to parse command line arguments.
Usage¶
CHANfiG has great backward compatibility with previous configs.
No matter if your old config is json or yaml, you could directly read from them.
And if you are using yacs, just replace CfgNode
with Config
and enjoy all the additional benefits that CHANfiG provides.
Moreover, if you find a name in the config is too long for command-line, you could simply call self.add_argument
with proper dest
to use a shorter name in command-line, as you do with argparse
.
All you need to do is just run a line:
Bash | |
---|---|
You could also load a default configure file and make changes based on it:
Note, you must specify config.parse(default_config='config')
to correctly load the default config.
Bash | |
---|---|
If you have made it dump current configurations, this should be in the written file:
YAML | |
---|---|
JSON | |
---|---|
Define the default arguments in function, put alterations in CLI, and leave the rest to CHANfiG.
Installation¶
Install the most recent stable version on pypi:
Bash | |
---|---|
Install the latest version from source:
Bash | |
---|---|
It works the way it should have worked.
License¶
CHANfiG is multi-licensed under the following licenses:
- The Unlicense
- GNU Affero General Public License v3.0 or later
- GNU General Public License v2.0 or later
- BSD 4-Clause “Original” or “Old” License
- MIT License
- Apache License 2.0
You can choose any (one or more) of these licenses if you use this work.
SPDX-License-Identifier: Unlicense OR AGPL-3.0-or-later OR GPL-2.0-or-later OR BSD-4-Clause OR MIT OR Apache-2.0
-
fun fact: time is always uncountable. ↩