Experiments

Imports

from keras_tuner import RandomSearch

environ["TF_FORCE_GPU_ALLOW_GROWTH"] = "true"

Experiments

For our experiments, we employ the datasets used by the authors of Certified Monotonic Network [1] and COMET [2]. We use the exact train-test split provided by the authors. Their respective repositories are linked below in the references. We directly load the saved train-test data split which have been saved after running the codes from respective papers’ authors.

References:

1. Xingchao Liu, Xing Han, Na Zhang, and Qiang Liu. Certified monotonic neural networks. Advances in Neural Information Processing Systems, 33:15427–15438, 2020

Github repo: https://github.com/gnobitab/CertifiedMonotonicNetwork

1. Aishwarya Sivaraman, Golnoosh Farnadi, Todd Millstein, and Guy Van den Broeck. Counterexample-guided learning of monotonic neural networks. Advances in Neural Information Processing Systems, 33:11936–11948, 2020

Github repo: https://github.com/AishwaryaSivaraman/COMET

_download_data("auto", force_download=True)

!ls -l data

assert (Path("data") / "train_auto.csv").exists()

train_auto.csv: 49.2kB [00:01, 48.4kB/s]                            
test_auto.csv: 16.4kB [00:00, 20.2kB/s]

total 257812
-rw-rw-r-- 1 davor davor    11161 Jun  2 13:28 test_auto.csv
-rw-rw-r-- 1 davor davor 11340054 May 25 04:48 test_blog.csv
-rw-rw-r-- 1 davor davor   101210 May 25 04:48 test_compas.csv
-rw-rw-r-- 1 davor davor    15798 May 25 04:48 test_heart.csv
-rw-rw-r-- 1 davor davor 13339777 May 25 04:48 test_loan.csv
-rw-rw-r-- 1 davor davor    44626 Jun  2 13:28 train_auto.csv
-rw-rw-r-- 1 davor davor 79478767 May 25 04:48 train_blog.csv
-rw-rw-r-- 1 davor davor   405660 May 25 04:48 train_compas.csv
-rw-rw-r-- 1 davor davor    62282 May 25 04:48 train_heart.csv
-rw-rw-r-- 1 davor davor 79588030 May 25 04:48 train_loan.csv
-rw-rw-r-- 1 davor davor 79588030 May 29 13:57 {prefix}_{name}.csv

_sanitize_col_names(pd.DataFrame({"a b": [1, 2, 3]}))

| | a_b | |-----|-----| | 0 | 1 | | 1 | 2 | | 2 | 3 |

---

source

get_train_n_test_data

 get_train_n_test_data (dataset_name:str,
                        data_path:Union[pathlib.Path,str,NoneType]='./data
                        ')

Download data

Args: dataset_name: name of the dataset, one of “auto”, “heart”, compas”, “blog”, “loan” data_path: root directory where to download data to

train_df, test_df = get_train_n_test_data("auto")
display(train_df)
display(test_df)

Upload skipped, file /home/davor/work/projects/airt/mono-dense-keras/nbs/data/train_auto.csv exists.
Upload skipped, file /home/davor/work/projects/airt/mono-dense-keras/nbs/data/test_auto.csv exists.

| | Cylinders | Displacement | Horsepower | Weight | Acceleration | Model_Year | Origin | ground_truth | |-----|-----------|--------------|------------|-----------|--------------|------------|-----------|--------------| | 0 | 1.482807 | 1.073028 | 0.650564 | 0.606625 | -1.275546 | -1.631803 | -0.701669 | 18.0 | | 1 | 1.482807 | 1.482902 | 1.548993 | 0.828131 | -1.452517 | -1.631803 | -0.701669 | 15.0 | | 2 | 1.482807 | 1.044432 | 1.163952 | 0.523413 | -1.275546 | -1.631803 | -0.701669 | 16.0 | | 3 | 1.482807 | 1.025368 | 0.907258 | 0.542165 | -1.806460 | -1.631803 | -0.701669 | 17.0 | | 4 | 1.482807 | 2.235927 | 2.396084 | 1.587581 | -1.983431 | -1.631803 | -0.701669 | 15.0 | | ... | ... | ... | ... | ... | ... | ... | ... | ... | | 309 | 0.310007 | 0.358131 | 0.188515 | -0.177437 | -0.319901 | 1.720778 | -0.701669 | 22.0 | | 310 | -0.862792 | -0.566468 | -0.530229 | -0.722413 | -0.921604 | 1.720778 | -0.701669 | 36.0 | | 311 | -0.862792 | -0.928683 | -1.351650 | -1.003691 | 3.184131 | 1.720778 | 0.557325 | 44.0 | | 312 | -0.862792 | -0.566468 | -0.530229 | -0.810312 | -1.417123 | 1.720778 | -0.701669 | 32.0 | | 313 | -0.862792 | -0.709448 | -0.658576 | -0.423555 | 1.060475 | 1.720778 | -0.701669 | 28.0 |

314 rows × 8 columns

| | Cylinders | Displacement | Horsepower | Weight | Acceleration | Model_Year | Origin | ground_truth | |-----|-----------|--------------|------------|-----------|--------------|------------|-----------|--------------| | 0 | -0.862792 | -1.043066 | -1.017947 | -1.027131 | 1.272841 | 1.162014 | 1.816319 | 40.8 | | 1 | 1.482807 | 1.177880 | 1.163952 | 0.526929 | -1.629489 | -1.631803 | -0.701669 | 18.0 | | 2 | 1.482807 | 1.482902 | 1.934034 | 0.794143 | -1.629489 | -0.793657 | -0.701669 | 11.0 | | 3 | 0.310007 | 0.529707 | -0.119518 | 0.346443 | -0.213718 | -1.352421 | -0.701669 | 19.0 | | 4 | -0.862792 | -1.004939 | -0.863931 | -1.243949 | -0.567661 | 0.882633 | 0.557325 | 31.9 | | ... | ... | ... | ... | ... | ... | ... | ... | ... | | 73 | -0.862792 | -0.699916 | 0.188515 | -0.062582 | -0.390690 | -1.073039 | 0.557325 | 18.0 | | 74 | -0.862792 | -0.518809 | -0.838261 | -0.686081 | 1.379024 | -0.793657 | -0.701669 | 21.0 | | 75 | 0.310007 | -0.251914 | 0.701903 | -0.089538 | -1.487912 | 1.162014 | 1.816319 | 32.7 | | 76 | 1.482807 | 1.492434 | 1.138283 | 1.580549 | -0.390690 | 0.323869 | -0.701669 | 16.0 | | 77 | 0.310007 | -0.375829 | 0.060168 | -0.602870 | -0.567661 | -0.793657 | -0.701669 | 21.0 |

78 rows × 8 columns

---

source

peek

 peek (ds:tensorflow.python.data.ops.dataset_ops.DatasetV2)

Returns the first element of the dataset

Args: ds: dataset

Returns: the first element of the dataset

---

source

df2ds

 df2ds (df:pandas.core.frame.DataFrame)

Converts DataFrame to Dataset

Args: df: input DataFrame

Returns: dataset

x, y = peek(df2ds(train_df).batch(8))
display(x)
display(y)

expected = {
    "Acceleration",
    "Cylinders",
    "Displacement",
    "Horsepower",
    "Model_Year",
    "Origin",
    "Weight",
}
assert set(x.keys()) == expected
for k in expected:
    assert x[k].shape == (8,)
assert y.shape == (8,)

{'Cylinders': <tf.Tensor: shape=(8,), dtype=float32, numpy=
 array([1.4828068, 1.4828068, 1.4828068, 1.4828068, 1.4828068, 1.4828068,
        1.4828068, 1.4828068], dtype=float32)>,
 'Displacement': <tf.Tensor: shape=(8,), dtype=float32, numpy=
 array([1.0730283, 1.4829025, 1.0444324, 1.0253685, 2.235927 , 2.474226 ,
        2.3407786, 1.8641808], dtype=float32)>,
 'Horsepower': <tf.Tensor: shape=(8,), dtype=float32, numpy=
 array([0.65056413, 1.5489933 , 1.1639522 , 0.9072582 , 2.3960838 ,
        2.9608107 , 2.8324637 , 2.1907284 ], dtype=float32)>,
 'Weight': <tf.Tensor: shape=(8,), dtype=float32, numpy=
 array([0.6066247, 0.828131 , 0.5234134, 0.5421652, 1.5875812, 1.602817 ,
        1.5535934, 1.0121336], dtype=float32)>,
 'Acceleration': <tf.Tensor: shape=(8,), dtype=float32, numpy=
 array([-1.2755462, -1.4525175, -1.2755462, -1.8064601, -1.9834315,
        -2.3373742, -2.5143454, -2.5143454], dtype=float32)>,
 'Model_Year': <tf.Tensor: shape=(8,), dtype=float32, numpy=
 array([-1.6318026, -1.6318026, -1.6318026, -1.6318026, -1.6318026,
        -1.6318026, -1.6318026, -1.6318026], dtype=float32)>,
 'Origin': <tf.Tensor: shape=(8,), dtype=float32, numpy=
 array([-0.7016686, -0.7016686, -0.7016686, -0.7016686, -0.7016686,
        -0.7016686, -0.7016686, -0.7016686], dtype=float32)>}

<tf.Tensor: shape=(8,), dtype=float32, numpy=array([18., 15., 16., 17., 15., 14., 14., 15.], dtype=float32)>

train_df, test_df = get_train_n_test_data("auto")
train_ds = df2ds(train_df)
test_ds = df2ds(test_df)

build_model_f = lambda: _build_mono_model_f(
    monotonicity_indicator={
        "Cylinders": 0,
        "Displacement": -1,
        "Horsepower": -1,
        "Weight": -1,
        "Acceleration": 0,
        "Model_Year": 0,
        "Origin": 0,
    },
    final_activation=None,
    loss="mse",
    metrics="mse",
    train_ds=train_ds,
    batch_size=8,
    units=16,
    n_layers=3,
    activation="elu",
    learning_rate=0.01,
    weight_decay=0.001,
    dropout=0.25,
    decay_rate=0.95,
)
model = build_model_f()
model.summary()
model.fit(train_ds.batch(8), validation_data=test_ds.batch(256), epochs=1)

Upload skipped, file /home/davor/work/projects/airt/mono-dense-keras/nbs/data/train_auto.csv exists.
Upload skipped, file /home/davor/work/projects/airt/mono-dense-keras/nbs/data/test_auto.csv exists.
Model: "model"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 Acceleration (InputLayer)      [(None, 1)]          0           []

 Cylinders (InputLayer)         [(None, 1)]          0           []

 Displacement (InputLayer)      [(None, 1)]          0           []

 Horsepower (InputLayer)        [(None, 1)]          0           []

 Model_Year (InputLayer)        [(None, 1)]          0           []

 Origin (InputLayer)            [(None, 1)]          0           []

 Weight (InputLayer)            [(None, 1)]          0           []

 dense_Acceleration (Dense)     (None, 4)            8           ['Acceleration[0][0]']

 dense_Cylinders (Dense)        (None, 4)            8           ['Cylinders[0][0]']

 mono_dense_Displacement_decrea  (None, 4)           8           ['Displacement[0][0]']           
 sing (MonoDense)

 mono_dense_Horsepower_decreasi  (None, 4)           8           ['Horsepower[0][0]']             
 ng (MonoDense)

 dense_Model_Year (Dense)       (None, 4)            8           ['Model_Year[0][0]']

 dense_Origin (Dense)           (None, 4)            8           ['Origin[0][0]']

 mono_dense_Weight_decreasing (  (None, 4)           8           ['Weight[0][0]']                 
 MonoDense)

 preprocessed_features (Concate  (None, 28)          0           ['dense_Acceleration[0][0]',     
 nate)                                                            'dense_Cylinders[0][0]',        
                                                                  'mono_dense_Displacement_decreas
                                                                 ing[0][0]',                      
                                                                  'mono_dense_Horsepower_decreasin
                                                                 g[0][0]',                        
                                                                  'dense_Model_Year[0][0]',       
                                                                  'dense_Origin[0][0]',           
                                                                  'mono_dense_Weight_decreasing[0]
                                                                 [0]']

 mono_dense_0 (MonoDense)       (None, 16)           464         ['preprocessed_features[0][0]']

 dropout (Dropout)              (None, 16)           0           ['mono_dense_0[0][0]']

 mono_dense_1_increasing (MonoD  (None, 16)          272         ['dropout[0][0]']                
 ense)

 dropout_1 (Dropout)            (None, 16)           0           ['mono_dense_1_increasing[0][0]']

 mono_dense_2_increasing (MonoD  (None, 1)           17          ['dropout_1[0][0]']              
 ense)

==================================================================================================
Total params: 809
Trainable params: 809
Non-trainable params: 0
__________________________________________________________________________________________________
40/40 [==============================] - 4s 12ms/step - loss: 150.6315 - mse: 150.6315 - val_loss: 342.8134 - val_mse: 342.8134

<keras.callbacks.History>

def hp_params_f(hp: HyperParameters):
    return dict(
        units=hp.Fixed(name="units", value=3),
        layers=hp.Fixed(name="units", value=1),
    )


with TemporaryDirectory() as d:
    tuner = RandomSearch(
        hypermodel=TestHyperModel(
            monotonicity_indicator={
                "Cylinders": 0,
                "Displacement": -1,
                "Horsepower": -1,
                "Weight": -1,
                "Acceleration": 0,
                "Model_Year": 0,
                "Origin": 0,
            },
            hp_params_f=lambda hp: {"units": hp.Fixed(name="units", value=3)},
            final_activation=None,
            loss="mse",
            metrics="mse",
            train_ds=train_ds,
            batch_size=8,
        ),
        directory=d,
        project_name="testing",
        max_trials=2,
        objective="val_loss",
    )
    tuner.search(
        train_ds.shuffle(len(train_ds)).batch(8).prefetch(2),
        validation_data=test_ds.batch(256),
        epochs=2,
    )

Trial 2 Complete [00h 00m 03s]
val_loss: 28.08372688293457

Best val_loss So Far: 28.08372688293457
Total elapsed time: 00h 00m 07s
INFO:tensorflow:Oracle triggered exit

---

source

find_hyperparameters

 find_hyperparameters (dataset_name:str,
                       monotonicity_indicator:Dict[str,int], final_activat
                       ion:Union[str,Callable[[Union[tensorflow.python.typ
                       es.core.Tensor,tensorflow.python.types.core.TensorP
                       rotocol,int,float,bool,str,bytes,complex,tuple,list
                       ,numpy.ndarray,numpy.generic],Union[tensorflow.pyth
                       on.types.core.Tensor,tensorflow.python.types.core.T
                       ensorProtocol,int,float,bool,str,bytes,complex,tupl
                       e,list,numpy.ndarray,numpy.generic]],Union[tensorfl
                       ow.python.types.core.Tensor,tensorflow.python.types
                       .core.TensorProtocol,int,float,bool,str,bytes,compl
                       ex,tuple,list,numpy.ndarray,numpy.generic]]], loss:
                       Union[str,Callable[[Union[tensorflow.python.types.c
                       ore.Tensor,tensorflow.python.types.core.TensorProto
                       col,int,float,bool,str,bytes,complex,tuple,list,num
                       py.ndarray,numpy.generic],Union[tensorflow.python.t
                       ypes.core.Tensor,tensorflow.python.types.core.Tenso
                       rProtocol,int,float,bool,str,bytes,complex,tuple,li
                       st,numpy.ndarray,numpy.generic]],Union[tensorflow.p
                       ython.types.core.Tensor,tensorflow.python.types.cor
                       e.TensorProtocol,int,float,bool,str,bytes,complex,t
                       uple,list,numpy.ndarray,numpy.generic]]], metrics:U
                       nion[str,Callable[[Union[tensorflow.python.types.co
                       re.Tensor,tensorflow.python.types.core.TensorProtoc
                       ol,int,float,bool,str,bytes,complex,tuple,list,nump
                       y.ndarray,numpy.generic],Union[tensorflow.python.ty
                       pes.core.Tensor,tensorflow.python.types.core.Tensor
                       Protocol,int,float,bool,str,bytes,complex,tuple,lis
                       t,numpy.ndarray,numpy.generic]],Union[tensorflow.py
                       thon.types.core.Tensor,tensorflow.python.types.core
                       .TensorProtocol,int,float,bool,str,bytes,complex,tu
                       ple,list,numpy.ndarray,numpy.generic]]], hp_params_
                       f:Optional[Callable[[keras_tuner.engine.hyperparame
                       ters.hyperparameters.HyperParameters],Dict[str,Any]
                       ]]=None, max_trials:int=100, max_epochs:int=50,
                       batch_size:int=8, objective:Union[str,keras_tuner.e
                       ngine.objective.Objective], direction:str,
                       dir_root:Union[pathlib.Path,str]='tuner',
                       seed:int=42, executions_per_trial:int=3,
                       max_consecutive_failed_trials:int=5,
                       patience:int=10)

Search for optimal hyperparameters

Args: monotonicity_indicator: monotonicity indicator as used in MonoDense.__init__ final_activation: final activation of the neural network loss: Tensorflow loss function metrics: Tensorflow metrics function hp_params_f: a function constructing sampling hyperparameters using Keras Tuner max_trials: maximum number of trials max_epochs: maximum number of epochs in each trial batch_size: batch size objective: objective, typically f”val_{metrics}” direction: direction of the objective, either “min” or “max” dir_root: root directory for storing Keras Tuner data seed: random seed used to guarantee reproducibility of results executions_per_trial: number of executions per trial. Set it to number higher than zero for small datasets max_consecutive_failed_trials: maximum number of failed trials as used in Keras Tuner patience: number of epoch with worse objective before stopping trial early

Returns: An instance of Keras Tuner

shutil.rmtree("tuner", ignore_errors=True)

tuner = find_hyperparameters(
    "auto",
    monotonicity_indicator={
        "Cylinders": 0,
        "Displacement": -1,
        "Horsepower": -1,
        "Weight": -1,
        "Acceleration": 0,
        "Model_Year": 0,
        "Origin": 0,
    },
    max_trials=2,
    final_activation=None,
    loss="mse",
    metrics="mse",
    objective="val_mse",
    direction="min",
    max_epochs=1,
    executions_per_trial=1,
)

Trial 2 Complete [00h 00m 03s]
val_mse: 32.87412643432617

Best val_mse So Far: 32.87412643432617
Total elapsed time: 00h 00m 06s
INFO:tensorflow:Oracle triggered exit

---

source

create_tuner_stats

 create_tuner_stats (tuner:keras_tuner.engine.tuner.Tuner,
                     num_models:int=10, max_epochs:int=50,
                     batch_size:int=8, patience:int=10, verbose:int=0)

Calculates statistics for the best models found by Keras Tuner

Args: tuner: an instance of Keras Tuner num_models: number of best models to use for calculating statistics max_epochs: maximum number of epochs used in runs batch_size: batch_size patience: maximum number of epochs with worse objective before stopping trial early verbose: verbosity level of Model.fit function

Returns: A dataframe with statistics

stats = create_tuner_stats(tuner, verbose=0)

Upload skipped, file /home/davor/work/projects/airt/mono-dense-keras/nbs/data/train_auto.csv exists.
Upload skipped, file /home/davor/work/projects/airt/mono-dense-keras/nbs/data/test_auto.csv exists.
WARNING:tensorflow:6 out of the last 8 calls to <function Model.make_test_function.<locals>.test_function> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.

| | units | n_layers | activation | learning_rate | weight_decay | dropout | decay_rate | val_mse_mean | val_mse_std | val_mse_min | val_mse_max | params | |-----|-------|----------|------------|---------------|--------------|----------|------------|--------------|-------------|-------------|-------------|--------| | 0 | 9 | 2 | elu | 0.265157 | 0.196993 | 0.456821 | 0.560699 | 12.738773 | 1.8673 | 10.745923 | 15.125115 | 173 |

| | units | n_layers | activation | learning_rate | weight_decay | dropout | decay_rate | val_mse_mean | val_mse_std | val_mse_min | val_mse_max | params | |-----|-------|----------|------------|---------------|--------------|----------|------------|--------------|-------------|-------------|-------------|--------| | 0 | 9 | 2 | elu | 0.265157 | 0.196993 | 0.456821 | 0.560699 | 12.738773 | 1.86730 | 10.745923 | 15.125115 | 173 | | 1 | 23 | 1 | elu | 0.004715 | 0.265345 | 0.175923 | 0.816107 | 21.378424 | 1.74334 | 18.393272 | 22.992588 | 106 |