Evaluation and Visualization

---

Advice for developers if needed: Evaluation and visualization

Algorithms in CADIMULC simply represent the causation among variables, for both ground-truth and learning results, as the directed pairs in an adjacency matrix with only two elements 0 and 1.

If you incline to this representation of data structure in your work or research, then Evaluator in CADIMULC might provide you convenience for evaluating the causal graph directly.

Class: Evaluator

cadimulc.utils.evaluation.Evaluator

Given an instance as to causal discovery, the Evaluator defines the classification errors between an actual graph and a predicted graph, which is corresponding to, in the field of machine learning, the four of the categories within a confusion matrix.

• TP (True Positives): The number of the estimated directed pairs that are consistent with the true causal pairs. Namely, TP qualifies the correct estimation of causal relations.

• FP (False Positives): The number of the estimated directed pairs that do not present in the true causal pairs.

• TN: (True Negatives): The number of the unestimated directed pairs that are consistent with the true causal pairs. TN reflects the correct prediction of unpresented causal relations.

• FN (False Negatives): The number of the unestimated directed pairs that do present in the true causal pairs.

The only assessment of directed causal relations

The Evaluator focuses on the assessment of estimated directed pairs (TP) extracted from an adjacency matrix, treating the rest as unpresented pairs (FP) relative to the ground-truth.

In other words, Evaluator in CADIMULC does not explicitly consider bi-directed pairs or undirected pairs.

---

Primary Method: precision_pairwise

Causal pair precision refers to the proportion of the correctly estimated directed pairs in all the estimated directed pairs (EDP):

\[ Precision = TP \ / \ (TP + FP) = TP \ / \ EDP. \]

The higher the precision, the larger the amount of the causal pairs, compared to EDP, that are identified, without considering the amount of unestimated pairs.

Parameters:

Name	Type	Description	Default
true_graph	ndarray	True causal graph, namely the ground-truth.	required
est_graph	ndarray	Estimated causal graph, namely the empirical causal graph.	required

Returns:

Name	Type	Description
precision	float	Precision of the "causal discovery task".

Source code in cadimulc\utils\evaluation.py

@staticmethod
def precision_pairwise(true_graph: ndarray, est_graph: ndarray) -> float:
    """
    **Causal pair precision** refers to the proportion of the correctly estimated
    directed pairs in all the **estimated directed pairs** (EDP):

    $$
        Precision = TP \ / \  (TP + FP) = TP \ / \ EDP.
    $$

    The higher the precision, the larger the amount of the causal pairs,
    compared to EDP, that are identified,
    without considering the amount of **unestimated** pairs.

    Parameters:
        true_graph: True causal graph, namely the ground-truth.
        est_graph: Estimated causal graph, namely the empirical causal graph.
        learned from data.

    Returns:
        precision: Precision of the "causal discovery task".
    """

    _, dict_directed_parent = Evaluator.get_pairwise_info(true_graph)
    est_pairs = Evaluator.get_directed_pairs(est_graph)
    num_est_pairs = len(est_pairs)

    if num_est_pairs > 0:
        tp = 0

        for est_pair in est_pairs:
            child = est_pair[1]
            parent = est_pair[0]

            if parent in dict_directed_parent[child]:
                tp += 1

        precision = round((tp / num_est_pairs), 3)

    else:
        precision = float(0)

    return precision

Primary Method: recall_pairwise

Causal pair recall refers to the proportion of correctly estimated directed pairs in all true causal pairs (TCP):

\[ Recall = TP \ / \ (TP + FN) = TP \ / \ TCP \]

The higher the recall, the larger the amount of the causal pairs, compared to TCP, that are identified, without considering the amount of incorrectly estimated pairs.

Parameters:

Name	Type	Description	Default
true_graph	ndarray	True causal graph, namely the ground-truth.	required
est_graph	ndarray	Estimated causal graph, namely the empirical causal graph.	required

Returns:

Name	Type	Description
recall	float	Recall of the "causal discovery task".

Source code in cadimulc\utils\evaluation.py

@staticmethod
def recall_pairwise(true_graph: ndarray, est_graph: ndarray) -> float:
    """
    **Causal pair recall** refers to the proportion of correctly estimated directed
    pairs in all **true causal pairs** (TCP):

    $$
        Recall = TP \ / \  (TP + FN) = TP \ / \  TCP
    $$

    The higher the recall, the larger the amount of the causal pairs,
    compared to TCP,
    that are identified, without considering the amount of **incorrectly** estimated pairs.

    Parameters:
        true_graph: True causal graph, namely the ground-truth.
        est_graph: Estimated causal graph, namely the empirical causal graph.

    Returns:
        recall: Recall of the "causal discovery task".
    """

    num_directed_pairs, dict_directed_parent = (
        Evaluator.get_pairwise_info(true_graph))

    est_pairs = Evaluator.get_directed_pairs(est_graph)
    num_est_pairs = len(est_pairs)

    if num_directed_pairs == 0:
        recall = float(1)

    elif num_est_pairs == 0:
        recall = float(0)

    else:
        tp = 0

        for est_pair in est_pairs:
            child = est_pair[1]
            parent = est_pair[0]

            if parent in dict_directed_parent[child]:
                tp += 1

        recall = round((tp / num_directed_pairs), 3)

    return recall

Primary Method: f1_score_pairwise

Causal pair F1-score, the concordant mean of the precision and recall, represents the global measurement of causal discovery, bring together the advantages from both the precision and recall.

\[ F1 = (2 * Precision * Recall)\ / \ (Precision + Recall). \]

Parameters:

Name	Type	Description	Default
true_graph	ndarray	True causal graph, namely the ground-truth.	required
est_graph	ndarray	Estimated causal graph, namely the empirical causal graph.	required

Returns:

Name	Type	Description
f1_score	float	F1-score of the "causal discovery task".

Source code in cadimulc\utils\evaluation.py

@staticmethod
def f1_score_pairwise(true_graph: ndarray, est_graph: ndarray) -> float:
    """
    **Causal pair F1-score**, the concordant mean of the precision and recall,
    represents the global measurement of causal discovery, bring together the
    advantages from both the precision and recall.

    $$
        F1 = (2 * Precision * Recall)\  / \ (Precision + Recall).
    $$

    Parameters:
        true_graph: True causal graph, namely the ground-truth.
        est_graph: Estimated causal graph, namely the empirical causal graph.

    Returns:
        f1_score: F1-score of the "causal discovery task".
    """

    precision = Evaluator.precision_pairwise(true_graph, est_graph)
    recall = Evaluator.recall_pairwise(true_graph, est_graph)

    if (precision + recall) != 0:
        f1_score = round(
            (2 * precision * recall) / (precision + recall), 3
        )

    else:
        f1_score = float(0)

    return f1_score

Primary Method: evaluate_skeleton

Note

Construction of a network skeleton is the fundamental part relative to the procedure of hybrid-based approaches. CADIMULC also provides simply way to evaluate the causal skeleton. Notice that performance of the hybrid-based approach largely depends on the initial performance of the causal skeleton learning.

The evaluate_skeleton method evaluates a network skeleton based on an assigned metric. To this end, available metrics mirroring to the causal pair evaluation are list as the following:

• Skeleton Precision = TP (of the estimated skeleton) / all estimated edges.
• Skeleton Recall = TP (of the estimated skeleton) / all true edges.
• Skeleton F1-score = (2 * Precision * Recall) / (Precision + Recall).

Parameters:

Name	Type	Description	Default
true_skeleton	ndarray	True causal skeleton, namely the ground-truth.	required
est_skeleton	ndarray	Estimated causal skeleton, namely the empirical causal skeleton.	required
metric	str	selective metrics from ['Precision', 'Recall', or 'F1-score'].	required

Returns:

Type	Description
float	The evaluating value of the causal skeleton in light of the assigned metric.

Source code in cadimulc\utils\evaluation.py

@staticmethod
def evaluate_skeleton(
        true_skeleton: ndarray,
        est_skeleton: ndarray,
        metric: str
) -> float:
    """
    The `evaluate_skeleton` method evaluates a network skeleton based on an assigned
    metric. To this end, available metrics mirroring to the causal pair evaluation
     are list as the following:

    * **Skeleton Precision** = TP (of the estimated skeleton) / all estimated edges.
    * **Skeleton Recall** = TP (of the estimated skeleton) / all true edges.
    * **Skeleton F1-score** = (2 * Precision * Recall) / (Precision + Recall).

    Parameters:
        true_skeleton: True causal skeleton, namely the ground-truth.
        est_skeleton: Estimated causal skeleton, namely the empirical causal skeleton.
        metric: selective metrics from `['Precision', 'Recall', or 'F1-score']`.

    Returns:
        The evaluating value of the causal skeleton in light of the assigned metric.
    """

    true_skeleton_nx = nx.from_numpy_array(true_skeleton)
    est_skeleton_nx = nx.from_numpy_array(est_skeleton)

    true_edges = list(true_skeleton_nx.edges())
    est_edges = list(est_skeleton_nx.edges())

    tp_skeleton = 0
    for est_edge in est_edges:
        if est_edge in true_edges:
            tp_skeleton += 1

    if len(est_edges) > 0:
        precision = round(tp_skeleton / len(est_edges), 3)
    else:
        precision = float(0)

    if len(true_edges) > 0:
        recall = round(tp_skeleton / len(true_edges), 3)
    else:
        recall = float(0)

    if precision + recall > 0:
        f1_score = round(
            2 * ((precision * recall) / (precision + recall)), 3
        )
    else:
        f1_score = float(0)

    if metric == 'Precision':
        return precision

    elif metric == 'Recall':
        return recall

    elif metric == 'F1-score':
        return f1_score

    else:
        raise ValueError("Please input established metric types: "
                         "'Precision', 'Recall', or 'F1-score'.")

Secondary Method: get_directed_pairs

Extract directed pairs from a graph.

Parameters:

Name	Type	Description	Default
graph	ndarray	An adjacency bool matrix representing the causation among variables.	required

Returns:

Name	Type	Description
direct_pairs	list[list]	A list whose elements are in form of [parent, child], referring to the causation parent -> child.

Source code in cadimulc\utils\evaluation.py

@staticmethod
def get_directed_pairs(graph: ndarray) -> list[list]:
    """
    Extract directed pairs from a graph.

    Parameters:
        graph: An adjacency bool matrix representing the causation among variables.

    Returns:
        direct_pairs:
            A list whose elements are in form of [parent, child], referring to the
            causation parent -> child.
    """

    dim = graph.shape[0]
    directed_pairs = []

    for j in range(dim):
        for i in range(dim):
            if graph[i][j] == 1 and graph[j][i] == 0:
                directed_pairs.append([j, i])

    return directed_pairs

Secondary Method: get_pairwise_info

Obtain information related to a given directed graph: (1) number of the directed pairs; (2) parents-child pairing relationships.

Parameters:

Name	Type	Description	Default
graph	ndarray	An adjacency bool matrix representing the causation among variables.	required

Returns:

Type	Description
(int, dict)	num_directed_pairs as the number of directed pairs and directed_parent_dict as the dictionary representing the parent-child pairing relationships.

Source code in cadimulc\utils\evaluation.py

@staticmethod
def get_pairwise_info(graph: ndarray) -> (int, dict):
    """
    Obtain information related to a given directed graph:
    (1) number of the directed pairs; (2) parents-child pairing relationships.

    Parameters:
        graph: An adjacency bool matrix representing the causation among variables.

    Returns:
        `num_directed_pairs` as the number of directed pairs and `directed_parent_dict`
         as the dictionary representing the parent-child pairing relationships.
    """

    directed_parent_dict = {}
    num_directed_pairs = 0
    dim = graph.shape[0]

    for i in range(dim):
        directed_parent_dict[i] = set()

    for i in range(dim):
        for j in range(dim):
            if graph[i][j] == 1:
                directed_parent_dict[i].add(j)
                num_directed_pairs += 1

    return num_directed_pairs, directed_parent_dict

Function: draw_graph_from_ndarray

cadimulc.utils.visualization.draw_graph_from_ndarray(array, graph_type='auto', rename_nodes=None, testing_text=None, save_fig=False, saving_path=None)

Draw the directed or undirected (causal) graph that is in form of adjacency matrix (implementation based on NetworkX).

Parameters:

Name	Type	Description	Default
array	ndarray	the causal graph (directed) or causal skeleton (indirected) in form of adjacency matrix.	required
graph_type	str	use directed to forcedly plot a directed graph.	'auto'
rename_nodes	list | None	Rename the nodes consisting with the column of dataset (n * d).	None
testing_text	str | None	Add simple text to the figure.	None
save_fig	bool	Specify saving a figure or not. Make sure to enter the saving path if you specify save_fig=True.	False
saving_path	str | None	The image saving path along with your image file name. e.g. ../file_location/image_file_name.	None

Source code in cadimulc\utils\visualization.py

def draw_graph_from_ndarray(
    array: ndarray,
    graph_type: str = "auto",
    rename_nodes: list | None = None,
    testing_text: str | None = None,
    save_fig: bool = False,
    saving_path: str | None = None
):
    """
    Draw the directed or undirected (causal) graph that is in form of adjacency matrix
    (implementation based on NetworkX).

    Parameters:
        array: the causal graph (directed) or causal skeleton (indirected) in form of adjacency matrix.
        graph_type: use `directed` to forcedly plot a directed graph.
        rename_nodes: Rename the nodes consisting with the column of dataset (n * d).
        Default as "X1,...Xd".
        testing_text: Add simple text to the figure.
        save_fig: Specify saving a figure or not. Make sure to enter the saving path if you specify `save_fig=True`.
        saving_path: The image saving path along with your image file name. e.g. ../file_location/image_file_name.
    """

    # ensure / convert into nx.graph
    if not isinstance(array, np.ndarray):
        raise TypeError("The expected type of input object should be numpy array.")

    if graph_type == 'auto':
        if (array == array.T).all():
            directed = False
        else:
            directed = True
    elif graph_type == 'directed':
        directed = True
    else:
        raise ValueError("Choose the graph type as 'auto' or 'directed'.")

    # directed graph
    if directed:
        # conventional causal direction / transpose of matrix
        G = nx.from_numpy_array(array.T, create_using=nx.DiGraph)
    # undirected graph
    else:
        # conventional causal direction / transpose of matrix
        G = nx.from_numpy_array(array.T)

    # rename graph node name
    if rename_nodes is None:
        node_id = 0
        for node in G.nodes():
            rename_node = f'X{node_id + 1}'
            G = nx.relabel_nodes(G, mapping={node: rename_node})
            node_id += 1
    else:
        for node, rename_node in zip(G.nodes(), rename_nodes):
            rename_node = str(rename_node)
            G = nx.relabel_nodes(G, mapping={node: rename_node})

    # fix position
    pos = nx.circular_layout(G)
    # setting parameters
    plt.figure(figsize=(5, 2.7), dpi=120)

    if directed:
        nx.draw(
            G=G,
            pos=pos,
            with_labels=True,
            node_color='black',
            font_color='white',
            font_size=15,
            width=1.25,
            arrowsize=20,
            node_size=500
        )

    else:
        nx.draw(
            G=G,
            pos=pos,
            with_labels=True,
            node_color='black',
            font_color='white',
            font_size=15,
            width=1.25,
            node_size=500
        )

    if testing_text is not None:
        plt.text(x=0.5, y=0.5, s=testing_text, fontsize=12, color='red')
        print('* Figure Label: ', testing_text)

    if save_fig:
        plt.savefig(
            saving_path + ".png",
            dpi=200,
            transparent=False,
            bbox_inches="tight"
        )

Running examples

CADIMULC is a light Python repository without sophisticated library API design. Documentation on this page is meant to provide introductory materials of the practical tool as to causal discovery. For running example, please simply check out Quick Tutorials for the straightforward usage in the "micro" workflow of causal discovery.