edsnlp.tune

Check if Plotly is installed. If not warn the user. Plotly is needed by optuna.visualization to produce tuning visual results.

Returns: bool: True if Plotly is installed, False otherwise.

Compute the time for the first trial or the EMA (Exponential Moving Average) across all trials in the study.

Parameters:

study : optuna.study.Study An Optuna study object containing past trials. ema : bool If True, computes the EMA of trial times; otherwise, computes the time of the first trial. alpha : float, optional Smoothing factor for EMA. Only used if ema is True.

Returns:

float: Time for the first trial or the EMA across all trials, in seconds.

Estimate the maximum number of trials that can be executed within a given GPU time budget.

Parameters:

gpu_hours : float The total amount of GPU time available for tuning, in hours. time_per_trial : float Time per trial, in seconds.

Returns:

int: The number of trials that can be run within the given GPU time budget.

Update a configuration dictionary with tuned hyperparameter values.

This function modifies a given configuration dictionary by updating the specified hyperparameters with values from either a dictionary or an Optuna trial object. The updated configuration and training keyword arguments are returned.

Parameters:

config : dict The configuration dictionary to be updated. tuned_parameters : dict A dictionary specifying the hyperparameters to tune. values : dict, optional A dictionary of parameter names and their corresponding values to update the configuration. Used when trial is not provided. trial : optuna.trial.Trial, optional An Optuna trial object to sample parameter values. Used when values is not provided.

Returns:

tuple - kwargs : dict The resolved training keyword arguments from the updated configuration. - updated_config : dict The modified configuration dictionary.

Perform two-phase hyperparameter tuning using Optuna.

This method executes a two-phase tuning strategy. In the first phase, all specified hyperparameters are tuned. Based on their computed importance, only the most important hyperparameters (top 50% by importance) are selected for fine-tuning in the second phase, while the less important hyperparameters are frozen to their best values from phase one.

Parameters:

config : dict The configuration dictionary for the model and training process. hyperparameters : dict A dictionary specifying the hyperparameters to tune. output_dir : str Directory where tuning results, visualizations, and best parameters will be saved. checkpoint_dir : str, Path to save the checkpoint file. n_trials : int The total number of trials to execute across both tuning phases. This number will be split between the two phases, with approximately half of the trials assigned to each phase. viz : bool Whether or not to include visual features (False if Plotly is unavailable). metric : Tuple[str] Metric used to evaluate trials. study : optuna.study.Study, optional Optuna study containing the first trial that was used to compute n_trials in case the user specifies a GPU hour budget. is_fixed_trial : bool, optional Whether or not the user specified fixed n_trials in config. If not, recompute n_trials between the two phases. In case there was multiples trials pruned in phase 1, we raise n_trials to compensate. Default is False. gpu_hours : float, optional Total GPU time available for tuning, in hours. Default is 1 hour. skip_phase_1 : bool, optional Whether or not to skip phase 1 (in case of resuming from checkpoint). Default is False.

Compute the remaining number of trials possible within the GPU time budget that was not used by the study (in cases where multiple trials were pruned).

Parameters:

study : optuna.study.Study An Optuna study object containing past trials. gpu_hours : float The total amount of GPU time available for tuning, in hours.

Returns:

int: The remaining number of trials possible.

Perform hyperparameter tuning for a model using Optuna.

Parameters:

config_meta : dict Metadata for the configuration file, containing at least the key "config_path" which specifies the path to the configuration file. hyperparameters : dict A dictionary specifying the hyperparameters to tune. The keys are the parameter names, and the values are dictionaries containing the following fields: - "path": List[str] representing the path to the parameter in config. - "type": The type of parameter ("float", "int", "categorical"). - "low": (optional) Lower bound for numerical parameters. - "high": (optional) Upper bound for numerical parameters. - "step": (optional) Step size for numerical parameters. - "log": (optional) Whether to sample numerical parameters on a log scale. - "choices": (optional) List of values for categorical parameters. output_dir : str Directory where tuning results, visualizations, and best parameters will be saved. checkpoint_dir : str, Path to save the checkpoint file. gpu_hours : float, optional Total GPU time available for tuning, in hours. Default is 1 hour. n_trials : int, optional Number of trials for tuning. If not provided, it will be computed based on the gpu_hours and the estimated time per trial. two_phase_tuning : bool, optional If True, performs two-phase tuning. In the first phase, all hyperparameters are tuned, and in the second phase, the top half (based on importance) are fine-tuned while freezing others. Default is False. seed : int, optional Random seed for reproducibility. Default is 42. metric : str, optional Metric used to evaluate trials. Default is "ner.micro.f". keep_checkpoint : bool, optional If True, keeps the checkpoint file after tuning. Default is False.