• Bean Machine: Meta model uncertainty estimation system

    Meta introduced Bean Machine, a probabilistic programming system that allows you to determine the errors of deep learning models. Bean Machine can be used to detect hidden properties of a model.

    Deep learning models are characterized by two types of errors.  The epistemic error describes what the model does not know because the training data was not suitable, while the aleatoric error is an error that occurs due to the natural randomness of observations. If there are enough training samples, the epistemic error will decrease, but the aleatory error cannot be reduced even if more data is provided.

    Probabilistic modeling— an artificial intelligence method used by the Bean Machine, can measure such types of errors, taking into account the influence of random events in forecasting. This allows analysts using the tool not only to understand the forecast of the artificial intelligence system, but also the relative probability of other possible forecasts. Probabilistic modeling also makes it easier to compare the structure of the model with the structure of the problem. With its help, users can interpret why certain predictions were made, which can help in the development of the model.

    The error measured by the Bean Machine allows you to identify the limitations of the model, for example, the permissible error of the housing price forecasting model or the level of confidence of the model designed to predict whether the new function of the application will work better than the old one.

    Notify of
    Inline Feedbacks
    View all comments